Download:
pdf |
pdfORNL/TM-2011/87
Evaluation of the National
Weatherization Assistance Program
during Program Years 2009-2011
(American Reinvestment and Recovery
Act Period)
Draft Date: April 21, 2011
Prepared by
Bruce Tonn
Erin Rose
Richard Schmoyer
Joel Eisenberg
Mark Ternes
Martin Schweitzer
Timothy Hendrick
DOCUMENT AVAILABILITY
Reports produced after January 1, 1996, are generally available free via the U.S. Department of
Energy (DOE) Information Bridge.
Web site http://www.osti.gov/bridge
Reports produced before January 1, 1996, may be purchased by members of the public from the
following source.
National Technical Information Service
5285 Port Royal Road
Springfield, VA 22161
Telephone 703-605-6000 (1-800-553-6847)
TDD 703-487-4639
Fax 703-605-6900
E-mail [email protected]
Web site http://www.ntis.gov/support/ordernowabout.htm
Reports are available to DOE employees, DOE contractors, Energy Technology Data Exchange
(ETDE) representatives, and International Nuclear Information System (INIS) representatives from
the following source.
Office of Scientific and Technical Information
P.O. Box 62
Oak Ridge, TN 37831
Telephone 865-576-8401
Fax 865-576-5728
E-mail [email protected]
Web site http://www.osti.gov/contact.html
This report was prepared as an account of work sponsored by an
agency of the United States Government. Neither the United States
Government nor any agency thereof, nor any of their employees,
makes any warranty, express or implied, or assumes any legal
liability or responsibility for the accuracy, completeness, or
usefulness of any information, apparatus, product, or process
disclosed, or represents that its use would not infringe privately
owned rights. Reference herein to any specific commercial product,
process, or service by trade name, trademark, manufacturer, or
otherwise, does not necessarily constitute or imply its endorsement,
recommendation, or favoring by the United States Government or
any agency thereof. The views and opinions of authors expressed
herein do not necessarily state or reflect those of the United States
Government or any agency thereof.
ORNL/TM-2011/87
Environmental Sciences Division
NATIONAL EVALUATION OF THE WEATHERIZATION ASSISTANCE
PROGRAM DURING THE ARRA PERIOD: PROGRAM YEARS 2009-2011
Bruce Tonn
Erin Rose
Richard Schmoyer
Joel F. Eisenberg
Mark Ternes
Martin Schweitzer
Tim Hendrick
Date Published: 4/4/2011
Prepared by
OAK RIDGE NATIONAL LABORATORY
Oak Ridge, Tennessee 37831-6283
managed by
UT-BATTELLE, LLC
for the
U.S. DEPARTMENT OF ENERGY
under contract number DE-AC05-00OR22725
THIS PAGE LEFT INTENTIONALLY BLANK
ii
CONTENTS
Page
LIST OF FIGURES ..................................................................................................................... ix
LIST OF TABLES .........................................................................................................................x
LIST OF ACRONYMS ............................................................................................................... xi
ACKNOWLEDGMENTS .......................................................................................................... xii
1. INTRODUCTION AND OVERVIEW ....................................................................................1
1.1 PURPOSES AND RESEARCH QUESTIONS ............................................................................ 2
1.1.1 Network Planning Committee.............................................................................................. 2
1.1.2 Program Logic Model ........................................................................................................... 9
1.1.3 Program Evaluation Design Matrix .................................................................................. 12
1.2 EVALUATION ORGANIZATION ............................................................................................ 19
1.3 COMPARISON OF WAP RETROSPECTIVE AND ARRA PERIOD EVALUATIONS ... 19
2. IMPACT ASSESSMENT ........................................................................................................23
2.1 PROGRAM CHARACTERIZATION STUDY ........................................................................ 24
2.1.1 Data and Sampling Frames ................................................................................................ 24
2.1.2 Low-Income Weatherization Market Analysis ................................................................ 27
2.1.3 State and Agency Characterization Analysis ................................................................... 28
2.1.4 Detailed Characterization of Program and Analysis of Implementation ...................... 28
2.1.5 Program Funding and Costs Analysis ............................................................................... 29
2.2. ENERGY AND COST SAVINGS .............................................................................................. 30
2.2.1 Sampling Frame and Data ................................................................................................. 32
2.2.2 Energy Analysis................................................................................................................... 35
2.2.3 Measures Analysis ............................................................................................................... 39
2.2.4 Attribution Methodology.................................................................................................... 39
2.3 NON-ENERGY IMPACTS ......................................................................................................... 40
2.3.1 Monetized Data Collection and Analysis .......................................................................... 49
2.3.2 Special IAQ Radon Remediation Cost Study ................................................................... 61
2.3.3 Non-Monetized Data Collection and Analysis .................................................................. 63
2.3.4 Social Network Study ......................................................................................................... 66
2.4. COST-EFFECTIVENESS .......................................................................................................... 68
2.4.1 General Assessment ............................................................................................................ 68
2.4.2 In-Depth Cost Assessment of Weatherization Measures ................................................. 70
2.5 EXPLANATORY FACTORS ..................................................................................................... 71
iii
2.5.1 Regression Analysis ............................................................................................................ 72
2.5.2 Cross-Tabulation Results ................................................................................................... 73
2.5.3 Data ...................................................................................................................................... 73
3. PROCESS ASSESSMENT......................................................................................................74
3.1 PROGRAM OPERATIONS AND IMPLEMENTATION ....................................................... 74
3.1.1 Data and Surveys ................................................................................................................ 74
3.1.2 Analysis ................................................................................................................................ 75
3.1.3 Deferral Study ..................................................................................................................... 76
3.1.4 Post-ARRA Weatherization Network Strategies ............................................................. 81
3.1.5 Post-ARRA Training Assessment ...................................................................................... 82
4. SPECIAL STUDIES ................................................................................................................83
4.1 UNDERPERFORMING WEATHERIZED UNITS ................................................................. 83
4.1.1 Introduction ......................................................................................................................... 83
4.1.2 Potential Causes of Underperformance and Over-performance .................................... 83
4.1.3 Evaluation Approach: Single Family and Mobile Homes .............................................. 84
4.1.4 Evaluation Approach: Large Multifamily Buildings ....................................................... 85
4.2 TERRITORIES ............................................................................................................................ 86
4.2.1 Introduction ......................................................................................................................... 86
4.2.2 Evaluation Methods ............................................................................................................ 87
4.2.3 Puerto Rico Case Study ...................................................................................................... 88
4.3 WEATHERIZATION INNOVATION PILOT PROGRAM ................................................... 90
4.3.1 Introduction ......................................................................................................................... 90
4.3.2. WIPP Programmatic Questions – For all grantees/approaches .................................... 92
4.3.3. WIPP Project Activities and Evaluation Questions and Approaches ........................... 92
4.3.4 Evaluation Plans.................................................................................................................. 99
4.3.5 Technical Assistance ......................................................................................................... 100
4.4 SUSTAINABLE ENERGY RESOURCES FOR CONSUMERS........................................... 101
4.4.1. Comprehensive Evaluation Approach ........................................................................... 102
4.4.2 Technologies Selected for High-Rigor Approach ........................................................... 105
4.4.3 SERC Randomized Controlled Trials (RCTs) ............................................................... 105
4.5 ENCOURAGEMENT DESIGN ................................................................................................ 106
4.6 GREENHOUSE GAS EMISSION ANALYSIS....................................................................... 106
4.6.1 Introduction ....................................................................................................................... 106
4.6.2 Carbon Offset Providers .................................................................................................. 106
4.6.3 Voluntary Carbon Markets and Weatherization Assistance ........................................ 110
4.6.4 Research Approach to Assessing GHG Emission Reductions ...................................... 113
4.7 PERSISTENCE OF ENERGY SAVINGS ............................................................................... 115
4.7.1 Background ....................................................................................................................... 115
4.7.2 Research Design ................................................................................................................ 115
4.7.3 Tasking Statement ............................................................................................................ 116
5. SYNTHESIS ...........................................................................................................................119
iv
6. SCHEDULE............................................................................................................................122
7. REFERENCES .......................................................................................................................124
APPENDIX A. NATIONAL WEATHERIZATION NETWORK COMMITTEE .............126
APPENDIX B. DOE SURVEY .................................................................................................128
APPENDIX C: S1: ALL STATES PROGRAM INFORMATION SURVEY .....................129
APPENDIX D -- S2: ALL AGENCIES PROGRAM INFORMATION SURVEY .............182
APPENDIX E. S3: SUBSET OF AGENCIES DETAILED PROGRAM INFORMATION
SURVEY .....................................................................................................................................220
APPENDIX F. DF2: HOUSING UNIT INFORMATION SURVEY ....................................263
APPENDIX G. DF3: BUILDING INFORMATION SURVEY .............................................286
APPENDIX H. DF4: ELECTRIC & NATURAL GAS BILLING INFORMATION FROM
AGENCIES DATA FORM .......................................................................................................314
APPENDIX I: DF5A - NATIONAL WEATHERIZATION ASSISTANCE PROGRAM
EVALUATION HOUSEHOLD ELECTRICITY USAGE FORM .......................................321
APPENDIX J. DF5B: NATIONAL WEATHERIZATION ASSISTANCE PROGRAM
EVALUATION HOUSEHOLD NATURAL GAS USAGE FORM ......................................327
APPENDIX K. S4: OCCUPANT SURVEY ............................................................................333
APPENDIX L – DF9 OCCUPANT SURVEY INFORMATION DATA FORM ................393
APPENDIX M: WEATHERIZATION STAFF SURVEY ....................................................419
APPENDIX N: DF11 - WEATHERIZATION STAFF SURVEY DATA FORM ..............426
APPENDIX O. SAMPLE SIZE JUSTIFICATION ................................................................428
O.1 POPULATIONS SAMPLED.................................................................................................... 428
O.2 AGENCY SAMPLING ............................................................................................................. 429
O.2.1 Agency sampling for energy benefits (billing data analysis) ........................................ 429
O.2.2 Agency sampling for program characterization and process assessment................... 431
O.2.3 Agency staff subsampling ................................................................................................ 431
O.3 WEATHERIZATION INNOVATION PILOT PROGRAM SAMPLING.......................... 432
v
O.4 SUSTAINABLE ENERGY RESOURCES FOR CONSUMERS SAMPLING ................... 437
As discussed above, all 92 SERC grantees will be included in the sub-sample of 450 subgrantees that will be asked to provide information re weatherized homes and utility account
information........................................................................................................................................ 437
O.5 DEFERRAL STUDY SAMPLING .......................................................................................... 437
Ten states and ten agencies will be sampled for this a deferral study, and the deferral
incidence and process (e.g., quality assurance) will be examined for a random sample of
weatherized units from each sampled agency. The number of units to be sampled is (at
least tentatively) specified in the draft Evaluation Plan as "200 occupants engaged in the
deferral process." Deferral rates encountered by some agencies are reckoned by agency
staff to be as high as twenty percent.
Agency selection will be purposive for agencies for
which deferrals are understood to be troublesome. Inferences will therefore be restricted
to the ten sampled agencies. However the ten sampled agencies will serve collectively as
anecdotal evidence about the extent to which deferrals can be a problem, and the overall
deferral rate for the ten sampled agencies will thus be a parameter of primary interest in
the analysis. Since we'll ask substantially the same question to each of the agencies, we
need to develop a formal OMB 83-i submission for this. Note that if (1) we sampled only
nine agencies instead of ten, and (2) the information about the deferral could be obtained
directly from agency staff, without posing additional questions to home owners, we might
be able to circumvent the OMB submission. Whether or not we pursue such an approach,
however, we still need to decide about how many of each agency's units to sample for the
deferral analysis. That statistical question is considered in the next paragraph. The
specifications can be tweaked, of course, but suppose for the moment that we would like to
estimate the overall deferral rate for the ten agencies to within five percentage points with
90% confidence. Given that deferral rates encountered by some agencies are reckoned by
agency staff to be as high as 20%, suppose that the deferral rate for any sampled agency is
no higher than 25%. This implies that the variance of the overall deferral rate estimate is
maximum when the deferral rate is 25% for each sampled agency. That is, the worst-case
(i.e., maximum) variance of the overall deferral rate estimate is .25(1-.25)/(10*n), where n is
the number of units sampled per agency. Hence, letting Z = 1.645 (95th percentile of the
standard normal distribution), the worst-case required sample size for the 5-percentagepoint-90%-confidence specification is n = Z*Z*.25(1-.25)/(10*(.05)**2) = 20.3 units per
vi
agency. So the sample size of 200 units seems reasonable. ....................................................437
O.7 ADDITIONAL NOTES ............................................................................................................ 441
O.7.1 Pretest/Pilot Studies ......................................................................................................... 441
O.7.2 Influential information .................................................................................................... 441
O.7.3 Response rates .................................................................................................................. 441
O.8 REFERENCES .......................................................................................................................... 442
APPENDIX P. BUILDING TYPE DEFINITIONS ................................................................444
APPENDIX Q. UNIT/BUILDING LEVEL ENERGY ANALYSIS .....................................446
Q.1 SINGLE-FAMILY HOUSES AND MOBILE HOMES ........................................................ 446
Q.2 LARGE MULTIFAMILY BUILDINGS................................................................................. 447
Q.2.1 Buildings with central building heating systems ........................................................... 447
Q.2.2 Buildings with apartment-level heating systems ........................................................... 449
Q.3 SMALL MULTIFAMILY BUILDINGS................................................................................. 450
APPENDIX R. ORNL AGGREGATE MODEL ....................................................................452
R.1 PRISM ........................................................................................................................................ 454
R.2 AGGREGATE PRISM ............................................................................................................. 455
R.3 ORNL AGGREGATE MODEL............................................................................................... 455
APPENDIX S. ASSESSMENT OF A POTENTIAL SPLIT-WINTER RCT
CONFIRMATORY PROJECT ................................................................................................458
APPENDIX T. S6: ALL STATES POST-ARRA SURVEY ..................................................466
APPENDIX U: S7: ALL AGENCIES POST-ARRA SURVEY ............................................472
APPENDIX V. S8: WEATHERIZATION TRAINING CENTERS POST-ARRA SURVEY479
vii
THIS PAGE LEFT INTENTIONALLY BLANK
viii
LIST OF FIGURES
Figure
Page
Fig. 3.1. Deferral Action Tree. ....................................................................................................93
Fig. 3.2. Defferral Process Assessment Timeline. ......................................................................95
Fig. 6.1. WAP--ARRA Period Evaluation schedule. ...............................................................132
ix
LIST OF TABLES
Table
Page
Table 1.1. Logic model for the Weatherization Assistance Program ......................................10
Table 1.2. Evaluation design matrix for the Weatherization Assistance Program ................28
Table 1.3. Design matrix questions addressing each logic model outcome.............................32
Table 1.4. Comparison of WAP Retrospective and WAP-ARRA Peirod Evaluations .........34
Table 2.1. Data fields for this study from RECS, CPS, AHS databases ................................39
Table 2.2. Total annual Program energy savings......................................................................50
Table 2.3. Non-engery impacts and the household-level data and metrics required to
calculate their value .....................................................................................................................55
Table 2.4. Impacts, metrics, and factors required to determine need for new data ..............63
Table 3.1. Deferral Protocol for All States and Territories .....................................................90
Table 3.2. Deferral Observation Assessment Criteria ..............................................................91
Table 3.3. Unit or Agency Attributes .........................................................................................92
Table 4.1. WAP funding allocations, key measures, and evaluation methods for U.S.
territories under ARRA ............................................................................................................102
Table 4.2. WIPP Funded Projects ............................................................................................103
Table 4.3. Evaluation Methods Applied to Categories of WIPP Activities ..........................111
Table 4.4. Schedule of Evaluation Activities ...........................................................................112
Table 4.5. Schedule of Categories and Tasks ..........................................................................114
Table 4.6. Technology Categories .............................................................................................115
x
LIST OF ACRONYMS
AFUE
Annual Fuel Utilization Efficiency
AHS
American Housing Survey
ANOVA
Analysis of variance
ASHRAE
American Society of Heating, Refrigerating, and Air Conditioning Engineers
Btu
British thermal unit
CATI
Computer assisted telephone interviewing
CDA
Conditional demand analysis
cfm
Cubic feet per minute
CFR
Code of Federal Regulations
CH4
Methane
CO
Carbon monoxide
CO2
Carbon dioxide
CPS
Current Population Survey
CV
Coefficient of variance
DOE
Department of Energy
EERE
Energy Efficiency and Renewable Energy
GPRA
Government Performance and Results Act
IMT
Inverse Modeling Toolkit
kWh
Kilowatt hours
LIHEAP
Low-Income Home Energy Assistance Program
NAC
Normalized annual consumption
NASCSP
National Association For State Community Services Programs
NCAF
National Community Action Foundation
NEI
Non-energy impact
NOx
Nitrogen oxides
O2
Oxygen
OMB
Office of Management and Budget
ORNL
Oak Ridge National Laboratory
PBA
Planning, Budget, and Analysis
PM
Particulate matter
PRA
Paperwork Reduction Act
PRISM Princeton Scorekeeping Method
PVE
Petroleum Violation Escrow
PY
Program year
R²
Coefficient of determination
RECS
Residential Energy Consumption Survey
SAE
Statistically Adjusted Engineering
SIR
Savings-to-investment ratio
SOx
Sulfur oxides
SSI
Supplemental Security Income
TANF
Temporary Assistance for Needy Families
xi
ACKNOWLEDGMENTS
We wish to thank the National Weatherization Network committee for their input to this plan to evaluate
WAP during the ARRA period. We also wish to thank Bob Adams, Jennifer Somers, Tyler Huebner,
Brian Levy, Frank Norcross, and Jon Muckey of DOE for their inputs as well. Comments on drafts of this
report provided by Inga Treitler were very much appreciated. We also appreciate the editorial touch
provided by Em Turner Chitty. Lastly, we thank Tracy Clem for her hard work on this report as well.
xii
THIS PAGE LEFT INTENTIONALLY BLANK
xiii
1. INTRODUCTION AND OVERVIEW
The U.S. Department of Energy‘s (DOE‘s) Weatherization Assistance Program (WAP) was
created by Congress in 1976 under Title IV of the Energy Conservation and Production Act. The purpose
and scope of the Program as currently stated in the Code of Federal Regulations (CFR) 10CFR 440.1 is
―to increase the energy efficiency of dwellings owned or occupied by low-income persons, reduce their
total residential expenditures, and improve their health and safety, especially low-income persons who are
particularly vulnerable such as the elderly, persons with disabilities, families with children, high
residential energy users, and households with high energy burden‖ (Code of Federal Regulations, 2005).
DOE sponsored the first comprehensive evaluation of the Program in the early 1990's to provide
policy makers and Program implementers with the up-to-date and reliable information they needed for
effective decision-making and cost-effective operations. Oak Ridge National Laboratory (ORNL)
managed the five-part study, which was based primarily on data from Program Year (PY) 1989 and
supplemented by data from 1991–92 (Brown, Berry, and Kinney, 1994). ORNL has also conducted four
meta-evaluations1 of the Program‘s energy savings using studies conducted by individual states between
the years 1990–1996 (Berry, 1997), 1996–1998 (Schweitzer and Berry, 1999), 1993–2002 (Berry and
Schweitzer, 2003), and 1993-2005 (Schweitzer, 2005).
In April 2009, DOE directed ORNL and its team of independent energy program evaluators to
initiate a second, now retrospective, evaluation of the Program for PYs 2007 and 2008 (Ternes et al.
2007). The Program changed significantly during the almost two-decade period between these
evaluations. In response to findings and recommendations resulting from the 1989 National Evaluation,
the Weatherization Plus strategic planning process, and other federal, state, and local initiatives, the
Program incorporated new funding sources, management principles, audit procedures, and energyefficiency measures. In particular, the use of computerized audits was increased, cooling and baseload
measures were added, and weatherization approaches that were tailored to the unique construction
characteristics of mobile homes were developed; in addition, the weatherization of large multifamily
buildings was expanded and became more sophisticated, while greater flexibility to improve ―energyrelated‖ health and safety was provided. Finally, the Program‘s ability to leverage influence with utilities,
other state programs, and owners of large multifamily buildings increased considerably. The retrospective
evaluation is expected to be completed by Fall 2012.
This report describes the third major evaluation of the Program, encompassing program years
2009 to 2011. In this report, this period of time is referred to as the ARRA Period. This is a special period
of time for the Program because the American Recovery and Reinvestment Act (ARRA) of 2009 has
allocated $5 billion of funding for the Program. In normal program years, WAP‘s annual appropriation is
in the range of $200-250 million, supporting the weatherization of approximately 100,000 homes. With
the addition of ARRA funding during these program years, the expectation is that weatherization activity
will exceed 300,000 homes per year. In addition to saving energy and reducing low-income energy bills,
expanded WAP funding is expected to stimulate the economy by providing new jobs in the
weatherization field and allowing low-income households to spend more money on goods and services by
spending less on energy.
During the ARRA period, the Weatherization Assistance Program is a much different program
than it was as recently as PY 2008 and also likely different than it will be in the future. Among the key
1
The term ―meta-evaluations‖ refers to the analysis of analyses; these are a more rigorous alternative to the narrative
discussion of research studies. Meta-evaluations involve the statistical analysis of a collection of analysis results
from individual studies for the purpose of integrating the findings.
1
differences are the following: First, a greatly expanded weatherization workforce has been recruited,
trained, organized, and sent into the field. In order to support this expansion, the percentage of spending
allowed for training and technical assistance has been raised from 10 percent to 20 percent.
Second, all states and U.S. territories have received unprecedented increases in their
weatherization funding and some grantees have grappled with budgets that were several times larger than
anything they had previously managed. Some states, faced with this massive program expansion, have
used WAP funds for weatherization, while others have implemented other approaches, including
innovations in Program delivery and management.
Third, substantial amounts of funding have been set aside to support innovations in Program
funding and design. The first of these, for the Sustainable Energy Resources for Consumers (SERC)
grants, sets aside up to 2% of funds to encourage innovative projects by Weatherization subgrantees (i.e.,
local weatherization agencies) to further weatherization efforts that are outside the scope of existing
Program regulations and restrictions. The second, the Weatherization Innovation Pilot Program (WIPP),
sets aside $30 million to encourage the formation of partnerships with both traditional and non-traditional
weatherization providers so that non-federal resources can be leveraged to pursue the Program‘s purposes.
Lastly, to accommodate the expansion of the weatherization program, several major changes in
Program administration were made. Eligibility requirements were eased: The household income
threshold increased from 150% to 200% of the Poverty Income Guidelines. Also, the average cost ceiling
(the average amount of money that can be spent by grantees to weatherize homes) was increased from
$2,500 to $6,500. Additionally, for the first time, the wages for weatherization workers were adjusted to
conform to Davis-Bacon Act prevailing wage requirements. All of these factors affect energy and cost
savings and have an impact on Program delivery; they may also have relevance to future Program design
and are included as topics in this evaluation.
1.1 PURPOSES AND RESEARCH QUESTIONS
ORNL reconvened a National Weatherization Network Committee to provide comments and
input for the evaluation of the WAP during the ARRA period (Section 1.1.1). The formalized planning
process used for the retrospective evaluation, based on the concept of a program logic model and
evaluation design matrix as developed by the W. K. Kellogg Foundation (2001), was again undertaken
(see Sections 1.1.2 and 1.1.3, respectively). This section concludes with an overview of the WAP-ARRA
period evaluation.
1.1.1 Network Planning Committee
One of the evaluation‘s most important goals is to meet the needs of the weatherization
community, since that community, also referred to as the weatherization network, will be a primary
beneficiary and user of the evaluation‘s findings. In addition, the network of state offices and over one
thousand local agencies will be relied upon to collect and provide significant amounts of the data needed
for the evaluation. Therefore, ORNL felt that it was important to involve the weatherization community
early in the planning process in order to establish open communications with them, get them actively
engaged in the evaluation, strengthen their voice in the planning process, clearly identify their
expectations of the evaluation, and increase their participation in the evaluation‘s implementation.
ORNL convened a National Weatherization Network Committee to provide input for the
retrospective evaluation in 2009 and reconvened a reconstituted Network Committee in January 2010 in
Washington, DC to provide input for the WAP-ARRA period evaluation.
2
After receiving recommendations from DOE headquarters and regional program staff, ORNL
selected 36 people from the National Association for State Community Services Programs (NASCSP) and
the National Community Action Foundation (NCAF) to serve on the committee. The committee
members are identified in Appendix A. The committee members include state weatherization officials,
local weatherization officials, DOE staff, ORNL staff, and independent evaluators.
The committee was tasked by ORNL to identify the information that they would find most useful
from the national evaluation; to identify data available at national, regional, state, and local levels that
would be pertinent to the evaluation; and to provide insight into how the evaluation and specifically the
data collection could be best conducted. This information was solicited to assist ORNL in developing the
evaluation‘s research questions, identifying the various studies that would need to be performed under the
evaluation, and formulating details of the implementation.
At the January 2010 meeting, a moderator led the committee through several discussions to
identify numerous research questions. These research questions may be broken into five main groups
relating to the following areas: (1) energy savings and cost-effectiveness; (2) process issues; (3) nonenergy-related benefits; (4) indications for post-ARRA research; and (5) broad research questions.
1) Energy Savings and Cost-effectiveness: It is important to measure energy savings attributable to
WAP during the ARRA period, but it is equally important to study changes ―on the ground‖ resulting
from the ARRA and those of its provisions related to weatherization assistance. Therefore, evaluation
questions related to energy savings include these:
How much in household energy savings is attributable to WAP during the ARRA period?
How cost-effective are these savings?
Were changes in the prices of weatherization measures possibly attributable to the uniqueness of
the ARRA-period impact cost-effectiveness?
Did the ARRA-period change in the formula for distribution of WAP funds to grantees (i.e.,
states and territories) affect energy savings and cost-effectiveness?
Did the expansion of existing weatherization crews and the establishment of new ones have an
impact on energy savings and cost-effectiveness?
How much energy was saved in the studied initiatives to weatherize public housing units, and
were those savings cost-effective?
How did the change in the assistance eligibility standard impact energy savings?
How did the increase in average expenditures on weatherization measures from $2,500 to $6,500
impact measure selection and energy savings?
What are the energy savings attributable to the SERC and WIPP projects?
2) Process Issues: ARRA funding for the WAP has had a significant impact on Program operation and
management. Process issues abound. These issues have been grouped into four categories: (a)
management and oversight; (b) prevailing wages (Davis-Bacon Act), (c) the national weatherization
network, and (d) labor force and training issues.
(a) Management and Oversight: The expansion of WAP has led to many management challenges and
opportunities. Scrutiny of the Program has also increased substantially. Here are suggested evaluation
questions that address Program management and oversight during the ARRA period:
At the state level, what programmatic changes and innovative approaches were implemented to
disburse weatherization funds, and how effective were those approaches (e.g., changes in
reporting requirements, changes in subgrantee participation, changes in training and technical
assistance procedures, changes in audit approaches)?
3
What programmatic changes and innovative approaches at the local level were implemented to
deliver weatherization services, and how effective were those innovations (e.g., changes in intake
procedures, changes in the mix of buildings weatherized, etc.)?
What were the results of experiments allowing some weatherization funds to be used in public
housing?
What types of projects were funded under SERC and WIPP?
How did hot-climate states manage substantial increases in weatherization funding?
How did U.S. territories deal with Program initiation?
Did walk-away policies (i.e., deferrals of weatherization efforts on unsuitable properties) and the
frequency of such deferrals change due to this increase in the amount of available funds for
weatherization measures?
How satisfied were clients with the weatherization services provided during the ARRA period?
Have the demographic characteristics of clients receiving weatherization and those on
weatherization waiting lists changed during the ARRA period?
Has DOE managed WAP effectively during the ARRA period (i.e. in terms of clarity and
timeliness of guidance);
What have been the actual monetary administrative costs associated with increased oversight
during the ARRA period for states and local weatherization agencies (e.g., from DOE IG, GAO)?
Has ARRA funding both allowed states and agencies to afford new technologies and pushed them
to use new information to increase operational and reporting efficiencies? If so, what new
technologies are being implemented?
Has ARRA funding allowed the purchase of new field technologies? If so, what new field
technologies are being implemented?
To what extent have the weatherization costs used in savings-to-investment ratio (SIR)
calculations differed from actual, possibly highly fluctuating measure costs endured during
ARRA?
To what extent have other large DOE programs competed with WAP for labor during the ARRA
period (e.g. SEP, EECGB)?
Were there any material, equipment or other supply-chain bottlenecks that hampered or prevented
weatherization production during the ARRA period?
What new state regulations were enacted with respect to weatherization during ARRA, and to
what extent did these regulations have unintended consequences?
(b) Prevailing Wages (Davis-Bacon Act): ―Davis-Bacon‖ is the common name applied to a 1931 Act that
requires all federal construction projects to pay prevailing wages to their workers. As part of the ARRA
legislation, Congress stipulated that projects funded with ARRA money must follow Davis-Bacon rules.
The U.S. Department of Labor (DoL) has the responsibility for identifying ―prevailing wages‖ in the
construction industry. These wages are identified for a set of construction industry jobs and are estimated
for each county in the United States. Prior to ARRA, weatherization activities have not been subject to the
requirements of Davis-Bacon. However, under ARRA, it was realized that weatherization-related jobs did
not overlap with construction-industry jobs. Therefore, DoL needed to establish prevailing wages for
weatherization-related jobs in every county in the country and DOE needed to develop new guidance
related to Davis-Bacon. Predictably, much confusion and many delays resulted. An entire set of
evaluation questions is devoted to process issues surrounding Davis-Bacon:
Did Davis-Bacon, on balance, lead to positive job creation?
Did the application of Davis-Bacon lead to changes in weatherization wages?
What were the actual monetary administrative costs for complying with Davis-Bacon?
Did paperwork requirements lead some experienced weatherization contractors to leave the lowincome weatherization field?
4
Has Davis-Bacon led local weatherization agencies to change their mix of in-house vs. contractor
labor and crews?
How did multi-county weatherization agencies deal with county-specific Davis-Bacon wage-rate
requirements?
How has Davis-Bacon affected weatherization costs associated with multifamily buildings of four
stories and higher?
Have changes in weatherization costs associated with Davis-Bacon altered residents‘ choices of
measures installed in homes?
Overall, how did Davis-Bacon implementation impact the Program‘s cost-effectiveness?
(c) National Weatherization Network: The unprecedented flow of federal funds into low-income home
weatherization efforts has changed the national weatherization network in several ways: firstly, the size of
the labor force has necessarily increased; secondly, these funding increases have naturally drawn new
stakeholders into the network. The new funding has also affected long-standing leveraging relationships,
in which states and agencies are able to leverage DOE funding in order to attract non-DOE funding both
positively and negatively; thus, it has increased the visibility of low-income weatherization. The
following evaluation questions are designed to document and evaluate changes in the national
weatherization network during the ARRA period.
Has the composition of the national weatherization network changed during the ARRA period?
What types of newcomers have joined the network during ARRA?
Has the influx of ARRA funding negatively affected existing leveraging relationships?
Are new leveraging relationships forming?
Has the public‘s perception of low-income weatherization changed during the ARRA period?
Has ARRA brought low-income weatherization more attention from state and local elected
officials and administrators? If so, has the attention been generally positive or negative?
How has the media portrayed low-income weatherization during the ARRA period?
To what extent have inexperienced and unqualified entities entered the weatherization network
and attempted to reap benefits from the increases in WAP funding?
How have private companies tried to change state and local weatherization procedures to benefit
themselves (e.g., by selling more insulation or energy-efficient lights)?
How have relationships between state and local weatherization agencies changed during ARRA?
How did local non-profit weatherization agencies deal with Program expansion (i.e., what models
did they use and which were more successful than others)?
Did expanding local weatherization agencies result in any economies of scale;
Did ARRA change the way local agencies procured weatherization services under contract (e.g.,
changes in using requests for proposals [RFPs] vs. bids)?
How did the weatherization community (including federal, state, and local stakeholders) interact
with federal agencies during the ARRA period?
(d) Labor Force and Training Issues: As mentioned above, the weatherization labor force necessarily
increased to meet the increase in the weatherization production rate. The following evaluation questions
address how the weatherization community handled this challenge:
What approaches did local agencies and/or contractors use to recruit qualified, reliable, and
trustworthy weatherization crew members, and how effective were these recruitment approaches?
What approaches did states and local agencies use to train the expanded weatherization workforce
and how effective were these approaches?
Did staff turnover and retention rates change during the ARRA period?
How did states manage the creation and training of staff associated with new subgrantees?
How did states and agencies manage increasing workloads and performance expectations?
5
How well did new weatherization staff perform?
How have certification programs changed during the ARRA period?
What are the strengths and weaknesses of national weatherization certification practices?
How have certification requirements changed weatherization staff hiring and retention practices?
3) Non-Energy-Related Benefits: The national evaluation currently being implemented will assess nonenergy benefits associated with WAP for houses under the $2,500 average expenditure level per house. It
is also important to assess non-energy benefits at the higher $6,500 level. Additionally, the greatly
increased scale of the ARRA weatherization effort should also increase the scale of the non-energy
benefits, especially with respect to employment and other economic issues. The following are potential
evaluation questions related to non-energy benefits:
Did utilities experience fewer problems with arrears and shut-offs associated with weatherized
homes because the Program participants‘ utility bills were more manageable as a result of the
increase in measure expenditures and the number of homes weatherized?
In households whose homes were weatherized at the higher level, how much more affordable
were their energy bills?
What other non-energy benefits to households increased at the higher level of expenditures (e.g.,
home value increases, improvements in health)?
Are low-income households whose homes have been weatherized less vulnerable to the impacts
of climate change, and if so, to what extent?
Nationally, how many new weatherization-related businesses were created during the ARRA
period?
Nationally, how many new jobs were created and existing jobs retained during this period?
Did the increased scale of WAP assistance affect local unemployment rates directly? Indirectly?
To what extent have people who received weatherization training under ARRA been able to
transfer their new skills to other sectors of the green economy in particular and the larger
economy in general?
Has WAP during ARRA affected the market for non-low-income weatherization?
Has WAP during ARRA affected the market for building related energy-efficiency products; and
What amount of greenhouse gas emissions was avoided during this period?
4) Indications for Post-ARRA Research Questions: This section poses additional policy-related and
research questions whose answers could benefit the low-income weatherization community in the period
after ARRA. The questions fall into four groups: (a) fundamental Program management and regulation
questions; (b) post-ARRA challenges; (c) technical research questions; and (d) broader research
questions. This evaluation of WAP during the ARRA period will not be able to pursue all of the questions
listed below due to time and budget constraints. In addition, answers to some questions will not be
available until the results from the retrospective and the WAP-ARRA period evaluations can be
synthesized. Lastly, many of the questions are not evaluative questions per se; rather, they represent
important research and analytical questions that should be pursued through other projects. However, these
questions are included in this report to ensure that important points made during the Network Committee
meeting are documented.
(a) Program Management and Regulation Questions: Numerous policy decisions shape the
implementation of the Program ranging from what measures are allowed to be installed in homes to who
is eligible for the Program. Some of these provisions were changed during ARRA. Moving forward,
several policy-related questions such as these could be considered:
Should WAP endeavor to go ―deeper‖ into home-weatherization efforts?
6
What are the benefits of having greater Program flexibility diversity in state and local
weatherization agency administration of WAP?
What are the benefits and costs of various income eligibility thresholds (e.g., 150%, 200% or
more of poverty level)?
Is it possible for homeowners‘ participation in a weatherization program to help forestall home
foreclosure?
Should the policy on re-weatherization be reconsidered in light of Program changes and new
technology developments?
What are the benefits and costs of various average per-house weatherization investment levels
(e.g., $2500, $6500)?
How should certification efforts move forward after the ARRA period?
What are the benefits and costs of using e-learning programs in weatherization training?
Should the fundamental funding mechanism of WAP (block grants) be reconsidered;
How might anticipated retirements in the next five or so years impact the leadership of the
national weatherization community?
Could WAP formally incorporate water conservation into its Program? Should it?
(b) Post-ARRA Challenges: An important issue for the weatherization community is what will happen
after ARRA. States and subgrantees are gearing up to weatherize up to twice as many homes per year as
they have in recent years. The capacity of the national weatherization network is expected to expand
greatly. Will efforts to retain this capacity be made? If so, what might those efforts be at the federal, state,
and local levels? Post-ARRA evaluation questions include the following:
What are states and agencies planning to do, if anything, to maintain their expanded capacities for
weatherization after ARRA?
What options are there at the federal, state, and local levels for marshalling additional resources
to maintain the expanded weatherization capacity?
How many weatherization jobs created during ARRA may be lost after ARRA?
What might the costs to states and local weatherization agencies associated with workforce
reductions (e.g., workers compensation) be?
How can leveraging relationships that were damaged or lost during ARRA because abundant
ARRA money took the place of the leveraging partners‘ involvement during the Program be
rebuilt?
What leveraging opportunities, such as opportunities to tap into voluntary carbon-reduction
markets, might expand after ARRA?
Will quality issues identified, rightly or wrongly, by the media during ARRA have lasting
impacts on weatherization funding?
What level of emission reductions (for carbon and other pollutants) is necessary for the
weatherization program to attract other funding?
Will the training capacity that has been created by states, agencies, community colleges, etc., be
sustainable after the expiration of ARRA?
Will expenses for new equipment, software, etc. purchased during ARRA be sustainable after
ARRA?
What strategies can be used to retain young weatherization staffers hired during ARRA?
(c) Technical Research Questions: The availability of data from two national evaluations of the Program
brings up many interesting research questions whose answers could have an impact on future Program
design. These questions include the following:
7
What lessons about the delivery of weatherization services can be learned through insights gained
from the retrospective and WAP-ARRA evaluations (e.g., through comparing energy savings,
cost-effectiveness, weatherization staff training and retention etc.)?
How long do energy savings attributable to weatherization last?
What are the benefits and costs of using computer audits vs. priority lists?
Are there any differences in energy savings and cost-effectiveness between homes weatherized
using only DOE WAP funds vs. a combination of DOE WAP and LIHEAP funds;
What are the benefits and costs of various new information technologies that allow real-time
reporting of audits and weatherization activities from the field?
Can energy savings attributable to weatherization be estimated or ―normalized‖ without taking
human behavior into account?
What opportunities exist to use random control trial methodologies to evaluate aspects of the lowincome weatherization program;
What are the impacts of weatherization on ―whole-service‖ utility bills;
Have any areas in the United States been ―saturated‖ with WAP-funded low-income
weatherization (i.e., are there any areas where most eligible homes have been weatherized with
WAP-funded assistance)?
Should weatherization decisions take into account local or regional peculiarities of utility rate
structures and other whole billing provisions?
Has the consolidation of the natural gas and electric utility industries impacted low-income
weatherization and/or evaluation?
8
5) Broad Research Questions: Lastly, numerous questions can be asked about the Program in the larger
national context. Suggested questions include the following:
Can weatherization programs be used to increase the market penetration of new energy efficiency
and renewable technologies in the residential sector, whether or not they meet current costeffectiveness standards (i.e., the currently defined SIR)?
What are the prospects for building a low-income weatherization portfolio within the voluntary
carbon markets?
How has the proliferation of historic preservation programs and historic districts affected the
implementation of low-income weatherization?
What might be the impact of the potential Home Score program on low-income weatherization?
How can states and other governmental entities build their own evaluations upon the retrospective
and ARRA period evaluations?
Has the expanded weatherization effort under ARRA transformed markets for any energy
efficient products?
What other weatherization programs serving the low-income community exist in the United
States?
Are the terms ―weatherization,‖ ―green jobs,‖ and ―clean energy‖ confused in the minds of the
public?
1.1.2 Program Logic Model
A program logic model is a tool used to provide complete analysis of a program‘s inputs,
activities, outputs (products), and outcomes. In following the W. K. Kellogg Foundation‘s formalized
evaluation planning process, development of a program logic model is an integral first step before a set of
program evaluation questions within the framework of a design matrix can be formulated. The program
logic model shown in Table 1.1 shows how the WAP is intended to work by systematically identifying
first, the resources available to operate the Program; second, the activities the Program is intended to
perform; and third, the results the Program is intended to accomplish. The program logic model for the
WAP shown in Table 1.1 comprises four sections:
1. Resources/Inputs — The first column identifies the resources (―inputs‖) of various kinds—human,
financial, organizational, and community—available to operate the Program. The first input identified
is the Federal legislation authorizing the Program and stipulating the Program‘s mission and overall
objectives. Financial inputs include direct DOE funding of the Program, funding from other Federal
sources such as the Low-Income Home Energy Assistance Program (LIHEAP), Petroleum Violation
Escrow (PVE) funds, and other leveraged sources, such as state public benefits funds or utility
programs. The organizations involved with the Program include Department of Energy (DOE)
program staff; state grantees and local agency subgrantees that directly implement the Program, along
with their related national organizations; a network of support groups such as DOE‘s national
laboratories, state and regional training centers, and various support contractors; and other
organizations, such as utilities and national and state energy organizations.
2. Activities—The processes, techniques, tools, events, technologies, and actions that the Program
conducts using the resources/inputs are considered its ―activities‖ and are listed in the second column.
The Program‘s primary activities are performed by three groups: DOE, the state grantees, and the
local agency subgrantees. DOE‘s activities focus on administering and running the Program; these
activities involve developing policy, guidance, and regulations; making and monitoring grants;
providing training; maintaining technical capabilities and tools; performing periodic evaluations; and
coordinating with other organizations. The states‘ activities are also administrative in nature, as they
involve making and monitoring contracts with the local agencies, establishing goals and
implementation procedures for the agencies, providing training, and establishing partnerships to
9
leverage resources. Local agencies implement the Program at a basic level, identifying clients and
performing all the tasks needed to select and install weatherization measures. The local agencies also
perform some administrative functions, such as providing client education, referring clients to other
programs and services, and teaching crews the correct procedures needed to perform these tasks.
3. Outputs—The Program‘s outputs are the direct products and services delivered as a result of its
activities. DOE‘s activities result in guidance and regulations being published and audits being
developed, improved, and approved. Through the activities of DOE, the states, and local agencies, a
known number of homes are weatherized, priority households weatherized, weatherization staff
trained, and clients referred to other programs or services. Other important services resulting from the
Program include the installation of cost-effective measures in the weatherized homes, the mitigation
of health and safety deficiencies in these homes, and the education of clients on energy usage.
Through the combined efforts of all organizations, partnerships with the Program are established.
10
Table 1.1. Logic model for the Weatherization Assistance Program
Resources/
Inputs
Roles/Activities
Outputs
Outcomes
Short-Term
Federal authorizing
legislation
Direct funding
from DOE,
LIHEAP, PVE,
and leveraged
sources
DOE Program staff
State grant
administration
agencies and
related national
organizations
Local service
network of 900
agencies and
related national
organizations
Support network in
national
laboratories,
training centers,
and support
contractors with
special technical
skills
Utilities and
national and state
energy
organizations
DOE
- Establish and explain national policy direction
- Formulate annual budgets and grant guidance, and make grants
- Formulate Program rules and regulations
- Initiate and coordinate strategic planning with network
- Approve and monitor state plans and their implementation
- Create, coordinate, and conduct technical training and assistance to state and
local agencies
- Develop and maintain core capabilities of the Program including audit tools
and standards, evaluations, and assessments
- Coordinate Program relations with other Federal agencies, programs, and
institutions
States
- Set eligibility requirements and priorities for participants
- Contract with local agencies and allocate funding
- Establish production goals (number of units weatherized) and schedule
- Specify diagnostic, audit, and inspection procedures and allowable measures
for local agencies
- Determine extent of allowable repair, health, and safety work
- Provide training and assistance to local agencies
- Establish leveraging programs and expand resources and partnerships
- Monitor local agency work
Local Agencies
- Solicit and process applications and select low-income residents to receive
weatherization services
- Train crew members
- Perform home energy diagnostics, audits, and inspections
- Determine most cost-effective weatherization measures and other work needed
for each home
- Purchase, store, and maintain equipment, materials, and supplies
- Install measures and perform other specified work
- Perform quality assurance work
- Meet with clients to review improvements and provide educational materials
- Support advocacy and leveraging
- Link clients to other programs and services
- Track and report client status, expenditures, and funding
11
Number of lowincome homes
weatherized
Number of priority
households
weatherized
Cost-effective
measures installed in
weatherized homes
Health and safety
deficiencies mitigated
in weatherized houses
Clients receive
education on energy
savings
Number of
weatherization staff
trained
Number of clients
referred to social
programs
Guidance and
regulations published
Audits developed,
improved, and
approved
Partnerships
established
Medium-Term
Long-Term
Weatherized
homes,
particularly those
of priority
populations, have
increased energy
efficiency
Reduced energy
consumption in
weatherized houses
Reduced gap
between lowincome energy
needs and actual
consumption of
energy services
Health and safety
of those living in
weatherized
homes improved
Reduced emissions of
pollutants and
greenhouse gases
involved in energy
production and
consumption
Indoor comfort of
those living in
weatherized
homes improved
Clients have
increased
knowledge of
energy savings
strategies
Reduced energy bills
and burdens for
clients
Other non-energy
benefits for clients,
utility rate payers,
and society
Robust
weatherization
network
Increased Program
leveraging
Reduced impact of
energy price
inflation and
market disruptions
on low-income
communities
Improved health
and safety for
communities
Improved local
housing stock
Workforce
enhancement in
local communities
Creation of
sustainable
weatherization
services market
Increased nonenergy purchases
in low-income
communities
Transform market
for weatherization
products
4. Outcomes
4a and 4b. Short- and Medium-Term Outcomes—Program outcomes are those short-term (1–3
months) and medium-term (1-year) changes that occur as a result of the Program‘s activities that impact
the Program‘s participants, participating households, and the Program itself. The immediate results of the
Program are that the energy efficiency of the weatherized homes is increased; that the health, safety, and
comfort of those living in the weatherized houses are improved; and that clients know more about energysaving strategies. In the medium term, energy consumption in the weatherized houses is reduced, leading
to reduced energy bills and energy burdens for the clients as well as non-energy benefits realized by the
clients, utility ratepayers, and society as a whole (especially benefits related to reductions in pollution and
greenhouse gas emissions from reduced energy use). In addition, a more robust weatherization network
community should result and the ability of the program to leverage additional resources should increase.
4c. Long-Term Outcomes—The fundamental, long-term (3–7 years) changes in organizations,
communities, or systems that result from the program‘s activities are its ―long-term‖ outcomes. By
reducing low-income clients‘ energy use and energy burdens, the gap between the energy needs of the
low-income community and the available resources to meet this need should be reduced; in addition, the
low-income community should be less susceptible to rising energy prices and market fluctuations; and
finally, clients should have more funds available to make non-energy purchases within their communities.
Non-energy benefits realized by the community as a result of the program include improved health and
safety, better housing stock, greater job creation, and a more skilled work force. Finally, the program
would be expected to encourage market transformation for weatherization products.
1.1.3 Program Evaluation Design Matrix
The evaluation design matrix shown in Table 1.2 identifies the general questions the program
evaluation will address. These questions were developed by examining the program‘s logic model (see
Table 1.1 and Section 1.1.2) and incorporating the input received from the Network Planning Committee
(see Section 1.1.1). The evaluation questions are organized into three categories in the design matrix:
1. Context: Relationships and Capacity—The context questions explore how the program functioned
within the economic, social, and political environment of the weatherization community; these
questions also address issues regarding the program‘s relationships and capacity. In terms of the
program‘s logic model, the context questions focus on how the program‘s resources and inputs led to
its activities. The evaluation questions dealing with the program‘s context focus primarily on
characterizing the low-income weatherization market, the weatherization network/community and
how it operates, and the partnership and leveraging opportunities available to the program and how
well the program is taking advantage of these opportunities. Context questions also deal with whether
the program has the capacity and structure to fulfill the mission and objectives established for it by
law, and put into context the role the program plays in the larger low-income energy assistance effort.
2. Implementation: Quality and Quantity—Implementation questions assess the extent to which the
activities listed in the program‘s logic model were executed as planned, and whether the outputs listed
in the program‘s logic model were achieved. Implementation questions deal with the characterization
of the clients and households served by the program, the services the program delivered to these
clients and households and how well these services were provided, and the costs associated with
delivering the program. An important implementation question based on the input received from the
Network Planning Committee is to fully determine the best approaches to implementing audits, client
education, training, and technical monitoring. A final implementation question deals with whether the
states and local agencies are fulfilling their obligations under federal regulations and the state plans
they have submitted.
12
3. Outcomes: Effectiveness, Magnitude, and Satisfaction—Outcome questions focus on the extent to
which progress was made toward the desired changes in the program participants, participating
households, and the low-income community and systems. In terms of the logic model, these questions
examine how well the program‘s outputs led to its desired outcomes. The Outcomes questions focus
on the energy savings achieved under the Program, the non-energy impacts that are being realized, the
Program‘s cost-effectiveness, how well individual measures work, and process variables that affect
these outcomes. These outcome questions include Items 1–4 raised by the Network Planning
Committee (see Section 1.1.1). Several final outcomes questions bring all the results of the evaluation
together, asking whether the Program is meeting the legislative missions and objectives identified
previously in the context questions, to what extent the program is meeting the needs of the lowincome weatherization community, and how the program and the weatherization network can be
improved.
Table 1.3 compares the program outcomes identified in the logic model (Table 1.1) to the
program evaluation questions listed in the design matrix (Table 1.2) to make sure that the evaluation is
addressing and measuring all the outcomes associated with the program. As shown by Table 1.3, all the
program outcomes are being addressed by the questions posed in the design matrix with the exception of
the market transformation activity anticipated for the program, which is beyond the scope of this
evaluation.
The evaluation as planned takes and evaluates a snapshot of the program‘s performance as it was
implemented in PYs 2009-2011. The evaluation does not focus most directly on the long-term outcomes,
instead focusing on the short- and medium-term outcomes listed in the logic model (Table 1.1). However,
longer-term outcomes are also being addressed, in some cases by assuming that short- and medium-term
results will have larger impacts as they are sustained over time. The snapshot-type evaluation being
planned does not allow long-term market transformation activity to be evaluated. Although this outcome
could be addressed by looking back in time at how the Program helped transform the weatherization
market, such an effort is not being planned at this time.
In the final synthesis (see Section 5), the evaluation should recommend how a longer-term, more
continuous evaluation of the Program could be implemented by DOE so that the longer-term outcomes of
the program could be more fully addressed. One process that should be explored is to identify other
government programs that are evaluating community and public-welfare issues (e.g., the Health
Department, the Census Bureau) and determine how the program‘s long-term outcomes might be
evaluated from these existing sources.
13
Table 1.2. Evaluation design matrix for the Weatherization Assistance Program
Evaluation focus
area
Context:
Relationships and
Capacity
Question
1. What are the mission and associated
objectives of the Program as established
by law?
2. Does the Program have the capacity
and structure (e.g., funding, staffing) to
meet its objectives?
3. What are the characteristics of the
national low-income weatherization
market?
4. Which segments of this market are
being served by the Program and other
parties?
5. What organizations are involved in
national low-income weatherization (e.g.,
agencies, states, utilities, private sector
firms)?
6. What are the characteristics of the
weatherization network?
7. How does the weatherization network
work?
8. What are the core leveraging and
partnership opportunities for the
Program?
9. Is the Program exploiting its leveraging
and partnership opportunities?
10. Are the Program‘s regulations
enhancing and/or inhibiting leveraging
and partnership opportunities?
Audience
Information use
Study
DOE – WAP
Weatherization network
Establish mission context
DOE – EERE
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
Program administration
Strategic planning; Program
design and marketing
Impact
Assessment
DOE – WAP
Weatherization network
Strategic planning; Program
design and marketing
Impact
Assessment
White House
Congress
DOE – Secretarial
DOE – EERE
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
Establish Program context;
Program support and
marketing
Process
Assessment
Strategic planning; Program
design and marketing
Organization and participation
decisions
Program design and marketing
Impact
Assessment
Process
Assessment
Synthesis
DOE – WAP
Weatherization network
Congress
DOE – Secretarial
DOE – EERE
DOE – WAP
Weatherization network
Program design and marketing
Process
Assessment
Process
Assessment
14
Program design
Process
Assessment
Synthesis
Table 1.2. Evaluation design matrix for the Weatherization Assistance Program
Evaluation focus
area
Implementation —
Quality and
Quantity
Outcomes —
Effectiveness,
Magnitude, and
Satisfaction
Question
Audience
Information use
Study
1. What are the characteristics of those
receiving Program services?
DOE – WAP
Weatherization network
Program design, planning, and
implementation
Impact
Assessment
2. What Program services are being
delivered to low-income households?
3. How well is the Program delivering its
services, including from the client
perspective?
4. What are the costs associated with the
Program services?
5. What are the best approaches to
implementing audits and measure
selection tools, client education, training,
and monitoring?
6. Are the Program‘s characterization and
process results valid and reliable?
7. Are the states and local agencies
fulfilling their obligations under federal
regulations and state plans?
1. What are the Programs average energy
benefits (heating, cooling, and baseload)
nationally and by climate region, housing
type, and fuel type?
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
Program design, planning, and
implementation
Program design, planning, and
implementation
Impact
Assessment
Process
Assessment
DOE – WAP
Weatherization network
Weatherization network
Program design, planning, and
implementation
State- and agency-level
Program design, planning, and
implementation
Impact
Assessment
Process
Assessment
Evaluation community
Generalize results to other
contexts
Program design, planning, and
implementation
Peer
Review
Synthesis
Budget justification; Program
marketing; utility business
planning; rule making
Impact
Assessment
Energy savings and GPRA
metrics
Program marketing; utility
business planning; rule
making
Impact
Assessment
Impact
Assessment
DOE – WAP
Weatherization network
OMB
DOE – Secretarial
DOE – EERE
DOE – WAP
Weatherization network
Utilities
Commissioners
2. How much energy is saved in aggregate DOE – EERE (PBA)
by the Program?
DOE – EERE
What are the Program‘s non-energy
DOE – WAP
impacts?
Weatherization network
Utilities
Commissioners
15
Table 1.2. Evaluation design matrix for the Weatherization Assistance Program
Evaluation focus
area
Question
Audience
Information use
Study
4. How do clients feel about the
Program‘s impact on their comfort, health
and safety, and energy costs?
5. Is the Program cost-effective?
DOE – WAP
Weatherization network
Program design, planning, and
implementation
Impact
Assessment
White House
Congress
OMB
DOE – Secretarial
DOE – EERE
DOE – WAP
Weatherization network
Cost-benefit analysis; future
funding decisions; Program
design, planning, and
implementation
Impact
Assessment
State and agency-level
Program design, planning, and
implementation
Impact
Assessment
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
Program design, planning, and
implementation
Program design, planning, and
implementation
Impact
Assessment
Impact
Assessment
DOE – WAP
Weatherization network
Program design, planning, and
implementation
Evaluation community
Generalize results to other
contexts
Budget justification; Program
marketing
Impact
Assessment
and
Special
Technical
Studies
Peer Review
6. What impact do alternative per
household investment levels (e.g., $2500
vs. $6500) have on key Program metrics
(e.g., units weatherized, average savings
per house, house and Program SIRs)?
7. How well do the selected measures
result in energy savings
8. What factors and measures explain
variation in energy savings and costeffective results?
9. How are the hot Southern climate
region market and performance unique?
10. Are the outcome estimates valid and
reliable?
11. Is the Program meeting its legislative
missions and objectives?
DOE – WAP
Weatherization network
16
Synthesis
Table 1.2. Evaluation design matrix for the Weatherization Assistance Program
Evaluation focus
area
Question
12. How much have the emissions of
greenhouse gases been reduced?
13. How well have expanded
weatherization activities in the U.S.
territories succeeded?
14. What are the impacts and process
outcomes of the SERC and WIPP
projects?
15. To what extent is the Program
meeting the needs of the national lowincome weatherization market?
16. In what ways can the weatherization
network‘s performance be improved?
Audience
Information use
Study
White House
Congress
OMB
DOE – Secretarial
DOE – EERE
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
Cost-benefit analysis; future
funding decisions; Program
design, planning, and
implementation
Special
Studies
Budget justification; strategic
planning
Special
Studies
White House
Congress
OMB
DOE – Secretarial
DOE – EERE
DOE – WAP
Weatherization network
DOE – WAP
Weatherization network
Cost-benefit analysis; future
funding decisions; Program
design, planning, and
implementation
Special
Studies
Budget justification; strategic
planning
Synthesis
DOE – WAP
Weatherization network
Program planning
Synthesis
17
Table 1.3. Design matrix questions addressing each logic model outcome
Number of questions in the evaluation design matrix
related to context, implementation, and outcomes
(Table 1.2)
Outcomes listed in the logic model
(Table 1.1)
Context
Implementation
Outcomes
Short -Term Outcomes
1. Increased energy efficiency in homes
4
1, 2
2. Improved health and safety in homes
3, 4
3. Improved indoor comfort
3, 4
4. Increased client knowledge of energy
5
Medium -Term Outcomes
1. Reduced energy use in homes
2. Reduced bills and financial burden for
clients
3. Reduced emissions
7
1, 2, 8
7, 14
1, 2, 4, 8
3, 12, 14
4. Other non-energy benefits
3, 4
5. Robust weatherization network
2, 5, 6, 7
6. Increased Program leveraging
8, 9, 10
13
13, 16
Long-Term Outcomes
1. Reduced gap between energy need and use
1, 2, 4, 11, 15
2. Reduced impact of inflation/market
fluctuations
3. Improved health and safety in community
1, 2, 4, 15
3, 4
4. Improved local housing stock
3, 4
5. Workforce enhancements
3, 14
6. Creation of sustainable weatherization
service market
7. Increased non-energy purchases
2, 4-11
14,15
3
8. Transformed market for weatherization
products
Will not be addressed in this evaluation
18
1.2 EVALUATION ORGANIZATION
Based on a review of the evaluation design matrix (see Section 1.1.3), the evaluation of the
Program will include four studies, identified in Table 1.2, which will address each of the questions listed
in the evaluation design matrix. These studies and the organization of the remaining elements of the
report are outlined below:
Impact Assessment—Section 2 describes the plan for evaluating the Program‘s impact in PYs 20092011. The weatherization network will be characterized, along with the nature and scope of the
Program‘s implementation and weatherization processes. Energy and its subsequent costs savings will
be quantified, along with non-energy impacts in order that the Program‘s cost-effectiveness can be
determined. Explanatory factors pertinent to energy savings, energy costs savings, and costeffectiveness will be identified.
Process Assessment—Section 3 describes a process assessment that will examine how well the
weatherization network and Program operated in during the ARRA period in delivering
weatherization services, and how well the Program is exploiting opportunities for leveraging and
partnership. Case studies of weatherization programs in territories will be performed. The national
weatherization network will be approached to assess plans post-ARRA.
Special Studies—Section 4 describes special studies that will be performed. These studies include an
analysis of underperforming weatherized units; energy savings analyses for selected U.S. territories;
SERC; and WIPP. To explore the feasibility of employing random control trial methods to estimate
energy impact savings, an encouragement design study will be conducted. Lastly, an in-depth study of
greenhouse gas emission reductions and further potential reductions at the local level will also be
conducted.
Synthesis Study—Section 5 describes how results from the evaluation work performed under
Sections 2–4 will be synthesized to address how well the program is meeting its overall goals, the
extent to which the program is serving the weatherization needs of the low-income community; and
how the program‘s and weatherization network‘s performance can be improved. Lessons learned
from both the retrospective and ARRA period evaluations will be identified.
Schedule—Section 6 outlines a schedule for the evaluation.
It should be noted that under the terms of the Paperwork Reduction Act, the Office of
Management and Budget (OMB) must approve most of the sampling plans and survey instruments
associated with this evaluation. Therefore, the sampling plans and survey instruments presented in this
preliminary evaluation plan may be modified during the OMB review process. However, once approved
by OMB, they will not be subject to any substantive modifications.
1.3 COMPARISON OF WAP RETROSPECTIVE AND ARRA PERIOD EVALUATIONS
There are several important points to make regarding the similarities, differences, and overlaps
between the retrospective evaluation of WAP as described in Ternes et al. (2007) and the WAP-ARRA
evaluation described in the balance of this report. First, as Table 1.4 shows, the two evaluations generally
encompass the same research tasks. For example, the central component of each evaluation is the
collection of billing histories for homes heated with electricity and natural gas; these data provide the
basis for the national estimate of energy savings attributable to the program as well as associated costeffectiveness analyses. Non-energy impacts are assessed by both evaluations. Additionally, both
evaluations administer a core set of surveys and data forms (e.g., S1: All States Program Information
Survey). All of these tasks focus on the Program Years (PYs) indicated in the Table 1.4.
19
The third column of Table 1.4 shows that the retrospective evaluation overlaps with the ARRA
period in five research areas. When the retrospective evaluation was conceived in 2005 and when the plan
was written in 2006 and 2007, it was assumed that the Program would not undergo any major changes
that could change the evaluation results. Thus, the retrospective evaluation was designed to estimate
national energy savings and to constitute most of the process assessment for the immediately past
Program Year (PY2008) while simultaneously implementing several research tasks during the Program
Year in which the evaluation was to take place (i.e., PY 2009). When the retrospective evaluation began,
though it was decided to look back to pre-ARRA Program Years 2007 and 2008, the ARRA period had
already begun.
Thus, these five research tasks that are being funded by the retrospective evaluation are actually
assessing weatherization activities that took place during the ARRA period: analysis of sub-metered data
for homes heated with bulk fuels; a major indoor air-quality study; case studies of high-performing
agencies and exemplary client-education programs; and the administration of two major surveys, S4
(Occupant Survey) and S5 (Weatherization Staff Survey).
Table 1.4 also indicates that a few of the tasks undertaken by the retrospective evaluation will not
be duplicated by the WAP-ARRA period evaluation (and vice versa). For example, the retrospective
evaluation will fund case studies of high-performing agencies and exemplary client-education programs,
but the WAP-ARRA evaluation will not. Conversely, the WAP-ARRA period evaluation will fund case
studies of underperforming weatherized units and one U.S. territory that received new, substantial ARRA
funding (e.g., Puerto Rico), which, naturally, is not covered in the retrospective evaluation.
20
Table 1.4 Comparison of WAP Retrospective and WAP-ARRA Period Evaluations
Retrospective Evaluation
on Weatherization
Assistance Program
Retrospective Evaluation
Research Taking Place
during (Overlapping with)
ARRA Period
WAP-ARRA Period
Evaluation Research
Analysis of Billing Histories:
Homes heated with electricity
and natural gas
Analysis of Submetered Data:
Homes heated with propane
and fuel oil
Analysis of Persistence of
Energy Savings in
Weatherized Homes
Cost-effectiveness Analyses
Non-Energy Impacts
Social Network Study
GHG Emissions Study
Program Years 2007 and
2008
Program Years 2009,
2010, and 2011
S1: All States Program
Information Survey
S2: All Agencies Program
Information Survey
S3: Subset of Agencies
Detailed Program Information
Survey
S4: Occupant Survey
S5: Weatherization Staff
Survey
Program Characterization
Case Studies
Program Year 2008
Program Year 2010
Program Year 2010
Program Year 2011
Program Years 20072010
Program Year 2010
Program Year 2008
Program Year 2010
Program Year 2008
Program Year 2010
Winter 2010-2011 and
Winter 2011-2012
Program Years in the
1990s
Program Year 2008
Program Year 2008
CY‘s 2011 and 2012
CY 2011
Program Year 2008
Program Year 2008 – Six
High Performing Agencies
and Six Exemplary Client
Ed Programs
Weatherization Deferral Study
Under-Performers Study
DF2/3: Housing and Building
Information Data Forms
DF4: Utility Information from
Agencies Data Form
DF5: Utility Billing History
Data Forms
Indoor Air Quality
CY2012
CY 2012
Program Year 2010
Program Year 2010 –
One U.S. Territory
Program Year 2008
Program Year 2010
Program Years 20072009
Program Year 2010
Program Years 2007 and
2008
Program Years 2007 and
2008
Program Years 2009,
2010, 2011
Program Years 2009,
2010, 2011
Winter 2010-2011,
Summer 2011, Winter
2011-2012
CY 2011
Field Process Study
Special Studies
WIPP and SERC
Program Years 20102011 ;
Encouragement Design
– Program Year 2011
CY 2011
Post ARRA Surveys (S6,7,8)
21
22
2. IMPACT ASSESSMENT
The impact assessment portion of the evaluation will address many of the questions identified in
the evaluation design matrix (see Table 1.2), especially those dealing specifically with the following:
Context—Questions 3, 4, and 6;
Implementation—Questions 1, 2, and 4; and
Outcomes—Questions 1–9.
The context, implementation, and outcomes questions listed above deal with characterizing the
weatherization network, the market that the Program serves, and the households served by the Program;
identifying the services delivered by the Program and their costs; determining the Program‘s energy and
non-energy benefits and cost-effectiveness; and understanding factors that have an impact on savings,
cost-effectiveness, and other key Program metrics, such as the number of units weatherized.
In addition, the impact assessment will address the following high-priority and consensus goals
that were identified by the Network Planning Committee:
Energy savings analysis—reports energy savings by various subgroups and includes measured
savings from propane and fuel-oil heated houses in the evaluation;
Baseload measures—includes savings for all end uses in the measured savings from the
Program;
Non-energy impacts—quantifies non-energy impacts produced by the Program;
Cost-effectiveness—determines the impacts of alternative per-household investment levels on
cost-effectiveness; and
The impact assessment will be performed by executing five integrated studies, focusing on the
performance of the Program in Program years 2009-2011:
1. The Program Characterization study will characterize the low-income population eligible for and in
need of the Program and, for PY 2010, characterize the segment of this population served by the
Program; the housing units and clients served; the weatherization and other services performed by the
Program; and the Program‘s expenditures and funding sources.
2. The Energy and Costs Savings study will establish the total and per-household energy and cost
savings (heating, cooling, and baseload) being achieved nationally and by climate region under the
Program in Program years 2009-2011, classified by the principal building types served and primary
fuel types used.
3. Non-Energy Impacts study will ascertain the non-energy impacts attributable to the Program in PYs
2009-2011 (especially those benefits addressing health and safety) and the value of those impacts
from the client, utility, and societal perspectives;
4. The Program Cost-effectiveness study will estimate the cost-effectiveness of the Program in PYs
2009-2011 on a national and climate-region basis and will seek to clarify the impact that alternative
per-household investment levels can have on cost-effectiveness and other Program metrics; and
5. The Explanatory Factors study will identify how specific weatherization measures and process
variables correlate, both positively and negatively, with energy savings and cost-effectiveness.
Each of these studies is described in detail below, including an outline of the data that need to be
collected to perform the study and how these data will be analyzed. For each study, a final report
including all the details of the study will be written; a final summary report for the impact assessment will
also be written to draw all the findings from the studies together.
23
2.1 PROGRAM CHARACTERIZATION STUDY
As mentioned in the introduction, the WAP during the ARRA period has been very different from
the Program as it existed in the past. Not only are several key Program guidelines different (for example,
the average allowable investment in homes was increased from $2500 to $6500), but the grantees and
subgrantees faced numerous challenges with ramping up the weatherization production, complying with
the Davis-Bacon Act, and handling other issues unique to ARRA. The impact assessment will collect key
data on the Program‘s implementation and weatherization processes in order to describe the following:
the low-income population eligible for the Program, in need of it, and actually being served by it;
the weatherization network (community) and how it works; especially the organizations that
administer the Program at the state and agency level (e.g., organization features and structure,
staffing, operational processes, funding levels);
the housing units that are served (including descriptors of their condition, state of repair, health
and safety issues with respect to them, and the types of heating and cooling equipment installed),
the clients served by the Program, and how they were selected for inclusion in it;
the types of audit and diagnostic procedures used on the houses, the time when the diagnostics
were performed relative to when measures were installed, and by whom the diagnostics were
performed (e.g., auditor, crew, or inspector);
the weatherization measures installed in the weatherized units (including repairs made, health and
safety issues addressed, and client education provided), the installation methods employed, and
by whom the measures were installed (contractor vs. in-house crew);
other Program services performed on the weatherized houses and how they were delivered; and
the Program‘s expenditures, expenditures per household, and funding sources.
The data to be collected and the analysis to be performed for the characterization study are presented
below.
2.1.1 Data and Sampling Frames
The eligible low-income population will be characterized using data from the following three national
databases:
Residential Energy Consumption Survey (RECS) conducted by the U.S. Energy Information
Administration
Current Population Survey (CPS) from the U.S. Census
American Housing Survey (AHS) from the U.S. Department of Housing and Urban
Development.
A list of the data fields to be mined from these databases is provided in Table 2.1 below.
The entities that received WAP-ARRA period funding--all 50 states, the District of Columbia,
and U.S. territories--will be asked to complete the S1 (All States Program Information Survey, see
Appendix C) at the end of their PY 2010. As part of this survey, the following information on their PY
2010 and 2011 activities will be obtained from all states:
general information on the characteristics of each state
details on PY 2010 funding and expenditure
characteristic data compiled at the state level on housing units weatherized in PY 2010
characterization data on state staff experience and activity in PY 2010
characterization data on training and monitoring performed at the state level in PY 2010.
24
All of the approximately 1000 agencies2 (i.e., subgrantees) that have been or are being employed
to implement the Program will be surveyed at the end of PY 2010, using the S2 (All Agencies Program
Information Survey, see Appendix D), to collect information on
PY 2010 funding and expenditure details
agency-level compiled characteristic data on housing units weatherized in PY 2010.
Although agencies supply similar information to their respective states, this information will be
collected from the agencies, not from the states, in order to get the information directly from the original
source and to make sure the data are accurate and consistent across all states and agencies.
The 400 agencies included in the billing data sample (see Section 2.2.1) will be surveyed in the
S3 (Subset of Agencies Detailed Program Information Survey, see Appendix E) at the end of their PY
2010. The following information will be obtained:
general characteristic information on each agency,
data characterizing agency staff experience and activity in PY 2010,
data characterizing how the agencies implemented client selection in PY 2010, and
data characterizing house audits, client education, training, and monitoring performed at the
agency level in PY 2010.
2
The number of local weatherization agencies increased to over 1000 during the ARRA period, from just over 900
in the pre-ARRA period.
25
Table 2.1. Data fields for this study from RECS, CPS, AHS databases
Data field
Low-income status
State
Census region
Housing type
Tenure
Primary space-heating fuel
type
House energy features
Children
Elderly
Handicapped
Single parent
Ethnicity
Income
Source of income
Nature of income
Energy consumption
Energy expenditures
Energy burden
Participation in public
assistance programs
Definitions of data field terms
Defined by ARRA eligibility maximums (i.e., 200% of poverty level or 60% of
state median income, whichever is higher)
Ownership
Presence/absence of wall insulation, storm windows, etc.
Presence of at least one child in household as defined by Program regulations
Presence of at least one elderly person in the household as defined by Program
regulations
Presence of at least one handicapped person in the household as defined by
Program regulations
Total household income
Fixed or not
Total, heating, cooling, and baseload that are nominal and weather-adjusted
CPI adjusted; high energy expenditures as defined by Program regulations
Calculated from income and energy expenditures, with ―high‖ energy burden as
defined by Program regulations
LIHEAP, Food Stamps, Temporary Assistance for Needy Families (TANF),
Section 8, Public Housing, Medicaid, Supplemental Security Income (SSI)
For each weatherized housing unit or building included in the billing data sample (see Section
2.2.1), using the DF2 (Housing Unit Information Data Form, see Appendix F) or the DF3 (Building
Information Data Form, see Appendix G), the following data will be collected from the agencies:
detailed housing unit/building and occupant characteristics,
identification of the diagnostics performed,
diagnostic data measured by the agencies,
identification of the measures installed, and
costs for measures installed and other work performed.
The billing data sample includes data only on those housing units or buildings that use
natural gas or electricity as their primary heating fuel. In order to fully characterize all housing units and
buildings served by the Program (not just those heated by natural gas or electricity), information will be
collected from the same 400 agencies used in the billing data sample on 25% of the housing units and
buildings from each agency whose primary heating fuel is NOT natural gas or electricity. These data will
be collected using DF2 (Housing Unit Information Data Form, see Appendix F) or DF3 (Building
Information Data Form, see Appendix G).
The data requested in DF2 and DF3, above, are typically maintained in the records of each
agency, so no additional information will need to be collected by the agencies. Agencies that store these
data electronically will likely be able to provide it on all the housing units and/or buildings they
weatherize rather than the 33% sample required for units and buildings heated by natural gas or electricity
26
(see Section 2.21.) or the 25% sample required for units and buildings heated by other fuels. These data
will be collected just after the end of the agency‘s PY 2010.
2.1.2 Low-Income Weatherization Market Analysis
To get a broad picture of the low-income weatherization market, descriptive statistics on key
attributes of the eligible population will be developed using data from RECS, CPS, and AHS. Households
with incomes of 60% or less of their state‘s median income will be the focus of this analysis. The entire
low-income population will be characterized, as well as the five subsets of the population allowed by
DOE to receive priority service: households with elderly, children, or handicapped; and houses with high
energy expenditures or high energy burdens. Other subsets of houses that may be studied separately if
there are sufficient data in the databases include ―low-efficiency‖ houses (e.g., houses with no attic
insulation), houses of people on fixed incomes, and/or houses whose occupants receive a majority of their
income from Social Security. The key attributes that will be studied include the following:
housing characteristics (housing type, tenure),
type of primary heating fuel,
demographics (elderly, children, handicapped, single parent, and ethnicity),
income,
energy usage (total, heating, cooling, and/or baseload),
energy burden,
energy expenditures, and
participation in other public-assistance programs.
These attributes will be presented nationally and by climate region in terms of means, medians,
distributions, and other characteristics. They will also be cross-tabulated by other key attributes.
Comparisons will be made between the low-income population and the national population, and among
the findings from this evaluation, the retrospective evaluation, and the 1989 National Evaluation in order
to identify changes since 1989.
A literature review will be conducted to explore the impact of energy expenditures on households
eligible for the Program as well as on households with higher incomes that might also have difficulty
paying their energy bills. This literature review will examine the issue of energy affordability across
different income categories and will provide a description of the population in need of assistance in order
to place the objectives of the Program in their appropriate context.
Using data collected from all states and agencies nationwide via the web-based survey, all the
units weatherized by the Program in the program year will be characterized by the following key
attributes:
classification as DOE or non-DOE units,
housing type,
primary heating fuel,
tenure,
climate region,
participation in other federal assistance programs,
income,
ethnicity,
single-parent, and
priority traits of occupants and houses for weatherization.
27
These attributes will be presented nationally and by climate region in terms of means, medians,
distributions, and cross-tabulations with other key attributes. The results will then be compared to the
characterization of the eligible population to identify the segments of the eligible population and eligible
housing stock being served by the Program. Results will be presented in relative percentages and
proportions nationally and by climate regions.
2.1.3 State and Agency Characterization Analysis
Local and state agencies will be characterized by key attributes, including the following:
agency type and size,
funding (both DOE and non-DOE),
how funding is allocated by function (e.g., intake, auditing, training, weatherization, quality
assurance monitoring),
number of units weatherized (total and by funding source, with tagging to avoid duplicated
counts),
number of units on a waiting list,
number of units referred to other programs,
number of units receiving on-site services from non-energy programs, and
number of staff/employees by role, tenure, training, experience, and those needing certification.
The scope and scale of agency involvement with other energy, housing, and low-income
programs will be characterized and described. The location and status of the state agencies administering
the Program within their state government organizations will be described, and the relationship of the
state agencies to other energy, housing, and low-income programs will be characterized and described.
Descriptive statistics will be presented nationally and by climate region in terms of totals, means,
medians, and distributions, as appropriate.
2.1.4 Detailed Characterization of Program and Analysis of Implementation
The approaches used to select clients, audit houses, provide client education, train crews and
agencies, and monitor agencies will be thoroughly characterized as part of the in-depth analysis to be
performed on audits and client education under the impact assessment portion of this evaluation. Results
from these characterization analyses will be used and integrated with the other characterizations being
described in this section.
The client-selection process will be characterized by the outreach and marketing methods that are
used to get clients to apply for the Program (i.e., how a waiting list is developed) and the methods used to
select clients for weatherization from among the qualified applicants (i.e., from clients on a waiting list).
These characterizations will be organized nationally and by climate region.
Using the detailed data collected on the housing units that will be used in the energy analyses, the
houses and occupants weatherized under the Program will be further characterized by key attributes,
including:
building characteristics (e.g., building type, tenure, floor area, age, number of stories,
condition/state of repair, health and safety problems present),
equipment characteristics (e.g., primary and supplemental heating fuels, central heating system,
air conditioning type), and
occupancy characteristics (e.g., number of occupants; number of children, elderly, and/or
handicapped; income; energy burden).
28
The key attributes will be characterized nationally and by climate region, primary heating fuel,
and dwelling type. Distributions will be examined and reported as appropriate. Results will be integrated
into the market analysis described in Section 2.1.2.
The frequency with which various weatherization measures are installed in houses under the
Program will be reported and classified into eight major categories: air and duct leakage, insulation,
window and doors, space heating equipment, cooling equipment, baseload, client education, and health
and safety/repair. Subcategories will further refine these categories and will note different implementation
approaches. For example,
the insulation category will be broken down into attic, wall, and floor insulation, and wall
insulation will be further divided into high-density and standard installation techniques;
the baseload category will be broken down into specific water-heater measures, lighting, and
refrigerators;
the client education category will be divided by different client-education approaches; and
health and safety/repairs will be classified as replacements of roofs, floors, doors, and windows;
installation of smoke and CO detectors; electrical-system repairs; replacement of unsafe space
heaters; replacement of broken air conditioners; and plumbing repairs.
These frequencies will be reported nationally and by climate region, and by primary heating fuel,
dwelling type, and other subgroups. The frequency with which different measures are installed by
contractors vs. crews will also be tabulated.
The frequency with which diagnostic techniques are used in weatherized houses will be reported
for the various techniques (e.g., blower doors, infrared cameras). These frequencies will be reported
nationally and by climate region, as well as by primary heating fuel, dwelling type, and other subgroups.
For each technique, frequencies will also be tabulated on when the diagnostics were performed (e.g.,
during the audit, at time of measure installation, or during final inspection) and who performed the
diagnostics (e.g., auditor, crew member, or inspector).
2.1.5 Program Funding and Costs Analysis
Using agency- and state-level data, financial resources used for weatherization at the local level
(both DOE and leveraged, non-DOE) will be characterized, as well as how those resources are combined
to weatherize individual units by unduplicated counts (i.e., units weatherized with funds from more than
one funding source will not be double counted). These data will be presented nationally and by climate
region. Performance requirements for non-DOE funding sources will be analyzed to determine how these
compare and relate to requirements for the DOE program. A similar analysis has recently been performed
for the Program by Economic Opportunity Studies (Power, 2003). This section will expand that study‘s
analysis outside its formal evaluation budget form to meet the present needs of this evaluation.
Using house-level cost data collected for the energy savings analyses, the average installation
costs (labor plus materials) per house will be determined nationally and by the following:
climate region,
building type,
fuel type,
tenure (ownership),
type of installer (contractor or in-house crew),
funding source, and
possibly other categories.
29
Distributions will be examined and reported as appropriate. Prices paid for materials and
measures will be assessed against market rates. Average per-house labor and material costs will be
examined in a similar manner, as will material costs for individual measures (labor costs per individual
measure will only be studied if consistent, high-quality data can be obtained from agencies; however, the
availability of such data is not anticipated).
2.2. ENERGY AND COST SAVINGS
A major task for the impact evaluation is to estimate energy savings attributable to WAP in
homes heated by natural gas and electricity; this study implements a quasi-experimental approach. It is
understood that, all things being equal, a random control trial (RCT) approach would be preferred over a
quasi-experimental approach. However, there are compelling reasons, explained in this subsection, why a
quasi-experimental design has been chosen instead.
The retrospective study, which was also designed quasi-experimentally, was based on the
WAP evaluation conducted two decades ago (when the last national evaluation of the Program was
conducted). The quasi-experimental design for the retrospective evaluation implemented probabilityproportional-to-size sampling (PPS) to subsample 400 subgrantees (out of ~900) (discussed in more detail
below). Sampling one-third of the units weatherized by these agencies in a targeted program year (e.g.,
PY 2008) yields a treatment sample size of approximately 10,000 units (out of approximately 100,000
WAP weatherized per year pre-ARRA). The approach calls for an equal number of units to be in the
control group, to be drawn from homes weatherized during the following program year.
This choice of control group is reasonable because this group, like the treatment group, has selfselected to apply for weatherization services, and the two groups are likely to be similar in all variables
correlated to energy use. (Historically, WAP has only served a small percentage of eligible homes
[100,000 homes per year vs. a potential pool of approximately 35 million] and the observed homes going
through the Program have had quite similar and constant characteristics for many years.
According to a recent GAO report, ―program evaluation literature generally agrees that wellconducted randomized experiments are best suited for assessing effectiveness when multiple causal
influences create uncertainty about what caused results.‖3 The GAO report goes on to note, however, that
randomized experiments ―are often difficult, and sometimes impossible, to carry out,‖ and that ―requiring
evidence from randomized studies as sole proof of effectiveness will likely exclude many potentially
effective and worthwhile practices.‖4 When randomized studies are impractical or impossible to carry out,
quasi-experimental (QE) comparison group studies satisfactorily provide ―rigorous alternatives to
randomized experiments.‖ For legal and practical considerations, we believe that a classical randomized
control trial (RCT) approach cannot be implemented to evaluate WAP during the ARRA period.
Additionally, the WAP is administered by States (i.e., grantees) through subgrantees who must
prioritize WAP applicants in order to select them. The primary barrier to randomization in a WAP
evaluation is in fact legislative priority constraints on how the subgrantees should prioritize WAP
applicants. From the U.S. Department of Energy, Weatherization Assistance Program for Low-Income
Persons, Title 10, Part 440 (Direct Final Rule, Federal Register, June 22, 2006) 5:
3
―Program Evaluation: A Variety of Rigorous Methods Can Help Identify Effective Interventions,‖ GAO-10-30,
November 2009.
4
Ibid.
5
See http://www.waptac.org/sp.asp?id=1812#minimum.
30
Section 440.16 Minimum program requirements…(b) Priority is given to identifying and
providing weatherization assistance to:
(1) Elderly persons;
(2) Persons with disabilities;
(3) Families with children;
(4) High residential energy users; and
(5) Households with a high energy burden.
Thus, Title 10, Part 440 essentially prohibits the purely random assignment of WAP applicants to control
groups, meaning that the RCP approach is not possible.
In conjunction with Title 10, Part 440, there is also a practical and perceived moral obligation
among subgrantees to provide services to all applicants—and particularly to high-priority applicants—as
fairly and expediently as the Program will allow. This institutional resistance to random assignment to
and the consequential delay of service to control groups would have to be overcome before an RCT could
be correctly implemented. Changing existing practices would mean that the evaluation team would need
to vigorously engage DOE WAP management and all the grantees and subgrantees to convince them all
to change Program management processes to fit the needs of an RCT evaluation. This task is beyond the
responsibilities of the evaluation team and would be virtually impossible to implement in time, given that
we are already well into the ARRA period.
Despite the barriers to a classical RCT approach to WAP evaluation, we consider in Appendix S a
hypothetical ―split-winter‖ RCT that could be conducted in conjunction with the QE WAP evaluation
study. By ―split-winter,‖ we mean that weatherization would be performed during one particular winter,
and the total duration of the study would generally be less than in a full evaluation. Sample-size
calculations for the split-winter RCT suggest that, even if the legislative and cultural barriers could be
circumvented, this alternative RCT approach would still not be a good idea because of very large (and
therefore expensive) sample-size requirements. This further supports the assumption that an RCT is not
feasible in the WAP-ARRA context and that a carefully conducted QE study is a better approach.
The quasi-experimental approach that has been decided upon for this analysis includes these
components:
estimating variation in billing history data;
estimating treatment and control group sample sizes;
determining how many subgrantees (i.e., local weatherization agencies) need to be sampled;
asking the subgrantees to provide lists of units weatherized during the program year under study;
asking the sampled subgrantees to provide lists of units weatherized during the following
program year to act as a control group;
sampling units from these two lists to identify units for which billing histories will be collected;
and
contacting the appropriate natural gas and electricity utilities to collect the billing histories, one
year pre- and one year post-weatherization.
Adopting the quasi-experimental design approach, the evaluation will focus on estimating the
following two aspects of energy and cost savings:
total annual energy savings achieved from all units weatherized by the Program in PYs 2009,
2010 and 2011 (all fuels combined—natural gas, fuel oil, propane, electricity, etc.—representing
a combination of all space-heating, cooling, and baseload energy uses in the houses); and
31
average annual energy savings (calculated separately by electricity savings and energy savings for
all non-electric primary space-heating fuels combined) achieved per household in PYs 2009,
2010 and 2011 nationally and by climate region, housing type, primary space-heating fuel type,
and the five client groups that the Program is specifically instructed to focus on (i.e., the elderly,
persons with disabilities, families with children, high residential energy users, and households
with high energy burden).
The cost savings associated with the above energy savings will be calculated using regionallydependent fuel costs; the estimated energy and cost savings for PYs 2009, 2010 and 2011 will be
compared to results from the retrospective evaluation, the 1990 National Evaluation, and the metaevaluations performed between 1990 and 2005. Although average energy and cost savings will be
calculated in this study by region, housing type, and primary space-heating fuel type, a full analysis of
factors affecting energy consumptions and savings will be performed (see Section 2.5).
Energy savings will be estimated based on a sample of housing units or buildings selected from
each state in the nation. Billing data will be collected and analyzed on a majority of the housing
units/buildings sampled. Energy savings estimated for individual housing units or buildings will be used
to develop national estimates. Details are provided below.
2.2.1 Sampling Frame and Data
For the retrospective sampling approach, the following information were needed to design and
implement the energy analysis sampling frames described in this section:
DOE funding received by each agency from ARRA;
identification of agencies that weatherize a significant number of the following types of units:
large multifamily units, large multifamily buildings heated by fuel oil, single-family houses
heated by fuel oil, single-family houses heated by propane, or mobile homes heated by propane;
and
identification of agencies whose housing units or buildings are served by natural gas and electric
utilities that will be cooperative in providing billing data for the evaluation.
This information will be collected as part of S1 (All States Program Information Survey, see
Appendix C) administered to all states and territories and S2 (All Agencies Program Information Survey,
see Appendix D) administered to all agencies just after the end of PY 2010.
Billing Data Sample— For the retrospective evaluation, natural gas and electricity billing data
were collected on a sample of the single-family houses, mobile homes, and both small and large
multifamily housing units that were weatherized by 400 agencies in PYs 2007 and 2008. For each agency,
all units whose primary heating fuel was natural gas or electricity were sampled if utility account data and
other information can be easily provided electronically by the agency; otherwise 33 percent of such units
will be sampled (total number of units is estimated to be approximately 10,000). Natural gas billing data
were collected on those units whose primary heating fuel was natural gas. Electricity billing data were
collected from all the sampled units. Billing data were collected for at least 12 months before and 12
months after each unit‘s weatherization date. Billing data on a comparable number of control houses were
also collected.
The sample size of 400 agencies and 10,000 housing units was selected so that the nationwide
total annual energy savings (and average energy savings per housing unit) attributable to the Program can
be estimated to within ~15% of its actual value at a 90% confidence level after non-response and attrition
are taken into account (see Appendix O for a detailed justification for this sample size). Agencies (and
32
thus their housing units) were sampled rather than sampling housing units directly from among all
agencies nationwide because of the cost that would be involved in working cooperatively with ~900
separate agencies. The 400 agencies were selected in two steps: the number of agencies to be selected
from each state was determined first, and then agencies within each state were selected.
The selection of the 400 agencies was stratified by state because such stratification
controls for differences in geography, climate, housing stock, fuel types, and other factors;
controls for the fact that each state administers its program differently (i.e., savings for homes or
agencies are likely to be similar to other homes or agencies in the same state rather than a
different state);
ensures that each state will have at least one agency included in the sample; and
ensures that data provided by states that wish to contribute resources to extend the survey in their
states can be easily incorporated into the analysis, and the benefit to the state from doing so can
be clearly seen.
The number of individual agencies that were selected from each state were in proportion to the
amount (or ―size‖) of the weatherization activity that occurred in each state in PY 2008. For the
retrospective evaluation, ―size‖ was defined as the amount of DOE Program funding received by the state.
If, for example, a state received 10 percent of the Program‘s available funding, then 30 agencies (10
percent of 400) would be selected from that state. The number of agencies counted in a state was rounded
up to 2 even if its numerical proportion was 1.5 or less in order to ensure that an agency from each state
was included in the sample, that standard deviations could be calculated for each state, and that the 14
hot-climate states were adequately represented. It should be stressed that neither the retrospective nor the
WAP ARRA period evaluations are interested in comparing states, but that the method of stratification by
states is being used to improve the sampling randomization and to minimize the sampling error.
Agencies were selected within a state using PPS sampling, with ―size‖ again defined as the
amount of DOE Program funding received by the agency from ARRA funds. PPS sampling is a standard
statistical method that selected agencies that were representative of the entire state but which
preferentially selects larger agencies (i.e., agencies that received more DOE Program funding) with a
higher probability than smaller agencies. This sampling approach led to estimates of totals that are more
accurate than estimates based on simple random sampling (i.e., equal probability sampling).
In general, 33 percent of the housing units and buildings whose primary heating fuel is natural
gas or electricity and that were weatherized by each agency were randomly selected for inclusion in the
retrospective evaluation sample. If an agency was able to provide in electronic form utility account
information on all the natural gas and electrically heated units or buildings it weatherized in PYs 2007
and 2008, all such units and buildings weatherized by the agency were included in the billing data sample.
At least seven housing units will be selected from each agency to ensure that three housing units remain
for each agency after non-response and attrition are considered.
The agency sampling for this WAP ARRA period evaluation will differ from the agency
sampling approach for the retrospective study for three reasons. First, there are 129 more agencies
providing weatherization services during the ARRA period than during the retrospective period. The
sampling approach needs to include some new agencies in the sub-sample.
Second, as explained in depth in Section 4.4, an important component of this evaluation is an
assessment of the outcomes of the Sustainable Energy Resources for Consumers Program (SERC),
initiated during the ARRA period. This program awarded grants to 92 local weatherization agencies to
33
install renewable energy and advanced energy efficiency measures. For reasons explained in Section 4.4,
it is necessary to collect billing histories for the SERC homes for all 92 agencies. Thus, all 92 agencies
need to be part of the subsample. Of these 92 agencies, 35 were in the original sample of 400, 50 were
not, and 5 are new to the program.6
Third, experiences gained during the data collection phase of the retrospective evaluation suggest
that some of the subsampled agencies cannot be persuaded to participate in this project. On the other
hand, the identities of very willing agency participants are known. Thus, the revised methodology
incorporates known respondents and drops known non-respondents from the set of sub-sample agencies.
It should be noted that it is important to ensure that the WAP ARRA period sub-sample of agencies
includes a substantial number of agency respondents for the retrospective evaluation in order to facilitate
comparisons across a large number of variables between the two time periods.
Combining these factors together yields the following approach to developing the set of sub-sampled
agencies:
The 344 agencies that responded during the retrospective study will be included in the sub-sample
(344 out of 847 equals 41% of the original set of retrospective agencies);
Included in this set of 344 agencies are 35 agencies that received SERC grants;
Fifty-six new agencies will be included in the sample (56 out of 129 new agencies equals 43% of
the new agencies);
All five of the new agencies that also received SERC grants will be included in the set of 56 new
agencies;
The other 50 agencies that received SERC grants but were not part of the retrospective subsample of agencies will also be included in the new set of sub-sampled agencies; and
The sample size for the sub-sampled agencies for the WAP ARRA period evaluation will increase
to 450 for this reason.
Just like the retrospective evaluation, for each house and building included in the billing data
sample, the names of the electric and gas (if applicable) utilities, account numbers, weatherization period,
and waiver (release) forms will be collected from the agencies, along with the other housing unit and
building information described in Section 2.1.1 using the DF2 (Housing Unit Information Survey, see
Appendix F) or the DF3 (Building Information Survey, see Appendix G).
The evaluation team will collect the actual electricity and natural gas billing data directly from
the utilities (at least 12 months of bills before weatherization and 12 months of bills after weatherization),
although agency assistance may be needed in the collection of these data. Natural gas billing data will be
collected on those housing units and buildings whose primary heating fuel was natural gas. Electricity
billing data will be collected on all housing units and buildings sampled (i.e., both those whose primary
heating fuel was electricity and those whose primary heating fuel was natural gas). For multifamily
buildings, natural gas and electricity bills will be collected for all master meters as well as for all meters
serving the individual apartment units. Utilities will be asked to provide data for each address regardless
of occupancy changes and to note when occupancy changes occurred, as these data will be used in the
study of non-energy impacts (see Section 2.3.1). The following will be done in order to improve the
response rate for the billing data requests:
an appropriate person at each utility will be contacted to smooth out the process,
6
As of this writing, the identities of the other two agencies are uncertain because of conflicting information in
programmatic records.
34
the billing data requests will be planned so that data for multiple housing units and buildings are
requested from a utility at one time,
billing data will be requested at regular intervals to reduce the chance that the utilities will not be
able to provide data that have already been archived and no longer readily accessible, but the
number of such requests will be limited so that utilities have to provide data a limited number of
times during the course of the evaluation, and
assistance from utility regulatory commissions and similar organizations will be solicited as
needed.
A control group for the billing data sample will be developed using housing units and buildings
weatherized by the same agencies in the PY immediately following. So for example, the control group for
homes weatherized in PY 2010 would be homes weatherized in PY 2011. Such a control group will have
characteristics that are similar to the weatherized group because they are houses and buildings served by
the same agencies; in addition, the client self-selection that led to clients‘ applying for weatherization
assistance will be the same, and the selection process used by the agencies will be the same. The number
of control housing units and buildings selected from each agency will be approximately the same as the
number of weatherized units sampled from that agency. Controls will be selected from each agency
throughout the PY so that pre- and post-weatherization periods for the control units will be similar to
those for the weatherized units. Controls will be randomly selected within each agency after building type
and primary heating fuel are considered, so that these general characteristics closely match those for the
weatherized group.
2.2.2 Energy Analysis
For the energy analysis, energy savings for individual housing units and buildings, normalized to
a typical-weather year, will be estimated using data and approaches that depend on the building type (see
Appendix P for a detailed definition and description of the building types that will be used in the
evaluation) and on whether billing data or sub-metered data were collected. Weather-normalized savings
estimates for individual houses and buildings will then be used to estimate the total annual energy savings
or average annual per-household energy savings for the Program. Energy-savings estimates will be
converted to cost savings using known fuel costs.
Energy Analyses Using Billing Data—Billing data collected on housing units or buildings will
be analyzed using three different methods:
the Princeton Scorekeeping Method (PRISM) as outlined in more detail in Appendix R or suitable
substitution;
the ORNL Aggregate Method as outlined in Appendix R; and
a third method based on a review of the state-of-the-art techniques such as Statistically Adjusted
Engineering (SAE) models, Analysis of Covariance (ANOVA) models, Conditional Demand
Analysis (CDA), and fixed-effect models (Hall, 2006; Hall 2004).
For houses or buildings using natural gas as the primary space-heating fuel, two analyses will be
performed using each of the three methods: one analysis to determine the weather-normalized savings in
the space-heating fuel and another analysis to determine the weather-normalized electricity savings.
PRISM, one of the analysis methods used in the retrospective evaluation, was also the primary
method used in the 1989 National Evaluation. It has been recommended that methods other than PRISM
be used to supplement and/or serve as an overall check on the PRISM analysis (especially in the hotclimate region, where attrition has been high in previous studies and statistically significant savings have
been difficult to measure). Simple methods, such as using simple degree-day adjustments or summing up
35
seasonal usage, have been suggested to reduce model failures when PRISM is used and to avoid the
subsequent bias that can be introduced. The ORNL aggregate model was selected as a primary alternative
method that will be used because, like PRISM, it identifies baseload consumption and allows
uncertainties in estimated parameters and calculated values to be determined with a statistical basis. A
second alternative method will be selected after a review of the most current methods.
Annual Program Energy Savings—The total annual energy savings achieved by the Program in
PYs 2009, 2010 and 2011 will be estimated using the weather-normalized saving estimates for the
individual houses and buildings sampled and a statistical approach based on how these houses and
buildings were sampled. As outlined below, the total annual energy savings achieved by each state in PYs
2009, 2010 and 2011 will be estimated, and these state values will then be summed to calculate the total
annual energy-savings estimate for the Program. State savings are being estimated as an intermediate
value to estimating the Program savings because the selection of agencies (and hence housing units and
buildings) was stratified by state. Also, the best estimator for savings achieved in housing units and by
agencies within a state are savings measured in other housing units and by other agencies within that
same state because of differences in how states implement the Program (e.g., what measures are installed,
how measures are installed, etc.). The total energy-savings estimates for each state and the Program will
be calculated on both a site and source basis.
For each state, the cells in Table 2.2 will be filled in and summed to calculate the total annual
energy-savings estimate for the state as follows:
The savings estimate for each cell involving natural gas or electricity use will be calculated using
the weather-normalized savings estimates for the individual houses and buildings sampled in the
state, along with appropriate weighting factors, which are based on how the agencies were
sampled, the size of the agencies, the number of houses sampled, the number of houses
weatherized in the state, etc. Weather-normalized savings estimates for the individual housing
units and buildings will be those calculated using PRISM, the ORNL Aggregate Method, the third
method chosen, or a combination of these, especially for cell entries based on normalized annual
consumptions (NACs) that cannot be well determined, as, for example, analyzing the electricity
use in homes where electricity is not the primary heating fuel or fuel use in homes in hot climates
with little heating load).
For cells involving fuel oil and propane, energy savings will be estimated based on the results of
the fuel-oil and propane monitored samples from the retrospective evaluation.
For the ―other‖ cells, engineering estimates will be made based on savings measured for other
cells in that state. It is anticipated that engineering estimates will only be required for cells that
represent a small percentage of the units weatherized in a state because of the breadth of the
proposed sampling plan.
Electricity savings on all sampled houses will be estimated (in part to address space cooling,
especially in the hot-climate region) and to include savings from baseload energy uses in the total
Program energy-savings estimate. The analysis approach presented above accomplishes this. Electricity
consumptions and savings will be estimated in all sampled houses and buildings (not just those that are
electrically heated). The analysis of natural gas and electricity billing data will include baseload uses as
well as space heating and space cooling.
The average annual energy savings per household achieved by the Program will be estimated in a
manner similar to that for the total annual energy savings described above, except that savings will be
normalized by the number of units weatherized. Average energy savings will be calculated on both an
absolute and percentage basis, and separately by electricity and all other primary space-heating fuels
combined. Average annual per household energy savings will be calculated by climate region, housing
36
type, primary space-heating fuel type, and various combinations of these categories, as well as by the five
client groups that the Program is specifically encouraged to focus on (i.e., the elderly, persons with
disabilities, families with children, high residential energy users, and households with high energy
burden). Four climate regions consistent with those used in the retrospective evaluation and 1989
National Evaluation will be used: cold, moderate, hot-humid, and hot-dry.
Table 2.2. Total annual Program energy savings
Building type/
Primary space-heating fuel
Number of units
served by the
Program
Non-electric
fuels
(Btu)
Electricity
(kWh)
Total
(Btu)
Site
Single-family:
Natural gas
Electricity
Fuel oil
Propane
Other
Mobile home:
Natural gas
Electricity
Fuel oil
Propane
Other
Small multifamily:
Natural gas
Electricity
Fuel oil
Propane
Other
Large multifamily:
Natural gas
Electricity
Fuel oil
Other
Total
37
Source
The total annual energy savings and the average annual per household energy savings described
above will be calculated two ways when PRISM results are used. First, results will be calculated using
only those houses or buildings that have typical indicators of model reliability (coefficient of
determination (R²) and coefficient of variance (CV) of the normalized annual consumption that pass
standard PRISM criteria (or equivalent for the sub-metered models). This is consistent with past
evaluations and is done to eliminate houses and buildings that have models with poor predictive ability
from the analysis. Secondly, because of concerns that eliminating such houses and buildings may
introduce bias into the results, additional results will be calculated using those houses that pass a more
relaxed set of criteria and/or a minimum set of criteria (essentially all houses and buildings). In all cases, a
―flatness index‖ available in PRISM will be used to pass additional houses and buildings that would
otherwise fail the PRISM criteria. The flatness index identifies houses and buildings with neither a strong
heating nor cooling signal (where R² is very low) but with a normalized annual consumption that is well
determined. This occurs, for example, in examining the space-heating fuel use in a house in a warm
climate that has little heating load, the electricity use of a house in a cold climate that has little cooling
load, and the electricity use of a house in any climate without an air conditioner. Also, in all cases,
outliers will be identified, data quality will be carefully checked, and outliers possibly screened.
In calculating the total annual energy savings and the average annual per-household savings as
described above, occupancy changes (and the subsequent large fluctuation in energy consumption that
may result) will not cause a house or building to be removed from the analysis. The Program is intended
to increase the energy efficiency of low-income housing, and occupancy changes occur naturally in such
houses. This is consistent with the approach taken in the retrospective evaluation and 1989 National
Evaluation but somewhat atypical of other weatherization evaluations. If desired and deemed necessary,
separate analyses with and without occupancy changes will be performed. One concern in automatically
dropping such housing units is that large sample attrition may result because low-income housing can
have high turnover rates. Another concern is that bias could be introduced because housing units with
occupancy changes may have different energy-related characteristics than housing units without
occupancy changes and the characteristics and behaviors of movers could be different from that of people
who do not move.
The total annual energy savings and the average annual per household savings calculated above
will be calculated with and without adjustments for savings in a comparable set of control homes and
buildings (i.e., both gross and net results will be presented). Inclusion of a control group (i.e., adjustment
of savings for weatherized housing units by the savings for the control group) allows estimation of energy
consumption changes that would have occurred in the absence of the Program and controls for factors
such as occupant behavior and fuel prices that influence housing-unit energy consumption.
It is desirable to help states that want to determine the savings in their specific state using both
data collected under this evaluation and supplemental data collected specifically for that state. The
evaluation approach presented in this section easily allows for this since data are being collected in each
state and savings are built up by state. Additional funding will be provided by the states or from some
other sources to (1) determine what additional data need to be collected needed to perform a state-level
analysis, (2) develop the necessary sampling plans and survey instruments, (3) collect the additional data,
(4) analyze the additional data together with information already collected under this evaluation in the
state, and (5) write a report for the state. The supplemental information collected for an individual state
will be incorporated into the analysis performed for the national evaluation by using it, together with data
collected in that state under the evaluation, to develop state values.
Cost Savings—The energy savings estimated above will be converted to cost savings using the
best available fuel-cost data, which are based on the actual costs incurred by the weatherized homes used
in the analysis discussed in this section. Average published fuel-cost data are unlikely to match the
38
climate regions being used in the evaluation and are likely representative of all households rather than just
low-income households. Therefore, fuel-cost data obtained for the homes in the energy analyses should be
used to convert energy savings into cost savings. Special care will be taken in converting energy savings
into cost savings if costs were especially volatile over the Program year.
Sensitivity Analysis—After all energy and cost savings are calculated, a sensitivity analysis will
be conducted to see how out-year estimates of energy and costs savings might change in response to
variation in key driving factors, such as changing demographics in the houses, loss of housing stock,
volatility in fuel costs, new technology, and climate change. The results of this analysis will be used in the
sensitivity analyses performed for non-energy impacts (see Section 2.3.1) and cost-effectiveness (see
Section 2.4).
2.2.3 Measures Analysis
Lastly, an in-depth analysis of the measures installed will be conducted. This is a particularly
important task for this WAP-ARRA period evaluation because during this time, the average investment in
homes increased from $2500 to $6500. This task will answer the question: How did the packages of
measures installed in homes change from PY 2008 to PY 2010?
Several analytical techniques will be applied. First, descriptive statistics that compare the absolute
number of measures by type of measure installed in PY 2008 vs. PY 2010 will be produced. Second,
percentages of each measure installed (out of all measures installed) will be calculated for both program
years. Third, the average number of measures installed per home for both program years will be
calculated. Fourth, the probabilities that any particular measure will be installed in a home will be
calculated for both program years. Fifth, cluster analyses will be done to explore if there are regular
groupings of installed measures and to assess whether the most common measure packages changed
between program years. These statistics will be calculated nationally and also by climate region and house
type (e.g., single family, mobile home). This information should provide a comprehensive picture of
changes in measure installation from PY 2008 to PY 2010.
2.2.4 Attribution Methodology
While the Program is the major driving force behind the weatherization of low-income homes in
the United States, the Program‘s resources are leveraged by several other parties and programs, including
LIHEAP, PVE, public benefits funds, states, utilities, and non-profit organizations. It is important to
properly attribute energy savings and energy cost savings to those parties that, along with DOE,
contribute financial and in-kind resources to weatherize low-income homes.
This evaluation will develop a methodology to allow energy savings and energy cost savings to
be attributed to the set of parties mentioned above based on well-known concepts found in the field of
decision analysis. Generally, the methodology will be based on concepts used in multi-criteria decisionmaking, which includes such tools as decision matrices and evaluation criteria. More specifically, the
methodology will categorize weatherization into a finite set of activities and functions (program
management, outreach and marketing, client selection, audit and measure selection, measure installation,
and training). The contributions of the parties to these activities and functions will be estimated using
information collected from all the states as part of S1 (All States Program Information Survey, see
Appendix C) and from the 400 agencies included in the billing data sample as part of S3 (Subset of
Agencies Program Information Survey, see Appendix E). The influence of these activities and functions
on energy savings and cost savings will be estimated by a panel of experts. Using the two sets of
estimates and a decision-matrix approach, the accurate attribution of energy savings and cost savings
appears to be fairly technically straightforward . If the panel of experts feels that the influences of the
activities and functions on savings vary by known state characteristics (e.g., states with that have utility
39
weatherization involvement vs. those that do not), then the analysis could be performed by categories of
states to build up the appropriate national attribution values.
The challenges to implementing an attribution methodology are likely not to be so much technical
as related to the process. For example, one important question relates to who should be involved in
making the two sets of estimates described above (although an approach is outlined above). How should
the estimates be generated if several parties are involved? How should disagreements among the parties
about the estimates be resolved? Lastly, the scale of the attribution methodology needs to be carefully
considered. It is assumed that the methodology will be developed within a national context. However,
various parties may request that the methodology be applied on a state-by-state basis. This latter approach
may require considerably more data collection and would certainly require much more effort to generate
the two sets of estimates for every state. Therefore, the process parameters must be clearly established in
order to achieve a meaningful, accurate, and efficient survey.
2.3 NON-ENERGY IMPACTS
As part of the impact assessment, the non-energy impacts (NEIs) attributable to the Program that
affect the clients served, the ratepayers, the utilities, and society will be ascertained. Table 2.3 shows the
primary non-energy impacts that have been identified to date and that will be quantified in this evaluation.
Schweitzer and Tonn (2002) identified most of these non-energy impacts as being applicable to the
Program and provide a detailed discussion of each. It is important to note that the project team will have
the flexibility to consider new impacts, new metrics, and new values for existing metrics, as long as such
investigation does not involve the collection of primary data not previously approved by OMB under the
terms of the Paperwork Reduction Act (PRA).
In addition to the non-energy impacts identified in Table 2.3, the number of actions taken by
weatherization providers to improve health and safety (e.g., fix broken flues, replace cracked heat
exchangers) will be reported as part of this evaluation.
Table 2.3 also quantifies each of the primary non-energy impacts as a monetized or non-monetized
value. Definitions of these terms for the purpose of this evaluation follow.
Monetized value—For most of the Program-generated non-energy impacts, a monetary value
(annual dollar value and lifetime net present dollar value) will be calculated nationally (and
possibly by climate region) from different perspectives (client, utility/ratepayer, and society)
using a computer model or some other mechanism for performing the necessary calculations. The
major inputs for these calculations include household-level data gathered for this national
evaluation, a large set of performance metrics describing key Program outputs, and a set of
monetized metrics that converts performance measures into dollar values. The dollar value of
each monetized impact is calculated by taking the number of relevant household-level activities
reported, multiplying that number by the appropriate performance metric, and multiplying that
product by the matching monetized metric. Both a point estimate and a confidence interval are
expected to be calculated for each impact, in recognition of the uncertainty surrounding these
estimates. The ―monetized value‖ will represent the net economic value of the impact, as both
costs and benefits associated with the impact will be included. However, monetized values will be
calculated only where a specific identifiable expense is avoided or incurred, or where a clear
monetary impact is obtained. Subjective approaches to calculating the dollar value of non-energy
impacts (e.g., using willingness-to-pay or relative-valuation approaches) will not be used in this
evaluation.
40
Non-monetized value—For a sizable minority of Program-generated non-energy impacts, all of
which fall under the broad umbrella of ―safety, health, and comfort,‖ a non-monetary value will
be calculated. Most of these non-monetary values will come from surveys of occupant
perceptions, but some will come from the direct measurement of such key factors as indoor air
temperature and humidity levels. In assessing the value of these non-monetized impacts, the
performance metrics will be calculated directly from the relevant household-level data.
Table 2.3 shows the household-level data that will be used as a basis for calculating each nonenergy impact, as well as for calculating the performance metric and the monetized metric (where
applicable) associated with each specific impact.
Under the impact assessment, data will be collected to update some of the performance and
monetized metrics needed from earlier estimates before calculating values for the monetized non-energy
impacts. In updating these metrics, both costs and benefits will be considered so that net economic values
are developed. In addition, household-level data will be collected and analyzed to directly calculate values
for the non-monetized impacts. The required data collection and analyses are described more fully below.
41
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
Impact
Categories and
Specific Impacts
Type of
Value
Calculated
HouseholdLevel
Data (used as
basis for
calculating the
value of
impacts)
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
I. Utility/Ratepayer Impacts
A. Payment-Related Impacts
1. Rate subsidy
payments
avoided by state
aid agencies
2. Lower rate of
bad debt writeoffs
Monetized
Number of
households
weatherized
Monetized
Number of
households
weatherized
3. Reduced
carrying cost on
arrearages
4. Fewer notices
and customer
calls
Monetized
Number of
households
weatherized
Number of
households
weatherized
5. Fewer shutoffs and
reconnections for
delinquency
Monetized
Number of
households
weatherized
6. Reduced
Monetized
collection costs
for delinquent
payments
B. Service Provision Impacts
Number of
households
weatherized
1. Fewer
emergency gas
service calls
Monetized
Number of
households
weatherized
2. Reduction in
transmission and
distribution
losses
3. Insurance
savings
Monetized
Electricity
savings (in kWh)
in weatherized
houses
Number of
households
weatherized
Monetized
Monetized
Average reduction in
number of subsidized
units of energy sold per
weatherized household
Average reduction in
amount of bad debt
written off by utility per
weatherized household
Average dollar reduction
in arrearage per
weatherized household
Average reduction in
number of notices sent
and calls made to
customers, per
weatherized household
Average reduction in
number of customer shutoffs and reconnections
made by utility, per
weatherized household
Average reduction in
number of collections
made by utility per
weatherized household
Cost to utility per
subsidized unit of energy
sold
Average reduction in
number of emergency
service calls made per
weatherized household
Average amount of
electricity lost in
transmission and
distribution, per kWh sold
Average reduction in
utility‘s cost for insurance
to cover household fires
and explosions, per
weatherized household
Average cost to utility per
service call
42
Same as Performance
Metric
Interest due utility per
dollar of arrearage
Average cost to utility per
notice sent and call made
Average cost to utility per
shut-off and reconnection
Average cost to utility per
collection
Average cost to utility per
unit of lost electricity
Same as Performance
Metric
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
HouseholdLevel
Data (used as
basis for
Impact
Type of
calculating the
Categories and
Value
value of
Specific Impacts
Calculated
impacts)
4. Shifted fixed
Monetized
Number of
costs to utilities
households
weatherized
II. Impacts on Participating Households
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Average energy savings in
weatherized houses
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
Change in fuel cost per unit
of energy savings to cover
fixed costs
A. Affordable Housing Impacts
1. Water and
sewer service
savings
Monetized
2. Property value
impacts
Monetized
3. Avoided shutoffs and
reconnections
Monetized
4. Reduced
mobility
Monetized
5. Reduced
transaction costs
Monetized
Number of
water-saving
devices installed
in weatherized
houses
Number of
households
weatherized
Number of
households
weatherized
Number of
households
weatherized
Number of
households
weatherized
Average water savings (in
gallons) per device
installed
Cost of water and sewer
service per gallon of water
Average cost of structural
repairs per weatherized
household
Average reduction in
number of shut-offs and
reconnections, per
weatherized household
Average reduction in
number of moves per
weatherized household
Average number of hours
required to become
familiar with energysaving products per
household
Same as Performance
Metric
Average reduction in
number of fires per
weatherized household
Perceived changes in
safety of heating system
and electrical wiring in
weatherized houses
Average monetary loss to
household (property,
injury, and death) per fire
Not applicable
Perceived change in
health problems in
weatherized houses
Not applicable
Average cost to customer
per shut-off (for ―lost rent‖
and restart fee)
Average cost per move
Average cost per hour of
time (use minimum wage
for this calculation)
B. Safety, Health, and Comfort Impacts
1. Fewer fires
Monetized
Nonmonetized
2. Changes in
frequency of
health problems
Nonmonetized
Number of
households
weatherized
Occupant
perceptions of
household fire
safety before and
after
weatherization
Occupant
perceptions of
general health
and safety before
and after
weatherization
43
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
Impact
Categories and
Specific Impacts
Type of
Value
Calculated
Monetized
Nonmonetized
3. Enhanced
prevention and
treatment of
health problems
Nonmonetized
Nonmonetized
4. Changes in
indoor air quality
Nonmonetized
Nonmonetized
HouseholdLevel
Data (used as
basis for
calculating the
value of
impacts)
Number of
households
weatherized
Occupant reports
on incidence of
symptoms or
occurrences of
specific health
problems before
and after
weatherization
Occupant reports
on number of
times food
purchases were
postponed or not
made in order to
pay utility bills
before and after
weatherization
Occupant reports
on access to
health care and
medication
before and after
weatherization
Measured CO
levels before and
after
weatherization
Measured levels
of indoor
airborne mold
spores relative to
outdoor levels
before and after
weatherization
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Average reduction in
number of workdays lost
due to health problems per
weatherized household
Change in incidence of
symptoms or occurrences
of specific health
problems in weatherized
houses
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
Average cost to household
per lost work day
Not applicable
Reduction in number of
times food could not be
purchased due to size of
utility bill in weatherized
houses
Not applicable
Change in access to and
ability to pay for health
care and medication in
weatherized houses
Not applicable
Measured change in CO
levels in weatherized
houses
Not applicable
Measured change in level
of indoor airborne mold
spores relative to outdoor
levels in weatherized
houses
Not applicable
44
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
Impact
Categories and
Specific Impacts
Type of
Value
Calculated
Nonmonetized
Nonmonetized
5. Changes in
household
moisture levels
Nonmonetized
6. Decreased
incidence of
hypothermia and
hyperthermia
Monetized
Nonmonetized
7. Improved food
safety
Nonmonetized
HouseholdLevel
Data (used as
basis for
calculating the
value of
impacts)
Measured levels
of indoor
airborne pollen
relative to
outdoor levels
before and after
weatherization
Occupant
perceptions of
odors that could
indicate a
problem with
indoor air quality
Measured levels
of indoor
humidity before
and after
weatherization
Number of
households
weatherized
Occupant reports
on incidence of
students‘
disrupted study
due to excessive
heat or cold
before and after
weatherization
Measured
temperature in
refrigerator
before and after
weatherization
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Measured change in level
of indoor airborne pollen
relative to outdoor levels
in weatherized houses
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
Not applicable
Perceived change in
frequency of odors within
weatherized houses
Not applicable
Measured change in
humidity levels in
weatherized houses
Not applicable
Average reduction in
number of times
emergency medical care is
sought due to heat stress
or overexposure to cold
per weatherized
household
Change in incidence of
students‘ disrupted study
in weatherized houses
Average cost of emergency
medical care at hospital,
emergency room, or urgent
care facility
Measured change in
refrigerator temperature in
weatherized houses
Not applicable
45
Not applicable
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
Impact
Categories and
Specific Impacts
Type of
Value
Calculated
Nonmonetized
8. Improved
household safety
and security
Monetized
Monetized
Number of
households
weatherized
Nonmonetized
Occupant
perceptions of
security of home
from criminal
intrusion before
and after
weatherization
Number of
households
weatherized
Measured levels
of asbestos and
radon in houses
before and after
weatherization
Reports on
incidence of
poisoning from
household
chemicals before
and after
weatherization
Monetized
9. Change in
presence of
environmental
hazards
HouseholdLevel
Data (used as
basis for
calculating the
value of
impacts)
Occupant reports
on number of
incidents of
gastrointestinal
problems and
food poisoning
before and after
weatherization
Number of
households
weatherized
Nonmonetized
Nonmonetized
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Change in incidence of
gastrointestinal problems
and food poisoning in
weatherized houses
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
Not applicable
Average reduction in
number of times
emergency medical care is
sought for injuries from
tripping and falling in the
home
Average reduction in
number of times
emergency medical care is
sought for burns from
scalding from domestic
hot water
Perceived change in
security from criminal
intrusion in weatherized
houses
Average cost of emergency
medical care at hospital,
emergency room, or
urgent-care facility
Average reduction in
number of break-ins per
weatherized household
Measured change in levels
of asbestos and radon in
weatherized houses
Average value of items
stolen in break-in
Change in number of
poisonings from
household chemicals in
weatherized houses
Not applicable
46
Average cost of emergency
medical care at hospital,
emergency room, or
urgent-care facility
Not applicable
Not applicable
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
Impact
Categories and
Specific Impacts
Type of
Value
Calculated
Nonmonetized
10. Improved
comfort
Nonmonetized
Nonmonetized
11. Improved
appearance
Nonmonetized
12. Reduced
noise inside
dwelling
Nonmonetized
HouseholdLevel
Data (used as
basis for
calculating the
value of
impacts)
Occupant reports
on level of
household
infestation with
vermin before
and after
weatherization
Occupant
perceptions of
indoor comfort
(temperature and
draftiness) before
and after
weatherization
Measured indoor
air temperature
before and after
weatherization
Occupant
perceptions of
appearance of
dwelling before
and after
weatherization
Occupant
perceptions of
noise level
within dwelling
before and after
weatherization
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Change in level of vermin
infestation in weatherized
houses
Units of energy
saved in
weatherized
houses
Units of energy
saved in
weatherized
houses
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
Not applicable
Perceived improvement in
indoor comfort
(temperature and
draftiness) in weatherized
houses
Not applicable
Measured change in
indoor air temperature in
weatherized houses
Not Applicable
Perceived improvement in
appearance of weatherized
dwellings
Not Applicable
Perceived reduction in
noise within weatherized
dwellings
Not applicable
Pounds of CO2 emitted
per unit of energy saved
Value of CO2 emission
reduction in dollars per
pound
Pounds of SOx emitted
per unit of energy saved
Value of SOx emission
reduction in dollars per
pound
III. Societal Impacts
A. Environmental Impacts
1. Air emissions:
CO2
Monetized
2. Air emissions:
SOx
Monetized
47
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
Impact
Categories and
Specific Impacts
3. Air emissions:
NOx
Type of
Value
Calculated
Monetized
4. Air emissions:
CO
Monetized
5. Air emissions:
CH4
Monetized
6. Air emissions:
PM
Monetized
7. Air emissions:
heavy metals
Monetized
8. Fish
impingement
Monetized
9. Wastewater
and sewage in
electricity
production
B. Social Impacts
Monetized
1.Jobs for
unemployed
workers
Monetized
HouseholdLevel
Data (used as
basis for
calculating the
value of
impacts)
Units of energy
saved in
weatherized
houses
Units of energy
saved in
weatherized
houses
Units of energy
saved in
weatherized
houses
Units of energy
saved in
weatherized
houses
Units of energy
saved in
weatherized
houses
Units of
electricity saved
in weatherized
houses
Units of
electricity saved
in weatherized
houses
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Pounds of NOx emitted
per unit of energy saved
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
Value of NOx emission
reduction in dollars per
pound
Pounds of CO emitted per
unit of energy saved
Value of CO emission
reduction in dollars per
pound
Pounds of CH4 emitted
per unit of energy saved
Value of CH4 emission
reduction in dollars per
pound
Pounds of PM emitted per
unit of energy saved
Value of PM emission
reduction in dollars per
pound
Pounds of heavy metals
emitted per unit of energy
saved
Value of heavy metal
emission reduction in
dollars per pound
Number of fish impinged
at power plants per unit of
electricity saved
Dollar value per impinged
fish
Amount of wastewater
and sewage (in gallons)
produced per unit of
electricity saved
Cost per gallon of treating
wastewater and sewage
Dollars spent to
weatherize client
homes
Average number of
unemployed workers
given jobs per dollar spent
on weatherization
Average cost of
unemployment benefits
paid per unemployed
worker
Average number of direct
and indirect jobs created
per dollar spent on
weatherization
Average amount of unpaid
rent per weatherized rental
household before and after
weatherization
Taxes paid (local, state,
and federal) and dollars
spent locally, per job
created
Same as Performance
Metric
C. Economic Impacts
1. Direct and
indirect
employment
Monetized
Dollars spent to
weatherize client
homes
2. Lost rental
Monetized
Number of rental
households
weatherized
48
Table 2.3. Non-energy impacts and the household-level data and metrics required to calculate
their value
Impact
Categories and
Specific Impacts
3. National
security
Type of
Value
Calculated
Monetized
HouseholdLevel
Data (used as
basis for
calculating the
value of
impacts)
Units of source
energy saved in
weatherized
houses
Performance Metric (to
be multiplied by
household-level data for
monetized impacts and
calculated from
household-level data for
non-monetized impacts)
Average proportion of
source energy used for
residential purposes that is
normally imported
Monetized Metric
(to be multiplied by
Performance Metric
unless the two are
identical)
―Premium‖ paid in higher
prices and disturbance to
economy per unit of
imported energy
Care will be taken to avoid double-counting any non-energy impact and to make sure that
measured impacts are truly attributable to the Program (e.g., by use of control groups). In addition, the
non-energy impacts addressed in this evaluation will not include the impact of market transformation,
which can be thought of as additional energy savings that ―spill over‖ from direct program effects. The
possible differences between non-energy impacts achieved in urban and in rural areas are also not
subjects of this evaluation.
2.3.1 Monetized Data Collection and Analysis
To calculate the monetary values of selected non-energy impacts, coefficients for the
performance metrics and monetized metrics will be acquired either from previous research on non-energy
impacts or from new primary and secondary data gathered for this evaluation The default values for the
performance and monetized metric coefficients will be those used in ORNL‘s 2002 review of non-energy
impacts of the Weatherization Program (Schweitzer and Tonn 2002). The default values will be replaced
with new, updated values that reflect current conditions for the Program under the following
circumstances:
coefficients from newer studies or computerized models are judged to be superior,
existing coefficients do not adequately represent program impacts nationwide or the net economic
value of the impact, or
the impact was quantified with a new household level variable.
To the extent possible, coefficients that disaggregate non-energy impacts by geographic and/or
climate region should be used, and the coefficients used for this study, whether existing or newly
developed, will be of this type. It is likely (and acceptable) that region-specific coefficients (such as the
cost of water service) will be used for some non-energy impacts and not for others. In addition, the types
of housing units to which the available data apply (e.g., single-family dwellings, mobile homes,
multifamily units) will be tracked and, where appropriate (and data allowing), separate coefficients for the
different housing types will be developed.
Table 2.4 identifies the monetized non-energy impacts for which new performance and/or
monetized metrics will be developed in this evaluation. Table 2.4 also shows the factors to be considered
in determining what new data are needed. The determination about what new data to acquire is guided by
how much uncertainty surrounds current metrics, the potential magnitude of a metric‘s salience, and how
closely the metric is tied to primary Program purposes. It is possible that after existing performance and
monetized metric coefficients are examined, it will be concluded that other new coefficients are also
49
needed. As noted earlier, it is acceptable for new impacts, new metrics, and new values for existing
metrics to be added as long as no data are collected that have not been previously approved by OMB.
For each monetized non-energy impact, the data to be collected to update its performance and/or
monetized metric are described below, as well as the methods that will be used to collect and analyze
those data:
50
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
Performance
Metric and
Uncertainty
(L, M, H)
Monetized Metric
and Uncertainty
(L, M, H)
Potential
Magnitude of
Monetized
Value
(L, M, H)
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
New Data to
Collect for
This Study
Average reduction in
number of
subsidized units of
energy sold per
weatherized
household (L)
2. Lower bad
Average reduction in
debt write-off
amount of bad debt
written off by utility
per weatherized
household (L)
3. Reduced
Average dollar
carrying cost on reduction in
arrearages
arrearage per
weatherized
household (L)
4. Fewer notices Average reduction in
and customer
number of notices
calls
sent and calls made
to customers, per
weatherized
household (L)
5. Fewer shutAverage reduction in
offs and
number of customer
reconnections
shut-offs and
for delinquency reconnections made
by utility, per
weatherized
household (L)
6. Reduced
Average reduction in
collection costs number of
for delinquent
collections made by
payments
utility per
weatherized
household (H)
B. Service Provision Impacts
Cost to utility per
subsidized unit of
energy sold (L)
L
N
None
Same as
Performance
Metric (L)
M
N
None
Interest due utility
per dollar of
arrearage (L)
L
N
None
Average cost to
utility per notice
sent and call made
(L)
L
N
None
Average cost to
utility per shut-off
and reconnection
(L)
L
N
None
Average cost to
utility per
collection (M)
L
N
None
1. Fewer
emergency gas
service calls
Average cost to
utility per service
call (L)
M
N
None
I. Utility/Ratepayer Impacts
A. Payment-Related Impacts
1. Rate
subsidies
avoided
Average reduction in
number of
emergency service
calls made per
weatherized
household (L)
51
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
Performance
Metric and
Uncertainty
(L, M, H)
Monetized Metric
and Uncertainty
(L, M, H)
L
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
N
None
L
N
None
L
N
Monetized
Metric
Cost of water and
sewer service per
gallon of water (H)
M
N
Performance
and Monetary
Metrics
Same as
Performance
Metric (L)
M
N
Performance
Metric (data
collected in
samples)
Average cost to
customer per shutoff (for ―lost rent‖
and restart fee) (L)
L
Y
M
N
L
N
2. Transmission and
distribution loss
reduction
Average amount of
Average cost to
electricity lost in
utility per unit of
transmission and
electricity lost (L)
distribution, per
kWh sold (L)
3. Insurance
Average reduction in Same as
savings
utility's cost for
Performance
insurance to cover
Metric (M)
household fires and
explosions, per
weatherized
household (M)
4. Shifted utility Average energy
Change in fuel cost
fixed costs
savings in
per unit of energy
weatherized houses
savings to cover
(L)
fixed costs (M)
II. Impacts to Participating Households
Potential
Magnitude of
Monetized
Value
(L, M, H)
New Data to
Collect for
This Study
A. Affordable Housing Impacts
1. Water and
sewer savings
2. Property
value impacts
3. Avoided
shut-offs and
reconnections
Average water
savings (in gallons)
per device installed
(M)
Average cost of
structural repairs per
weatherized
household (L)
Average reduction in
number of shut-offs
and reconnections,
per weatherized
household (L)
4. Reduced
mobility
Average reduction Average cost per
in number of
move (M)
moves per
weatherized
household (H)
5. Reduced
Average number
Average cost per
transaction
of hours required
hour of time (use
costs
to become familiar minimum wage
with energyfor this
saving products
calculation) (M)
per household (L)
B. Safety, Health, and Comfort Impacts
52
Performance
and Monetary
Metrics
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
Performance
Metric and
Uncertainty
(L, M, H)
Monetized Metric
and Uncertainty
(L, M, H)
Potential
Magnitude of
Monetized
Value
(L, M, H)
1. Fewer fires
Average reduction
in number of fires
per weatherized
household (M)
Average
monetary loss to
household
(property, injury,
and death) per
fire (M)
Not applicable
M
Y
Performance
and Monetary
Metrics
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Average cost to
household per
lost work day (L)
M
Y
Performance
Metric
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Perceived changes
in safety of
heating system and
electrical wiring in
weatherized
houses (H)
2. Changes in Perceived change
frequency of
in health problems
health
in weatherized
problems
houses (H)
Average reduction
in number of
workdays lost due
to health problems
per weatherized
household (H)
Change in
incidence of
symptoms or
occurrences of
specific health
problems in
weatherized
houses (H)
3. Enhanced
Reduction in
prevention and number of times
treatment of
food could not be
health
purchased due to
problems
size of utility bill
in weatherized
houses (H)
Change in access
to health care and
medication in
weatherized
houses
53
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
New Data to
Collect for
This Study
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
4. Changes in
indoor air
quality
5. Changes in
household
moisture
levels
6. Decreased
incidence of
hypothermia
and
hyperthermia
Performance
Metric and
Uncertainty
(L, M, H)
Measured change
in CO levels in
weatherized
houses (H)
Measured change
in level of indoor
airborne mold
spores relative to
outdoor levels in
weatherized
houses (H)
Measured change
in level of indoor
airborne pollen
relative to outdoor
levels in
weatherized
houses (H)
Perceived change
in frequency of
odors within
weatherized
houses (H)
Measured change
in humidity levels
in weatherized
houses (H)
Average reduction
in number of times
emergency
medical care is
sought due to heat
stress or over
exposure to cold
per weatherized
household (H)
Change in
incidence of
students‘ disrupted
study in
weatherized
houses (H)
Monetized Metric
and Uncertainty
(L, M, H)
Potential
Magnitude of
Monetized
Value
(L, M, H)
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Average cost of
emergency
medical care at
hospital,
emergency room,
or urgent care
facility (L)
M
Y
Performance
and Monetary
Metrics
Not applicable
Not
applicable
Y
Householdlevel Data
54
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
New Data to
Collect for
This Study
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
7. Improved
food safety
8. Improved
household
safety and
security
9. Change in
presence of
environmental
hazards
Performance
Metric and
Uncertainty
(L, M, H)
Measured change
in refrigerator
temperature in
weatherized
houses (H)
Change in
incidence of
gastrointestinal
problems and food
poisoning in
weatherized
houses (H)
Average reduction
in number of times
emergency
medical care is
sought for injuries
from tripping and
falling in the home
(H)
Average reduction
in number of times
emergency
medical care is
sought for burns
from scalding
from domestic hot
water (H)
Perceived change
in security from
criminal intrusion
in weatherized
houses (H)
Average reduction
in number of
break-ins per
weatherized
household (H)
Measured change
in levels of
asbestos and radon
in weatherized
houses (H)
Monetized Metric
and Uncertainty
(L, M, H)
Potential
Magnitude of
Monetized
Value
(L, M, H)
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Average cost of
emergency
medical care at
hospital,
emergency room,
or urgent care
facility (L)
M
Y
Performance
and monetary
Metrics
Average cost of
emergency
medical care at
hospital,
emergency room,
or urgent care
facility (L)
M
Y
Performance
and Monetary
Metrics
Not applicable
Not
applicable
Y
Householdlevel Data
Average value of
items stolen in
break-in (H)
M
Y
Performance
and Monetary
Metrics
Not applicable
Not
applicable
Y
Householdlevel Data
55
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
New Data to
Collect for
This Study
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
Performance
Metric and
Uncertainty
(L, M, H)
Change in number
of poisonings from
household
chemicals in
weatherized
houses (H)
Change in level of
vermin infestation
in weatherized
houses (H)
10. Improved
Perceived
comfort
improvement in
indoor comfort
(temperature and
draftiness) in
weatherized
houses (H)
Measured change
in indoor air
temperature in
weatherized
houses (H)
11. Improved
Perceived
appearance
improvement in
appearance of
weatherized
dwellings (H)
12. Reduced
Perceived
noise inside
reduction in noise
dwelling
within weatherized
dwellings (H)
III. Societal Impacts
Monetized Metric
and Uncertainty
(L, M, H)
Potential
Magnitude of
Monetized
Value
(L, M, H)
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
New Data to
Collect for
This Study
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Not Applicable
Not
applicable
Y
Householdlevel Data
Not Applicable
Not
applicable
N
Householdlevel Data
Not applicable
Not
applicable
Y
Householdlevel Data
Value of CO2
emission
reduction in
dollars per pound
(H)
H
N
Performance
and Monetary
Metrics
A. Environmental Impacts
1. Air
emissions:
CO2
Pounds of CO2
emitted per unit of
energy saved (M)
56
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
Performance
Metric and
Uncertainty
(L, M, H)
Monetized Metric
and Uncertainty
(L, M, H)
Potential
Magnitude of
Monetized
Value
(L, M, H)
2. Air
emissions:
SOx
Pounds of SOx
emitted per unit of
energy saved (M)
M
N
Performance
and Monetary
Metrics
3. Air
emissions:
NOx
Pounds of NOx
emitted per unit of
energy saved (M)
H
N
Performance
and Monetary
Metrics
4. Air
emissions: CO
Pounds of CO
emitted per unit of
energy saved (M)
M
N
Performance
and Monetary
Metrics
5. Air
emissions:
CH4
Pounds of CH4
emitted per unit of
energy saved (M)
M
N
Performance
and Monetary
Metrics
6. Air
emissions: PM
Pounds of PM
emitted per unit of
energy saved (M)
M
N
Performance
and Monetary
Metrics
7. Air
emissions:
heavy metals
Pounds of heavy
metals emitted per
unit of energy
saved (M)
H
N
Performance
and Monetary
Metrics
8. Fish
impingement
Number of fish
impinged at power
plants per unit of
electricity saved
(L)
Amount of
wastewater and
sewage (in
gallons) produced
per unit of
electricity saved
(M)
Value of SOx
emission
reduction in
dollars per pound
(H)
Value of NOx
emission
reduction in
dollars per pound
(H)
Value of CO
emission
reduction in
dollars per pound
(H)
Value of CH4
emission
reduction in
dollars per pound
(H)
Value of PM
emission
reduction in
dollars per pound
(H)
Value of heavy
metal emission
reduction in
dollars per pound
(H)
Dollar value per
impinged fish (L)
L
N
Cost per gallon of M
treating
wastewater and
sewage (M)
N
9. Wastewater
and sewage in
electricity
production
57
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
New Data to
Collect for
This Study
Performance
and Monetary
Metrics
Table 2.4. Impacts, metrics, and factors required to determine need for new data
Impact
Categories and
Specific
Impacts
Performance
Metric and
Uncertainty
(L, M, H)
Monetized Metric
and Uncertainty
(L, M, H)
Potential
Magnitude of
Monetized
Value
(L, M, H)
Metric is
Closely
Tied to
Program
Purposes
(Y, N)
1. Avoided
Average number
unemployment of unemployed
impact
workers given jobs
per dollar spent on
weatherization (L)
C. Economic Impacts
Average cost of
unemployment
benefits paid per
unemployed
worker (L)
M
N
1. Direct and
indirect
employment
Taxes paid (local,
state, and federal)
and dollars spent
locally, per job
created (L)
H
N
Same as
Performance
Metric (L)
L
N
―Premium‖ paid
in higher prices
and disturbance
to economy per
unit of imported
energy (H)
H
Y
New Data to
Collect for
This Study
B. Social Impacts
2. Lost rental
3. National
security
Average number
of direct and
indirect jobs
created per dollar
spent on
weatherization
(M)
Average amount
of unpaid rent per
weatherized rental
household before
and after
weatherization (L)
Average
proportion of
source energy used
for residential
purposes that is
imported (L)
Performance
Metric
Performance
and Monetary
Metrics
Shifted utility fixed costs—Monetized Metric: Change in fuel cost per unit of energy savings to
cover fixed costs. Information from the literature will be collected on how fuel cost prices increase as
a result of reduced consumption to cover utilities‘ fixed costs.
Water and sewer savings—Performance Metric: Average water savings (in gallons) per device
installed; Monetized Metric: Cost of water and sewer service per gallon of water. For the
performance metric, information will be collected from the literature on the amount of water saved
through the installation of low-flow showerheads and faucet aerators. From those secondary sources,
average water savings per device will be calculated. For the monetary metric, primary data on the cost
of water and sewer service (i.e., costs per unit of consumption) will be collected from a nationwide
sample of 30 to 50 water utilities serving the houses used in the energy study by examining published
information on their websites. From this, average costs will be calculated for the entire nation and, if
possible, for individual geographic and/or climate regions. The rates of the utilities chosen will be
representative of climate regions and housing types.
58
Property value impacts—Performance Metric: Average cost of structural repairs per
weatherized household. Information on the dollar value of the structural repairs performed for each
unit weatherized in PY 2010 will be collected as part of the Program characterization study (see
Section 2.1). The average cost of structural repairs per weatherized household will be calculated by
summing the value of repairs performed on all units and dividing by the number of dwellings
weatherized. An alternative Performance Metric for evaluating property value impacts might be to
collect information from realtors or appraisers to determine the increase in property value based on
the amount of structural repairs performed.
Reduced mobility—Performance Metric: Average reduction in number of moves per
weatherized household; Monetized Metric: Average cost per move. If the billing data gathered for
the billing data sample indicate when the occupants of a dwelling move, the average number of
moves per household will be calculated for the treatment group in the year following weatherization;
that number will then be compared to the average number of moves during that same period for the
control group (using an appropriate statistical procedure). If the billing data do not identify when
occupants moved, program participants and a control group of non-participants will be surveyed via
telephone regarding the number of times they changed residences in the year after the weatherization.
As before, the mean number of moves for the treatment and control groups will be compared. Only
the post-weatherization period will be studied because it is expected that prospective participants will
move much less frequently than non-participants during the pre-weatherization period because of the
process of applying for and awaiting the weatherization. The average cost of moving for a typical
low-income family will be gathered from secondary sources.
Fewer fires—Performance Metric: Average reduction in number of fires per weatherized
household; Monetized Metric: Average monetary loss to household per fire. Data on
weatherization-induced changes in the number of fires will be gathered using the Occupant Survey,
which is being conducted by the retrospective evaluation during WAP-ARRA PYs 2011 and 2012
(Ternes et al. 2007). Additional data on the reduction in fires will be taken from reliable secondary
sources. (It is important to augment survey findings through the use of national statistics when
examining relatively rare events such as fires.) The average monetary loss per fire will also be
gathered from secondary sources. The most difficult aspect of quantifying monetary loss is, of course,
assigning an acceptable value to the loss of a human life.
Changes in frequency of health problems—Performance Metric: Average reduction in number
of workdays lost per weatherized household due to health problems. Occupants of the same
weatherized and control units mentioned above will be surveyed over the phone using a portion of the
Occupant Survey to determine the number of days they were absent from work during the pre- and
post-weatherization periods due to health problems. Net change in number of lost workdays from the
pre- to the post-weatherization period will be determined by comparing change for the treatment and
control groups using an appropriate statistical procedure.
Decreased incidence of hypothermia and hyperthermia—Performance Metric: Average
reduction in number of times emergency medical care is sought due to heat stress or
overexposure to cold per weatherized household; Monetized Metric: Average cost of emergency
medical care at hospital, emergency room, or urgent care facility. A portion of the Occupant
Survey will be used to determine the number of times a household member sought emergency
medical care due to heat stress or overexposure to cold during the pre- and post-weatherization
periods. Changes between the two periods will be compared for the treatment and control groups
using an appropriate statistical procedure. The average cost of emergency medical care at a hospital,
emergency room, or urgent-care facility will be gathered from secondary sources.
59
Improved household safety and security—Performance Metric: Average reduction in number
of times emergency medical care is sought for injuries from tripping and falling in the home;
Monetized Metric: Average cost of emergency medical care at hospital, emergency room, or
urgent care facility. Once again, the Occupant Survey will be administered to the same weatherized
and control units described above. During the pre-weatherization period and again in the postweatherization period, the subjects will be asked to report the number of times a household member
has sought emergency medical care for injuries from tripping and falling in the home. Net change
from the pre- to post-weatherization period will be determined by comparing change for the treatment
and control groups using an appropriate statistical procedure. Additional data on the frequency of
serious trip and fall injuries will be taken from reliable secondary sources. The average cost of
emergency medical care at a hospital, emergency room, or urgent care facility will also be gathered
from secondary sources.
Improved household safety and security—Performance Metric: Average reduction in number
of times emergency medical care is sought for burns from scalding from domestic hot water in
weatherized houses; Monetized Metric: Average cost of emergency medical care at hospital,
emergency room, or urgent care facility. The Occupant Survey will be administered to the same
weatherized and control units described above. The subjects will be asked to report the number of
times a household member sought emergency medical care as a result of burns from scalding water
from a faucet or showerhead in their home during the pre- and post-weatherization periods. Changes
between the two periods will be compared for the treatment and control groups using the same
general approach described above. As noted previously, the average cost of emergency medical care
at a hospital, emergency room, or urgent care facility will be gathered from secondary sources.
Improved household safety and security—Performance Metric: Average reduction in number
of break-ins per weatherized household: Monetized Metric: Average value of items stolen per
break-in. Houses that are weatherized often get new windows and doors that have better security
features such as locks and bolts than did the unweatherized original fixtures. This can result in better
security and household safety. For both the performance and monetary metric, the necessary data will
be collected through the previously described Occupant Survey. In both the pre- and postweatherization periods, the subjects will be asked to report the number of break-ins to their residence
during the previous year and the value of the items stolen during those incidents. An appropriate
statistical procedure will be used to compare changes from the pre- to post-weatherization period for
the treatment and control groups.
ALL air emissions—Performance Metrics: Pounds of substances (CO2, SOx, NOx, CO, CH4,
PM, heavy metals) emitted per unit of energy saved; Monetized Metrics: Value of substances
emitted in dollars per pound. Necessary data pertaining to these metrics can be collected from
secondary sources. Specifically, a literature review will be conducted regarding the amount of each
relevant substance typically emitted per unit of energy saved. This review will focus on emissions for
those households that are ―on the margin,‖ meaning that their fuel consumption is most likely to be
cut when energy use is reduced. Getting region-specific numbers for these factors should be relatively
straightforward, and emissions reductions will be able to be calculated from the energy savings
findings (see Section 2.2). To determine the monetary value of the emissions reductions, information
on the values established by the market through emissions trading for each substance will be
gathered. It should be noted that getting good, up-to-date information on the monetary metrics is even
more important than gathering data on the performance metrics because the former is surrounded by
greater uncertainty.
60
Wastewater and sewage—Performance Metric: Amount of wastewater and sewage produced
per unit of electricity; Monetized Metric: Cost per gallon of treating wastewater and sewage.
Necessary data pertaining to these metrics will be gathered through a literature review.
Direct and indirect employment—Performance Metric: Average number of direct and indirect
jobs created per dollar spent on weatherization. Secondary sources on economic multipliers will
be used to calculate the average number of direct and indirect jobs created per dollar spent on
weatherization in the geographic areas under study. Input/output models utilizing the best available
data should be useful for this purpose. To the extent possible, the analysis will attempt to identify the
net impact, which is the effect the Weatherization Program had on employment minus the
employment effect that might have resulted from the same magnitude of expenditure on likely
alternative projects.
National security—Performance Metric: Average proportion of source energy used for
residential purposes that is imported; Monetized Metric: ―Premium‖ paid in higher prices and
disturbance to economy per unit of imported energy. Data on energy imports will be derived from
the most up-to-date secondary sources. The value of the imported energy ―premium‖ will be taken
from a study currently being performed by ORNL researchers.
Once the full data-collection effort is complete and new coefficients have been developed, an
analysis will be performed to calculate values for all monetized non-energy impacts. In the final report for
this study, each coefficient used to calculate the total monetized value of non-energy impacts will be
described, and the reason it was selected will be explained. Impacts will be reported separately for each
major category shown in Tables 2.3 and 2.4 (Utility/Ratepayer Impacts, Impacts to Participating
Households, and Societal Impacts) and the total impact for all categories combined will also be given.
After all monetized impacts have been calculated, a quick sensitivity analysis will be conducted
to see how out-year estimates of non-energy impacts might change in response to variation in key driving
factors and assumptions made in the calculations, such as changing demographics in the houses, loss of
housing stock, energy prices, discount rates, new technology, and climate change. This analysis will use
the results of a prior sensitivity analysis of how energy savings may change in response to variance in the
same driving factors (see Section 2.2.2). The results of this analysis will be used in the sensitivity
analyses performed for cost-effectiveness (see Section 2.4).
Additional analyses will be performed to explore the effects of specific agency actions on various
monetized health- and safety-related impacts. This can be done by (1) performing regression analysis to
search for relationships between various impacts and agency actions (e.g., installation of smoke alarms,
security measures) that have the potential to affect health and safety; and (2) doing a literature review on
the relationships between selected agency actions and health effects.
2.3.2 Special IAQ Radon Remediation Cost Study
One of the tasks of the retrospective evaluation is to study the potential impacts of weatherization
on indoor air quality. To accomplish this goal, a large-scale field study of approximately 550 homes (the
IAQ study) is being implemented nationally. Homes for the study were selected based on a two-stage
sampling strategy that first selected geographic areas of the country, and then sampled single-family
households scheduled for weatherization by local agencies within each geographic region. The
geographic sampling was based on areas defined by the U.S. Census Bureau, known as super-PUMAs
(Public Use Microdata Areas), which are areas with a population of at least 400,000. The 532 superPUMAs in the U.S. were stratified regionally and by radon level by the Census Bureau, and a national
sample of 80 super-PUMAs was drawn with probability proportional to the Census 2000 population of
61
single-family, weatherization-eligible households and weatherization funding in PY 2008. High-radon
areas were explicitly over-sampled in order to ensure that our sample contained enough homes to
statistically explore relationships between the installation of various weatherization measures and changes
in radon in the home. 7 Local weatherization agencies serving sampled super-PUMAs were then contacted
to obtain lists of homes soon to be weatherized. Homes were randomly chosen from these lists and
contacted to be part of the study. The goal was to enroll six to eight homes in this IAQ study from each
sampled super-PUMA, with two to three of these homes randomly assigned to a control group of homes
that will not be weatherized until the study is completed.
This study is measuring pre- and post-weatherization levels of the following indoor air pollutants:
carbon monoxide (CO), radon, formaldehyde, and indoor humidity.
CO is produced by incomplete combustion from fossil-fueled heating systems, appliances, and
other combustion sources and can be a serious problem in homes. Home-energy audits under WAP
commonly assess CO production by combustion appliances in post-weatherization inspections to ensure
that the installation of weatherization measures, especially those related to furnace work, have not caused
CO problems.
Naturally occurring radon gas can accumulate in confined spaces in homes and, being
radioactive, is often responsible for the majority of a person‘s exposure to background radiation. Radon
and formaldehyde issues are not formally addressed by energy audits at the present time. Radon problems
can be mitigated by weatherization measures that air-seal unheated basements and crawlspaces from the
living areas; conversely, it can be exacerbated by overall home weatherization ―tightening.‖ At least one
prior study found no relationship between weatherization and radon risks, although that study was
conducted over two decades ago and involved a relatively small sample of homes.
Formaldehyde was included in this study to represent the larger class of VOCs that could be
present in homes. Additionally, home-energy auditors regularly make note of moisture and mold issues in
homes, and ventilation measures are often implemented to deal with these issues.
As mentioned above, the retrospective study‘s design called for over-sampling in high radon
areas. This is because the impacts of weatherization on radon in the home are a point of emphasis of this
study. The retrospective study will conduct analyses to identify any relationships between weatherization
and changes in radon levels in homes from pre- to post weatherization. This study will also assess the
relationships between various weatherization measures and changes in radon levels.
The WAP-ARRA period study will take this radon work one step further. The project team will
remediate all homes that test for radon levels over the recommended federal threshold of 4pC/L postweatherization and in a second test post-weatherization. Low-cost weatherization measures will be
implemented first, and if after re-testing the threshold is still exceeded, standard remediation actions will
be taken. Accurate cost information will be collected for all remediation work performed. This
information will be useful in estimating the costs of radon remediation performed in the low-income
weatherization context and providing insights as to whether there are any correlations between radon
levels post-weatherization and the cost of radon remediation.
7
We used county-level estimates results from both Lawrence Berkeley National Laboratory‘s radon research and the
Environmental Protection Agency to define four radon-level strata based on the estimated proportion of homes at the
county level with radon levels exceeding 4 pCi/l: see http://eetd.lbl.gov/iep/high-radon/hr.html and
http://www.epa.gov/radon/zonemap.html. County level data were then aggregated to the super-PUMA level using
Census 2000 population counts.
62
2.3.3 Non-Monetized Data Collection and Analysis
Performance metrics to determine the value of all non-monetized impacts will be calculated
directly from the relevant household-level data shown in Table 2.3. The appropriate performance metric
for each non-monetized non-energy impact is identified in Table 2.4. The methods that will be used to
collect and analyze the relevant data for each non-monetized impact are described below. As with the
monetized non-energy impacts, to the extent possible, results will be developed that are specific to
geographic region, climate region, and/or housing type.
For most of the non-monetized impacts described below, information will be obtained using a
portion of the S4 Occupant Survey. The survey will be administered immediately after each house is
audited and again a year after weatherization, and an incentive will be provided to improve the response
rate.
The other main data-collection approach used for the study of non-monetized health, safety, and
comfort impacts is the direct measurement of key indoor air quality factors. These data will be collected
by means of the Indoor Air Quality study, part of the retrospective evaluation being conducted in PYs
2010, 2011 and 2012.
Non-monetized impacts include the following:
Fewer fires—Performance Metric: Perceived changes in safety of heating system and electrical
wiring in weatherized houses. Using the Occupant Survey described above, program participants
and a control group of non-participants will be asked for their perceptions of the safety of their
dwelling‘s heating system and electrical wiring before and after the period in which weatherization
work is performed. Net change in perceptions from the pre- to post-weatherization period will be
determined by comparing change for the treatment and control groups using an appropriate statistical
procedure.
Changes in frequency of health problems—Performance Metric: Perceived change in health
problems in weatherized houses. Program participants and a control group of non-participants will
be surveyed using the Occupant Survey regarding the perceived condition of their (and their family‘s)
health in the pre- and post-weatherization periods. An appropriate statistical procedure will be used to
compare the change from the pre- to post-weatherization period for the weatherized and control
groups.
Changes in frequency of health problems—Performance Metric: Change in incidence of
symptoms or occurrences of specific health problems in weatherized houses. Through the
Occupant Survey, program participants and non-participants will be asked to report the frequency
with which they experience certain health problems or the principal symptoms of those problems
during the pre-weatherization period and again during the post-weatherization period. Subjects will be
asked specifically about occurrences of asthma and other respiratory problems, as well as colds and
flu. They will also be asked to report symptoms of these and other conditions possibly resulting from
exposure to allergens, mold, and CO. Such symptoms include wheezing, shortness of breath,
coughing, congestion, headaches, and nausea.
Enhanced prevention and treatment of health problems—Performance Metric: Reduction in
number of times food could not be purchased due to size of utility bill in weatherized houses.
The Occupant Survey will ask weatherization clients and a control group about the frequency with
which they have foregone the purchase of food in order to pay utility bills and the frequency with
which they have not paid utility bills in order to purchase food. The same questions will be asked
63
during the pre- and post-weatherization periods, and any changes from the former to the latter will be
calculated. The net change from the pre- to post-weatherization period will be determined by
comparing change for the treatment and control groups using an appropriate statistical procedure.
Enhanced prevention and treatment of health problems—Performance Metric: Change in
access to health care and medication in weatherized houses. The Occupant Survey will ask
residents of weatherized and non-weatherized households questions similar to those described above
for food purchases about the trade-offs they have made between paying utility bills and purchasing
prescription medicines. Changes from the pre- to post-weatherization period will be compared for the
weatherized and control groups. In addition, the weatherized and non-weatherized groups will be
compared in terms of how their number of visits to emergency rooms, primary physicians, or other
primary health-care providers changed from the pre- to post-weatherization period. The frequency
with which respondents report not having a primary-care physician or other primary health-care
provider will also be examined.
Changes in indoor air quality—Performance Metric: Measured change in CO levels in
weatherized houses. As part of the retrospective evaluation‘s Indoor Air Quality task, mentioned
above, equipment will be installed in weatherized housing and control units to measure carbon
monoxide (CO) levels during both the pre- and post-weatherization periods. The non-weatherized
houses will be measured at the same time as the weatherized units to control for any possible changes
in outdoor temperature or other climatic conditions. Net change in CO levels from the pre- to the
post-weatherization period will be determined by comparing change for the treatment and control
groups using an appropriate statistical procedure. In addition, descriptive statistics showing the
frequency with which dangerously high levels of CO were found during the pre-weatherization period
will be generated and an appropriate statistical test will be used to calculate the frequency with which
those high concentrations were reduced to safe levels in weatherized units.
Changes in indoor air quality—Performance Metric: Perceived change in frequency of odors
within weatherized houses. The Occupant Survey will ask a sample of weatherization participants
and non-participants for their perceptions of how often there are odors inside their home that could
indicate a problem with indoor air quality. The occupants will be asked to describe the situation
separately for the winter and the summer, and the survey will be administered both before and after
weatherization. Net change from the pre- to post-weatherization period will be determined by
comparing changes for the treatment and control groups using an appropriate statistical procedure.
Changes in household moisture levels—Performance Metric: Measured change in humidity
levels in weatherized houses. Relative humidity will be measured in the same housing units for
which the concentrations of CO, indoor mold spores, and pollen will be examined, during both the
pre- and post-weatherization periods. The non-weatherized units will be measured at the same time as
the weatherized units, and careful attention will be paid to the season in which the measurements are
taken. Net change in humidity levels from the pre- to post-weatherization period will be determined
by comparing changes for the treatment and control groups using an appropriate statistical procedure.
Decreased incidence of hypothermia and hyperthermia—Performance Metric: Change in
incidence of students‘ disrupted study in weatherized houses. The Occupant Survey will ask how
frequently household residents find it hard to study at home because of excessive heat or cold. The
net change in frequency of study disruption from the pre- to post-weatherization period will be
determined by comparing change for the treatment and control groups using an appropriate statistical
procedure.
64
Improved food safety—Performance Metric: Measured change in refrigerator temperature in
weatherized houses. In the same housing units for which CO, indoor mold spores, pollen, and
relative humidity are examined, the change in the internal temperature of refrigerators will also be
measured. Temperatures will be taken inside the household refrigerator both before and after
weatherization, and the net change in refrigerator temperature from the pre- to post-weatherization
period will be calculated from both the treatment and control group data using an appropriate
statistical procedure. In addition, descriptive statistics will be generated showing the frequency with
which unsafe temperatures were found inside refrigerators during the pre-weatherization period, and
an appropriate statistical test will be used to calculate the frequency with which those high
temperatures were reduced to safe levels in weatherized units.
Improved food safety—Performance Metric: Change in incidence of gastrointestinal problems
and food poisoning in weatherized houses. The Occupant Survey will ask people in weatherized
and non-weatherized households whether they have experienced serious gastrointestinal problems in
the previous month or have suffered from food poisoning during the past six months. The same
questions will be asked both before and after weatherization, and the changes will be compared for
the weatherized and control groups.
Improved household safety and security—Performance Metric: Perceived change in security
from criminal intrusion in weatherized houses. Using the Occupant Survey, program participants
and a control group of non-participants will be asked for their perceptions of how secure their home is
from intrusion by criminals both before and after weatherization. The change over time will be
compared for the weatherized and control groups using an appropriate statistical procedure.
Change in presence of environmental hazards—Performance Metric: Measured change in
levels of radon in weatherized houses. In the same dwellings for which CO and other factors will be
measured, the levels of radon will be measured during both the pre- and post-weatherization periods;
net change will be calculated as described previously. In addition, descriptive statistics will be
generated showing the frequency with which unsafe levels of asbestos and radon were found during
the pre- weatherization period, and an appropriate statistical test will be used to calculate the
frequency with which those high concentrations were reduced to safe levels in weatherized units.
Change in presence of environmental hazards—Performance Metric: Change in number of
poisonings from household chemicals in weatherized houses. Through the Occupant Survey,
subjects in both weatherized and non-weatherized dwellings will be asked to report whether the
members of their household had been poisoned by household chemicals during the past year and, if
so, to identify the substance with which they had been poisoned. Net change from the pre- to postweatherization period will be determined by comparing change for the treatment and control groups
using an appropriate statistical procedure. Additional data on the frequency of poisoning from
household chemicals will be taken from reliable secondary sources.
Change in presence of environmental hazards—Performance Metric: Change in level of vermin
infestation in weatherized houses. The Occupant Survey will solicit information from both
weatherization participants and non-participants on the extent to which their dwelling is infested with
rats, cockroaches, and other vermin. This question will be asked both before and after weatherization.
The net change in this factor from the pre- to post-weatherization period will be determined by
comparing change for the treatment and control groups.
Improved comfort—Performance Metric: Perceived improvement in indoor comfort in
weatherized houses. A sample of Program participants and a control group of non-participants will
65
be asked about their perceptions of the indoor comfort of their dwelling unit during the pre- and postweatherization period via the Occupant Survey. ―Comfort‖ will include both indoor temperature and
draftiness, as in the 1989 Weatherization Program national evaluation, as well as the floor area or
number of rooms that can be conditioned. Responses will be solicited during both the heating and
cooling seasons. Net change from the pre- to post-weatherization period will be determined using an
appropriate statistical procedure.
Improved comfort—Performance Metric: Measured change in indoor air temperature in
weatherized houses. Equipment will be installed in the previously mentioned homes in the
previously mentioned Indoor Air Quality study to measure indoor winter and, if possible, summer air
temperatures during both the pre- and post-weatherization periods. The non-weatherized houses will
be measured at the same time as the weatherized units to control for any possible changes in outdoor
temperature or other climatic conditions, and careful attention will be paid to the season in which the
measurements are taken. Net change from the pre- to post-weatherization period will be determined
by comparing change from the treatment and control groups using an appropriate statistical
procedure.
Improved appearance—Performance Metric: Perceived improvement in appearance of
weatherized dwellings. Using the Occupant Survey, a sample of Program participants and a control
group of non-participants will be asked for their perceptions of the appearance of their dwellings both
before and after weatherization. Changes from the pre- to post-weatherization period will be
compared for the treatment and control groups.
Reduced noise inside dwelling—Performance Metric: Perceived reduction in noise within
weatherized dwellings. The Occupant Survey will ask a sample of weatherization participants and
non-participants for their perceptions of the noise level within their dwellings both before and after
weatherization. Once again, net change from the pre- to post-weatherization period will be
determined by comparing change for the treatment and control groups using an appropriate statistical
procedure.
As described above, the magnitude of each non-monetized impact will be calculated separately.
In addition, the effects of specific agency actions on various non-monetized health- and safety-related
impacts will be explored. This can be done by (1) performing regression analysis to search for
relationships between various impacts and agency actions (e.g., plumbing repairs, improved ventilation)
that have the potential to affect health and safety; and (2) doing a literature review on the relationships
between selected agency actions and health effects.
2.3.4 Social Network Study
The non-energy benefits research described above addresses many additional ways that the
Program can generate benefits to clients and society. This section addresses yet another potential benefit
of the Program: the potential for two groups, clients and the weatherization staff, to influence energy
savings beyond their homes and their day jobs. Every individual and every household has a network of
social contacts that may be visualized as a map of nodes (individuals/households) and lines (connections).
The analysis of social networks (Social Network Analysis or SNA) will be employed to explore linkages
between individual households, weatherization staff, and agencies as nodes within a multi-level and
multi-relational social system that could influence energy savings beyond the Program.
66
Taking advantage of the natural tendency to build social networks to create a system of
information flow in which information about weatherization can be shared, SNA will allow evaluators to
visually capture the structural attributes of the network and relational data in order to provide both
statistical and visual analysis.
The clients will be studied to determine whether and how they communicate with neighbors,
acquaintances, friends, and family about the results of their home audits, the measures installed, any
changes in their energy bills, and any other energy and non-energy benefits.
In the case of weatherization staff, technicians have social networks of their own. This study will
serve to identify and map ways in which the WAP program encourages secondary impacts on home
energy savings through social networks. The timeframe for this project extends from March 2010 through
September 2011 to allow time for culturally sensitive implementation plans for each agency, hiring and
training of research staff, sampling, interviewing, and analysis.
The communities in our pilot, though economically marginal, are likely to be sophisticated in
social media usage. Any network analysis must start with a population‘s access to modes of
communication from letter writing to modern social media such as online forums, listservs, instant
messaging, Twitter, and Facebook.
This project will leverage the results from a task being undertaken by the retrospective evaluation
to conduct case studies on approximately six high performing local agencies. The retrospective
evaluation‘s case studies will describe local agency operations, philosophies, and approaches, and how
the local agencies reach out to surrounding communities to promote weatherization and other social
services. In addition, these case studies will describe how the agencies act as nods of communication
about energy efficiency within the community.
The SNA employed in this evaluation will add to the retrospective study‘s results, enhancing the
description of the network by adding clients and weatherization staff as nodes in the community network.
Upon identification of the high performance agencies and communities to be assessed, census
tracking, and other forms of structural data will be collected for the WAP-ARRA evaluation to reveal
community demographics with regards to socio-economic status of the community as a whole, individual
income levels within particular neighborhoods, racial or ethnic make-up, median age, disability rates,
number of single parent homes, unemployment, population density, and education statistics. This
information will provide potential factors relating to the high performance rate of the agency, as well as
cultural implications related to social networking systems. Quantitative and qualitative data will be
evaluated for a rich ethnographic understanding of the community as a whole. This reading of the
community will help understand how information within the community is shared (possibly feeding a
viral spread of interest in weatherization), and what the core values are in place that might support or
hinder adoption of new energy practices.
To conduct the client-based SNA, a list of weatherized homes will be supplied from the agencies.
A visual spatial map will be created using this list. This visual aid will assist with identifying high
utilization clusters within the community and will assist with identifying initial client (i.e., household)
nodes for this analysis. Upon selection of the preliminary household nodes to initiate the study of social
networking around the household, in-person interviews will be conducted with the clients to collect
information to build frameworks of their social networks. Snowball sampling methods will be used to
contact people within the clients‘ social networks via telephone or in person. Researchers will talk with
members of the client‘s network to learn more about their interest in weatherization. By soliciting broad
information, rather than targeted ―source of influence,‖ the respondent may volunteer unanticipated
67
information beyond the scope of initial researcher knowledge. Researchers will attempt to determine the
impact the clients‘ weatherization activities had on the behavior of other individuals, measured by the
number of applications for Program services, household energy-use behavior, and influence on
investment in energy-efficiency measures. Individuals one step away from the initial client nodes will be
asked for a list of other nodes in their own social networks and a second round of snowball interviews
will be conducted. The goal is to conduct approximately 100 interviews per agency.
To gather information about the household nodes and to create a social network map, local
recipients of the program will be recruited and hired through the local agencies. This technique, known as
participatory research, will allow information to be collected starting with the initial client as a node.
Efforts will be made to recruit researchers who are well versed in the culture of the area, have good
interpersonal skills, and are familiar with the WAP application process and with energy conservation.
These researchers will be trained by research professionals before collecting data within their local areas,
and once trained, will make contact with the initial client nodes, paying special attention to whether or not
the household either may have served as a catalyst for energy change within the community or its own
social network, or conversely, whether the household was influenced by another household that could
better be identified as a catalyst. SNA of the household nodes will present a visual analysis at the
conclusion of the study and will offer insight into program utilization process; this analysis will further
provide data for a secondary impact analysis of the Program‘s influence on home energy conservation.
To conduct the SNA of the weatherization staff, local workers employed by the high-performing
agencies within the retrospective study‘s Case Study Analysis will be interviewed to gain insight
regarding the impact these local workers have on the rate of homes weatherized within their own social or
professional networks or communities. Weatherization staff will be interviewed to initiate a second
snowball sampling with follow-up telephone surveys. Additionally, the weatherization staff survey will
be modified for this study to include questions related to communication of the program within social
networks.
This approach (use of participatory techniques within client populations and partnering with
community agencies) is reflective of OMB‘s Open Government Directive, which emphasizes the need to
incorporate transparency, participation, and collaboration with multi-disciplinary approaches into the
design, implementation and analysis of federal programs and projects. The participatory research
technique also offers the benefit of empowering clients and workers and work experience that may extend
those individuals‘ employment opportunities.
2.4. COST-EFFECTIVENESS
2.4.1 General Assessment
The impact assessment will determine the cost-effectiveness of the Program as implemented in
PYs 2009, 2010 and 2011 on a national basis and by climate region, housing type, primary space-heating
fuel type, and the five client groups that the Program is specifically instructed to focus on (the elderly,
persons with disabilities, families with children, high residential energy users, and households with high
energy burden). The cost-effectiveness of the Program in PYs 2009, 2010 and 2011 will then be
compared to results from the retrospective evaluation (PYs 2007 and 2008) and the 1989 National
Evaluation and from the meta-evaluations performed between 1990 and 2005. It should be noted that,
although cost-effectiveness will be calculated by climate region, housing type, and type of primary spaceheating fuel, a full analysis of factors affecting cost-effectiveness will be performed as described in
Section 2.5.
68
The cost-effectiveness of the Program will be calculated using appropriate methods for
coordinated programs (Brown and Hill, 1994; Hill and Brown, 1994). Specifically, cost-effectiveness for
WAP-ARRA will be calculated using the total costs spent on the house from all funding sources as
collected and analyzed in Section 2.1, the energy cost savings calculated in Section 2.2, and the monetary
values of the non-energy impacts (which may include both benefits and costs) estimated in Section 2.3.
Cost-effectiveness will be determined using savings-to-investment ratio (SIR), the same indicator as used
in the retrospective and the 1989 National Evaluation. Standard formulas for this indicator will be used.
Cost-effectiveness will be examined from three perspectives:
Installation perspective—―savings‖ are limited to energy savings (all heating, cooling, and
baseload energy savings combined), and ―investments‖ (i.e., costs) are limited to installation
expenditures (on-site labor and materials),
Program perspective—―savings‖ are limited to energy savings as above, but ―investments‖ are
expanded to include management and overhead costs along with installation expenditures, and
Societal perspective—―savings‖ include both energy savings and monetary values for nonenergy impacts (which may include both benefits and costs and, therefore, are net economic
values), and ―investments‖ include installation, management, and overhead expenditures.
To calculate the SIR, it is necessary to include information about the average lifetime of
measures, such as refrigerators or windows, that are implemented. This information will be determined by
weighting the individual lifetimes of each measure (as determined from secondary sources) by the
frequency of its installation and relative energy savings. The monetary values of the non-energy impacts
used in the calculations will be the net present value of the impact and, thus, will already have taken into
account the lifetime of the impact and how the impact varies over time. Real discount rates and fuel
escalation rates as recommended by the Department of Commerce will be used in the calculations.
A sensitivity analysis will be performed to determine the impact of key assumptions used in the
calculation of SIR. These key assumptions include energy savings, fuel costs, measure lifetime, real
discount rate, fuel escalation rate, and the monetary value of non-energy impacts. The risk analysis
modeling approach to be used allows the uncertainty in model inputs to be defined by probability
distributions, so that distributions of likely SIR outcomes can be developed. Results from the sensitivity
analysis performed specifically for energy savings (see Section 2.2.2) and non-energy impacts (see
Section 2.3.1) will be used in this analysis.
As part of the cost-effectiveness analysis, the impact that alternative per-household investment
levels can have on Program cost-effectiveness and other key Program metrics, such as the number of units
weatherized and average energy savings, should be examined (i.e., it should be examined whether there
are investment levels that optimize the SIR at an agency or state level and, if so, how this subsequently
impacts the number of units weatherized by the agency or state and the average energy savings per
weatherized unit). The analysis method to determine this impact has not yet been determined. However,
in addition to a review of the relevant research, insight into the impact of household investment levels
might be obtained by two possible methods:
Calculating SIRs for different expenditure categories of weatherization jobs (e.g., those costing
between $1,500 and $2,000, between $2,000 and $2,500, up to $6500 etc.) and then comparing
the means of each category using appropriate statistical methods. Information on the houses used
in the billing data sample would be sufficient to perform such an analysis
69
Using an audit program such as the Weatherization Assistant (Gettings, 2006). A significantly
large number (perhaps 100 units) of real or typical houses could be modeled for a real or example
agency to identify all the measures with an SIR greater than 1.0 for each house and then to
estimate the cost and SIR for each of these individual measures. The cost-effectiveness and
average investment level for this agency can be calculated for a fixed budget that is sufficient to
install all the measures with an SIR greater than 1.0 in, perhaps, the first 50 houses, as well as to
perform all administrative functions associated with these 50 houses (e.g., intake, auditing, and
post-weatherization inspections). The cost-effectiveness and average investment level for the
agency would then be recalculated as the SIR cutoff, which is used to determine which measures
are installed in each house, is raised (or the average investment level per house is decreased),
such that fewer measures are installed in these first 50 houses but measures with high SIRs are
installed in additional houses until the same fixed budget is expended. If desired, houses used in
the billing data sample could be modeled, and the model predictions could be calibrated to actual
energy data. The analysis could be repeated using different costs to perform the administrative
functions to gain insight into how the optimal SIR cutoff and average investment level changes as
these fixed costs change.
2.4.2 In-Depth Cost Assessment of Weatherization Measures
It is possible that ARRA funding for weatherization and other energy-related programs (e.g.,
State Energy Program, Energy Efficient Community Block Grant Program) had an impact on the cost and
availability of weatherization measures. This study will assess the changes in the costs of weatherization
measures, as well as bottlenecks in their availability, during the ARRA period and related impacts on
cost-effectiveness. This study will also compare the costs used in SIR calculations to those actually
encountered in the field.
One way to perform this assessment is by using linear regression analysis to find the Total
Installed Measures Costs (TIMC). The general form of the equation to be estimated is as follows:
TIMCi = f( b1*M1 + b2*M2 + b3*M3 + b4*M4 + ….. bn*Mn)
where TIMCi stands for Total Installed Measures Costs for home, and Mi is a ―dummy‖ variable
indicating whether or not that measure was installed in any given home i. Up to n different measures could
be installed in a home. The beta coefficients represent estimates of each measure‘s cost.
Information about the Total Installed Measure Costs and the measures installed in each home will
be available through the DF2 (Housing Unit Information Data Form), which local agencies will complete
for each home (single-family detached, single-family attached, and mobile home) included in the national
sample of weatherized homes heated with electricity or natural gas. The database is expected to include
approximately 10,000 records, which is a database large enough to estimate regression models that are
stratified by climate region at the very least and maybe also by housing type (e.g., single-family detached
vs. mobile home by climate region).
Models will be estimated for PY2008 and PY2010. Beta coefficients for each measure will be
compared between the models to estimate possible measure price changes. It is possible that the PY 2008
coefficients will also be adjusted for inflation.
Controls to make sure that the beta coefficients are valid as accurate estimates of measure prices
will be done in three manners. First, specific coefficients for specific measures will be compared to prices
for analogous measures compiled by the Producer Price Index (PPI) (http://www.bls.gov/ppi/) and
possibly the Consumer Price Index (CPI) (http://www.bls.gov/cpi/). The PPI tracks prices related to air
70
conditioners, furnaces, lights, and insulation, and the CPI tracks prices related to such consumer goods as
lights and energy. Second, the estimated regression models will be run on the data agency-by-agency to
estimate total measure costs by sampled agencies. These results will be compared to total reported
measure costs by agency. Third, a small number of questions about the cost of weatherization measures
will be included in S3 (Subset of Agencies Detailed Program Information Survey, see Appendix E) to
collect subjective estimations of changes in measure costs from PY 2008 to PY 2010.
A similar approach will be used to explore possible changes in labor costs. Regression models
will be run with Total Labor Costs per home as the dependent variable and ―dummy‖ variables for
measure installation. The beta coefficients will represent the costs for installing each measure. Beta
coefficients will be compared between PY 2008 and PY 2010 (the former may be adjusted for inflation).
As a check on the validity of these beta coefficients, the database Occupational Employment Statistics
(http://www.bls.gov/oes/), which has data on contractor wages, will be consulted.
Validated measure costs will be compared to measure costs used in SIR calculations during PY
2010, as appropriate by climate region and house type. NEAT will be used as the source of the SIR
calculations. Bottlenecks in measure availability will be explored through the addition of several
questions to S3 (Subset of Agencies Detailed Program Information Survey, see Appendix E).
2.5 EXPLANATORY FACTORS
Although average energy and cost savings will be calculated in the impact assessment by climate
region, housing type, and primary space-heating fuel type (see Sections 2.2 and 2.4), a full analysis of
factors that explain variations in energy savings and cost-effectiveness will also be performed. The impact
assessment will assess how the energy savings achieved by the Program and the cost-effectiveness of the
Program are affected by the various organizational features and operational processes of the Program, the
households the Program serves, the measures installed, and the environment in which the Program
operates. Some specific factors that will be examined include the following:
household pre-weatherization energy consumption,
installation of particular weatherization measures,
key house characteristics (e.g., type, size),
key occupant characteristics (e.g., age, disability),
fuel prices,
climate zone,
training methods for weatherization crews,
type of audit used,
client education approach used,
monitoring procedure employed,
total investment levels,
funding sources,
low and high material expenditures (as opposed to total expenditures, which include labor costs),
weatherization using only DOE funds vs. funds from multiple sources,
air-leakage reduction,
duct-leakage reduction, and
increased furnace steady-state efficiency.
A broad range of potential explanatory variables will be examined using regression analysis. In
addition, average savings associated with and without a single factor will be compared using all houses,
and mean savings for explanatory factors will be compared between high-saving and low-saving houses.
71
Those factors that explain the most variation and are controllable by state and local weatherization
agencies will be given the most attention, because results in those areas can suggest potentially valuable
changes in program implementation. Special emphasis will be placed in these analyses on identifying
variables that explain why the performance in the hot-climate region is unique. Details on these analyses
are provided below in Section 2.5.1.
2.5.1 Regression Analysis
The primary analytical approach that will be used to study explanatory factors will be regression
analysis. The regression analysis will explore the relationships between household energy savings and
cost-effectiveness and a broad variety of factors; this form of analysis has the potential to explain
variations in those two performance measures, including many of those identified above. By examining
the possible influence of pre-weatherization energy use, it will be possible to identify whether any of the
observed relationship is due to the regression-to-the-mean phenomenon (in which extreme measurements
are subsequently followed by measurements closer to the norm).
Hypotheses will be developed concerning a priori expectations of the influences of each
independent (explanatory) variable on the dependent variables (savings and cost-effectiveness). Energy
savings will be measured first in absolute terms, secondly as a percentage of pre-weatherization wholehouse energy use, and thirdly as a percentage of the pre-weatherization energy used for space heating.
Cost-effectiveness will be defined as energy savings divided by the cost required to achieve those savings.
The results of the regression analyses will be examined, and significant beta coefficients of the proper
sign will provide support for the hypotheses. Insignificant variables will be dropped from the regression
models.
Separate regression analyses will be run for houses heated by natural gas, electrically-heated
houses, and houses heated by non-metered fuels (fuel oil and propane). Within those categories, the
factors influencing energy savings and those influencing cost-effectiveness will be examined separately.
For all dwellings other than those heated with electricity, further analyses will be performed to focus on
factors affecting savings of the primary heating fuel only, baseload electricity only, and both fuels
(heating and baseload electricity) combined.
Houses Heated by Natural Gas—As part of the regression analysis, a large multiple regression
model that includes all potential explanatory variables will be run . It is likely that this will be a
―stepwise‖ regression, in which independent factors are added to the model in the order of their
explanatory significance. In addition, a series of simple regressions will probably be conducted using one
independent variable at a time, and a factor analysis will likely be done to examine what sets of
explanatory variables are associated with each other.
The above analyses will be run first using energy savings as the dependent variable and then
again using cost-effectiveness as the dependent variable. Actually, the analysis of energy savings will
consist of three different analyses, one for each of the above-mentioned definitions of savings (absolute
household savings, household savings as a percentage of pre-weatherization whole-house energy use, and
household savings as a percentage of the pre-weatherization energy used for space-heating). Further
complexity will be added by the fact that energy savings will be first defined as savings of the primary
heating fuel only, then as baseload electricity savings, and finally as savings of both fuels (heating and
baseload electricity) combined.
Once the above analyses are run for all weatherized households, they will be run again for
relevant subsets of households. These subsets will include, but are not limited to, different geographic
and/or climate regions, agency sizes, and housing types.
72
Finally, regression analyses will be run using average household savings and cost-effectiveness
per weatherization agency as the dependent variable. The same independent variables and analytical
procedures listed above will be used. The theory behind undertaking this analysis is that using agency
averages is likely to reduce the variability of results.
Electrically-Heated Houses—The regression analyses conducted for electrically-heated homes
will be the same as those described above for gas-heated homes, except that they will be run only for the
primary heating fuel because the dwellings involved use electricity both for heating and baseload
purposes.
2.5.2 Cross-Tabulation Results
Average savings associated with a single factor (e.g., savings in houses that received wall
insulation compared to houses that did not) will be compared for all houses and by subgroups of houses
depending on primary heating fuel. Mean values for key explanatory factors (e.g., floor area, preweatherization energy use, installation of attic insulation) for high-saving and low-saving houses and for
high-saving and low-saving agencies will also be calculated and compared.
2.5.3 Data
This analysis of explanatory factors will use pre-weatherization energy consumptions and energy
savings as described in Section 2.2, and cost-effectiveness calculated for individual houses as described in
Section 2.4. This analysis will also draw upon data relating to house, occupant, and program
characteristics as described in Section 2.1. Information on some potential explanatory factors will be
gathered for all weatherized houses, while data on other factors will be collected from a subset of houses
if they are not available for all dwellings served by the program. Average data for a number of factors will
be calculated for all weatherization agencies, and some additional information will be collected from a
selected group of agencies.
73
3. PROCESS ASSESSMENT
The process assessment portion of the evaluation will address many of the questions that were
identified in the evaluation design matrix (see Table 1.2), but which the impact assessment did not
address:
Context—Questions 1, 5, 7 and 9-11; and
Implementation—Questions 3 and 5.
The above questions relate to identifying the missions and associated objectives of the Program and the
organizations that the Program works with; understanding how the weatherization network works and the
role of the Program in serving the low-income weatherization market; investigating the leveraging and
partnership opportunities for the Program; and determining how well the Program is delivering services as
well as how essential elements of the Program can be improved.
3.1 PROGRAM OPERATIONS AND IMPLEMENTATION
This assessment will aim to understand the context in which the Program operates by
identifying the legislatively mandated goals (missions and associated objectives) of the Program
and determining the Program‘s ability to meet these goals;
identifying how states implement the Program relative to the logic model developed for the
Program (see Section 1.1.2);
determining how well DOE manages and administers the Program;
determining how well the Program and the weatherization network are delivering services to the
low-income weatherization market;
identifying the leveraging and partnership opportunities the Program is exploiting (i.e.,
identifying the degree to which states and agencies coordinate the implementation of the Program
with other federal, state, utility, and similar programs) and determining whether the Program‘s
regulations are enhancing or inhibiting leveraging and partnership opportunities (or doing both);
and
determining the role the Program plays in the larger low-income energy-assistance effort.
3.1.1 Data and Surveys
Information on process improvement will be collected from all states as part of S1 (All States
Program Information Survey, see Appendix C) and from the 400 agencies involved in the billing data
energy-savings portion of the impact assessment (see Section 2.2.1) as part of S3 (Subset of Agencies
Detailed Program Information Survey, see Appendix E). Both surveys will also ascertain states and
agencies views of how DOE manages and administers the Program and will collect detailed input on how
the states and agencies implement house audits, client education, training, and monitoring, as previously
described in Section 2.1.1. The state survey will also collect information that defines state
―implementation models.‖ These surveys will be implemented after each state and agency completes PYs
2010 and 2011.
74
DOE will be surveyed using the DOE Survey (see Appendix B) to collect information on staffing,
costs, and their enforcement of state and local agency data collection, storage, and data mining
capabilities. This survey will be given when PYs 2010 and 2011 are complete for all states.
Open-ended interviews will be held with DOE and a subset of states and agencies via telephone
or by means of one or more group meetings to solicit needed process information for this study. The
sampling plan and survey instruments associated with this data collection will need to be developed;
however, the survey instrument should include the following information:
major strengths/positive traits at each implementation level (DOE, state, and agency),
major weaknesses/negative traits at each level,
major barriers to effective operation posed by each level and outside partners,
adequacy of current resources,
suggested reallocation of current resources,
suggested allocation of additional resources,
coordination,
communications, and
Program administration.
The Occupant Survey that is being implemented as part of the retrospective evaluation contains a
customer satisfaction section, so this survey will collect information such as the timeliness of the work
performed (e.g., audit, measure installation, inspection), whether agency staff reported at scheduled times,
the condition in which workers left the house following their work (e.g., cleanliness, debris removal), and
the courtesy of the agency personnel.
3.1.2 Analysis
The mission and objectives of the Program and DOE‘s management structure and responsibilities
will be described. This will include a summary of:
the Program‘s legal authority and its regulatory framework;
the goals, objectives, and key performance measures as viewed by Congress, the Department, and
the Administration; and
Federal, state, and local responsibilities as described by both regulation and network participants.
In addition, particular attention will be paid to leveraging activities allowed under the Programs
rules and regulations.
An analysis of the management structure, responsibilities, and resources for each of the
Program‘s management levels (i.e., headquarters, regions, state, and local agencies) will be performed.
This analysis will focus on operations at each management level, the allocation of human and funding
resources to various functions, and whether each management level perceives the resources it has to
perform its functions both within and between management levels. A secondary focus of this analysis will
be the perceived barriers to effective operations imposed by each management level on other management
levels and barriers at each management level to effective operations with outside partners.
Organizational activities relating to how well Program services were delivered will be identified,
and a measurable indicator for each activity will be developed (e.g., whether an audit was performed on
time, whether measures were installed when scheduled, etc.). Information collected from all agencies or a
75
subset of agencies on all houses or a subset of houses weatherized will be used to determine the average
values for these indicators.
Approximately five typical models of how states administer and implement the Program will be
developed based on information collected from the state survey and from reviewing state weatherization
plans submitted to DOE. How well these models work, and how well they fit the theory and logic of the
Program, will be discussed using the logic model developed for the Program (see Section 1.1.2) as a
guide. Lessons learned from the various approaches, model dependencies, and key issues and
administrative concerns affecting model effectiveness are expected to be identified.
Occupants‘ perceptions on how well Program services were delivered will be compared to the
perceptions provided by the agency staff and crews and the comparisons analyzed.
The role of the Program in the overall low-income energy assistance effort will be assessed by
drawing together information from the impact assessment on the Program characteristics (see Section 2.1)
with information collected and analyzed in this study. Program characteristics that will be used include
information on the national low-income population, the segment of the market currently being served by
the Program, and the characterization of the local and state agencies and the scope of their programs.
3.1.3 Deferral Study
The Process Assessment for the WAP-ARRA Evaluation includes an exploratory study of
federal, state, and local weatherization agencies‘ policies, procedures, incidence rates, and final
weatherization service status regarding deferrals, (i.e., situations where agencies decide to postpone
weatherization till a later date or ultimately where weatherization services are not provided). Reasons for
deferral of weatherization treatment are generally categorized into two areas:
Health and Safety – conditions in the home may pose health and/or safety risks to weatherization
workers; and
Cost-effectiveness – home may be in such disrepair that weatherization measures might not result
in cost effective energy savings.
Regional and state field guides or manuals provide guidance for policy and acceptable reasons for
deferment. OSHA standards for Health and Safety for weatherization auditors, contractors, and crew
members must be adhered to and are referred to within the Health and Safety section of the OSHA field
manual.
This exploratory research study will utilize information from the state, agency, weatherization
staff/occupant surveys, the retrospective case-study process evaluation, and data collected from a sample
of 10 states and 10 local agencies that describes the processes experienced by 200 occupants engaged in
the deferral process. Appendix O, Section 5, contains an in-depth discussion of the sampling plan for this
study.
The exploratory design involves documenting the accessibility and use of field manuals (which
may have deferral guidelines, reported deferral frequency rates, and agency protocols and standards. The
first task to be undertaken is to identify each state‘s deferral protocols for PY2010. Table 3.1 presents the
structure for preliminary data collection in order to shape the exploratory design of the process
assessment; this structure will be updated throughout the data-collection process.
76
Table 3.1 Deferral Protocol for All States and Territories
State or Territory
Regional Guide
Only
State Guide
Agency Guide
Standard Forms
Appeal Procedure
Appeal Forms
Grievance Forms
Referral Plan
Referral
Documentation
Tracking System
for Deferral
Incidence
The conditions under which deferrals may be granted vary from state and state, and potentially,
from agency to agency. The reasons that people abandon a housing unit will initially be categorized by
the evaluators based on the list of deferral conditions posted by the Weatherization Assistance Program
Technical Assistance Center (WAPTAC) and on additional conditions found in regional protocols.
Additional conditions suggested by weatherization staff or surveyed occupants, or from state or agency
survey instruments, may be added.
The retrospective evaluation‘s versions of S1 (All States Program Information Survey) and S2
(All Agencies Program Information Survey) will be modified to include requests for data designed
specifically to capture tracking of deferral rates and to confirm sources for deferral policies and
procedures nationally.
The weatherization staff survey, also being conducted as part of the retrospective evaluation, will
be administered nationally to 271 auditors between Spring 2011 and Summer 2011. Specific questions
about deferral policy, procedure, and frequency of specific conditions or reasons for deferring services
have been included in a revised version of this staff survey. Responses collected from the survey will be
compared to state deferral-protocol standards, while incidence rates reported (if they were in fact tracked)
for PY2010.
Before beginning to collect data for the records analysis, evaluators will observe at least four
deferrals in process. The four agencies selected for observation will be based on the criteria in Table 3.2.
Evaluators will be instructed to note the following:
cause of deferral,
assessment of impact of accessibility to standard deferral plans,
utilization of standard forms,
auditor knowledge of community resources (e.g., other social services available to the household)
targeting the specific cause(s) for the deferral,
innovative techniques for preventing potential deferrals,
range of subjective or ethnocentric responses to home conditions (i.e., auditors and/or clients
judgments about conditions in the homes) or client behavior, and
77
other characteristics of the homes and weatherization personnel. A case-study process evaluation
will be conducted to record the overall assessment of the observed deferral occurrences; this
process evaluation will entail interviews with clients and weatherization staff, review of available
documentation, and analysis of collected data.
Table 3.2. Deferral Observation Assessment Criteria
Formal Deferral Plan; Low Incidence
Formal State Deferral Plan; High Incidence
No State Formal Deferral Plan; Low Incidence
No State Formal Deferral Plan; High Incidence
Assessment of the deferral process will follow a sequence of events during PY 2010, beginning
with the home audit through to the final deferral decision. Figure 3.1 illustrates the potential actions
within deferral processes and is subject to change according to differing processes among states, agencies,
or households.
Data for analysis will be collected after the deferral decision is made. A sample of 10 states and
10 local weatherization agencies will be selected based on the varying levels of historical commitment to
successfully resolving deferral situations and adoption of individual vs. regional deferral protocols. The
technical aspects of this study are flexible due to these variables. Each state and agency is presumed to
organize its deferral documentation differently, since there is no nationally accepted streamlined approach
to tracking and monitoring in place. The sample of agencies and their corresponding state grantees will
be stratified to allow for comparison between states that have written deferral policies and procedures
codified in state field guides and states that rely only on regional field manuals. It is presumed that states
with state field guides and protocols are also states with historical commitment to WAP; if so, a second
comparison can be made between those states with established weatherization programs and states that
have historically low levels of Program participation. The study will compare quality assurance,
frequency rates, and documentation of deferrals between those sub-sampled states and agencies that have
standard deferral plans and those that do not; additionally, the study will consider how unit or agency
attributes correlate with adopted deferral plans (Table 3.3).
78
Figure 3.1: Deferral Action Tree
Reason for deferral
not addressed.
Home not
weatherized
Services Suspended.
Referred to other
agency
Other agency is
contacted for
assistance
Reason for deferral
addressed. Home
Weatherized
Services suspended
with no referral to
other agency
Home Audit
Services Deferred
Reason for deferral
not addressed.
Home not
weatherized
Reason for deferral
not addressed.
Home not
weatherized
Home Weatherized
Application
Approved
Reason for deferral
addressed. Home
Weatherized
Reason for deferral
addressed.
Applicant reapplies
Services Denied.
Need to reapply
after problem
addressed
Home re-audited.
Home weatherized
Reason for deferral
not addressed.
Home not
weatherized
Services Suspended.
Referred to other
agency
Applicant Appeals
Deferral
Appeal supported
by oversight
management
Reason for deferral
addressed. Home
weatherized
Appeal overturned
by oversight
management.
Home Weatherized
Reason for deferral
not addressed.
Home not
weatherized
Reason for deferral
not addressed.
Home not
weatherized
Other agency is
contacted for
assistance
Reason for deferral
addressed. Home
Weatherized
Reason for deferral
not addressed.
Home not
weatherized
Table 3.3 Unit or Agency Attributes
Urban
vs
Rural
Hot
vs.
Cold
Single Family vs.
Multifamily vs. Mobile
Home
Age of
Housing
Unit
Wx Agency
Experience
Formal Plan
No Formal Plan
Evaluators will retrieve documents detailing deferral forms or deferral notes from individual
applicant files or from agency data files organized either electronically or in paper form. Oversight and
tracking processes, if observed, will also be documented. To reduce the burden on local weatherization
agencies, methods of data collection may include site visits to the sampled agencies by evaluators, as well
as the methods used to implement the staff survey.
Evaluators will follow the deferral process in real time for a sample of 203 households. This
process assessment will involve phone interviews with occupants and weatherization staff, as well as
records collection. The 203 households will not be randomly selected due to the unpredictable nature of
deferrals and the value of collecting data in real time. The purpose of the household survey is to document
the occupant‘s account of the deferral and his or her perception of the overall process. The evaluators
will administer the survey as a pre-tested telephone questionnaire, which may require a one-time call or
an initial call with additional follow-up if treatment has been deferred for a certain amount of time, or if
an appeal is in process.
79
The following information will be collected from the sample of 10 local weatherization agencies
and the 200 sampled households:
Standard or informal deferral policy or protocol
Documentation of deferral process
Reasons for suspended and deferred treatment
Incidence of suspended and deferred treatment
Efforts to communicate with applicant
Appeal process for deferred applicant
Incidence of appeals
Incidence of weatherization post-deferral
Tracking mechanisms imposed by local agency or state office.
This study seeks to explore the ways in which agencies and states differ in their deferral
strategies, and the limitations these differences place on the representativeness of results for the nation as
a whole. The following attribute data will be collected and factored into the analysis of results across
local, state, and federal systems:
Variations in determination of health and safety standards
Determination of what measures are deemed to be cost-ineffective
Accessibility of state or regional field manuals and whether or not these field guides are
consulted at the local agency level
Whether there is a lack of a streamlined documentation and tracking strategy to
effectively capture the incidence of walk-aways on both agency-to-agency and state-tostate levels
Objectivity of the determinations made by auditors, contractors and crew members
Cultural influence on home conditions
Cultural or language barriers between auditor or crew members and occupants
Prevalence of health and safety barriers specific to geographic areas
Other attribute and relational data discovered though process assessment.
Because there is no current tracking system in place, the data that will be collected to provide
insight into the frequency of deferrals will be reported as an estimation of the incidence of deferrals.
Incidence estimates within the 10 states sampled will help determine the need for policy and for
standardized procedures aimed at minimizing deferral rates within the states. The data collection and
analysis aim to
Calculate estimates of incidence rates of interrupted or suspended weatherization as a
result of health and safety issues or cost-effectiveness concerns
Provide estimates of the rate of final deferrals of treatment, in which the applicant is
advised to reapply after addressing extensive concerns observed in the home
Assess policy and protocols for deferral of treatment as defined in regional, state, and
local field manuals or guides
Compare deferral protocols and OSHA standards
Assess the impact of standard deferral forms on incidence and documentation
Review quality assurance issues regarding the deferral or suspended work phase to ensure
fair treatment of applicants
Assess the potential benefit of streamlining the deferral process and documentation on a
national level to ensure the health and safety of weatherization auditors, contractors and
crew members
Assess policy and procedure with regard to documentation, tracking, and monitoring of
individual deferral occurrences and collective incidents on state and national levels
80
Identify effective model strategies for tracking deferrals for future studies aimed at
seeking more accurate incidence records of deferrals on a national level
Determine the incidence of referrals made to alternate resource(s) for assistance, with
reasons(s) for deferred services
Provide discussion of needed resources from government, private, and non-profit sectors
to better address the underlying causes for home deferrals discovered through the data
collection and analysis
Ascertain whether reasons for deferrals are acceptable and are subject to appropriate
management oversight
Determine how ―user-friendly‖ the appeal process is for deferred applicants
Determine the impact of Program mandated limitations on weatherization funds
allowable for health and safety conditions of the home
This study has two primary objectives. The first is to determine deferral incidence rates and to
explore impacts on the Program The second is to identify both strengths and weaknesses in the deferral
process at the local agency level, in the monitoring system established by the state grantees of WAP, and
in oversight of the grantees by DOE Project Officers assigned to each state. It is hoped that existing and
effective tracking mechanisms with the potential to serve as a model for state monitoring of local agencies
or subgrantees will be identified. The timeline for tasks associated with the Deferral Process Evaluation is
shown in Figure 1.2.
Figure 3.2 Deferral Process Assessment Timeline
Fall 2010 | Winter 2011| Spring 2011 | Summer 2011 | Fall 2011 | Winter 2012|
Spring 2012|
1 – Policy research
3 – Case Studies
2 – S1 and S2
4 – Weatherization Staff Survey
5 – Records Collection
Analysis
7 – Final
Report
6 – Occupant Survey
3.1.4 Post-ARRA Weatherization Network Strategies
As mentioned above, ARRA funding required the national weatherization network to expand
greatly in a short period of time, essentially doubling the annual number of homes weatherized during
PY‘s 2009, 2010, and 2011. This increase in production required associated increases in training and
manpower. As this document is being written, the activity of the national weatherization network is
peaking and then is expected to plateau. States and agencies are already considering what to do when the
WAP-ARRA period ends in March 2012.
The purpose of this task is to collect information for DOE and the national weatherization
network on strategies being developed by grantees and subgrantees post-ARRA. It is envisioned that
strategic decisions about in-house staffing will need to be made; equally, the existing leveraging
81
relationships, future implementation of innovative financing programs (e.g., revolving loan funds,
participation in carbon markets), and ongoing maintenance of the weatherization training infrastructure
(see 3.1.5) will have to be considered, among many topics. Suggestions about how the transition could be
softened will also be collected. The following activities will be undertaken:
Winter 2011 – ORNL will arrange for a national conference call with its National Weatherization
Network Committee (see Appendix A) to discuss the types of information ORNL should attempt
to collect to inform the Network and DOE about post-ARRA strategies. ORNL will use these
inputs to develop new questions to be included in these two surveys: S1 (All States Program
Information Survey) and S2 (All Agencies Program Information Survey). ORNL will take the
opportunity during this call to also update the Network Committee on the progress that has been
made on the retrospective evaluation and on the planning for the WAP-ARRA period evaluation.
Winter/Spring 2011 – ORNL staff will meet with members of the national weatherization
network in various forums to discuss evolving strategies. It is expected that ORNL will arrange
for time to meet with network members at the NASCSP Mid-Winter Training Conference in early
March 2011, and at a similar NCAF event. ORNL will also arrange conference calls with
DOE/OWIP program officers to discuss what they may be seeing during their visits and
discussions with grantees.
Summer 2011 – The revised versions of S1 and S2 will be administered to all grantees and
subgrantees.
Fall 2011 – The survey results and inputs from various discussions will be summarized into a
stand-alone report.
3.1.5 Post-ARRA Training Assessment
This task specifically focuses on strategies and plans that address the national weatherization
training infrastructure. This infrastructure significantly expanded during the ARRA period (for example,
the number of DOE-supported training centers increased from single to double digits). There is some
evidence that spillover into related but separate endeavors may be occurring; for example, it has been
reported anecdotally that the training centers may be serving purposes beyond training people to do lowincome weatherization. Some of those who have received DOE-supported training may be equipping
themselves to work in the non-low-income retrofit sector as well. Some high-school students are
receiving training as part of after-school career development programs. The training centers may be able
to play a role in supporting other federal home-retrofit initiatives, such as Home Score.
With these thoughts in mind, training issues will be integrated in the four tasks listed under 3.1.4.
That is, training infrastructure issues will be addressed with the National Weatherization Network
Committee, in meetings at NASCSP and NCF, and in revisions of S1 and S2. Information from grantees
and subgrantees will be collected to anticipate training needs beyond the ARRA period. Use of the DOEsupported training centers by individuals not directly involved in low-income weatherization is being
tracked by the Weatherization Staff survey which is being used in the retrospective evaluation. Results
from this survey, along with discussions with appropriate national stakeholders, will be used to project
those national weatherization training needs beyond low-income weatherization that could be met by the
DOE-supported training infrastructure. A summary report on post-ARRA training strategies will also be
prepared during Fall 2011.
82
4. SPECIAL STUDIES
4.1 UNDERPERFORMING WEATHERIZED UNITS
4.1.1 Introduction
In spite of the marked decreases in energy consumption and utility cost savings that low-income
home weatherization has notably achieved, some weatherized units produce less-than-expected changes in
levels of energy consumption after weatherization. Dubbed ―underperformers,‖ these cases occur across
all building types (single-family, multifamily, and mobile homes) and regions. DOE, weatherization
agencies, and ORNL evaluators have all identified the need for an investigation into multifamily units and
the potential causes for their underperformance. Homes weatherized in PY 2009 and large multifamily
buildings weatherized in PYs 2007, 2008, and 2009 will be included in this study.
4.1.2 Potential Causes of Underperformance and Over-performance
There are several possible causes for underperformance in weatherized units; these include
Take-back or Rebound Effect – After weatherization, some households spend their energy cost
savings on measures to increase the comfort of their homes (for example, buying new climatecontrol technology), thereby increasing their energy consumption to pre-weatherization levels.
Change in Household Demographics – After weatherization, the size and/or composition of the
household may change. For example, grandparents may take in grandchildren. Household
members previously employed may become unemployed and, as a result, spend more time in the
home. This could lead to increases in energy use during the day.
More energy-consuming equipment may be bought or used after weatherization – It is possible
that some households purchase or use more energy-consuming equipment (e.g. deep freezers,
new televisions or computers, clothes dryers, etc.) after their home has been weatherized, thereby
affecting the predicted energy performance of the home.
Simple Regression to Mean – There is no reason to believe that every year‘s energy consumption
by a household should be the same. There may be natural variations in energy use that would only
become apparent through analysis of billing histories several years before and after
weatherization. This hypothesis suggests, for example, that households might actually show
savings two or more years after weatherization.
Inaccurate Audit Tool – The variations in home designs, internal and external conditions,
household energy consumption patterns, and household compositions are considerable. No
computerized audit tool or priority list can be expected to accurately model energy savings
associated with potential weatherization measures for every conceivable home situation
encountered in the field. It is possible that some audits recommended less-than-optimal sets of
measures to install in some homes.
Poorly Done Audit – It could be that some auditors need additional training or that they tend to
recommend measures that occupants want but that do not contribute much to the energy
efficiency of the homes.
Substandard Installation of Measures – Some weatherization measures may not be installed
properly; recommended insulation levels may not be met; furnaces may not be properly tuned up;
and some recommended measures are simply not installed. Such issues could reduce observed
energy savings in homes.
Failure of Measures – It is possible that some properly installed measures could fail during the
one-year post-weatherization period during which bills are collected. In these cases, households
may be forced to use old, inefficient technologies and/or revert to previous, energy-inefficient
behaviors.
83
Poor Operation of Measures – In the large multifamily building context, it could be that those
responsible for energy management may not be operating new technologies as prescribed. For
example, building supervisors may be neglecting to use inputs from new temperature sensors
located in select apartments.
Lack of Education – Anyone, from multi-family building owners and managers to residents in
single-family units, may lack effective and comprehensive energy education.
No significant Weatherization Measures Installed – If no major weatherization measures are
installed in a house/unit, then expected energy savings will be small. A unit with a small level of
expected savings could appear as multifamily when compared to all weatherized units.
It is useful to consider whether some of the causes of underperformance could, if reversed, lead to
over-performance. There are several possible causes for over-performance in weatherized units, which
include:
Reverse take-back effect – Households extend their energy cost savings by changing behaviors
within the home to reduce energy consumption (e.g. changing the thermostat settings).
Less energy-consuming equipment – Some households may discard energy-consuming
equipment after weatherization.
Simple regression to the mean – ―Over-performing‖ households could actually show less savings
two or more years after weatherization.
Reverse change in household demographics – If one or more residents moves out of a house or
unit after weatherization, higher energy performance in the unit as compared to estimated savings
could result.
Inaccurate audit tool – The audit of a unit may have underestimated the potential energy savings
after weatherization.
High occupant education – Multifamily building owners and staff and residents of weatherized
homes/units may have received a higher-than-expected level of client education.
The evaluation approaches discussed below are designed to test these types of hypotheses.
4.1.3 Evaluation Approach: Single Family and Mobile Homes
The evaluation methods discussed in this section are designed to assess major potential causes for
underperformance among the three major types of weatherized units—single-family, multifamily, and
mobile homes.
The first task is to define underperforming (and over-performing) homes. ―Underperforming‖
homes are defined to be in the bottom 10% when audit-estimated energy savings are compared to
measured energy savings post-weatherization as set out in Equation (1). ―Over-performing‖ homes are
defined to be in the top 10%.
Equation (1):
Level of performance = (weather-adjusted audit-estimated energy savings minus weatheradjusted measured energy savings)/
(weather-adjusted audit-estimated energy savings)
Distributions of the level of performance will be estimated for all of the approximately 450 subsampled agencies. The top and bottom 10% of homes will be identified, and a list of agencies with eight
or more multifamily and four or more over-performing homes will be compiled. Twenty agencies will be
randomly selected from this list to be part of this study. For each selected agency, the approximately four
multifamily homes (total of 73) and two over-performing homes (total of 42), (a total of 115 homes) will
84
be randomly selected for inclusion in this study. The study will start with a larger sample and pre-screen
to ensure that home inspections are warranted (i.e., that major demographic changes in the home are not
the sole cause for under- or over-performance.
The project team will already have a good deal of information about these homes. In addition to
the billing histories (only homes heated with natural gas or electricity will be included in this part of the
study), the project team will have housing unit, installed weatherization measure information, and auditestimated savings from the DF2 Housing Unit Information Dataform and additional information from the
DF4b form. From S2 and S3, the project team will have retrieved information on how the agencies
conduct their audits, the extent to which the agencies use contractors, and how client education is
delivered. Each classification will be crossed-referenced against building and weatherization
characteristics, identifying numbers of multifamily units of each building type and particular technologies
present in multifamily units. Data on client-education measures will also be compared to unit
performance classifications and building types to ascertain the effect that client education can exert on
level of performance.
Each underperforming and over-performing home will be thoroughly inspected, and the
inspectors will review audit procedures and recommended measures. In the cases of priority lists, the lists
will be evaluated to determine the level of appropriateness for the multifamily unit. In instances of
computer audit, the auditing process will be evaluated to determine whether the audit software made
mistakes (such as overestimates), whether calculated savings estimates were unreliable, or whether
auditors misused software. In addition to inspecting the auditing process, evaluators will inspect for
incorrect installation of measures. Thoroughness of installation will be compared against the
recommended list of measures to be installed, thus allowing evaluators to determine whether any
measures were forgotten during installation.
Occupant surveys will also be administered to collect further data on those living in the
weatherized units that have been identified as either poorly performing or over-performing. The occupant
surveys will also show whether residents are correctly utilizing the newly installed measures. The survey
data will expose any behavioral change resulting from the weatherization process, and if any change is
noted, may provide insights into under- and over-performing units. Changes in household demographics
and composition will be noted.
Lastly, this study will develop regression models to help explain variations in energy savings
among weatherized homes. The regression models will contain variables such as installed measures and
demographics. The 120 homes included in this study will be run through the appropriate regression
models to ascertain their predicted savings and whether the models also predicted the homes to be underand over-performers. The project team will address the significant variables in the models during its
investigations to generate more detailed support or non-support for the inclusion of the variables in the
models. Conversely, the project team may find in the field causal factors that might have been included in
the models but were not.
4.1.4 Evaluation Approach: Large Multifamily Buildings
The complexities of large multifamily units require a different evaluation approach. This project
will work with two organizations in the New York City region that routinely perform large multifamily
building audits: Community Environmental Center (CEC) and the Association for Energy Affordability,
Inc. (AEA). Each organization will identify five large multifamily buildings and two ―over-performing‖
buildings, based on their audit results and post-weatherization energy-savings analyses, to be included in
this study. The study will also include five multifamily and two over-performing buildings in the greater
Chicago-Milwaukee region.
85
Each of the twenty-one buildings in the study will be treated as a case study. The project team
will interview building owners and managers about the weatherization process and the operation of newly
installed measures (including boilers and energy-management systems). The project team will collect
information to ascertain whether the occupancy of the building changed from pre-weatherization to postweatherization. Each building will be inspected to determine the quality of the weatherization measures‘
installation and will compare what was actually done in each building to the audit recommendations.
The case studies will aim to answer the following questions:
How effective is the whole-building approach?
What effect does poor implementation of installed measures have on the whole-building
approach?
How well is energy monitoring being implemented, and what effect does it have on the
whole-building approach?
Do any buildings utilize off-site energy management tools, and if so, how effective are
they?
How and to what level are building staff trained to use newly installed measures, and how
effective is this training?
Is building age a factor in performance?
Are occupants correctly utilizing the newly installed measures?
Are occupants educated in how to use the measures correctly?
4.2 TERRITORIES
4.2.1 Introduction
In March 2009, a Final Rule was published in the Federal Register amending DOE‘s definition of
―state‖ to include the Commonwealth of Puerto Rico, Guam, the US Virgin Islands, the Commonwealth
of the Northern Marianas Islands, and American Samoa. The new definition is consistent with
modifications made to Section 411(c) of the Energy Independence and Security Act (EISA) of 2007. The
final rule extended all federal regulations and guidances of the Weatherization Assistance Program
(WAP) to these US territories under both regular and ARRA program years.
The proposed evaluation approach discussed in this document allows for a comparative analysis
of the results achieved in the tropical and semi-tropical climates typical of the U.S. territories and the
results from other US climate zones.
Table 4.1 shows the current WAP funding allocation for each US territory during the ARRA
program years, along with the number of projected housing units to be weatherized, assuming a maximum
cost allocation of $6500 as per WAP-ARRA guidance. Although households in hot-weather climates
typically use much less energy than the average US household, residents of island states can pay up to
double in electricity per kWh, resulting in a near comparable home energy cost burden.
Hot-climate weatherization projects tend to focus on base-load electric measures from household
appliances and efficient cooling systems, both of which tend to be less costly than heating system
replacements or installations. For example, approximately 24% of home energy use in Puerto Rico is
attributed to air conditioning; heating system expenditures are not allowable in some territories‘ WAP
program plans (www.energystar.gov, 2009). As a result, the per-unit cost of retrofits in the tropical and
sub-tropical marine climate zones of these US territories could be lower than in the rest of the United
States, allowing an increased number of local housing units to be weatherized. However, it is also
possible that a greater number of housing units in the territories will be fitted with renewable energy
technologies due to the ample availability of renewable energy in the forms of solar, wind, and
86
geothermal resources. Currently, two of the five US territories have placed solar hot-water heaters on
their priority lists for weatherization (Table 4.1). The use of this technology could increase the cost per
unit, creating the potential for those dwellings to reach the maximum allowable cost and resulting in the
final number of homes weatherized being similar to initial projections.
4.2.2 Evaluation Methods
The evaluation methods for this study are designed to determine average energy and cost savings
associated with weatherization and calculate program efficiency. Variations in technology priorities and
the use of weatherization measures that are appropriate for these tropical and sub-tropical territories in
light of possible increases in the frequency of extreme weather events due to climate change will also be
assessed.
Prior to the implementation of evaluation tasks, members of the evaluation team will complete an on-site
field visit to Puerto Rico. Puerto Rico was singled out for this because it is by far the most populous
territory and its funding allocation under ARRA is more than 10 times that of all the other U.S. territories
combined. This site visit is expected to allow evaluators the opportunity to accomplish the following
goals:
Introduce the evaluation team
Observe the Program in process
Collaborate with weatherization network members regarding selected aspects of the
evaluation approach, methods and tasks
Disseminate information regarding the proposed evaluation tasks impacting network
members.
Report anticipated burden of the evaluation on network members
Allow network members the opportunity to ask questions in person regarding the
evaluation process
Complete a preliminary assessment of cultural factors which may enhance or challenge
the evaluation process
Complete a preliminary assessment of variations in priorities regarding weatherization
technologies and measures for tropical marine climates.
All territories will receive the All State Program Information Survey (S1) for the WAP process
evaluation and program characterization at the same time as other U.S. states participating in the National
evaluation. The evaluation will collect additional information from representatives of both the Pacific and
Caribbean island territories at various stages in PY 2010, 2011 and 2012 (Table 4.1). Guam has been
selected to represent the Pacific islands and Puerto Rico has been selected to represent the Caribbean
islands. Both are located in tropical climate zones. It is presumed that the geographic and cultural
differences between the two regions may present varying logistical issues and possible variations in
energy efficiency practices and results so both regions will be studied with equal amounts of rigor
regarding energy and cost savings.
For both Puerto Rico and Guam, a sample of housing units will be selected for home energy
savings analysis. For those households, monthly billing data will be collected and analyzed for the
purpose of calculating energy and cost savings. As shown in Table 4.1, the Subset of Agencies Detailed
Program Information Survey (S3), Housing Unit Information Surveys (DF2), and Building Information
Surveys for Multifamily housing (DF3) (if applicable) will also be administered in those two territories.
87
4.2.3 Puerto Rico Case Study
A case study will be conducted for Puerto Rico only. Puerto Rico was selected for this based on
the level of awarded funding, number of homes projected for weatherization services, eligible population
size, and geographic location. In addition to the energy savings analysis, the S4: Occupant Survey will be
administered to 258 occupants. The Case Study aims to answer the following questions:
What are the measures taken by Puerto Rico and their one sub-grantee to manage over
$65M in ARRA funding for the WAP?
What challenges are the client, sub-grantee, grantee, and DOE project officer facing with
the new eligibility and ARRA ramp up?
What are the overall strengths and deficits of the Puerto Rico WAP?
How well does the Puerto Rico State Plan for WAP under ARRA harmonize with the
practical application of services and with expectations from DOE?
What is the demand for services?
How efficient and client friendly are the application and implementation processes from
the agency and client perspectives?
What is the inspection protocol for homes weatherized and does it assure quality
What unique or innovative measures or technologies are installed addressing marine
climate characteristics and how are these measures/technologies influenced and accepted
by the client population?
What client education techniques are being employed to ensure long-term impact on
energy efficiency?
Is the weatherization network taking advantage of available renewable or sustainable
resources inherent to island characteristics?
Are measures taken to create a cultural norm around home energy efficiency?
What measures are being implemented to address the expected post-ARRA decrease in
funding and resources?
Does installation of solar water heaters in weatherized homes influence others in the
household‘s social network or other moderate and high income families not eligible for
the WAP services to invest in solar water heaters or other renewable resource
technologies, especially those eligible for a federal tax credit or other economic
incentives?
88
*S1, S3, DF2, DF3, and DF4 are surveys and data forms contained in the appendices.
Table 4.1. WAP funding allocations, key measures, and evaluation methods for U.S.
territories under ARRA
US
Funding
Primary
Evaluation Methods and
Territory
Allocation
installations/measures
Time Frames for Task
under
Completion
ARRA
American
$896,449
CFLs, solar water heaters,
Preliminary review of State Plan
Samoa
refrigerator replacement,
(Fall 2010)
window air conditioners, and Survey: S1 (Spring 2011)*
low-flow faucets and
showerheads
Puerto Rico $65,262,581 Reflective films, air leakage
Preliminary review of State Plan
in air conditioned areas, solar
(Fall 2010)
hot water heaters, Energy
Surveys:
Star water heaters and air
S1, S2, S3, DF2, DF3, DF4 (if
conditioners, refrigerator
applicable) (Spring 2011)
replacement, CFLs, shower
Billing Data Collection
head replacements, smart
(Summer 2011, 2012, and 2013)
power strips, and mitigation
1 Week of QA Inspections
of energy related health and
(PY 2011)
safety concerns
Case Study (Spring/Summer 2011)
Northern
$997,686
Baseload efficiency
Preliminary review of State Plan
Mariana
appliances, and cooling
(Fall 2010)
Islands
systems. (Details will need to Survey: S1 (Spring 2011)
be requested)
Guam
$1,431,132
Weather-stripping and
Preliminary review of State Plan
sealing of air leaks, CFLs,
(Fall 2010)
replacement of refrigerators
Surveys:
and air conditioners with
S1, S2, S3, DF2, DF3, DF4 (if
energy efficient appliances,
applicable) (Spring 2011)
shower head and faucet
Billing Data Collection
aerator replacement,
(Summer 2011, 2012, and 2013)
installation of smoke and
1 Week of QA Inspections
carbon monoxide detectors
(PY 2011)
US Virgin
$1,827,182
CFLs, refrigerator
Preliminary review of State Plan
Islands
replacement, timers on water
(Fall 2010)
heaters, low-flow
Survey: S1 (Spring 2011)
showerheads and aerators
89
4.3 WEATHERIZATION INNOVATION PILOT PROGRAM
4.3.1 Introduction
This piece addresses the evaluation of OWIP‘s Weatherization Innovation Pilot Program (WIPP).
Recently, OWIP announced awards to sixteen grantees of approximately $30M, with another $90M of
leveraged funding pledged by grantee partners. Table 4.2 summarizes the sixteen projects and funding
levels.
Most of the sixteen projects contain multiple activities. Each separate activity could be the focus
of specific evaluation activities. The next section identifies nine important project activities that are
contained in the sixteen projects, proposes meta-evaluation questions (i.e., questions framed within the
broader policy context), lists specific evaluation questions to be explored via a comprehensive evaluation,
and the provides the evaluation activities that would be conducted to answer the evaluation questions. The
following section summarizes the evaluation activities across the ten project activities.
90
Table 4.2. WIPP Funded Projects
Financing Approaches (mostly Multifamily)
Organization
DEPARTMENT OF
COMMERCE (Washington
State)
LEAP: Local Energy Alliance
Program
UTAH DEPARTMENT OF
COMMUNITY AND
ECONOMIC DEVELOPMENT
COMMUNITY
ENVIRONMENTAL CENTER
INC
STEWARDS OF AFFORDABLE
HOUSING
ENERGY PIONEER SOLUTIONS
Other
Green & Healthy
Homes
New Technologies
Workforce Development /
Volunteer
DANVILLE, CITY OF
YouthBuild USA, Inc.
HABITAT FOR HUMANITY
INTERNATIONAL (HFHI)
State
Type (Non-Profit,
State, Private,
Utility, etc)
WA
State / WAP
Agency
VA(1)
Non-Profit
UT
Program Structure
DOE Funds
Requested
Leveraged
Funds
Multifamily owner loans through
CDFI's
$ 3,000,000
$ 9,386,008
Multifamily ESCO approach
$ 1,898,938
$ 5,950,000
State / WAP
Agency
Revolving Loan Fund and
Performance Contracting to home
owners
$ 850,000
$ 2,550,000
NY
Non-Profit, WAP
Subgrantee
Low-rise multifamily approach with
on-bill financing
$ 3,000,000
$ 9,000,000
DC
Non-Profit
Multifamily ESCO approach
$ 2,590,523
$ 8,750,000
$ 812,418
$ 2,959,150
NE
Private Company
Private company offering 50% loans
to customers, with on-bill
repayment
VA(2)
City
City-wide collaboration with housing
agencies for rental multifamily;
smart meters; PACE or RLF
$ 1,015,746
$ 1,290,000
Non-Profit
Geographically diverse pilots for
workforce development and
volunteer models
$ 1,374,020
$ 4,020,593
Non-Profit
Geographically diverse pilots for
workforce development and
volunteer models
$ 3,000,000
$ 9,010,000
In-home metering and behavioral
interventions
$ 719,380
$ 1,200,000
In-home metering
$ 2,449,607
$ 9,291,200
CT, NY,
MN, MD,
WV, VA
AL, CA, DC,
FL, IA, IL,
ME, MI,
MN, MS,
NC, PA, TN,
TX
EFFICIENCY VERMONT
VT
Non-Profit
Commission on Economic
Opportunity
PA
Non-Profit, WAP
Subgrantee
UNIVERSITY OF NORTH
CAROLINA CHARLOTTE
NC
College /
University
Non-traditional provider with new
technologies: ductless heat pump,
whole house fan, in-home meter
(Google/TED)
$ 2,005,945
$ 6,214,400
The United Illuminating
Company
CT
Private, Utility
Green and Healthy Homes Initiative
led by Private utility
$ 3,000,000
$ 11,047,475
Coalition to End Childhood
Lead Poisoning
MD
Non-Profit
$ 1,287,598
$ 3,862,793
PWC
OH
Non-Profit, WAP
Subgrantee
$ 1,500,000
$ 4,529,536
New Hampshire Community
Loan Fund, Inc.
NH
State / WAP
Agency
$ 600,000
$ 2,500,000
$ 29,104,175
$ 91,561,155
91
Green and Healthy Homes
Inititiative, led by non-traditional
provider
Standard weatherization with
volunteer element and freezers,
solar PV technologies.
Manufactured/mobile housing
approach in targeted communities,
leverage RGGI grant
4.3.2. WIPP Programmatic Questions – For all grantees/approaches
Meta-Policy Questions
1. Evidence to answer public policy question: should we be using DOE dollars for multifamily
buildings? Discuss effectiveness of single-family vs multifamily innovative approaches.
Statement of Project Objectives (SOPO) Questions
1. How does the project compare to formula WAP program in terms of: job quality, monitoring and
quality assurance, cost effectiveness, energy savings per unit, low-income household benefits and
non-energy benefits?
2. Did the leveraged resources perform as proposed?
3. What are the primary lessons learned in terms of project scalability and replicability?
4.3.3. WIPP Project Activities and Evaluation Questions and Approaches
Questions Associated with Finance Projects (Danville [VA2], UT, WA)
Loans and Loan Funds
Meta-Policy Questions – What is the current use of revolving loan funds for non-low income
residential energy efficiency programs? How much money is out there in private sector financial markets
that could be diverted to revolving weatherization loan funds? What is preventing money from flowing
into revolving low-income weatherization loan funds? How much DOE money is needed to seed lowincome weatherization loan funds?
Detailed Evaluation Questions –
a) What is an optimal size for a revolving loan fund?
b) How many weatherization jobs annually can be supported by what size of loan fund?
c) Do loan funds shrink or grow over time?
d) What happens when a loan recipient defaults?
e) How is loan worthiness judged?
f) How is the amount of the loan judged?
g) How are loan terms determined?
h) Do loan recipients fully understand the terms of the loans?
i) How much extra financial burden is placed on landlords because of the loan payments?
j) What are the benefits to renters in properties where the landlord has taken out a loan to pay for
weatherization measures, and how certain are those benefits?
k) How are landlords involved and what are the benefits and risks (if any) to them?
l) What are the options when the client moves, is unable or unwilling to pay bills, or household size
changes?
m) Should the funds be only focused on the multi-family sector or also include the single-family
sector?
n) What is the average loan amount and repayment schedule?
o) What marketing approaches work best to attract clients to the loan program?
Evaluation Approaches – Conduct literature review of current status of use of revolving loan
funds in the residential energy efficiency arena. Conduct case study process evaluation that includes
interviews with key participants, examination of primary documents, and associated analyses. Survey
loan program participants to ascertain their viewpoints on the program. Conduct detailed economic
analyses of program operations to assess soundness of financial decisions. This will include billing
analysis to ascertain energy savings and bill reduction in order to determine benefits and cash flows for all
participants.
92
Carbon Offsets and RECS (LEAP [VA1], WA, CEO, PA)
Meta-Policy Questions – What investments are voluntary carbon markets currently making in
non-low income energy efficiency projects? How much money is out there in the carbon offset markets
that could be diverted to low-income weatherization investments? What is preventing money flowing
from the carbon offset world into low-income weatherization? What investments must DOE make to
catalyze this leveraging potential?
Detailed Evaluation Questions
a. How much money was attracted from the carbon markets for the project(s)?
b. What were the constraints and barriers to getting this money?
c. Who ultimately received the money from the carbon markets?
d. Who owns the carbon offsets?
Evaluation Approaches – Conduct literature review of current status of carbon offset market
investments in energy efficiency projects. Conduct case-study process evaluation that includes interviews
with key participants, examination of primary documents, and associated analyses. Conduct detailed
economic analyses of program operations to assess financial feasibility and sustainability.
ESCOs (SAFH [DC], LEAP [VA1])
Meta-Policy Questions – What is the current involvement of ESCOs in the non-low income
residential marketplace? How much money is out there in the ESCO world that could be diverted to lowincome weatherization? What has been preventing this money from flowing into (multifamily) low
income weatherization? What investments are needed from DOE to get ESCOs to enter this market?
Detailed Evaluation Questions –
a) How successful was the project in engaging ESCOs? What were the primary constraints and
barriers?
b) What appear to be the financial thresholds that must be crossed in order to get ESCOs engaged?
c) What are the main barriers to ESCO engagement in (multi-family) low-income weatherization?
d) How do building owners view working with ESCOs? How do ESCOs view working with lowincome building owners?
e) What are the most effective financial approaches to incentivize multi-family owners to invest in
building retrofits?
f) What are the primary HUD barriers to engaging in multifamily retrofit work?
g) How are energy performance contracts best structured?
Evaluation Approaches – Conduct literature review of current status of ESCO involvement in the
residential sector. Conduct case study process evaluation that includes interviews with key participants,
examination of primary documents, and associated analyses. Survey ESCO program participants to
ascertain their viewpoints on the program. Conduct detailed economic analyses of program operations to
assess financial soundness and sustainability. This will include billing analysis to ascertain energy savings
and bill reduction in order to determine benefits and cash flows for all participants.
Utility Bill Repayments (EPS [NE])
Meta-Policy Questions – What is the current status of utility bill re-payment programs outside of
the low-income residential sector? How much money is out there in the utility world that could be
diverted to weatherization/utility bill re-payment programs for low-income households? What is
93
preventing the spread of these programs? How much money might DOE need to invest to increase the
number of these programs?
Detailed Evaluation Questions –
a) How are weatherization investment levels in homes optimally set?
b) How are monthly re-payment schedules optimally set for the client base?
c) How is client eligibility best determined to assure program sustainability?
d) How much does the program add to utility bills on average?
e) What happens when the client moves or is unable to pay bills?
f) How does the program adapt in the case of sharp utility prices changes during the re-payment
period?
g) How does the program adapt in the case of fluctuating weather patterns year to year?
h) Do clients express clear understanding of the program?
i) What are the primary costs, benefits, and risks for clients?
j) What are the primary motivations for utility company involvement, and what are their
primary concerns?
Evaluation Approaches – Conduct literature review of current status of non-low-income utility
bill re-payment programs. Conduct case study process evaluation that includes interviews with key
participants, examination of primary documents, and associated analyses. Survey utility bill re-payment
program participants to ascertain their viewpoints on the program. Conduct detailed economic analyses of
program operations to assess financial soundness and sustainability. This will include billing analysis to
ascertain energy savings and bill reduction in order to determine benefits and cash flows for all
participants.
Questions Associated with Alternative Weatherization Workforce and Workforce Building Projects
(Habitat [national], YouthBuild [national], UNC ([NC])
Meta-Policy Questions – What are the costs and benefits of conducting weatherization work with
non-professionals? What is the national capacity of the alternative weatherization workforce? What
resources might DOE provide to train this workforce? Is the program replicable within other affiliates of
the non-profit (i.e. can it become a new line of business)? Is the program replicable to other
organizations, or is success specific to the nature of the implementing organization (i.e. Youthbuild,
Habitat)? If so, does it require further DOE investment? Can DOE establish a standard that outside
organizations, without DOE funds, would weatherize to and assure quality?
94
Detailed Evaluation Questions
a) What are best practices among the XX affiliates in terms of training and providing weatherization
services?
b) What are the experiences of the youth and/or volunteers?
c) Does client satisfaction vary between XX and WAP clients?
d) Are there specific workforce issues with respect to XX-selected workers, and how are they best
addressed?
e) What is the success rate of placing Habitat graduates into weatherization and home retrofit jobs?
Evaluation Approaches – Conduct case study process evaluation that includes interviews with key
participants, examination of primary documents, and associated analyses. Collect billing histories for
homes weatherized by non-professionals and compare energy savings to a control group of homes
weatherized during the ARRA period and pre-ARRA. Inspect a sample of homes. Administer portions of
the Weatherization Staff survey and attempt to track non-professional weatherization workers. Implement
customer satisfaction portion of Occupant Survey. Have agencies provide detailed information about
measures installed in each weatherized home.
Questions Associated with In-Home Metering Projects [CEO (PA), VEIC [VT], UNC [NC])
Meta-Policy Questions – How much energy is saved beyond the installation of weatherization
measures in homes that have in-home energy meters/monitors? Should this measure be on Appendix A?
If so, at what standard, and with what provisions? If not, why not? Do low-income clients see benefits
equal to, greater, or less than other studies have shown for pilots/programs in the U.S.?
Detailed Evaluation Questions –
a) What types of meters were installed, and where?
b) Do any homes refuse to have meters installed and if so, why?
c) What are the most common client misunderstandings about the meters and the information
being conveyed by them?
d) Are homes with meters installed different in some way from the larger population of
weatherized homes?
e) Were there any technical issues associated with meter installation?
f) How do energy savings vary by a) presence of a simple and end-use in-home energy displays
b) intervention strategy to groups of clients, c) type of meter, d) placement of meter within
the home.
VEIC/CEO specific questions:
a) General questions
a. Did the leveraged resources perform as proposed?
b) Meter questions
a. What types of meters were installed, and where?
b. Do any homes refuse to have meters installed and if so, why?
c. Were there any technical or customer issues associated with meter
installation?
d. Are homes with meters installed different in some way from the larger
population of weatherized homes?
e. How do energy savings vary by: (a) presence of simple and end-use, in-home
energy displays? (b) intervention strategy to groups of clients? (c) type of
meter?
f. Given the project‘s mix of low-income rural, suburban, and urban settings,
and this project‘s implementation by different types of utilities, what—given
95
any lessons learned—are the limits and the potential for project scalability
and replicability?
UNC-specific questions:
1. New technology project questions:
a. What are the costs and energy savings associated with ductless heat pumps and whole
house fans?
b. What types of meters were installed, and where?
c. Do any homes refuse to have meters installed and if so, why?
d. What are the most common client misunderstandings about the meters and the
information being conveyed by them?
e. Are homes with meters installed different in some way from the larger population of
weatherized homes?
f. Were there any technical issues associated with meter installation?
g. How do energy savings vary by a) presence of a simple and end-use in-home energy
displays b) intervention strategy to groups of clients, c) type of meter, d) placement of
meter within the home.
Evaluation Approaches – Collect billing histories for homes weatherized with meters and
compare energy savings to a control group of weatherized homes. Inspect a sample of meter installations.
Implement energy knowledge/behavior and meter portions of Occupant Survey. Have agencies provide
detailed information about measures installed in each weatherized home.
Questions Associated with Client Outreach, Education and Follow-up Projects (CEC [NY], NHCLF
(NH), LEAP (VA1), PWC (OH), CEO (PA)]
Meta-Policy Questions – How much more energy is saved beyond the installation of
weatherization measures in homes that have had extensive client education? Should DOE mandate client
education and additional contact with clients for a lengthy period of time post-weatherization? What type
of client education works, and how can DOE ensure that a further rollout would continue to work? How
do projects incentivize and empower building owners to engage in client education?
Detailed Evaluation Questions –
a) What client education methods were used?
b) How does the client outreach and education strategy affect energy savings?
c) How do demographic characteristics affect energy savings (with breakdown by outreach strategy,
as appropriate)?
CEC further revised their plan as follows:
a) Community-based Outreach:
a. What aspects of outreach can be streamlined through community-based outreach?
b. Are there measurable benefits to the use of peer-led enrollment strategies?
c. What particular outreach methods are most effective in low-income communities?
d. Can program outreach efforts be effectively combined with environmental awareness
campaigns
e. How should strategies differ between rental properties and resident-owned
properties?
96
NHCLF further revised their plan as follows:
a) Client education questions
a. Are there characteristics of Resident-Owned Communities of manufactured homes
that make them particularly fruitful places for public investments in weatherization?
We will focus on the importance of two characteristics:
i. shared ownership and community networks as a tool to mobilize
homeowners to apply
ii. relative density of eligible homes of similar construction as a means to
achieve economies of scale in the application process and in the actual
weatherization work.
The key measure of success will be energy saved per dollar of public subsidy: a function of
efficiency in the sign-up phase, economies of scale in the weatherization work, and improvements in
building performance per dollar invested in each home.
a. Can neighborhood-based recruitment efforts increase the number of eligible residents
seeking LIHEAP and weatherization services? If so, by what percentage?
b.
If a resident-owned community is organized and its residents well informed about
the weatherization program, will CAP agencies respond to invitations to receive
applications in those communities (possibly even within the eligible homes) and will
that level of service increase participation and cost-effectiveness?
c. What are the economies of scale in working on four or more homes sequentially in a
single neighborhood? Labor, transportation, materials? Are the manufactured homes
themselves sufficiently similar to permit savings in bulk purchase of materials?
d. Does the neighborhood structure of most ROCs help make weatherization a
community norm? NHCLF assumes that recruitment efforts will induce many—but
not all—eligible homeowners to apply for weatherization. NHCLF also assumes that
the resulting weatherization projects will be highly visible and benefits widely
discussed within the community. We will try to determine if the first wave of
weatherization generates a second wave of applications for weatherization services or
private investments by homeowners who don‘t qualify for the federal program.
Evaluation Approaches – Conduct case study process evaluation that includes interviews with key
participants, examination of primary documents, and associated analyses. Collect billing histories for
homes weatherized by programs administering additional client education and re-contacts and compare
energy savings to a control group of weatherized homes. Observe a sample of client education situations.
Implement energy knowledge and consuming behavior portions of Occupant Survey. Have agencies
provide detailed information about measures installed in each weatherized home.
Questions Associated with Consolidated, Coordinated, One-Stop Whole-House Projects [United
Illuminated (CT), Coalition to End Childhood Lead Poisoning (MD), PWC (OH), WA]
This category of activities pertains to projects whose representatives are able to offer multiple
public program services to households during one point-of-contact. Within this category are two distinct
types of projects. The projects being conducted by The United Illuminating Company and the Coalition to
End Childhood Lead Poisoning are primarily concerned with one-stop shopping of programs that address
97
healthy homes. The PWC and Washington State projects act primarily to bring multiple funding sources
together to facilitate weatherization.
Meta-Policy Questions – Is it possible for local organizations to consolidate disparate federal and
state programs into a coordinated program from the clients ‗point-of-view‘? What are the benefits to the
clients? What are the benefits to DOE and WAP?
Detailed Evaluation Questions
a) Did the comprehensive approach to weatherization (and healthy homes) work? What worked
and what did not work?
b) Does program consolidation lead to better outcomes in terms of client health and other nonenergy benefits?
c) Are clients satisfied with the One-Stop program?
Evaluation Approaches – Conduct case study process evaluation that includes interviews with key
participants, examination of primary documents, and associated analyses. Collect billing histories for
homes weatherized by One-Stop programs and compare energy savings to a control group of weatherized
homes. Observe One-Stop program delivery. Implement customer satisfaction and health portions of
Occupant Survey. Have agencies provide detailed information about measures installed in each
weatherized home.
Other (CEC [NY], UT)
CEC added the following questions to their evaluation plan:
a) Alternate cost-effectiveness rationale:
a. How can meaningful measures of societal and community benefits be included in
cost-effectiveness determination?
b. How do building owners feel about pursuing non-direct weatherization benefits?
c. Are local residents/organizations willing to support strategies that benefit entire
communities?
d. What is the benefit to overall WAP-delivery of using these rationales?
e. How can the ‗return on investment‘ needs of owners be balanced with societal
benefits in WAP work scope determination?
f. What renewable generation strategies become feasible with alternate costeffectiveness determination?
g. What societal and building-owner benefits do renewable energy generation strategies
deliver?
b) Post-construction monitoring and technical support:
a. How does the actual performance of energy efficiency measures compare to preconstruction projections?
b. What are common causes of unmet savings potential?
c. What level of post-construction investment is necessary to remedy problematic
retrofits?
d. Do building operators of multifamily buildings or homeowners have the requisite
skills to maintain modern building systems?
e. What energy monitoring and management systems are most effective in supporting
post-retrofit support?
f. How do client and tenant education strategies affect energy savings?
98
Utah added the following questions to their evaluation plan:
f. What are best practices regarding performance-based contracting?
g. What issues interfere with achieving expected levels of post-retrofit energy savings?
4.3.4 Evaluation Plans
Table 4.3 maps the eight types of evaluation methods listed in the evaluation approaches
presented above to the nine types of activities that the WIPP grants are supporting, as organized above.
The second column in the table indicates which states/projects fall into the ten types of activity categories.
This evaluation plan can be considered a comprehensive approach to evaluating the most important
aspects of WIPP. It will incorporate four literature reviews, eight case studies and four detailed economic
analyses. Billing histories will be collected for a sample of homes weatherized associated with all activity
areas (for approximately 4000 units), as would the associated data from the grantees about measures
installed in the homes. Approximately 1000 of those receiving benefits from WIPP activities will be
surveyed. Approximately 225 members of the alternative workforce will also be surveyed. A sample of
approximately 50 homes of weatherized by the alternative workforce will be inspected for quality
assurance purposes.
Table 4.3 Evaluation Methods Applied to Categories of WIPP Activities
Loan Funds
Carbon Offsets
ESCOs
Utility Bills
Alt. Workforce
In-home Meters
Client Ed
One-Stop Shop
WA, UT, VA2*
WA, PA, VA1**
VA1**, DC
NE
YB***, HFHI, NC
VT, PA, NC
NY, NH, VA1**,
OH, PA
CT, MD, OH, WA
Lit.
Review
Case
Study
Econ
Anal.
●
●
●
●
●
●
●
●
●
●
●
●
●
Home
Inspect
Occupant
Survey
Wx
Staff
Survey
Bills
Anal.
Grantee
Surveys
●
●
●
●
●
●
●
●
●
§
●
●
●
●
●
●
§
●
●
●
●
●
●
●
●
●
* City of Danville, ** Local Energy Alliance Program, *** YouthBuild
§ Bills and other data will be collected as part of the in-home meters project analyses.
Table 4.4 outlines the schedule for the evaluation activities. The first evaluation activity to be
undertaken will be the four literature reviews, which will be conducted during the 2nd quarter of 2011.
Case studies will be initiated during the third quarter and will be completed early in the fourth quarter of
2011. Initial economic assessments that will focus on financial decision rules will also be conducted
during this time period. Independent reports will be prepared to document the literature reviews, case
studies, and initial economic assessments.
Units weatherized by the various projects through June 30, 2011 will be eligible for inclusion in
the billing analysis study. This date was chosen to piggyback upon the larger WAP-ARRA period
evaluation. Starting in July 2012, the larger project will ask utility companies nationwide for billing
histories. Billing histories needed for the WIPP evaluation will be rolled into this much larger task and
will yield tremendous cost efficiencies.
Given this timeframe, approximately 68 on-site inspections of homes weatherized by alternative
workforces will be conducted during the third quarter 2011. Collection of detailed weatherization
information from the grantees will also begin at this time and continue until all required information is
99
collected or until data collection ends on September 30, 2012. Billing history analyses will be conducted
as the various project-based and activity-based data sets are completed. The goal of this project is to
complete the energy savings analyses during the fourth quarter of 2012.
Occupant surveys, 267, and weatherization worker surveys, 243, will be conducted during the 2nd
quarter 2012. These surveys are delayed somewhat in order to give occupants some time to experience
program results (especially those participating in financial programs) and alternative weatherization
workforce members some time to possibly change their employment situations. A final report will be
completed by the end of CY 2012.
Table 4.4 Schedule of Evaluation Activities
1st Quarter
2011
Lit Reviews
Case Studies
Economic
Analyses
Home
Inspections
Occupant
Surveys
Wx Staff
Surveys
Grantee
Surveys
Billing
Analyses
Final Report
2nd Quarter
2011
XXXX
3rd Quarter
2011
4th Quarter
2011
XX
XX
XX
XX
1st Quarter
2012
2nd Quarter
2012
3rd Quarter
2012
4th Quarter
2012
XX
XX
XXXX
XXXX
XXXX
XX
XX
XX
XX
XX
XX
XX
XX
XXXX
Finally, it should be noted that the timely administration of the surveys and the collection of the
weatherization information and billing data are contingent upon OMB approval. ORNL plans to submit to
OMB an Information Collection Request (ICR) for the entire WAP ARRA period evaluation in early
2011. This request will include the already approved data forms (DF2/3 that collect weatherization
measure information; DF4A/B, which collects utility account information; DF5, which collects billing
histories from utilities) needed by this project and two surveys that are currently under OMB review (S4
[Occupant Survey] and S5 [Weatherization Staff Survey]). ORNL will be primarily asking to increase
burden numbers and for approval of any new methodologies.
4.3.5 Technical Assistance
ORNL will provide technical assistance to grantees on an as-needed basis. ORNL will assist
grantees implement appropriate data collection procedures. ORNL will also assist grantees implement
program administration procedures in cases where evaluation plans require random control trial designs.
Projects that might be appropriate for RCT designs include those involving in-home metering and
installation of non-standard measures (e.g., solar PV technologies).
100
4.4 SUSTAINABLE ENERGY RESOURCES FOR CONSUMERS
This segment describes the evaluation approach designed to assess OWIP‘s Sustainable Energy
Resources for Consumers (SERC) Grant authorized under the Energy Independence and Security Act of
2007 (EISA) Pub. L. 110-140, Section 411(b). In June 2010, OWIP requested funding proposals for
projects focused on efficient, and renewable technologies, as well as innovative or evidence-based
interventions aimed at reducing home energy consumption. The SERC grant gives 92 high-performing
local weatherization agencies in 27 states the opportunity to employ alternative measures in the residential
sector that may have otherwise been deemed lower priority as a result of low cost efficiency, or that may
have been considered not allowable under current WAP standards due to the inherent risks involved with
innovation. In selecting grantees, priority was given based on the following criteria outlined in EISA
Section 411(b)(2):
―(2) PRIORITY.—In selecting grant recipients under this subsection, the Secretary shall give
priority to—
(A) the expected effectiveness and benefits of the proposed project to low- and moderate-income
energy consumers;
(B) the potential for replication of successful results;
(C) the impact on the health and safety and energy costs of consumers served; and
(D) the extent of partnerships with other public and private entities that contribute to the
resources and implementation of the program, including financial partnerships.‖
This section of the evaluation describes approved technology measures and outreach approaches,
their attributes, and a selective evaluation methodology based on available funding allocated for analysis.
Evaluation of OWIP‘s SERC Program should deliver answers to the following broad areas and more
specific questions:
Area 1: Quantitative Evaluation of Impacts
How do the projected energy savings compare to resulting savings upon application of the
measures?
What is the SIR for the measures rigorously evaluated?
How can the weatherization community assist with raising the SIR of a measure to > 1, and
under what conditions?
How does modification of occupant energy consumption behavior factor into home energy
savings with sustainable energy technologies?
Which technologies deserve inclusion on the approved installation measures list for WAP
based on either SIR or cost-effectiveness?
How do these technologies impact energy security (defined as household access to home
energy and measured by total number of utility interruptions pre-and post installation) across
socio-economic status of the populations served?
Area 2: Process Evaluation
Were the approved agencies able to allot the proposed number of innovative or renewable
technology units to willing households?
What were the barriers to meeting project goals?
What are the issues associated with implementing innovative strategies or measures for
reducing home energy consumption?
Did the technologies operate as expected? Were there any installation problems?
How well did SERC projects harmonize with standard weatherization process and treatments
offered under WAP?
101
How much, if any, follow-up with occupants is necessary to promote maximum savings as a
result of the new technology, measure, or outreach?
Should WAP transition into the sustainable energy resources arena for home weatherization
retrofits after the expiration of ARRA?
4.4.1. Comprehensive Evaluation Approach
The evaluation of SERC projects involves the assignment and application of a selected
methodology with an associated level of rigor for each broad category of technology or outreach. The
more innovative technologies or those projects infused with substantial amounts of funding will be placed
in a highly rigorous category for statistical analysis based on energy billing data. Data collection for
approximately 4000 homes will be used for determining home energy savings and SIR results. Although
the selected technologies for this grant are not expected to meet the SIR of 1.0, SIR calculations will
initiate discussion on future potential for cost-effectiveness and under what conditions, such as
comparisons of the cost of replacement upon failure of technology vs. early replacement, or buy-downs
created with secondary funding sources.
Those measures selected for moderate levels of rigor will be subjected to methodologies making
use of a technique called ―deemed savings analysis.‖ For example, ―deemed savings‖ for a household
with newly installed LED lights would be determined by surveying agencies to get the total number of
installed lights and then calculating energy savings based on their associated energy output as reported in
the literature. Outreach activities and innovative technologies requiring process assessment will be
evaluated through the use of structured interviews or field observations to get insight into the overall
impact of the measure. TABLE 4.5 describes each category of evaluation by rigor and the associated tasks
involved. TABLE 4.6 exhibits the approved technology, measure, or outreach by level of evaluation rigor.
The evaluation will focus on 10 categories of SERC technologies (15 technologies to be
evaluated with high rigor, 29 measures to be evaluated with a moderate level of rigor, and two categories
of technologies or interventions appropriate for process analysis). Process analysis questions pertaining to
the other 44 (15 high rigor and 29 moderate rigor) technologies will require the use of structured
interview tools.
102
Table 4.5 Evaluation Categories and Tasks
Administer DF2, DF3, and DF4 at agency level. Submit ICR for
High Rigor
Moderate Rigor
Process
Evaluation
OMB approval. (Sampling based on scope of project/number of
units treated/appropriateness of random assignment/ aggregatelevel selection…)
Collection of billing data pre- and post –installation (one year of
data to be collected Summer/Fall 2012) (DF5 to be administered
at utility level)
Control group billing data pulled from sub-sample of agencies
selected from general evaluation (impact analysis)
Post-treatment Quality Assurance review to ensure accurate
installation/consumer use of technology (103 homes)
Develop and administer client satisfaction survey with focus on
renewable technologies (384 clients)
Collection of cost of measures from agencies (projected and
actual)
Calculate SIR
Final Report
(Potential for RCT approach for unit assignment for
approximately 6 projects)
Administer DF2 and DF3
Deemed energy-savings estimates (based on number and types of
measures installed in unit, and recognized average energy savings
derived from the literature with consideration for varying
confounding variables: (climate region, housing characteristics,
etc.)
Collection of cost of measures from agencies
Estimate of cost-effectiveness
Final Report
Develop and Administer Structured Interview addressing
Evaluation Process Questions
Assess agency implementation strategies and obstacles to the
intervention
Synthesize interview results with results from other high- or
moderate-rigor evaluation tasks
103
Table 4.6 Technology Categories
Technology Category
Rigor
Level
Technology Category
1.0 RENEWABLE ENERGY
1.1 Solar PV
1.2 PV: shingles
5.0 APPLIANCES
5.1 Energy Star Clothes Washer
5.2 Energy-Efficient Clothes Dryer
1.3 Wind: small-scale residential
2.0 HOT WATER SYSTEMS
2.1 Solar HW
2.2 Tankless/On-demand HW
5.3 Energy-Efficient Refrigerator
5.4 Appliance Energy Meters
6.0 INSULATION
6.1 Insulation: Aerogel/super
6.2 Insulation: foam injection
technology
6.3 Insulation: Masonry Foam
6.4 Insulation: Radiant barrier attic
2.3 Condensing HW
2.4 Heat Pump/Hybrid HW
2.5 Combination HW and Boiler
2.6 Other HW (specify)
6.5 Insulation: Spray Foam
6.6 Insulation: Reflective attic
insulation
7.0 WHOLE-HOUSE RETROFIT
7.1 Centralized building controls
3.0 HVAC SYSTEMS
3.1 Heat Pumps: Geothermal
3.2 Heat Pumps: Air
3.3 Heat Pumps: mini split-system
ductless
3.4 Replacement of improperly sized
HVAC equipment
7.2 Deep energy retrofits
7.3 High-performance space
conditioning retrofits
7.4 High-performance building
envelope retrofits
3.5 Solar thermal (home heat)
3.6 Wood Pellet Stoves
3.7 Ultra Cooling Systems
3.8 Central AC units
3.9 Window AC units installed
7.5 Cold Energy Retrofits
7.6 Warm Energy Retrofits
8.0 OUTREACH
8.1 Home Energy Saver Workshops
8.2 Households touched by behavioral
change message
9.0 EQUIPMENT
9.1 Monitoring: In-home energy
monitors
10.0 OTHER
10.1 Units with Window upgrades
10.2 Outdoor solar security lighting
10.3 Ceiling Fans
10.4 LED Lights
3.10 Micro-combined heat and power
3.11 High-efficiency furnaces
3.12 Heat recovery ventilators
3.14 Evaporative Cooling System
4.0 ROOFING:
4.0 Cool roof technology installed
104
Rigor
Level
4.4.2 Technologies Selected for High-Rigor Approach
The proposed strategy for a highly rigorous evaluation of SERC projects involves analysis of 15
selected innovative or highly funded technologies. TABLE 4.6 displays the identified technologies
selected for this evaluation strategy. The technologies were selected based on the following factors:
Number of local agencies, number of projected installations, and estimated budget
associated with the technology
Potential for future inclusion in WAP
Potential for Randomized Controlled Trials (RCTs)
Each of these technologies will be approached with statistical rigor requiring the selection of a
random sample of households across projects. For example, approximately 550 geothermal heat pumps
are slated to be installed by 12 agencies from eight states across the country. A random sample of
households will be selected for evaluation from the eight states . Households selected to represent the
impact of the measure will be stratified for conditions such as climate region, high energy usage, and
housing characteristics. The evaluation will compare the national results of the impact analysis of the
installed technology with a control sample offering baseline weatherization results, and with the impact
analysis of the 14 other technologies selected to undergo highly rigorous evaluation.
Analysis of home energy billing data pre- and post- retrofit will be compared to a control group
of weatherized homes selected from the WAP-ARRA Evaluation sample. Pre-tested survey instruments
will be administered to agencies and occupants, and post-treatment quality assurance inspections will be
completed. The Occupant Survey will be administered to a sample of households with installed renewable
energy technologies, or in-home monitoring devises.
4.4.3 SERC Randomized Controlled Trials (RCTs)
Projects will be reviewed to identify likely candidates for randomized controlled trials (RCTs) or
randomized trials between like technologies. For example, Colorado received approval for the installation
of in-home energy monitoring devices as part of a 1600-household project. Project planners already
intend to split the sample into 800 treatment homes, which will receive the monitoring device and 800
control homes, which will not receive the device. If an RCT method is adopted, households will be
randomly assigned to either the treatment or control group, and the project‘s findings will be strengthened
by the power of the statistical method used. As another example, two agencies in Arizona have been
approved to install two different hot-water technologies, solar and heat pump, for a total of 470 units; this
offers the potential for a randomized trial between the two interventions. In the randomized trial cases,
household recipients of standard weatherization treatment would be sampled from the WAP-ARRA
Evaluation to serve as a baseline comparison group. Randomized trials serve as an alternative to RCTs in
projects or program areas with assignment challenges and still offer significant results. Evaluators would
review these types of methodologies and then communicate with the grantees to determine any challenges
regarding randomly assigning homes to either treatment or control groups. If the conditions prove ideal,
then an RCT will be coordinated and implemented with follow-up training and technical assistance
(T&TA) from the evaluation team for the duration of the study. RCT and randomized trial projects will be
evaluated with statistical rigor. Approximately six projects will be selected for RCT experimental design.
Upon identification of potential RCT projects by the evaluation team and DOE, implementation plans will
be designed with the help of project leaders at the sub-grantee level.
Analysis of the SERC projects offers OWIP the ability to better inform WAP stakeholders of
their options–clearly presented by strengths and deficits–for each of the evaluated SERC interventions,
with successes and challenges associated during the stages of eligibility, implementation, and analysis of
energy and cost savings. Detailed evaluation designs will be developed pending further information
105
received on project descriptions, implementation plans, and communication with the awarded local
weatherization agencies.
4.5 ENCOURAGEMENT DESIGN
Given the legal and practical barriers at this time to implementing a classical RCT to estimate
energy savings attributable to WAP, an alternative approach has been identified. The University of
California-Berkeley (UC-B) has proposed what is known as an ―encouragement design approach.‖
Simply, this approach entails identifying a group of low-income homes that are eligible for WAP and
―treating‖ them with intensive encouragement to apply for weatherization services. A similar number of
homes that have similar characteristics are identified, but are not treated with any extra encouragement.
This second group serves as the control group. UC-B has developed a theoretical approach to an
encouragement design study. The study is expected to be implemented in one or two states during PY
2011.
4.6 GREENHOUSE GAS EMISSION ANALYSIS
4.6.1 Introduction
As individuals, nations, and corporations become increasingly aware of the environmental
consequences of their actions and behaviors, there is more interest in reducing carbon footprints. Key
phrases such as ―reduce/reuse/recycle,‖ ―decrease consumption,‖ and ―lower the temperature on your
thermostat‖ form part of this national conversation. For those people or organizations who will not or
cannot reduce emissions themselves, carbon offset companies accept donations from concerned
individuals, countries striving to meet their Kyoto Protocol targets, and companies trying to be socially
responsible. The companies use these donations to create and/or invest in projects that will increase
carbon sequestration or decrease greenhouse gases (GHGs) emissions, also known as carbon dioxide
equivalents (CO2e). Essentially, patrons of carbon-offset companies are attempting to mitigate the
consequences of their own carbon emissions by buying ―credits‖ in a program somewhere else.
Carbon-offset portfolios almost always contain investment projects involving reforestation/forest
carbon sequestration, renewable energy, and industrial gases. Projects are very often located in
developing countries because of lower implementation expenses, higher participation rates, and because
the carbon offset projects themselves are ―charismatic‖ (―charismatic‖ meaning that a project seeks to
address other social goals or ills in addition to reducing carbon emissions or sequestering carbon. For
example, if a project is also responsible for teaching job skills or curbing famine, it is considered
charismatic).
The purpose of this subsection is, first, to explain the carbon-offset industry and its current
conditions–who is participating, how it works, and what they are doing. It then it presents a research
approach to estimating potential GHG reductions in low-income housing in the United States and costs
per ton of GHG emissions reduced.
4.6.2 Carbon Offset Providers
Today, there are approximately 60 carbon-offset companies around the world, of which about 30
operate in the voluntary or commercial (OTC) carbon market.8 Approximately 50% of voluntary market
providers and 68% of voluntary market customers are based in the United States, while 43% of the carbon
offsets sold in the voluntary market are sourced to projects located in North America (other locations for
8
Kanter, James. ―Carbon Footprint Offsets: False Sense of Satisfaction?‖ International Herald Tribune. February
19, 2007. http://www.iht.com/articles/2007/02/19/business/carbon.php. Accessed 10.10.07.
106
participation include England, Australia, Canada, Germany, Ireland, and the Netherlands).9 Third-party
evaluators endorse carbon-offset companies and have ranked non-U.S. companies the highest. This can
probably be attributed to the fact that countries participating in the Kyoto Protocol have higher and
universal standards for offsets and have generally been in business longer than most American offset
companies. However, some companies can simultaneously rank very high on one third-party evaluation,
and then receive an average ranking on another evaluation because of the lack of established standards in
the voluntary carbon market.
Approximately half of the carbon-offset companies are non-profit organizations (NPOs).10 NPOs
are generally more transparent than for-profit companies; they are usually willing (if not legally
obligated) to release information regarding their finances and project portfolios for evaluation. NPOs also
tend to devote more of their revenues towards offset projects and less towards administrative overhead.
The mean overhead for non-profit organizations is 18.4%, as opposed to a much higher 56.6% overhead
in for-profit businesses.11 According to Mark Trexler, however, higher overhead is not necessarily
demonstrative of a poor-performing company or offset portfolio: ―If a company is thoroughly
investigating the carbon reduction projects and closely monitoring the progress to ensure that the carbon
reduction occurs, then a higher administrative cost is acceptable.‖12 It would appear that other voluntary
carbon market experts agree, because third parties award higher rankings to for-profit businesses than to
NPOs, despite the significantly higher overhead costs.
Companies in the carbon-offset market give individuals the opportunity to purchase carbon
offsets in a retail environment, in much the same way corporations voluntarily participate in the Chicago
Climate Exchange. Consumers can choose from more than 30 offset companies, but the process for
participation is often very similar. At a company‘s website, consumers can use an online ―carbon
calculator‖ to estimate their annual carbon (equivalent) emissions based on the type and amount of
vehicle travel, use of energy-efficient appliances, the type and amount of energy used in the home, the
amount of air travel, and other factors. The calculator will then determine how many tons of CO2e the
consumer emits annually as a result of his or her lifestyle, establishing a basis for how many carbon
offsets should be purchased. On average, consumers pay approximately $12.55 per ton of carbon
offsets.13 However, purchase prices can range from $5-$99 per ton of offsets.14Carbon-offset companies
use their revenue from consumers to invest in carbon-offset projects, primarily forestry sequestration
(36%), renewable energy (33%), and industrial gases (20%).15 Other smaller projects can include
methane reduction and energy efficiency.
Forestry sequestration (also called Land Use, Land Use Change and Forestry or LULUCF) is
currently the most popular albeit the most controversial project type in the voluntary carbon offset market.
Reforestation projects have several advantages that increase their appeal to carbon-offset companies.
First, it is very inexpensive to plant and maintain these forests. According to the IPCC, mitigation costs
through LULUCFs can cost as little as 1 penny per ton. Second, LULUCF projects are easy for the
general public and consumers to understand because most Americans learn about the role of trees in the
carbon and oxygen cycles in grade school.16 Third, LULUCF projects are often ―gourmet‖ or
―charismatic,‖ meaning the program is designed to accomplish more than simply to offset carbon; it is
9
Trexler Climate + Energy Services. ―A Consumer‘s Guide to Carbon Offset Providers.‖ December 2006.
Tufts Climate Initiative. www.tufts.edu/tie/tci/pdf/TCI_Carbon_Offsets_Paper_April-2-07.pdf.
11
Tufts Climate Initiative. Ibid.
12
http://featured.matternetwork.com/2007/8/ranking-the-offset-providers.cfm
13
Average based on figures from Ecobusinesslinks.com Carbon Offset Survey
14
Ibid.
15
Hamilton, Katherine et al. ―Picking Up Steam: The State of the Voluntary Carbon Markets 2007.‖
16
Hamilton. Ibid.
10
107
also expected to create job skills, promote gender equality, provide infrastructure, and, in the case of
LULUCFs, improve community aesthetics and provide critical wildlife habitats.17
While LULUCF‘s advantages are both charismatic and important, there are critical disadvantages
to the projects. First, it is difficult to measure carbon sequestration in forests because of varied tree types,
changes in season temperature or precipitation, and the age of the forests. The amount of carbon
sequestered is constantly changing as a result of these variables, and facilitators of these projects must be
sensitive to these changes. Second, leakage in sequestration can easily occur. Leakage is the
unanticipated loss of carbon reductions, which in this case might be the loss of trees due to disease, pests,
drought, fire, or harvesting. Leakage is likely to occur in situations where the locals have not been taught
how to sustain the project, non-native species were planted, or in communities where the political climate
is unstable. Third, and most crucial, is the issue of permanence. If trees are lost through any of the
situations previously mentioned, the estimates of carbon sequestration may be wildly overstated. The
carbon stored in lost trees will consequently be released back into the atmosphere, ultimately resulting in
little to no reduction in carbon emissions.18
Consumers who are more interested in obtaining large quantities of offsets and less interested in
the ―charisma‖ of an offset usually invest in industrial gas projects. Industrial processes accounted for 28
percent of total U.S. energy-related carbon dioxide emissions in 2005.19 They were the third largest
producer of greenhouse gases in the United States, emitting methane, nitrous oxide, and
HFCs/PFC/SF6s.20 More importantly, the gases produced by industrial processes have high global
warming potential (GWP). The Environmental Protection Agency determines GWP by comparing ―the
radiative forcing or ability to trap heat of one metric ton of a greenhouse gas to a metric ton of CO2.‖21 In
other words, gases with a high GWP will create or intensify climate change faster than carbon dioxide
would. For example, the GWP of methane is 21 times that of carbon CO2.22 Industrial gas projects are
considered the ―low-hanging fruit‖ or easiest projects to implement of the carbon market. Sequestering or
eliminating gases in industrial processes is a quick and effective offset because those gases with a greater
ability to trap heat are removed from the byproducts, outputs, and life cycle of the industrial process.
Investing in industrial gas projects gives consumers the maximum benefit for their purchase.
However, the use of industrial gas projects in the voluntary carbon market may decline in the near
future because of Clean Development Mechanism (CDM) regulations for new facilities, the absence of
sustainable development project co-benefits, and the supposed limited supply of these Verified Energy
Reduction credits. As the CDM currently exists, it is more profitable for companies to invest in Kyoto
Protocol compliance than it is to invest in new and clean energy sources. Compliance means companies
spend a modest amount on initial investments and enjoy low operating costs. Investing in new energy
sources means lower profits because of substantial initial investments and high operating costs. As a
result, in the past, companies have only complied with demands to decrease their industrial gasses, but
they have not gone the extra distance to invest in new energy technologies. Their compliance has yielded
many industrial gas projects. But with the adoption of the Montreal Protocol, HFC emissions are to be
phased out of industrial processes in any case.23 As the number of projects declines and specific
17
www.begreennow.com
Tufts Climate Initiative. ―Voluntary Offsets for Air-Travel Carbon Emissions: Evaluations and
Recommendations of Thirteen Offset Companies.‖
19
Energy Information Administration. ―Emissions of Greenhouse Gasses in the United States 2005.‖ Released
November 2006. http://www.eia.doe.gov/oiaf/1605/ggrpt/carbon.html. Accessed October 3, 2007.
20
Environmental Protection Agency. ―U.S. Greenhouse Gas Inventory.‖
http://www.epa.gov/climatechange/emissions/usgginventory.html. Accessed October 14, 2007.
21
EPA. Ibid.
22
Tufts Climate Initiative.
23
Rainer, Mark. ―Kyoto‘s Clean Development Mechanism: Global Warming and Its Market Fix.‖ World Socialist
18
108
emissions are phased out, the number of Verifiable Energy Reductions from those projects and
compliance projects will decrease. Add to that the fact that these carbon offset projects are not
charismatic, and their future in the carbon offset industry looks dim.
Renewable energy projects today exist in two different forms in the voluntary market. The first
form is the simple investment in wind, biomass, or solar technologies to create clean methods of
generating electricity, with the hope that we will slowly decrease our reliance on fossil fuels for electricity
as a result. The second form is Renewable Energy Credits (RECs), ―which represent the environmental
attributes of 1 MWh of power generated from renewable sources.‖24 Purchasing RECs means that the
amount of electricity coming from renewable resources will increase and the amount provided by fossil
fuels will decrease. In 2005, Americans purchased the equivalent of 3,890 million kWh in Renewable
Energy Credits (RECs).25 It is important to note that RECs are not currently measured the same way
carbon offsets are measured. Therefore, a problem can easily arise in the conversion of RECs into
carbon-offset equivalents; this may discourage some from investing in these projects. Legislation, local
opposition to a project, and high initial capital costs also prove to be barriers in developing renewable
energy projects.
Carbon-offset projects are evaluated based on ―additionality,‖ double counting, type of credit, and
standards and verification. The first concept, additionality, is usually phrased as a question: Are the
carbon offsets of the program ―business as usual‖ or are they beyond business as usual? Would the project
have happened, holding everything else constant, if the carbon offsets from it could not be sold? If in fact
the carbon offsets are not so important that the project would have been canceled without them, carbon
emissions are not actually being reduced and money is essentially being wasted. The project is
―additional‖—that is, worthwhile—only if the answer is ―no.‖
The second point of evaluation, double-counting, occurs when companies claim one set of
reductions as multiple sets. Unfortunately, double counting can occur very easily when companies invest
in carbon-offset projects in other countries, especially if that country is held to reduction standards based
on the Kyoto Protocol. All stakeholders have an incentive to claim those reductions for themselves,
thereby creating an inaccurate figure of actual carbon reductions. Double-counting can be minimized by
articulating clear legal ownership of emission reductions, retiring offsets once sold, ensuring that
renewable energy project offsets are not also sold as RECs, and prohibiting utilities that sell RECs from
using that project to quality for Renewable Portfolio Standards. While all these mechanisms increase the
cost of an offset, they also help ensure that the offset provided is legitimate.
Carbon offset projects are often verified by third parties, which may utilize internationally
recognized criteria, such as the type of project, the impact on local communities, additionality, and
leakage. Well-known third party evaluators include Clean Development Mechanism (CDM), Gold
Standard and Voluntary Gold Standard, the Chicago Climate Exchange (CCX), Green-E, and
Environmental Resources Trust.26
The Clean Development Mechanism is the largest regulatory project-based mechanism and is
involved in setting standards and verifying projects. Standards are extremely stringent, making the
Website. January 13, 2007. http://www.wsws.org/articles/2007/jan2007/glob-j13.shtml. Accessed December 1,
2007.
24
www.begreennow.com . Ibid.
25
Bird and Sweezey. ―Green Power Marketing in the United States: A Status Report.‖ 2006.
26
All information on third party verification systems, except the Environmental Resource Trust, is taken from the
Tufts Climate Initiative.
109
transaction costs so high that normally, only large projects are registered. A project cannot be certified
unless it meets strict requirements regarding additionality.
The Gold Standard and Voluntary Gold Standard was developed by a network of nongovernmental organizations and is endorsed by 42 NGOs around the world. It does not accept
sequestration projects, and certifies only renewable energy or energy-efficiency projects. It not only
requires strict measures to address additionality, but it also monitors and verifies projects to ensure that
the claims of companies and projects are true. It also evaluates co-benefits and negative externalities of
projects. Certified projects are usually large because of the high transaction costs.
The Chicago Climate Exchange (CCX) is a voluntary, but legally binding, cap-and-trade emission
trading system that has members from the United States, Canada, and Mexico. It traded just over 10
million tons of carbon dioxide in 2006. It had reached nearly 12 million tons of traded carbon dioxide by
July 1, 2007. There are currently about 300 members in the CCX, including Ford Motor Company,
DuPont, Bank of America, Rolls Royce, Safeway, several counties and municipalities, and many colleges
and universities. Members join the CCX for several reasons: It is a proactive response to future
governmental regulations, it streamlines the ease of doing business in Europe, it increases the profitability
of being ―green‖ in many instances, and generally, the companies participate either for good public
relations or because they actually believe in the cause.27 Members commit to reduce their emissions
annually from their original baseline. Companies that exceed their committed reductions can sell the
excess as CCX commodities called Carbon Financial Investments (CFIs). CCX has been criticized
because it has loopholes, does not articulate its additionality criteria well, and is not effectively
transparent. Furthermore, many companies have exceeded their reduction commitments, creating an
excess of CFIs.
Green-e was created by the non-profit organization Center for Resource Solutions both to set
standards for American renewable-energy projects and to verify those projects. Power plants built post1997 can be certified with Renewable Energy Credits (RECs) as long as they are not also used to meet
regulatory portfolio standards. Green-e is currently working on formulating stricter standards for the
conversion of RECs into carbon offsets.
The Environmental Resource Trust is a non-profit organization in Washington, D.C. that works
toward cost-effective emission reductions through a GHG registry which will define the exchanged
commodity, establish an accounting language and protocols for measuring and verifying performance,
provide documentation of third-party validation of emission reduction performance for offset companies,
and create a credible mechanism for registration, trade, banking, and retirement of GHG emission
reductions. It is designed explicitly to help build the GHG trading market and focuses on emissions
performance.28
4.6.3 Voluntary Carbon Markets and Weatherization Assistance
Every year, Americans spend over $160 billion on heating, cooling, lighting, and otherwise
powering their homes.29 Residential energy use comprises nearly 21% of total energy consumption in the
United States and contributes about 17% of our total greenhouse gas emissions every year. Most homes
are not energy-efficient and lose energy in many ways. Because low-income families are less likely to be
able to afford home improvements and upkeep than middle- or high- income families, they spend a higher
27
Aster, Nick. ―Chicago Climate Exchange 101.‖ Written May 19, 2006.
http://www.treehugger.com/files/2006/05/chicago_climate.php. Accessed December 5, 2007.
28
Environmental Resource Trust. ―GHG Registry Program.‖ http://www.ert.net/ghg/full.html#3. Accessed October
11, 2007.
29
http://www.energysavers.gov/homeowners.html
110
percentage of their income on energy. Their homes often do not have proper insulation, energy-efficient
appliances, or high-quality windows. Consequently, they pay relatively higher energy bills, especially
with regard to percentage of total household income. Weatherization of low-income housing provides
substantial and cost-effective economic, financial, and societal benefits.
Wasted energy should be regarded not only as a local problem but as a global problem in terms of
GHG and carbon emissions. What are the effects of an under-weatherized home on the environment?
The Harvard School of Public Health estimates that approximately 60 million American homes have
attics that are under-insulated. Compared to adequately insulated homes, these under-insulated attics emit
an additional half-ton of CO2, or, cumulatively, 30 million tons of CO2 annually.30 Additionally, the
amount of energy that slips through poorly insulated windows and doors in American homes is roughly
equal to the amount of energy that we get from the Alaska pipeline.31 If every American home replaced
its five most frequently used bulbs with compact fluorescent light bulbs (CFLs), one trillion pounds of
greenhouse gasses would be kept out of the air over the course of the bulbs' 5- to 8-year lives. 32
Replacing a 20-year-old refrigerator with a new, energy-efficient model reduces a home's CO2
contribution by about one ton per year.33
As discussed below, one important goal of this task is to estimate how much money needs to be
invested in low-income homes to reduce GHG emissions (e.g., dollars per ton of carbon emissions
reduced). However, it should be noted that investments in low-income weatherization through voluntary
carbon markets have substantial charismatic value because these investments yield attractive energy and
non-energy benefits as well. Of course, the retrospective and WAP-ARRA period evaluations will
produce up-to-date energy-savings estimates. But for the sake of this discussion, according to a metaevaluation by Martin Schweitzer34, when low-income homes utilizing natural gas were weatherized, the
per-household energy savings equaled 22.9% of the pre-weatherization consumption levels for all end
uses of the primary heating fuel. Numerically speaking, the average home in the studies used 133 million
site Btus annually prior to weatherization, and after subtracting their post-weatherization energy savings
of 30.5 million site Btus, they were using 102.5 million site Btus each year. For every $1 the program
spent on weatherization costs in homes using natural gas, an energy savings of $1.34 is realized.35
The non-energy benefits of weatherization have a ripple effect throughout society. Of course,
weatherization can reduce carbon emissions when fossil-fuel use is reduced, but weatherization can also
reduce emissions of sulfur oxides, nitrogen oxides, carbon monoxide, methane, and particulate matter.
Other environmental benefits occur as well in the reduction of heavy metal contamination, fish
impingement, and wastewater/sewage contamination. Utilities and their ratepayers benefit financially
from avoided rate subsidies, lower bad-debt write-off, reduced carrying costs on arrearages, fewer notices
and customer calls, fewer shut-offs and reconnections for delinquency, and reduced collection costs,
fewer emergency gas service calls, transmission and distribution loss reduction, and insurance savings.
30
Owens Corning. ―Under-insulated American Homes Cause Three Million Blimps-worth of CO2 Emissions Each
Year, EVERY Year‖ Corporate Social Responsibility Press Release. 11/05/07.
http://www.csrwire.com/News/10070.html Accessed November 7, 2007.
31
PATH Partners. ―Easy Energy Efficiency Upgrades.‖ http://www.pathnet.org/sp.asp?id=15345. Accessed
October 23, 2007.
32
http://www.coolmayors.com/common/print_page.cfm?ClientID=11061&QID=4747&Type=ReportDetail&TopicI
D=0
33
Ibid.
34
Schweitzer, Martin, Estimating the National Effects of the U.S. Department of Energy's
Weatherization Assistance Program with State-Level Data: A Metaevaluation Using Studies from 1993
to 2005, ORNL/CON-493, Oak Ridge National Laboratory, Oak Ridge, Tennessee. September, 2005.
35
Good estimates for electrically heated homes are not available at the time of writing.
111
When low-income households do not have to spend as much on energy bills, they have more
money available for other household expenses, which decreases involuntary mobility or frequent need to
move as a result of affordability. Furthermore, they experience fewer disasters like fire, are generally
more comfortable in their dwellings, and may become ill less frequently, resulting in fewer work and
school absences, a benefit for both them and the greater community. Furthermore, weatherization can
also have an impact on the avoidance of unemployment benefits, increases in employment and retention,
decreases in lost rental, and better national security. The results of the retrospective and WAP-ARRA
period evaluations will also allow better estimates of the non-energy benefit accruable from voluntary
carbon investments in low-income housing.
In a low-income weatherization assistance program that leverages the participation of carbonoffset companies, the companies would contribute money directly to community-action agencies already
in contact with low-income households. The community-action agency would determine which houses
are eligible for weatherization assistance, perform the energy audits, and install the weatherization
measures.
Overall, there are many arguments in favor of having voluntary carbon markets include
investments in low-income weatherization in the United States in their portfolios.
Projects are located in many places around the world, and carbon-offset companies will invest
primarily in projects where their nation‘s currency is very strong. For example, European Union
companies may invest in American projects, whereas U.S.-based companies will invest in Latin American
or African projects. However, from the U.S. viewpoint, there are many advantages to locating future
projects domestically, especially those that focus on low-income American households--projects in lowincome American households can be just as charismatic as those located in developing countries; in fact,
as described above, there are numerous social goals that can be promoted by locating projects in the
United States.
First, locating projects domestically improves the accountability factor. Project operations and
efficacy can be monitored more easily and more often, and results such as less storm water runoff, lower
energy bills, less household garbage, and healthier clients will be more immediately evident. However, if
a local program isn‘t working, problems will be easier and faster to identify and correct because the
carbon-offset coordinators will be nearby and in frequent communication with participants. In carbonoffset projects located in developing countries where oversight is necessarily more sporadic, there is an
inherently higher risk for smaller and less verifiable CO2e savings, double-counting, lack of additionality,
and leakage. The person in charge of day-to-day operations may not completely understand the project or
process, opportunities for political corruption may be higher, and there may be confusion over who
actually owns the credit as different political entities attempt to meet their Kyoto Protocol goals. These
factors can significantly impact the amount of carbon reductions that actually occur within a project
located out-of-country, making such projects an undependable investment.
The retrospective and WAP-ARRA period evaluations will be able to provide up-to-date,
statistically valid estimates for first-year energy savings attributable to low-income weatherization. The
energy-savings persistence project discussed in the next sub-section (Section 4.7) will be able to show,
roughly, how far into the future GHG emissions reductions could achieved. It should also be noted that
additionality is not an issue with respect to investments in low-income weatherization because it can be
strongly argued that these households do not have the financial resources to make such investments.
Lastly, it needs to be stated that the needs for low-income weatherization far outweigh the resources
available to the Program, even under ARRA. Annually, there are approximately 35-40 million U.S.
households that are eligible for weatherization services compared to the ability of the Program to fund
about 100,000 weatherization jobs per year normally and 600,000 jobs with ARRA funds.
112
4.6.4 Research Approach to Assessing GHG Emission Reductions
The retrospective evaluation will estimate national reductions in carbon emissions attributable to
WAP. This WAP ARRA evaluation task will assess carbon emission reductions more in-depth. Emission
reduction estimates will be prepared for several different climate regions in the United States.
Additionally, a national map of emissions reductions by agency will be prepared along with a second map
that estimates potential emissions reductions from weatherizing low-income homes by agency. Assessing
emission reductions at the agency level is important because it is hypothesized that interactions between
the weatherization network and the voluntary carbon reduction market would probably take place at the
agency level.
The key tasks of this research are to estimate the number of low-income homes in each local
agency jurisdiction that have not yet been weatherized, estimate the GHG emissions reduction potential
achievable from weatherizing these homes, and then estimate average weatherization costs per ton GHG
emissions reduced. The key research challenges are to ‗downsize‘ estimates of low-income homes
generated on the national, state and super-puma levels through the Program Characterization research
discussed above to the local agency level and to do the same with estimates of GHG reductions. Average
weatherization costs per ton of GHG emissions reduced will be estimated twice. The first instance will be
based on results from the PY2008 evaluation and will represent typical weatherization jobs. The second
instance will include results from the evaluation of SERC and WIPP projects, which will include costs,
energy savings and GHG emission reductions also attributable to renewable energy measures.
EIA has calculated emissions reductions in metric tons of CO2e/MWh across the entire U.S. and
subdivided into 15 regions36. Their calculations have already taken into account transmission and
distribution losses and intensity factors, thus making the figures essentially tailor-made for the needs of
this study. Table 4.7 shows the calculations, and the far right column labeled ―Indirect Emissions‖ are the
specific regional data to be used for this study. When calculating data on the agency level, the regional
figure for that agency will be used.
Table 4.7 Emission Factors
Domestic Electricity Emission Factors, 1999-2002
Emission Inventory
Region
36
Carbon
Dioxide
(Metric
Methane
tons/MWh) (kg/MWh)
Emission Reductions
Nitrous
Oxide
(kg/MWh)
Avoided
Emissions
(Metric tons
CO2e/MWh)
Indirect
Emissions
(Metric tons
CO2e/MWh)
(1) New York, Connecticut,
Rhode Island, Massachusetts,
Vermont, New Hampshire, and
Maine
0.466
0.02647
0.00616
0.744
0.793
(2) New Jersey, Delaware,
Pennsylvania, Maryland, West
Virginia, Ohio, Indiana, and
Michigan
(3) Illinois and Wisconsin
0.782
0.638
0.01404
0.01231
0.01281
0.01048
0.900
0.900
1.002
1.151
http://www.eia.doe.gov/oiaf/1605/pdf/Appendix%20F_r071023.pdf
113
(4) Missouri, Kentucky,
Virginia, Arkansas, Tennessee,
North Carolina, South
Carolina, Louisiana,
Mississippi, Alabama, and
Georgia
(5) Florida
(6) Texas
(7) Oklahoma and Kansas
(8) North Dakota, South
Dakota, Nebraska, Minnesota,
and Iowa
(9) Colorado, Utah, Nevada,
Wyoming, and Montana
(10) New Mexico and Arizona
(11) Oregon, Washington, and
Idaho
(12) California
(13) Hawaii
(14) Alaska
(15) U.S. Territories
U.S. Average
0.690
0.678
0.730
0.867
0.02556
0.02437
0.01351
0.01315
0.01283
0.00856
0.00774
0.01236
0.900
0.788
0.782
0.900
1.005
0.840
0.833
0.990
0.875
0.01392
0.01414
0.900
1.160
0.909
0.658
0.01158
0.00762
0.01377
0.00941
0.900
0.900
1.009
0.970
0.147
0.350
0.858
0.749
0.858
0.676
0.01345
0.01831
0.03443
0.01163
0.03443
0.01815
0.00337
0.00299
0.00777
0.00461
0.00777
0.01053
0.781
0.618
0.849
0.859
0.849
0.900
0.833
0.659
0.905
0.916
0.905
0.959
The estimate of potential weatherization-eligible homes will be calculated from U.S. Census
Bureau data and information collected from the agencies. The number of homes that match the income
requirements for weatherization will be calculated at the local agency jurisdiction level then the number
of already weatherized homes will be subtracted from this total, showing the number of weatherizationeligible homes in each agency‘s jurisdiction. The agency totals can then be added to calculate regional
numbers of eligible homes. An average of energy savings from weatherized homes, in MWh and on a peragency level when possible, will be multiplied by the number of weatherization-eligible homes to
determine the potential energy savings on the local agency jurisdiction and then regional levels.
Following these calculations, the indirect emissions from Table 4.7 will be multiplied by the
calculated potential energy savings (in MWh) to determine the potential GHG emissions savings on
regional and agency levels, measured in metric tons of CO2e. These figures will be the actual potential
GHG emissions reductions for each agency and each region.
Using GIS software, the potential GHG emission reductions will be mapped across the U.S. to
show the areas with the greatest potential for emissions savings. Maps will be created on the national,
regional, and local agency jurisdiction levels. This savings information can then be passed on to third
parties in the carbon markets and will ideally draw interest from private industry to the low-income home
weatherization market.
114
4.7 PERSISTENCE OF ENERGY SAVINGS
Almost two decades ago, Ed Vine wrote: ―The persistence of energy and demand savings is an
important issue to many stakeholders: building owners, architects and engineers, utility program
managers and evaluators, regulators, utility shareholders, resource planners, forecasters, and
researchers.‖37
4.7.1 Background
The U.S. Department of Energy, as well as states and utilities, has funded the weatherization of
low-income homes for over three decades. As stated by Ed Vine in 1992 (see footnote above), it is
important for many reasons and from many viewpoints to understand how energy savings persist in
weatherized homes. With increasing concerns about global climate change, it is even more important to
estimate persistence because of the carbon emissions benefits from weatherizing homes. Many are
exploring how carbon credits can be generated from weatherizing low-income homes, and potential
purchasers of carbon credits need to know what amount of carbon emissions will be reduced for what
period of time.
Unfortunately, no credible persistence studies have been conducted in recent years and only a few
have ever been conducted. Those studies that were conducted date back to the late 1980s and early
1990s38 and only explored persistence two to three years after weatherization. Thus, the weatherization
and voluntary carbon market communities have little information about how to estimate persistence of
energy savings in weatherized low-income homes.
4.7.2 Research Design
Proposed herein is a pilot project to retrospectively estimate persistence of energy savings in
approximately 114 weatherized single-family and low-income mobile homes. The project team will work
with the State of Wisconsin (where several local weatherization agencies have been identified that have
comprehensive historical weatherization records) and possibly one or two additional states to identify
several hundred homes that were weatherized around 1995 by three to four different local agencies. From
this sample, the goal will be to identify 114 homes that have good weatherization records and whose
occupants will allow home inspections and will consent to participating in a short survey (see below).
Homes that were weatherized by the same agencies in Program Years 2007, 2008, and 2009 will
be used as a control group for this study. Specifically, treatment and control homes will be matched by
age. In other words, a fifteen year old home weatherized in 1995 of a general type and size would be
matched with a twenty-eight year old home weatherized in 2008 of the same general type and size,
making sure that the latter had not been weatherized previously. This matching technique allows
researchers to track changes in air leakage rates over time. A blower door test done pre-weatherization on
the 2008 home would represent the tightness not only of that home but what the tightness of the 1995
home might have been in 2008 had it not been weatherized in 1995. Since new blower tests will be
performed for all homes weatherized in the 1995 period, one can then estimate the persistence of 1995
weatherization measures on air leakage rates by taking the difference between the blower door test results
for the 2008 weatherized homes and the 2008 blower door test results for the 1995 weatherized homes.
An added bonus would be if blower door test results were also available pre- and post-weatherization for
the homes weatherized during the 1995 period.
37
Vine, E. 1992. Persistence of Energy Savings: What Do We Know and How Can It Be Ensured? Energy, Vol. 17,
No. 11, 1073-1084.
38
E.g., Brandis, P., and Haeri, H. 1992. The Persistence of Energy Savings Over Time: Two and Three Years After
Participation in a Retrofit Program. Policy Studies Review, Vol. 20, No. 1, 68-75.
115
An essential part of this proposed project is home inspection. As detailed in Task 4, each
treatment home will receive an on-site inspection to generate information about weatherization measures
installed in the home years in the past. The inspections will reveal the condition of the measures and
whether they have been replaced or discarded. Any changes in house structure and integrity will also be
noted.
The last component of this research design is a short interview. The current occupants of the
homes included in the study will be asked about their knowledge of the weatherization measures installed
in their homes, the occupant history of the homes (to the extent possible), and any changes in house
structure, integrity, occupancy, etc. that could be useful in understanding variations in energy use in the
home over time.
4.7.3 Tasking Statement
The proposed project will be composed of the following nine tasks:
Task 1. Agencies for Retrospective Study – The first task is to identify three or four local
weatherization agencies in Wisconsin to participate in this study. The focus of this pilot study
will be in cold-weather regions, under the assumption that weatherization activities save more
energy, and therefore reduce carbon emissions more, in cold regions. Future projects should
balance this information by focusing on homes in hot climates. The study will focus only on
single-family and mobile homes that heat with fossil fuels or electricity produced by fossil fuels.
Any local agencies selected to participate in this project will be required to have excellent
records of past weatherization activities the 1995 time period. Information needed will include
home addresses, audit reports, and installed weatherization measures. The records need not be in
electronic form, although that would be preferable.
Task 2. Collect Weatherization Records– Project members will visit the participating local
agencies to collect weatherization records. The goal will be to collect records for approximately
400 homes that were weatherized in the 1995 period.
Task 3. Sample Development – This task entails matching homes weatherized in the 2007, 2008,
and 2009 period to homes weatherized in the 1995 period. As mentioned above, each home in
the former list needs to be matched with a home in the latter list that was build in exactly the
same year and is essentially the same type and size. The goal is to end up with a matched sample
of approximately 125 homes.
Task 4. Conduct Inspections – Households will be paid an incentive to encourage them to
participate in this study. The incentive will encourage households, firstly, to allow their homes to
be thoroughly inspected. With the audit reports and list of installed weatherization measures in
hand, inspectors will visit the treatment homes to document the following:
Measure degradation – Weather stripping and other air-sealing measures will be
examined. Infrared technology may be used to measure any settling of insulation in the
walls. Measurements of the depth of insulation in ceilings will be taken. The condition of
insulation around water heaters and ducts will also be noted. It is understood that past
records may not include infrared measurements, for example, but using expert judgment,
some conclusions can be made about measure degradation in these cases;
116
Measure replacement – It will be determined whether various installed measures were
replaced and if so, whether they were replaced with measures that had better, worse or
the same energy-savings characteristics. Examples of measures that could be replaced
include furnaces, windows, doors, and refrigerators;
Measure retention – It will be determined whether various installed measures were
retained in the home. Examples of these types of measures include CFLs and low-flow
shower heads;
Change in the condition or size of the home – Inspectors will determine to the extent
possible whether the size of the home has changed (e.g., whether there have been any
additions). Also, the inspectors will note any possible changes in the condition of the
home (e.g., broken windows, holes in the roof) that could significantly impact energy
savings; and
Any other changes in the home that may impact energy-savings persistence estimates will
be noted. Such changes could include the addition of window air conditioners, portable
heaters, and wood furnaces.
It is proposed that blower door tests be conducted on treatment homes. If available, these
results can be compared to blower test results from 1995, allowing for the possibility that tests
conducted in 1995 were less accurate that those conducted today. Blower door test results from
pre-weatherization and immediately post-weatherization for recently weatherized homes can be
used as benchmarks. Additionally, it is proposed that the efficiency of furnaces be measured,
along with the emissions of CO in flu gases. Indoor temperatures and water-heater temperatures
may also be recorded.
Task 5. Conduct Household Interviews – These will be conducted on-site by the inspectors.
They will inquire about the occupancy history of the home and any major structural or other
changes in the home that may have significantly changed energy consumption over the past
decade and a half years.
Task 6. Collect Utility Bills – The project will attempt collect twelve years of billing histories for
all homes enrolled in the study (i.e., both treatment and control). It is anticipated that these data
will be available in various formats, from readily available electronic data bases to paper files
residing in filing cabinets in the basements of utility buildings.
Task 7. Energy Savings Data Analysis – PRISM and one or two other methods will be used to
weather adjust and analyze the billing histories. Energy use for each group of homes will plotted
over time (pre-and post-weatherization). Comparisons will be made between weatherized and
control homes and the persistence of gross and net energy savings will be estimated.
Task 8. Implications for Voluntary Carbon Markets – An exercise will be conducted to
estimate carbon emissions reductions that could be associated with each group of weatherized
homes. Costs associated with weatherizing the homes will be compared to typical prices for
carbon credits found in representative voluntary carbon markets. The cost-effectiveness of home
weatherization as a potential vehicle for voluntary carbon markets will be assessed.
117
Task 9. Develop a Forward-Looking Data-Collection Protocol on Persistence– The data
collection procedures described above probably only represent a first try at the types of data one
might actually wish to do in persistence studies. For example, thoroughly documenting the
condition and structural characteristics of homes at the time of weatherization and every three or
four years afterward, through complete inspection reports and photographic documentation,
would be much preferable to relying only on anecdotal evidence or the memories of occupants.
Also, more extensive testing and data collection at the time of weatherization could serve as
benchmarks (blower door tests, even infrared analyses of insulation in walls immediately after
weatherization) for future studies; those would then be supplemented by repeated testing and
data collection repeated every three or four years in the same homes. The output of this task will
be a comprehensive and detailed data collection protocol to be implemented on any homes
chosen to be enrolled in persistence studies.
Task 10. Draft and Final Reports – A draft report will be prepared and peer-reviewed. The final
report will be revised accordingly.
118
5. SYNTHESIS
After the impact assessment, process assessment, and special technical studies have been
performed, results from these studies will be drawn together via a synthesis study to address remaining
questions identified in the evaluation design matrix (see Table 1.2):
Context—Questions 2 and 8,
Implementation—Question 7, and
Outcomes—Questions 11 and 15-16.
Specifically, the synthesis study will determine:
whether the Program has the capacity and structure (e.g., funding, staffing) to meet its legislative
missions and objectives,
how well the Program is meeting its legislative missions and objectives,
whether the states and local agencies are fulfilling their obligations under federal regulations and
state plans,
the extent to which the Program is serving the weatherization needs of the low-income
community and meeting the needs of the national low-income weatherization market, and
how the weatherization network‘s performance can be improved to guide the Program into the
next decade.
The Program objectives as set by legislation will be identified in the impact assessment (see
Section 3.1). One or more measurable indicators will be developed under the synthesis study for each
identified objective and an expected value for each indicator will be established based on the legislative
intent. Evaluation data and results from the previous studies will be used to determine an actual value for
each indicator, and the actual value of each indicator will be compared to the expected value to determine
whether the legislative intent is being met. The key Program objectives that will be examined include:
the number of clients served by the Program,
the extent to which the Program focused on low-income persons who are particularly vulnerable
as defined by DOE (i.e., households with elderly or disabled persons, those with children, and
those with high residential energy use and high energy burden), and
the spending of Program financial resources according to federal regulations (e.g., adherence to
spending limits for training, overhead, and weatherization measures ; following rules concerning
materials purchased and measures installed with Program funds).
Using results from the impact and process assessments, a determination will be made as to
whether states and agencies are fulfilling their obligations under Federal regulations and state plans (e.g.,
units weatherized, average household expenditures, expenditures for training and overhead).
The extent to which the weatherization needs of low-income households are being met by the
Program will be assessed by examining (from the previous studies) the households being served by the
Program compared to the larger low-income population, the breadth of activity performed nationally
119
under the Program and differences in this activity by climate region, and the energy impacts of this
activity and the quality of the jobs performed.
The results and findings from the previous three study areas (i.e., impact assessment, process
assessment, and special technical studies) will be brought together and examined to develop
recommendations on how the Program and the weatherization network‘s performance can be improved.
In addition to synthesizing and distilling findings about Program outcomes and processes concerning PYs
2009, 2010 and 2011, this study will develop insights useful for guiding the Program into the next decade.
Future trends of many variables relevant to the Program will be assessed, including demographics, energy
prices and availability, housing stock, residential energy technologies, possibly new energy and
environmental legislation, restructuring of the electric utility industry, and workforce. Recommendations
will address the full breadth of the Program and the network‘s operation, including the delivery of the
Program, communications within the network, coordination with other programs, and reporting of this
coordination to DOE. Recommendations should also be developed on how a longer-term, more
continuous evaluation of the Program could be implemented by DOE so that the longer-term outcomes of
the Program and the long-term persistence of energy savings could be more fully addressed (see Section
1.1.3). The standardized data collection needed to support such an effort should also be addressed. Two
groups will be consulted to help develop the recommendations:
Network Committee—The Network Committee will be re-convened to consider the evaluation‘s
findings and trends into the future. The Committee will identify those trends that could most
impact the Program in the next decade and will also make recommendations to the Program with
respect to guiding the program into the next decade.
Expert Panel—An Expert Panel will be formed to solicit the opinions of about a dozen policy
and public administration experts (most are expected to be academics). Through an iterative
process, the Expert Panel members will provide (1) their opinions and insights about the
evaluation‘s findings and their reactions to the opinions and insights of others, (2) their responses
to policy-oriented and program administration questions and their reaction to the responses of
others, and (3) their recommendations and their reaction to the recommendations proposed by
others. It is expected that a well-run expert-panel process will find areas of consensus and
disagreement amongst the panel members. This expert panel process will be run virtually (i.e.,
without convening the participants in one physical place), both to minimize expenses and also to
maximize the time allowed to the panelists to provide answers, consider the inputs of the other
panelists, and provide their reactions to the opinions of the panel.
The results of the discussions of the Network Committee and the Expert Panel will be compiled
into a separate report and delivered to DOE for its use.
120
121
6. SCHEDULE
A schedule for the evaluation is shown in Figure 6.1. It should be noted that the intent of the
evaluation is to measure and evaluate PY 2009-2011 activities; PY 2009 starts in April 2009 and PY 2011
ends in June 2012.
The timely implementation of the data-collection aspects of this preliminary evaluation plan
depends upon receiving approval of the national evaluation from OMB in a timely manner. DOE and the
ORNL evaluation team are in the process of obtaining OMB‘s approval.
122
Figure 6.1. WAP-ARRA Period Evaluation schedule
Task Name
Calendar Year
Calendar Year
Calendar Year
2011
2012
2013
Q Q Q Q Q Q Q Q Q Q Q Q
1
2
3
4
1
2
3
4
1
2
3
4
Calendar Year
2014
Q Q Q Q
1
2
3
4
National
Energy
Savings
Analyses
States Survey
(S1)
Agency
Surveys
WIPP Study
SERC Study
UnderPerformers
Study
GHG
Emissions
Study
Deferral Study
Social
Network
Study
Weatherizatio
n Staff Survey
Follow-ups
Persistence
Study
123
7. REFERENCES
APPRISE Incorporated. ―Ohio REACH Baseline Survey Instrument.‖
APPRISE Incorporated. ―NEADA 2009 National Energy Assistance Survey.‖ November 2009
Center for Studying Health System Change (HSC). ―Community Tracking Study Household Survey
Instrument.‖ February 2005
Centers for Disease Control and Prevention (CDC). ―2010 Behavioral Risk Factor Surveillance System
Questionnaire.‖ 18 November 2009
Centers for Disease Control and Prevention (CDC). ―National Asthma Survey.‖ 18 April 2003
Centers for Disease Control and Prevention (CDC). ―National Health Interview Survey.‖ 3 March 2010
Code of Federal Regulations, National Archives and Records Administration, Office of the Federal
Register, Title 10, Part 440, Section 1, Revised January 1, 2005 (see www.gpoaccess.gov/cfr).
Berry, Linda, ―State-Level Evaluations of the Weatherization Assistance Program in 1990-1996: A
Metaevaluation that Estimates National Savings,‖ ORNL/CON-435, Oak Ridge National Laboratory,
January 1997.
Berry, Linda and Martin Schweitzer, ―Metaevaluation of National Weatherization Assistance Program
Based on State Studies, 1993–2002,‖ ORNL/CON-488, Oak Ridge National Laboratory, February 2003.
Brown, Marilyn A., Linda G. Berry, and Laurence F. Kinney, ―Weatherization Works: Final Report of the
National Weatherization Evaluation,‖ ORNL/CON-395, Oak Ridge National Laboratory, September
1994.
Brown, Marilyn A., Linda G. Berry, Richard A. Balzer, and Ellen Faby, ―National Impacts of the
Weatherization Assistance Program in Single-Family and Small Multifamily Dwellings,‖ ORNL/CON326, May 1993.
Brown, Marilyn A. and Lawrence J. Hill, ―Low-Income DSM Programs: The Cost-Effectiveness of
Coordinated Partnerships,‖ ORNL/CON-375, Oak Ridge National Laboratory, May 1994.
DOE, ―Weatherization Program Notice 05-1,‖ November 12, 2004 (see www.waptac.org).
Fels, M., K. Kissock, M. Marean, and C. Reynolds, ―PRISM Advanced Version 1.0 User‘s Guide,‖
Princeton University, Center for Energy and Environmental Studies, Princeton, NJ, 1995.
Gettings, Michael, ―The Weatherization Assistant Users Manual for Administrative Features (Version
8),‖ ORNL/TM-2005/236, Oak Ridge National Laboratory, January 2006 (see ―Energy Audits‖ under
www.waptac.org).
Hall, Nick et al., ―California Energy Efficiency Evaluation Protocols: Technical, Methodological, and
Reporting Requirements for Evaluation Professionals,‖ State of California Public Utilities Commission,
April 2006.
124
Hall, Nick et al., ―The California Evaluation Framework,‖ Southern California Edison Company, Project
K2033910, June 2004.
Hill, Lawrence J. and Marilyn A. Brown, ―Standard Practice: Estimating the Cost-Effectiveness of
Coordinated DSM Programs,‖ ORNL/CON-390, Oak Ridge National Laboratory, December 1994.
W. K. Kellogg Foundation, ―Logic Model Development Guide: Using Logic Models to Bring Together
Planning, Evaluation, & Action,‖ December 2001 (see www.wkkf.org).
Kissock, J. Kelly, Jeff S. Haberl, and David E. Claridge, ―Development of a Toolkit for Calculating
Linear, Change-Point Linear, and Multiple-Linear Inverse Building Energy Analysis Models,‖ RP-1050,
American Society for Heating, Refrigerating, and Air Conditioning Engineers, 2004.
Levins, William P. and Mark P. Ternes, ―Impacts of the Weatherization Assistance Program in Fuel-Oil
Heated Houses,‖ ORNL/CON-327, Oak Ridge National Laboratory, October 1994.
National Center for Healthy Housing. ―WATTS and Well-being Study Health Survey Questionnaire.‖ 7
November 2009
Oak Ridge National Laboratory. ―WAP Evaluation Occupant Survey.‖ 2006
Power, Meg, ―Weatherization PLUS Other Efficiency and Housing Investments Delivered by Local
Weatherizers in PY 2000,‖ Economic Opportunity Studies, June 13, 2003
(www.opportunitystudies.org/repository/File/weatherization/utility-wap-combined-programs.pdf).
Schweitzer, Martin, ―Estimating the National Effects of the U.S. Department of Energy‘s Weatherization
Assistance Program with State-Level Data: A Metaevaluation Using Studies from 1993 to 2005,‖
ORNL/CON-493, Oak Ridge National Laboratory, September 2005.
Schweitzer, Martin and Linda Berry, ―Metaevaluation of National Weatherization Assistance Program
Based on State Studies, 1996–1998,‖ ORNL/CON-467, Oak Ridge National Laboratory, May 1999.
Schweitzer, M. and B. Tonn, ―Nonenergy Benefits from the Weatherization Assistance Program: a
Summary of Findings from the Recent Literature,‖ ORNL/CON-484, Oak Ridge National Laboratory,
April 2002.
Ternes, M., Schweitzer, M., Tonn, B., Schmoyer, R., and Eisenberg, J. 2007. ―National Evaluation of the
Department of Energy‘s Weatherization Assistance Program (WAP): Program Year 2006 Experimental
Plan,‖ ORNL/CON-498, Oak Ridge National Laboratory, Oak Ridge, TN, February.
USAID. ―Household Food Insecurity Assessment Scale.‖August 2007
US Department of Energy (DOE). ―Residential Energy Consumption Survey.‖ 2009
125
APPENDIX A. NATIONAL WEATHERIZATION NETWORK COMMITTEE
Adams, Robert
Beachy, Bill
Bennett, Randy
Bensch, Ingo
Berger, Jackie
Bethke, Jack
Bowmar, Kip
Brady, Eugene
Carroll, David
Choate, JoAnn
Costello, Pat
Crisp, Jim
Cutchen, Kelly
Diggs, Jean
Doyle, Dawn
Eisenberg, Joel
Foote, Katherine
Gerardot, Ed
Hamilton, John
Harmelink, Suzanne
Hepinstall, Dave
Jacobson, Elliott
Peterson, Mike
Pitts, Keith
Power, Meg
Quenemoen, Kane
Ravesloot, Holly
Sabree-Sylla,Clarice
Schweitzer, Martin
Scott, Bob
Simonson, Cynthia
Smith, Carrie
Sunday, Nick
Tonn, Bruce
Van der Meer, Bill
Zamora, Christina
DOE/OWIP/WAP
Virginia-Community Housing Partners Corporation
State of IL, Dept. Commerce & Econ. Opp.
Apprise
Apprise
Community Action of Minneapolis
Community Action Kentucky
Pennsylvania-Commission on Economic Opportunity
Apprise
Maine State Housing Authority
NYS Division of Housing & Community Rene
Michigan Community Action Agency Assoc.
SMS
DOE/OWIP/WAP
Texas Department of Housing and Community Affairs
Oak Ridge National Laboratory
DOE/GO
Indiana Community Action Association
Illinois – CEDA
WI Energy Conservation Corporation
New York -Association for Energy Affordability
Massachusetts-Action, Inc
DOE/GO
Ohio-Corporation for Ohio Appalachian Develop.
NCAF
Montana DPHHS
DOE/OWIP/WAP
NJ Department of Community Affairs
Oak Ridge National Laboratory
NASCSP
SMS
Arizona – FSL
State of Ohio-Office of Community Service
Oak Ridge National Laboratory
SMS
Idaho –CAPAI
126
127
APPENDIX B. DOE SURVEY
Pending DOE approval, the Project Team will interview key OWIP managers concerning the challenges
to expanding and then ramping down WAP during the ARRA period. The interviews will be open-ended.
128
OMB Control Number: XXXX-XXXX
APPENDIX C: S1: ALL STATES PROGRAM INFORMATION SURVEY
This data is being collected to conduct a process evaluation of the Weatherization Assistance
Program at the state level. The data you supply will be used to characterize program activities
during Program Year 2010.
Public reporting burden for this collection of information is estimated to average sixteen hours
per response, including the time for reviewing instructions, searching existing data sources,
gathering and maintaining the data needed, and completing and reviewing the collection of
information. Send comments regarding this burden estimate or any other aspect of this
collection of information, including suggestions for reducing this burden, to Office of the Chief
Information Officer, Records Management Division, IM-11, Paperwork Reduction Project
(XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW, Washington, DC,
20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork Reduction
Project (XXXX-XXXX), Washington, DC 20503.
All of the information obtained from this survey will be protected and will remain confidential.
The data will be analyzed in such a way that the information provided cannot be associated back
to your state, your agencies, or the housing units and clients that your state served. Again, please
note that the questions refer to PY 2010 unless otherwise noted.
PROGRAM CHARACTERIZATION
1. Please identify your state: __________________________________
2. It is important to collect information about the weatherization of homes beyond the standard
single family homes that are heated with natural gas or electricity. Please review the following
information for accuracy about each of the local agencies (subgrantees) that you fund to provide
weatherization services in your state. Please add any additional grantees and funding amount in
the extra space at the bottom of the table:
Local Agency (Subgrantee) Name
Active
Subgrantee
(Y/N)
Amount of DOE
Funds Received
by Agency in
Program Year
2010
Correct Amount
(Y/N)
If NO, Please
Insert Correct
Amount
129
Local Agency (Subgrantee) Name
Active
Subgrantee
(Y/N)
Amount of DOE
Funds Received
by Agency in
Program Year
2010
Correct Amount
(Y/N)
If NO, Please
Insert Correct
Amount
3. During Program Year 2010, was the director of your State‘s Weatherization Program (Check
best answer):
_____ a civil servant
_____ political appointee
_____ elected official
4.During Program Year 2010, did the director of your State‘s Weatherization Program report to a
(Check best answer):
_____ civil servant,
_____ political appointee
_____ elected official
5. For how many years had the current director of your State‘s Weatherization Program served in
that capacity prior to PY 2010?
_____
6. Did your State‘s Weatherization Program set annual performance goals for PY 2010?
_____ Yes
_____ No (go to Question 7)
6a. What agency, office, or department was responsible for reviewing the annual performance
goals and achievement of goals of your State‘s Weatherization Program? _____
130
7. Please list other important housing and/or energy-related programs for low-income residents
that were administered by the same office that is in charge of your state‘s Weatherization
Assistance Program.
______________________________________________________________________________
______________________________________________________________________________
8. What weatherization program data did your state require its weatherization agencies to provide
in PY2010? (Check all that apply)
_____ Number of homes weatherized
_____ Number of homes weatherized for high priority categories
_____ DOE weatherization funds expended
_____ Non-DOE weatherization funds expended
_____ Descriptive statistics on demographics of households weatherized
_____ Descriptive statistics on weatherization measures installed in households weatherized
_____ Descriptive statistics on energy use/savings of households weatherized
_____ Copy of audits performed on the households weatherized
_____ Results of certain diagnostic tests
_____ Number of homes deferred for weatherization
_____ Other ___________________
9. Does your state maintain an electronic state weatherization program data base?
a. Yes
b. No (Go to Q13)
10. If what does this database contain these weatherization related-elements? (check all that
apply)
a. number of homes weatherized
b. number of homes weatherized by type
c. costs of measures installed
d. billing records for weatherized homes
e. audit records
f. measures installed by unit
g. other _______________________
11. Does your database also contains these types of information: (check all that apply)
a. LIHEAP records
b. household demographics
c. other ___________________________
12. Who has access to this database? (check all that apply)
a. state weatherization office staff
b. other state employees
c. local weatherization agency staff
d. other ______________________
13. Please indicate the number of staff that supported your State‘s Weatherization Program and
their work effort in Program Year 2010. In considering the number of staff, please include
131
everyone who worked full- or part-time or who worked with the weatherization program as well
as other state programs. Also include any non-agency staff supporting the state program that
work under contract to the state.
Type of Administrative Function
Number of
Staff (#
persons)
Total Staff Work
Effort (# of FTE)
Management/administration
Agency monitoring
Training and Technical Assistance
Other (specify)
TOTAL
14. For the in-house staff working on your state‘s weatherization program in each of the
following functional areas in Program Year 2010, please indicate their collective level of
experience with the weatherization program:
Very
High
High
Medium
Low
Very Low
Management/administration
Field monitoring/auditing
Training and Technical
Assistance
Other (specify)
15. For the in-house staff working in your state‘s weatherization program in each of the
functional areas listed below, please indicate the amount of turnover in staff from the
beginning of PY 2009 to the end of PY 2010. (Please check appropriate box representing the
level of turnover for each functional area.)
Management/
administration
Field monitoring/
Auditing
Training and Technical
Assistance
Other (specify)
____________________
No Turnover
Some turnover
(all staff in this functional
area at the beginning of
PY 2009 were in the same
functional area by the end
of PY 2010)
(1-15% of the staff in this
functional area at the
beginning of PY 2009 did
not remain in the same
functional area by the end
of PY 2010)
Substantial
turnover
(more than 15% of the
staff in this functional
area at the beginning of
PY 2009 remained in the
same functional area at
the end of PY 2010)
□
□
□
□
□
□
□
□
□
□
□
□
132
16. The Federal Regulations governing the Weatherization Assistance Program define children
as ―dependents not exceeding 19 years or a lesser age set forth in the State plan.‖ What age did
your state use in your state‘s definition of children in PY 2010? _____
17. Did your state use a high energy burden category to prioritize the provision of weatherization
services in PY 2010?
_____ Yes
_____ No (go to Question 18)
17a. How was ‗high energy burden‘ defined? ______________________________
18. Did your state use a high energy expenditure category to prioritize the provision of
weatherization services in PY 2010?
_____ Yes
_____ No (go to Question 19)
18a. How was ‗high energy expenditure‘ defined? ____________________________
19. What were the income guidelines for households to be eligible for your state‘s weatherization
program in PY 2010? (Check all that apply)
_____ 150% of Federal Poverty Guidelines
_____ 200% of Federal Poverty Guidelines
_____ More than 200% of Federal Poverty Guidelines
_____ 60% of state median income
_____ Other: ________________________
133
LEVERAGING RELATIONSHIPS
1. Please list weatherization funding received during PY 2010 by completing the table below.
Column A lists potential sources of weatherization funding. As a reference, Column B lists the amount of PY 2008
funding reported by your state in the retrospective evaluation S1 survey. If the PY 2008 funding amount is incorrect,
please list the correct amount in Column B. In Column C, please indicate whether your state administered each
funding source in PY 2010. In Column D, enter the total funding amount administered in PY 2010 from each source
received. Please allocate the total funding amount listed in Column D to the sub-categories listed in Columns E
through I.
A
Funding Source
DOE1
LIHEAP
Petroleum
Violation Escrow
(PVE)
Other Federal
Programs
State Public
Benefit Funds
Other State
Programs
Utilities
Program Income
In-Kind
Non-Profits
Third Party
(e.g., Foundations,
Lenders)
All other
(Please specify)
______________
TOTAL
1.
2.
B
C
D
Funds
administered
from this
source in PY
2008
Did state
administer
weatherization
funds from
this source in
PY 2010
Prepopulated
Prepopulated
Prepopulated
Yes
No
Yes
No
Yes
No
Prepopulated
Prepopulated
Prepopulated
Prepopulated
Prepopulated
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Prepopulated
Prepopulated
Prepopulated
Yes
No
Yes
No
Yes
No
Prepopulated
Yes
No
Total funds
administered
from this
source in PY
2010
E
F
Funds retained and spent by
state
G
H
I
Funds passed on to subgrantees
Grantee
administration
and
leveraging
Grantee
training and
technical
assistance
(T&TA)
Subgrantee
funding for
program
operations2
Selecting
―no‖ will
gray out all
columns to the
right in
Selecting
―yes‖ will
drive skip
patterns in
Q1b-13
Subgrantee
training and
technical
assistance
(T&TA)
Subgrantee
administration
and leveraging
online survey
Prepopulated
Include WAP and ARRA funds in this row.
List all funding for weatherization program operations that was passed on to subgrantees, including amounts spent on measures, labor, health and
safety, financial audits, liability insurance, vehicles and equipment.
134
1a. Of the Program Year 2010 funds retained by your state‘s weatherization program for
management functions, how much was used for each function listed below?
Type of Management Function
Total
Administration*
Agency monitoring
Other (specify)
TOTAL
* Includes planning, finance and accounting, clerical support, outreach, and evaluation.
[1b will be activated by ‗yes‘ responses in the associated funding categories of Question 1
and skipped for any ‗no‘ responses in those categories].
1b. In the categories below, please specify the name of each funder and the total amount of
weatherization funding provided in PY 2010.
Funding entity
(please enter names)
Total weatherization funding
administered in PY 2010
(please enter amounts)
Utilities
Utility 1
Utility 2
Utility 3
All other utilities
In-kind contributions
In-kind 1
In-kind 2
In-kind 3
All other in-kind
Nonprofits
Nonprofit 1
Nonprofit 2
Nonprofit 3
All other nonprofits
Other
Other 1
Other 2
Other 3
All others
135
[Q2 will be activated by ‗yes‘ responses in the LIHEAP category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
2a. When did you first receive weatherization funds from LIHEAP? _________
2b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
2c. Please describe the change in the leveraging relationship between your state‘s low-income
weatherization program and LIHEAP during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely negative
2d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
2e. If leveraged funding from LIHEAP decreased from PY 2008, was the change attributable to
increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
2f. Did you require local agencies to follow DOE rules when spending weatherization funds
from LIHEAP in PY 2010?
_____ Yes
_____ No
[2g will appear if respondent answered ‗no‘ in 2f]
2g. What were the major differences in the rules governing the expenditure of LIHEAP
funds in comparison to the rules governing the expenditure of DOE funds?____________
________________________________________________________________________
[Q3 will be activated by ‗yes‘ responses in the PVE category of Question 1 and deactivated
(skipped) for ‗no‘ responses in that category.]
3a. When did state first receive weatherization funds from Petroleum Violation Escrow (PVE)?
_________
3b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
136
3c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and PVE during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
3d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
3e. If leveraged funding from PVE decreased from PY 2008, was the change attributable to
increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
3f. Did you require local agencies to follow DOE rules when spending weatherization funds
from PVE in PY 2010?
_____ Yes
_____ No
[3g will appear if respondent answered ‗no‘ in 3f]
3g. What were the major differences in the rules governing the expenditure of PVE funds
in comparison to the rules governing the expenditure of DOE funds?
_____________________________________________________________
________________________________________________________________________
[Q4 will be activated by ‗yes‘ responses in the Other Federal Programs category of
Question 1 and deactivated (skipped) for ‗no‘ responses in that category.]
4a. When did state first receive weatherization funds from Other Federal Programs? _________
4b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
4c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and Other Federal Programs during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
137
4d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
4e. If leveraged funding from Other Federal Programs decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
4f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Other Federal Programs in PY 2010?
_____ Yes
_____ No
[4g will appear if respondent answered ‗no‘ in 4f]
4g. What were the major differences in the rules governing the expenditure of Other
Federal Programs funds in comparison to the rules governing the expenditure of DOE
funds? __________________________________________________________________
________________________________________________________________________
[Q5 will be activated by ‗yes‘ responses in the State Public Benefit Funds category of
Question 1 and deactivated (skipped) for ‗no‘ responses in that category.]
5a. When did state first receive weatherization funds from State Public Benefit Funds?
_________
5b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
5c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and State Public Benefit Funds during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
5d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
138
5e. If leveraged funding from State Public Benefit Funds decreased from PY 2008, was the
change attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
5f. Did you require local agencies to follow DOE rules when spending weatherization funds
from State Public Benefit Funds in PY 2010?
_____ Yes
_____ No
[5g will appear if respondent answered ‗no‘ in 5f]
5g. What were the major differences in the rules governing the expenditure of State
Public Benefit Funds in comparison to the rules governing the expenditure of DOE
funds? __________________________________________________________________
________________________________________________________________________
[Q6 will be activated by ‗yes‘ responses in the Other State Programs category of Question 1
and deactivated (skipped) for ‗no‘ responses in that category.]
6a. When did state first receive weatherization funds from Other State Programs? _________
6b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
6c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and Other State Programs during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
6d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
6e. If leveraged funding from Other State Programs decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
6f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Other State Programs in PY 2010?
139
_____ Yes
_____ No
[6g will appear if respondent answered ‗no‘ in 6f]
6g. What were the major differences in the rules governing the expenditure of Other State
Programs funds in comparison to the rules governing the expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
[Q7 will be activated by ‗yes‘ responses in the Utilities category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
7a. When did state first receive weatherization funds from Utilities? _________
7b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
7c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and Utilities during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
7d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
7e. If leveraged funding from Utilities decreased from PY 2008, was the change attributable to
increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
7f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Utilities in PY 2010?
_____ Yes
_____ No
[7g will appear if respondent answered ‗no‘ in 7f]
7g. What were the major differences in the rules governing the expenditure of Utilities
funds in comparison to the rules governing the expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
140
[Q8 will be activated by ‗yes‘ responses in the Program Income category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
8a. When did state first receive weatherization funds from Program Income? _________
8b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
8c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and Program Income during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
8d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
8e. If leveraged funding from Program Income decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
8f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Program Income in PY 2010?
_____ Yes
_____ No
[8g will appear if respondent answered ‗no‘ in 8f]
8g. What were the major differences in the rules governing the expenditure of Program
Income funds in comparison to the rules governing the expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
[Q9 will be activated by ‗yes‘ responses in the In-Kind Contributions category of Question
1 and deactivated (skipped) for ‗no‘ responses in that category.]
9a. When did state first receive weatherization funds from In-Kind Contributions? _________
9b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
141
9c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and In-Kind Contributions during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
9d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
9e. If leveraged funding from In-Kind Contributions decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
9f. Did you require local agencies to follow DOE rules when spending weatherization funds
from In-Kind Contributions in PY 2010?
_____ Yes
_____ No
[9g will appear if respondent answered ‗no‘ in 9f]
9g. What were the major differences in the rules governing the expenditure of In-Kind
Contributions in comparison to the rules governing the expenditure of DOE
funds?__________________________________________________________________
________________________________________________________________________
[Q10 will be activated by ‗yes‘ responses in the Non-profits category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
10a. When did state first receive weatherization funds from Non-profits? _________
10b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
10c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and Non-profits during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
142
10d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
10e. If leveraged funding from Non-Profits decreased from PY 2008, was the change attributable
to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
10f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Non-Profits in PY 2010?
_____ Yes
_____ No
[10g will appear if respondent answered ‗no‘ in 10f]
10g. What were the major differences in the rules governing the expenditure of NonProfits funds in comparison to the rules governing the expenditure of DOE funds?
__________________________________________________________________
________________________________________________________________________
[Q11 will be activated by ‗yes‘ responses in the Third Party category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
11a. When did state first receive weatherization funds from Third Parties (foundations, lenders)?
_________
11b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
11c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and Third Parties during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
11d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
143
11e. If leveraged funding from Third Parties decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
11f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Third Parties in PY 2010?
_____ Yes
_____ No
[11g will appear if respondent answered ‗no‘ in 11f]
11g. What were the major differences in the rules governing the expenditure of Third
Party funds in comparison to the rules governing the expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
[Q12 will be activated by ‗yes‘ responses in All Other, Specify category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
12a. When did state first receive weatherization funds from [insert text from ‗All Other,
Specify‘]?
12b. If known, number of years that state worked to achieve this leveraging relationship (leave
blank if unknown): _________
12c. Please describe the change in leveraging relationships between your state‘s low-income
weatherization program and [insert text from ‗All Other, Specify‘] during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
12d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
12e. If leveraged funding from [insert text from ‗All Other, Specify‘] decreased from PY 2008,
was the change attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
144
12f. Did you require local agencies to follow DOE rules when spending weatherization funds
from [insert text from ‗All Other, Specify‘] in PY 2010?
_____ Yes
_____ No
[12g will appear if respondent answered ‗no‘ in 12f]
12g. What were the major differences in the rules governing the expenditure of [insert
text from ‗All Other, Specify‘] funds in comparison to the rules governing the
expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
13. You indicated your state did not administer leveraged funding from the sources listed below
during PY 2010. For each funding source, please indicate your expectations for funding during
the post-ARRA period beginning in PY 2012.
Funding Source
Populate from ‗no‘ response
in Q1
Populate from ‗no‘ response
in Q1
Populate from ‗no‘ response
in Q1
Populate from ‗no‘ response
in Q1
Expectations for funding in the post-ARRA period beginning in
PY 2012
Funding will
Funding will
increase
No change
increase
greatly
Uncertain
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
14. How important were leveraged funds for your State‘s Weatherization Program in PY 2010
compared to program years without additional ARRA funding? (Check best answer)
_____Very important
_____ Important
_____ Not very important
_____ Not important at all
15. Did your state set aside funding to advocate for leveraged resources in PY 2010?
_____ Yes
_____ No (Go to Question 9)
16. What organizations advocated for leveraged resources in PY 2010? (Check all that apply)
_____ Your state office
_____ Your state‘s agencies
_____ Non-profit organizations funded by your state
_____ Other
17. How successful would you rate your state‘s efforts to acquire leveraged funds in PY 2010?
(Check best answer)
145
_____ Very successful
_____ Successful
_____ Not very successful
_____ Not successful at all
_____ State does not seek leveraged funds
18. What factors limited the success of your state‘s efforts to acquire leveraged funding in PY
2010? _______________________
19. Have you modified your state‘s weatherization program practices or regulations since PY
2008 to facilitate spending and reporting on leveraged resources?
_____Yes
_____ No
20. Overall, how has the influx of ARRA funding impacted previously existing leveraging
relationships?
a. Extremely positive impact
b. Positive impact
c. No Impact
d. Negative impact
e. Extremely negative impact
21. Can leveraging relationships damaged or lost during ARRA be re-built post-ARRA?
a. Yes, absolutely
b. Yes, probably
c. Uncertain
d. No, probably not
e. No, definitely not
22. Do you see state-level Renewable Portfolio Standards programs benefitting low-income
weatherization? If so, how? If not, why not? ____________________________________
23. What aspects about your state‘s low-income weatherization program are most misunderstood
by actual and potential leveraging partners? __________________
24. How has your program worked to overcome these misunderstandings? _________________
25. What information would your state‘s weatherization program like to have that could be used
to overcome these misunderstandings?
26. What other information would your state‘s weatherization program like to have that could be
used to ‗sell‘ leveraging relationships? __________________________
27. Overall, what is your expectation for total state leveraged funding for low-income
weatherization funding post ARRA in PY 2012 (post- ARRA) compared to PY 2008 (preARRA)?
a. Greatly increased
146
b. Increased
c. Same level
d. Decreased
e. Greatly decreased
28. On balance, how beneficial do you think ARRA funding will prove to be over the longerterm on your state‘s ability to leverage DOE WAP-program funding for low-income
weatherization?
a. Extremely beneficial
b. Beneficial
c. No long-term impact
d. Unbeneficial
e. Extremely unbeneficial
PROGRAM OPERATIONS AND IMPLEMENTATION
1. Using the following scale, how adequate was the Program Year 2010 funding received by
your state from ALL funding sources for weatherizing the stock of eligible low-income
dwelling units in your state in a timely fashion? (Check all that apply)
_____ Very Adequate
_____ Adequate
_____ Inadequate
_____ Very Inadequate
2. What was the quality of the management support that your state received from DOE and its
contractors in Program Year 2010? (Check all that apply)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
3. What was the quality of the training that your state received from DOE and its contractors in
Program Year 2010? (Check all that apply)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
4. What was the quality of the support and assistance on client education that your state
received from DOE and its contractors in Program Year 2010? (Check all that apply)
_____ very high quality
_____ high quality
_____ moderate quality
147
_____ low quality
_____ very low quality
_____ not applicable
4a. If appropriate, why did you rate the quality very low or low? ______________
5. What was the quality of the support and assistance on leveraging the Weatherization
Assistance Program funding provided by DOE with other funding sources in Program Year
2010? (Check all that apply)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
5a. If appropriate, why did you rate the quality very low or low? ______________
6. What was the quality of the technical support (e.g., measure selection and installation) that
your state received from DOE and its contractors in Program Year 2010? (Check all that
apply)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
6a. If appropriate, why did you rate the quality very low or low? ______________
7. How flexible did you find the DOE program rules that governed the weatherization program
in Program Year 2010? In other words, did the program rules allow your state to tailor your
program to your needs (very flexible) or proscribe your program to only one way of
operation (very inflexible)? (Check all that apply)
_____ Very Flexible
_____ Flexible
_____ Inflexible
_____ Very Inflexible
148
7a. Using PY 2010 as the reference point, how should the program rules change? (One)
_____ Become much more flexible
_____ Become more flexible
_____ Stay about the same
_____ Become more inflexible
_____ Become much more inflexible
7b. In what areas should the program rules become more flexible? ________________________
7c. In what areas should the program rules become less flexible? _________________________
8. Please describe any important political issues faced by your state‘s weatherization program in
Program Year 2010.
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
___________________________________________________________________________
9. At this point in time, does your state need the following (Check any needed items):
_____Additional administrative support and assistance from DOE
_____Improved training from DOE and its contractors
_____Additional assistance from DOE and its contractors with leveraging resources
_____Improved technical support from DOE and its contractors
_____Additional funding for the delivery of weatherization services
_____Improved data and information systems for the delivery of weatherization services
_____ Additional assistance to develop and administer client education programs
PROGRAM IMPLEMENTATION DURING THE ARRA PERIOD
1. During the ARRA period, has your state added any new local weatherization agencies (i.e.,
sub-grantees) to its low-income weatherization program?
a. Yes
b. No (Skip to Question 4)
2. How many new local weatherization agencies were added? ____________
3.Why were these agencies added? (check all that apply)
a. To meet increased production targets
b. To replace non-performing subgrantees
c. To build synergies between the state weatherization program and other community service
programs in community action agencies that did not provide weatherization services
d. Other _______________________
149
4. For many years, the national weatherization network and the state-level national
weatherization networks had been relatively stable. During the ARRA period, beyond subgrantees, did the composition of your state-level weatherization network change?
a. Yes
b. No (Skip to Question 8)
5. How would you characterize the magnitude of this change?
a. Extreme
b. Great
c. Moderate
d. Small
6. Using the table below, please list the names and contributions of up to ten major new players.
Name
Description of Weatherization Related Contributions (e.g.,
provides training, sells energy efficient products )
7. Please rate the quality of services and products provided by the new major players:
a. very high quality
b. high quality
c. fair quality
d. low quality
e. very low quality
8. Please provide your best estimate for how many new low-income weatherization-related
businesses were created in your state during the ARRA period:
a. none
b. 1-10
c. 11-20
d. 21-30
e. 30+
f. don‘t know
9. How have private companies new to the weatherization network tried to change state and local
weatherization procedures during the ARRA period? ___________________________________
150
10. On balance, how beneficial have these new players been to your state‘s weatherization
program?
a. Extremely beneficial
b. Beneficial
c. Neither beneficial nor unbeneficial
d. Unbeneficial
e. Extremely unbeneficial
Please Explain _____________________
11. During the ARRA period, did organizational responsibilities for your state‘s weatherization
program (e.g., reporting lines, organizational home) change?
a. Yes
b. No (Skip to Question 13)
12. How did the organizational responsibilities change? Please Explain _______________
13. How have relationships changed between your state weatherization office and local
weatherization agencies during ARRA?
a. become much more congenial
b. become more congenial
c. no changes
d. become more strained
e. become much more strained
14. Which statement best describes how the existing local weatherization agencies (i.e.,
subgrantees) dealt with program expansion during the ARRA period?
a. promoted existing weatherization staff to management positions and hired new entry level
workers
b. hired new weatherization managers and hired new entry level workers
c. promoted existing weatherization staff to management positions and hired contractors to ramp
up production
d. hired contractors to ramp up production
15. Were there any material, equipment or other supply chain bottlenecks during the ARRA
period that negatively impacted weatherization production in your state?
a. No
b. Yes
If yes, what was in short supply? ______________________
16. To what extent have large DOE/OWIP programs, WAP, State Energy Program, and Energy
Efficiency Community Block Grant Program competed for labor during the ARRA period?
a. Not at all
b. a small extent
c. a moderate extent
d. a large extent
e. a very large extent
151
17. During the ARRA period, how has state-level oversight of your state‘s weatherization
assistance program changed?
a. Greatly increased
b. Increased
c. Stayed the same
d. Decreased
e. Greatly decreased
18. Has ARRA funding allowed your state program to purchase and implement new information
technologies to improve program administration?
a. Yes
b. No
If yes, please describe these new information technologies. __________________________
19 During PY 2010, how much of a burden were your state program‘s administrative costs
allocated to dealing with increased oversight during the ARRA period (e.g., from DOE IG,
GAO)?
a. no burden
b. slight burden
c. moderate burden
d. substantial burden
e. extreme burden
20. Did your state change its audit approach during the ARRA-period from the approach used
pre-ARRA (i.e., PY2008)?
a. Yes, went from priority list to computerized audit
b. Yes, went from computerized audit to priority list
c. Yes, changed from one computerized audit tool to another
d. No
e. Other _________________________
21. During the ARRA period, has your state passed any new laws and/or implemented any new
regulations that have directly impacted your state‘s weatherization assistance program?
a. Yes
b. No (Skip to Question 25)
22. Please describe the new laws and/or regulations. __________________________
23. On balance, how beneficial have these new laws and/or regulations been to your state‘s
weatherization program?
a. Extremely beneficial
b. Beneficial
c. Neither beneficial nor unbeneficial
d. Unbeneficial
e. Extremely unbeneficial
Please explain ______________________________________
152
24. Would these laws and/or regulations have been passed without the increased attention on the
program attributable to ARRA?
a. No, Wholly due to ARRA
b. No, Primarily due to ARRA
c. No, Somewhat due to ARRA
d. Yes, Would have been passed anyway without ARRA
25. Did your state change the way it provides technical assistance to its subgrantees during the
ARRA period?
a. No
b. Yes
If yes, how? ____________________________________
26. Has your state implemented new certification requirements for weatherization workers
during the ARRA period?
a. No (go to Q28)
b. Yes
27. What new certifications are required? (Check all that apply)
a. BPI Building Analyst
b. BPI Envelope
c. BPI Residential Building Envelope Accessible Areas Air Leakage Control Installer
d. Residential Building Envelope Whole House Air Leakage Control Crew Chief
e. BPI Manufactured Housing
f. BPI Heating
g. BPI Air Conditioning and Heat Pump
h. BPI Multifamily
i. HERS
j. LEED
k. Lead Safe Weatherization
l. Lead Certified Renovator
m. NAHB Green Building
n. Other ___________________
28. Which statement best describes changes in the level of employment in your state resulting
from Davis-Bacon?
a. large number of jobs created
b. small number of jobs created
c. no change in the number of jobs
d. small number of jobs lost
e. large number of jobs lost
f. don‘t know
153
29. Which statement best describes the impact of Davis-Bacon on wages paid to weatherization
workers in your state?
a. large increase in wages (over 10% increase)
b. moderate increase in wages (between 5 and 10% increase)
c. small increase in wages (between 1 and 5% increase)
d. no increase in wages
e. small decrease in wages(between 1 and 5% decrease)
f. moderate decrease in wages(between 5 and 10% decrease)
g. large decrease in wages (over 10% decrease)
30. How much of a problem was there in your state in coordinating wages stipulated by DavisBacon for local weatherization agencies whose operations spanned multiple counties?
a. no problem
b. small problem
c. moderate problem
d. large problem
e. very large problem
31. In your state, on average, how much did Davis-Bacon rules increase the cost of weatherizing
low-income multifamily buildings four stories and higher?
a. no increase
b. 1-10%
c. 11-20%
d. 21-30%
e. 31+%
f. N/A
32. Overall, how did Davis-Bacon impact the costs of weatherization in your state?
a. greatly increased costs
b. increased costs
c. no change in costs
d. decreased costs
e. greatly decreased costs
33. Does your state allow the weatherization of large low-income multi-family buildings?
a. yes
b. no
If not, why not? _______________________________
154
34. What are the barriers to weatherizing large low-income multi-family buildings in your state?
(check all that apply)
a. lack of trained auditors
b. lack of trained crew
c. too expensive
d. building owners are uncooperative
e. energy savings are not high enough
f. unclear guidance from DOE or other agencies on owner contributions
g. other __________________
35. Does your state allow the weatherization of public housing, that is, housing owned by a
public housing authority?
a. Yes
b. No
If not, why not? _______________________________
36. Does your state allow the weatherization of HUD assisted housing?
a. Yes
b. No
If not, why not? _______________________________
37. How many units of each type were weatherized in your state in PY 2010?
Type of Housing
Large Multi-family
Public Housing Multi- HUD Assisted Multi(Not Public Housing or family
family
HUD Assisted
38. Are there any DOE rules that could be changed to make it easier to weatherize large lowincome multi-family buildings?
a. Yes. Describe:_____________________________________________________________
b. No
39. Does your state‘s weatherization program have an agreement with your State Historic
Preservation Office (SHPO)?
a. Yes
b. No
40. What percentage of units weatherized in PY 2010 fell under the SHPO agreement?
a. 0
b. 1-5
c. 6-10
d. 11-15
e. 15+
155
41. Has low-income weatherization had an impact in preventing home foreclosures in your state
during the ARRA period?
a. no
b. little impact
c. moderate impact
d. great impact
e. very great impact
f. don‘t know
42. Does your state have a deferral policy?
a. Yes
b. No
43. Please estimate the number of units where weatherization was deferred during PY 2010?
a. 0%
b. 1-5%
c. 6-10%
d. 10-20%
e. 21-30%
f. more than 30%
44. Please estimate what percentage of units where weatherization was deferred during PY 2010
will eventually be weatherized?
a. 0%
b. 1-25%
c. 26-50%
d. 51-75%
e. 76-90%
f. 91-100%
45. Please describe your states deferral rates in PY2010 compared to PY2008:
a. very great increase
b. great increase
c. increase
d. no change
e. decrease
f. great decrease
g. very great decrease
46. How has the visibility of your state‘s weatherization program changed vis-à-vis state elected
officials during the ARRA period?
a. greatly increased
b. increased
c. no change
d. decreased
e. greatly decreased
156
47. How has the visibility of your state‘s weatherization program changed vis-à-vis the general
public during the ARRA period?
a. greatly increased
b. increased
c. no change
d. decreased
e. greatly decreased
48. Please rate the quality of the media coverage your state‘s weatherization program has
received during the ARRA period?
a. very high quality
b. high quality
c. moderate quality
d. low quality
e. very low quality
f. N/A no media coverage (skip to 52)
g. no opinion
49. Please describe the overall media coverage of your state‘s weatherization program during the
ARRA period:
a. very positive
b. positive
c. neither positive or negative
d. negative
e. very negative
50. What topics did the media mainly focus on? (check all that apply)
a. Jobs created
b. Energy saved
c. Helping low income households
d. waste, fraud and abuse
e. Lack of energy savings
f. Organizational mis-steps
g. Other ______________________
51. Will weatherization issues identified rightly or wrongly by the media during ARRA have
lasting impacts on leveraging weatherization funding for your state in the future?
a. could substantially reduce funding
b. could reduce funding
c. no impact
d. could increase funding
e. could substantially increase funding
157
52. On balance, how beneficial has been the attention paid to your state‘s weatherization
program during the ARRA period?
a. Extremely beneficial
b. Beneficial
c. Neither beneficial nor unbeneficial
d. Unbeneficial
e. Extremely unbeneficial
53. Please explain ____________________________
TRAINING
1. For those staff working in your state‘s weatherization office who need to have knowledge
about the following list of weatherization topics, how well trained were they in each area in PY
2010? Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 –
moderately well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best
answer.
(1) Diagnostic procedures
(2) Insulation
-- single family dwellings
-- multifamily dwellings
-- mobile homes
(3) Space heating, ventilation, air conditioning
-- single family dwellings
-- multifamily dwellings
-- mobile homes
(4) Infiltration measures
-- single family dwellings
-- multifamily dwellings
-- mobile homes
(5) Doors and windows
-- single family dwellings
-- multifamily dwellings
-- mobile homes
(6) Hot water heating
-- single family dwellings
-- multifamily dwellings
-- mobile homes
(7) Baseloads (e.g., lighting, refrigerators)
-- single family dwellings
-- multifamily dwellings
-- mobile homes
1
2
3
4
5
6
1
1
1
2
2
2
3
3
3
4
4
4
5
5
5
6
6
6
1
1
1
2
2
2
3
3
3
4
4
4
5
5
5
6
6
6
1
1
1
2
2
2
3
3
3
4
4
4
5
5
5
6
6
6
1
1
1
2
2
2
3
3
3
4
4
4
5
5
5
6
6
6
1
1
1
2
2
2
3
3
3
4
4
4
5
5
5
6
6
6
1
1
1
2
2
2
3
3
3
4
4
4
5
5
5
6
6
6
158
1a. For those staff working in your state‘s weatherization office who need to have knowledge
about the following list of administrative-related topics, how well trained were they in each area?
Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 – moderately
well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best answer.
(1) Management
(2) Client education
(3) Auditing/estimating
-- single family dwellings
-- multifamily dwellings
-- mobile homes
(4) Monitoring/quality control
(5) Financial topics
(6) Outreach and communications
(7) Other (please specify)
1
1
2
2
3
3
4
4
5
5
6
6
1
1
1
1
1
1
1
2
2
2
2
2
2
2
3
3
3
3
3
3
3
4
4
4
4
4
4
4
5
5
5
5
5
5
5
6
6
6
6
6
6
6
1b. For those staff working in your state‘s weatherization office who need to have knowledge
about the following list of health and safety topics, how well trained were they in each area?
Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 – moderately
well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best answer.
(1) Fire safety
(2) Indoor air quality
(3) Measures to increase security of housing unit
(4) Measures to reduce common household hazards
(5) Mold and mildew
(6) Lead
(7) Asbestos
(8) Vermiculite
(9) General crew safety
(10) Other health and safety
(11) Other (please specify
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
1c. For categories receiving answers of (1)-not at all well trained, or (2)-not well trained to the
above questions, what were the barriers for receiving this training:
a. Funding
b. Time
c. Not a priority
d. Not available
e. Other___________________
159
2. On which of the following weatherization subjects did staff working in your state‘s
weatherization office receive training in Program Year 2010 from DOE, your state, or other
entities? (Check all that apply)
(1) Diagnostic procedures
_____
(2) Insulation
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(3) Space heating, ventilation, air conditioning
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Infiltration measures
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(5) Doors and windows
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(6) Hot water heating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(7) Baseloads (e.g., lighting, refrigerators)
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
2a. On which administration-related topics did staff working in your state‘s weatherization office
receive training in Program Year 2010 from DOE, your state, or other entities? (Check all that
apply)
(1) Management
_____
(2) Client education
_____
(3) Auditing/estimating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Monitoring/quality control
_____
(5) Financial topics
_____
(6) Outreach and communications
_____
(7) Other (please specify)
_____
160
2b. On which health and safety topics did staff working in your state‘s weatherization office
receive training in Program Year 2010 from DOE, your state, or other entities? (Check all that
apply.)
_____ Fire safety
_____ Indoor air quality
_____ Measures to increase security of housing unit
_____ Measures to reduce common household hazards
_____ Mold and mildew
_____ Lead
_____ Asbestos
_____ Vermiculite
_____ General crew safety
_____ Other health and safety
_____ Other (please specify
161
3. For those staff working in your state‘s weatherization office who need to have knowledge
about the following list of diagnostic topics, how well trained were they in each area in PY
2010? Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 –
moderately well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best
answer.
Pressure diagnostics:
Blower door (house air leakage rate)
1
Zonal pressure measurements
1
Room-to-room pressure measurements
1
Duct pressure pan measurements
1
Duct blower measurements (duct air leakage rate)
1
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)1
Heat rise measurements
1
CO measurements in flues
1
Draft/spillage (normal operation)
1
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
1
HVAC components and cross-cutting diagnostics:
Air handler flow rate
1
Thermostat anticipator current
1
Worst case draft/spillage (CAZ)
1
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)1
CO measurements in flues
1
Draft/spillage (normal operation)
1
Water flow rates (showerheads and faucets)
1
Other CO measurements:
CO measurements in equipment rooms
1
Cooking stove
1
CO measurements in living areas
1
Other diagnostics and inspections:
Refrigerator energy use
1
Exhaust fan air flow rate measurement
1
Infrared scanning (camera)
1
Radon testing
1
Lead testing
1
Mold and mildew testing
1
Moisture context testing
1
Other (please specify) ____________________
1
2
2
2
2
2
3
3
3
3
3
4
4
4
4
4
5
5
5
5
5
6
6
6
6
6
2
2
2
2
3
3
3
3
4
4
4
4
5
5
5
5
6
6
6
6
2
3
4
5 6
2
2
2
3
3
3
4
4
4
5 6
5 6
5 6
2
2
2
2
3
3
3
3
4
4
4
4
5
5
5
5
2
2
2
3
3
3
4
4
4
5 6
5 6
5 6
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
162
3a. For categories receiving answers of (1)-not at all well trained, or (2)-not well trained to the
above question, what were the barriers for receiving this training:
a. Funding
b. Time
c. Not a priority
d. Not available
e. Other___________________
4. On which of the following diagnostic procedures did staff working in your state‘s
weatherization office receive training in Program Year 2010 from DOE, your state, or other
entities? (Check all that apply.)
Pressure diagnostics:
Blower door (house air leakage rate)
_____
Zonal pressure measurements
_____
Room-to-room pressure measurements (distribution balancing)
_____
Duct pressure pan measurements
_____
Duct blower measurements (duct air leakage rate)
_____
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)
_____
Heat rise measurements
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
_____
HVAC components and cross-cutting diagnostics:
Air handler flow rate
_____
Thermostat anticipator current
_____
Worst case draft/spillage (CAZ)
_____
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Water flow rates (showerheads and faucets)
_____
Other CO measurements:
CO measurements in equipment rooms
_____
Cooking stove
_____
CO measurements in living areas
_____
Other diagnostics and inspections:
Refrigerator energy use
_____
Exhaust fan air flow rate measurement
_____
Infrared scanning (camera)
_____
Radon testing
_____
Lead testing
_____
Mold and mildew testing
_____
Moisture context testing
_____
163
Other (please specify) ___________________________________
_____
164
5. For each broad subject listed in the left-most column of the following table, put a check mark
in the appropriate cell(s) to indicate which training method(s) you believe were most effective
for imparting key skills and information in that area to your state‘s in-house staff and any nonagency staff supporting the state program who work under contract to the state in PY 2010:
Subject
Management
Weatherization skills
and methods
Auditing/Estimating/
Measure selection
Monitoring and
quality control
Financial topics
Outreach and
communications
Health and safety
Diagnostic
procedures
Client education
Other (specify)
Conferences
Primarily
Classroom
Training
Primarily
Field
Training
In-person
expert
visits
Web casts
Other (specify)
165
6. For each broad subject listed in the left-most column of the following table, please indicate the
quality of training received in Program Year 2010 at the training venues listed in the column
headings. Please leave cells blank were your in-house staff did not receive training during this
period of time. Please use the following scale: 1-very low; 2 - low; 3-medium; 4- high; 5-very
high
National
Weatherization
Program
Conference
Affordable
Comfort
Conference
Regional
Weatherization
Conference
State
Weatherization
Conference
State/
Regional
Training
Center
Training
Provided by
Your Own
State
Subject
Management
Weatherization skills
and methods
Auditing/
Estimating
Monitoring/
quality control
Financial topics
Outreach and
communications
Health and safety
Diagnostic
procedures
Procedures for
selecting
weatherization
measures
Client education
Other (specify)
166
7. On which of the following weatherization topics did your state provide training to your state‘s
local weatherization agencies or their contractors in Program Year 2010? (Check all that apply)
(1) Diagnostic procedures
_____
(2) Insulation
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(3) Space heating, ventilation, air conditioning
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Infiltration measures
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(5) Doors and windows
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(6) Hot water heating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(7) Baseloads (e.g., lighting, refrigerators)
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
7a. On which of the following administrative-related topics did your state provide training to
your state‘s local weatherization agencies or their contractors in Program Year 2010? (Check all
that apply)
(1) Management
_____
(2) Client education
_____
(3) Auditing/estimating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Monitoring/quality control
_____
(5) Financial topics
_____
(6) Outreach and communications
_____
(7) Other (please specify)
_____
167
7b. On which of the following health and safety topics did your state provide training to your
state‘s local weatherization agencies or their contractors in Program Year 2010? (Check all that
apply)
_____ Fire safety
_____ Indoor air quality
_____ Measures to increase security of housing unit
_____ Measures to reduce common household hazards
_____ Mold and mildew
_____ Lead
_____ Asbestos
_____ Vermiculite
_____ General crew safety
_____ Other health and safety
_____ Other (please specify)
168
8. On which of the following diagnostic procedures did your state provide training to your state‘s
local weatherization agencies or their contractors in Program Year 2010? (Check all that apply)
Pressure diagnostics:
Blower door (house air leakage rate)
_____
Zonal pressure measurements
_____
Room-to-room pressure measurements (distribution balancing)
_____
Duct pressure pan measurements
_____
Duct blower measurements (duct air leakage rate)
_____
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)
_____
Heat rise measurements
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
_____
HVAC components and cross-cutting diagnostics:
Air handler flow rate
_____
Thermostat anticipator current
_____
Worst case draft/spillage (CAZ)
_____
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Water flow rates (showerheads and faucets)
_____
Other CO measurements:
CO measurements in equipment rooms
_____
Cooking stove
_____
CO measurements in living areas
_____
Other diagnostics and inspections:
Refrigerator energy use
_____
Exhaust fan air flow rate measurement
_____
Infrared scanning (camera)
_____
Radon testing
_____
Lead testing
_____
Mold and mildew testing
_____
Moisture context testing
_____
Other (please specify) ___________________________________
_____
169
9. Which of the following types of personnel did your state use to provide training to your state‘s
local weatherization agencies or their contractors in Program Year 2010? (Check all that apply)
DOE staff
_____
DOE contractor
_____
State staff
_____
State contractor
_____
Staff from another state
_____
State training center staff
_____
Local agency staff from your state
_____
Agency staff from another state
_____
Manufacturer representative
_____
Utility staff
_____
Representative from trade organization
_____
Consultant
_____
Other (please specify) _____________________________________ _____
10. What types of credentials or experience were required of the personnel your state used to
provide training to your state‘s local weatherization agencies or their contractors in Program
Year 2010? (Check all that apply)
Technical certification
_____
Extensive weatherization field experience
_____
Construction experience
_____
Extensive management experience
_____
Extensive experience with financial matters
_____
Other (please specify) _____________________________________ _____
10a. Using the scale below, please indicate how important each credential was for trainers to
have in PY 2010?
1= Very Unimportant; 2=Unimportant; 3= Important; 4=Very Important
Technical certification
Extensive weatherization field experience
Construction experience
Extensive management experience
Extensive experience with financial matters
Other (please specify) _____________________________________
_____
_____
_____
_____
_____
_____
170
11. How many of your state‘s weatherization office staff acted as instructors at the following
training events that your state provided (e.g., funded, organized) to your state‘s local
weatherization agencies or their contractors in Program Year 2010?
State weatherization conference
_____
Other state conference
_____
State/regional training center class
_____
State-sponsored class taught at central location
_____
In-person expert visit (e.g., peer exchange, consultant)
_____
Instruction given to individual agency during an agency visit
_____
Web cast
_____
Other (please specify) _____________________________________ _____
12. For each broad subject listed in the left-most column of the following table, put a check mark
in the appropriate cell(s) to indicate which training method(s) you believe were most effective
for imparting key skills and information in that area to your local weatherization agencies or
their contractors in PY 2010:
Subject
Management
Weatherization skills
and methods
Auditing/Estimating/
Measure selection
Monitoring and
quality control
Financial topics
Outreach and
communications
Health and safety
Diagnostic procedures
Client education
Other (specify)
State
weatheriza
tion
conference
Other state
conference
State/
regional
training
center
class
Statesponsored
class
taught at
central
location
Inperson
expert
visit
Instruction
given to
individual
agency
Web
casts
Other
(specify)
171
13. On average, how well trained were local weatherization crews (both agency and contractor)
in your state in the following weatherization topics in PY 2010? Please use the following scale:
1– not at all well trained; 2 – not well trained; 3 – moderately well trained; 4 –well trained; 5 –
very well trained; 6 – not applicable Circle best answer.
(1) Diagnostic procedures
1
2
3
4
5 6
(2) Insulation
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(3) Space heating, ventilation, air conditioning
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(4) Infiltration measures
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(5) Doors and windows
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(6) Hot water heating
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(7) Baseloads (e.g., lighting, refrigerators)
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
13a. On average, how well trained were local weatherization crews (both agency and contractor)
in your state in the following administrative-related topics in PY 2010? Please use the following
scale: 1– not at all well trained; 2 – not well trained; 3 – moderately well trained; 4 –well trained;
5 – very well trained; 6 – not applicable Circle best answer.
(1) Management
1
2
3
4
5 6
(2) Client education
1
2
3
4
5 6
(3) Auditing/estimating
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(4) Monitoring/quality control
1
2
3
4
5 6
(5) Financial topics
1
2
3
4
5 6
(6) Outreach and communications
1
2
3
4
5 6
(7) Other (please specify)
1
2
3
4
5 6
172
13b. On average, how well trained were local weatherization crews (both agency and contractor)
in your state in the following health and safety topics in PY 2010? Please use the following
scale: 1– not at all well trained; 2 – not well trained; 3 – moderately well trained; 4 –well trained;
5 – very well trained; 6 – not applicable Circle best answer.
(1) Fire safety
(2) Indoor air quality
(3) Measures to increase security of housing unit
(4) Measures to reduce common household hazards
(5) Mold and mildew
(6) Lead
(7) Asbestos
(8) Vermiculite
(9) General crew safety
(10) Other health and safety
(11) Other (please specify
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
13c. For categories receiving answers of (1)-not at all well trained, or (2)-not well trained to the
above question, what were the barriers for receiving this training:
a. Funding
b. Time
c. Not a priority
d. Not available
e. Other___________________
14. Overall, how well trained were your state‘s weatherization crews in PY 2010? (Check best
answer)
_____ Very well trained
_____ Well trained
_____ Moderately well trained
_____ Poorly trained
_____ Very poorly trained
MONITORING
1. About how many state weatherization office staff went into the field to monitor local
weatherization agencies in your state in Program Year 2010? {Note: do not include people
who do quality assurance at the local agency level for the local agencies.}
State staff
_____
State contractors
_____
Other (please specify) _____________________________________ _____
173
2. Which of the following types of post-weatherization quality control inspection did your state
perform on weatherized dwelling units in Program Year 2010? (Check all that apply)
Visual inspection of installed measures
_____
Verification of insulation depths/quantities
_____
Verification of operation of measures installed
_____
Assessment of quality of measures installed
_____
Identification of needed measures that were not installed
_____
Blower door test
_____
Heating system efficiency test (flue gas analysis)
_____
Draft/spillage tests of heating systems
_____
Carbon monoxide (CO) monitoring
_____
Infrared scanning
_____
Identification of unresolved health and safety issues
_____
Discussion with occupants
_____
Other (specify)________________________________________
_____
3. Please indicate which types of post-weatherization quality control inspection listed below
were initiated since ARRA PY 2009. (Check all that apply)
Visual inspection of installed measures
_____
Verification of insulation depths/quantities
_____
Verification of operation of measures installed
_____
Assessment of quality of measures installed
_____
Identification of needed measures that were not installed
_____
Blower door test
_____
Heating system efficiency test (flue gas analysis)
_____
Draft/spillage tests of heating systems
_____
Carbon monoxide (CO) monitoring
_____
Infrared scanning
_____
Identification of unresolved health and safety issues
_____
Discussion with occupants
_____
Other (specify)________________________________________
_____
174
4. Please indicate the cost, the amount of training needed, the amount of time needed and the
effectiveness of the following types of post-weatherization quality control inspection
procedures relative to each other for PY 2010. Please use the following scale: 1 – very low; 2
– low; 3 – medium; 4 – high; 5 – very high.
Type of Post-Weatherization Quality
Control Inspection
Visual inspection of installed measures
Verification of insulation depths/quantities
Verification of operation of measures
installed
Assessment of quality of measures installed
Identification of needed measures that were
not installed
Blower door test
Heating system efficiency test (flue gas
analysis)
Draft/spillage tests of heating systems
Carbon monoxide (CO) monitoring
Infrared scanning
Identification of unresolved health and
safety issues
Discussion with occupants
Other (specify)
Cost
Training
Needed
Time
Needed
Effectiveness
5. On average, how many hours were spent by state weatherization office staff on-site
conducting post-weatherization quality control in a typical weatherized home in Program
Year 2010? __________
6. What types of credentials or experience were required of your post-weatherization quality
control inspectors in your state weatherization office in Program Year 2010? (Check all that
apply)
Technical certification
_____
Extensive experience performing pre-weatherization audits
_____
Extensive experience performing weatherization work
_____
Extensive experience supervising weatherization work
_____
Construction experience
_____
Other (please specify) _____________________________________ _____
175
7. Please indicate their level of experience of the post-weatherization quality control inspectors
in your state weatherization office for each of the following areas in Program Year 2010.
Very
High
High
Medium
Low
Very
Low
Performing pre-weatherization audits
Performing weatherization work
Supervising weatherization work
Working in construction
Performing post-weatherization
inspections
Other (specify)
8. On average, how frequently did state weatherization program office staff visit each local
agency to conduct post-weatherization quality control inspections in Program Year 2010?
(Check best answer)
_____ Weekly
_____ Monthly
_____ Quarterly
_____ Annually
_____ Other (please specify) _________________
9. On how many dwelling units did your state perform post-weatherization quality control
inspections in Program Year 2010? _____
9a. Of those inspected, approximately how many were found to have a problem significant
enough to require a return visit by local agency weatherization crews? _____
9b. Of those requiring a return visit, how many had work done that probably resulted in more
energy savings? _____
9c. What were the three most common problems found in the dwelling units inspected by
your state in Program Year 2010?
___________________________________________________________________________
___________________________________________________________________________
__________________________________________________________________
10. In those cases where a Program Year 2010 post-weatherization quality control inspection
revealed a problem with the job performed, what single action was most commonly taken in
response to that finding? (Check best answer)
Made agency send crew back to correct problem
_____
Made agency send crew supervisor to correct problem
_____
Sent someone from state office to correct problem
_____
No action taken
_____
Other (please specify) ______________________________________ _____
176
11. What other actions were taken in Program Year 2010 in response to the discovery of a
problem with the weatherization job performed? (Check all that apply)
Made agency send original crew back to correct problem
_____
Made agency send different crew to correct problem
_____
Made agency send crew supervisor to correct problem
_____
Sent someone from state office to correct problem
_____
No action taken
_____
Other (please specify) ______________________________________ _____
12. How often did the inspection reveal a misuse of the priority list?
a. Never
b. 1-5% of the time
c. 6-10% of the time
d. 11-20% of the time
e. More than 20% of the time
f. N/A
13. How often did the inspection reveal a misuse of the computer audit?
a. Never
b. 1-5% of the time
c. 6-10% of the time
d. 11-20% of the time
e. More than 20% of the time
f. N/A
14. Which of the following monitoring tasks did your state perform in Program Year 2010 to
check on the administration of local weatherization efforts? (Check all that apply)
Verification of number of dwelling units weatherized
_____
Verification of clients‘ income eligibility
_____
Verification of average expenditure per weatherized unit
_____
Verification of material expenditures
_____
Verification that installed measures had an SIR of 1.0 or greater
_____
Examination of vehicle costs
_____
Examination of other equipment costs
_____
Examination of training and technical assistance (T&TA) costs
_____
Examination of administrative costs
_____
Examination of material inventory
_____
Interviews with agency staff
_____
Interviews with agency contractor staff
_____
Interviews with agency clients
_____
Other (please specify) ___________________________________
_____
177
15. Please indicate which types of monitoring tasks listed below were initiated since ARRA to
PY 2009. (Check all that apply)
Verification of number of dwelling units weatherized
_____
Verification of clients‘ income eligibility
_____
Verification of average expenditure per weatherized unit
_____
Verification of material expenditures
_____
Verification that installed measures had an SIR of 1.0 or greater
_____
Examination of vehicle costs
_____
Examination of other equipment costs
_____
Examination of training and technical assistance (T&TA) costs
_____
Examination of administrative costs
_____
Examination of material inventory
_____
Interviews with agency staff
_____
Interviews with agency contractor staff
_____
Interviews with agency clients
_____
Other (please specify) ___________________________________
_____
178
16. Please indicate the cost, the amount of training needed, the amount of time needed and the
effectiveness of the following types of monitoring tasks relative to each other for PY 2010.
Please use the following scale: 1 – very low; 2 -- low; 3 – moderate; 4 –high; 5 – very high.
Type of Monitoring Tasks
Cost
Training
Needed
Time
Needed
Effectiveness
Verification of number of dwelling units
weatherized
Verification of clients‘ income eligibility
Verification of average expenditure per
weatherized unit
Verification of material expenditures
Verification that installed measures had an
SIR of 1.0 or greater
Examination of vehicle costs
Examination of other equipment costs
Examination of training and technical
assistance (T&TA) costs
Examination of administrative costs
Examination of material inventory
Interviews with agency staff
Interviews with agency contractor staff
Interviews with agency clients
Other (specify)
17. On average, how many hours were spent by state weatherization office staff on-site at each
local agency monitoring agency administrative activities in Program Year 2010? __________
18. What types of credentials or experience were required of those who monitored the
administration of local weatherization efforts in your state in Program Year 2010? Check all that
apply.
Technical certification
_____
Extensive experience performing pre-weatherization audits
_____
Extensive experience performing weatherization work
_____
Extensive experience supervising weatherization work
_____
Construction experience
_____
Extensive management experience
_____
Extensive finance experience
_____
Extensive experience administering local weatherization programs
_____
Other (please specify) _____________________________________ _____
179
19. Please indicate their level of experience of the local agency monitors in your state
weatherization office for each of the following areas in Program Year 2010.
Very
High
High
Moderate
Low
Very
Low
Management
Finance
Administration of local weatherization
programs
Other (specify)
20. On average, how frequently did state weatherization program office staff visit each local
agency to monitor administrative activities in Program Year 2010? (Check best answer)
_____ Weekly
_____ Monthly
_____ Quarterly
_____ Annually
_____ Other (please specify) _________________
21. For how many of the local weatherization agencies monitored in your state in Program Year
2010 was an administrative problem found that required corrective actions above and beyond
acceptable findings and recommendations? _____
22. What were the three most common problems requiring corrective actions above and beyond
acceptable findings and recommendations found in the local weatherization agencies monitored
in your state in Program Year 2010?
______________________________________________________________________________
______________________________________________________________________________
____________________________________________________________
23. In those cases where state monitoring of the administration of local weatherization efforts in
Program Year 2010 revealed an administrative problem requiring corrective actions above and
beyond acceptable findings and recommendations, what actions were taken in response? (Check
all that apply)
Sent written report to local agency
_____
Required corrective action
_____
Made presentation to local agency
_____
Sent someone from state office to help correct problem
_____
Sent state contractor to help correct problem
_____
No action taken
_____
Other (please specify) ______________________________________ _____
180
24 Did the observation of problems with the quality of weatherization work lead to changes in
weatherization training for local agency staff?
_____ Yes
_____ No
24a. If Yes, what changes were made? ______________
25. Does your state observe weatherization training sessions to help identify potential problem
areas for monitoring in the field (e.g., with respect to installation of measures that trainees seem
to have trouble understanding)?
_____ Yes
_____ No
25a. If Yes, briefly describe how your in-field monitoring activities were affected by your
training session observations.
____________________________________________________________
_____________________________________________________________________________
181
OMB Control Number: XXXX-XXXX
APPENDIX D -- S2: ALL AGENCIES PROGRAM INFORMATION SURVEY
This data is being collected to conduct a process evaluation of the Weatherization Assistance
Program at the local level. The data you supply will be used to characterize the program in
Program Year 2010, unless otherwise noted.
Public reporting burden for this collection of information is estimated to average eight hours per
response, including the time for reviewing instructions, searching existing data sources,
gathering and maintaining the data needed, and completing and reviewing the collection of
information. Send comments regarding this burden estimate or any other aspect of this
collection of information, including suggestions for reducing this burden, to Office of the Chief
Information Officer, Records Management Division, IM-11, Paperwork Reduction Project
(XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW, Washington, DC,
20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork Reduction
Project (XXXX-XXXX), Washington, DC 20503.
All of the information obtained from this survey will be protected and will remain confidential.
The data will be analyzed in such a way that the information provided cannot be associated back
to your state, your agencies, or the housing units and clients that your state served. Again, please
note that the questions refer to PY 2010 unless otherwise noted.
Part 1. General Information
1. Please identify your state. ________________________________________
2. Please identify your local agency. ________________________________________
3. Which of the following best characterizes your agency? (Please check the one answer that best
applies):
_____ Local Non-Profit Organization
_____ Local Government Agency
_____ County Government Agency
_____ Indian Tribe
_____ Other entity not eligible for CSBG funding
_____ Other (please specify) ____________________________
4. For how many years had the current director of your local Weatherization Program served in
that capacity prior to PY 2010? _____
182
5. What agency, office, or department was responsible for reviewing the performance of your
local Weatherization Program in PY 2010? _____
6. How many layers of management or supervision were there between your weatherization
crews and the director of your local Weatherization Program in PY 2010? _____ [If your
weatherization crews reported directly to the Program director, the answer should be 0.]
7. Please indicate other energy-related, housing, and other programs that cooperated with your
agency‘s weatherization program, by source of funding in PY 2010. Please check all that apply.
Type of Program
Federal
Funding
State
Funding
Utility
Funding
Other
Funding
Energy bill paying assistance
Housing re-habilitation
Home emergency repairs
Hardship funds (other than
for energy bill paying)
Fuel delivery in crisis
Fair housing
Health and safety
Energy education (other than
client education delivered by
weatherization program)
Home buying education
Rehabilitation loan
Mortgage loan
Emergency food
Emergency safety
Other (please specify)
Other (please specify)
Other (please specify)
Other (please specify)
Other (please specify)
Other (please specify)
183
Part 2. Leveraging
1. Please list weatherization funding received during PY 2010 by completing the table below.
Column A lists potential sources of weatherization funding. As a reference, Column B lists the amount of PY 2008
funding reported by your agency in the retrospective evaluation S2 survey. If the PY 2008 funding amount is
incorrect, please list the correct amount in Column B. In Column C, please indicate whether your agency received
funding from each source in PY 2010. In Column D, enter the total funding amount received in PY 2010 from each
source received. Please allocate the total funding amount listed in Column D to the sub-categories listed in Columns
E through G.
A
B
Funding source
Funds
received from
this source in
PY 2008
DOE1
LIHEAP
Petroleum
Violation Escrow
(PVE)
Other Federal
Programs
State Public
Benefit Funds
Other State
Programs
Utilities
Program Income
In-Kind
Non-Profits
Third Party
(e.g.,
Foundations,
Lenders)
All other
(Please specify)
______________
TOTAL
3.
4.
Prepopulated
Prepopulated
Prepopulated
C
Did agency
receive
weatherizati
on funds
from this
source in PY
2010?
Yes
No
Yes
No
Yes
No
Prepopulated
Prepopulated
Prepopulated
Prepopulated
Prepopulated
Yes
No
Yes
No
Yes
No
Yes
No
Yes
No
Prepopulated
Prepopulated
Prepopulated
Yes
No
Yes
No
Yes
No
Prepopulated
Yes
No
D
E
F
G
Total funds
received from
this source in
PY 2010
Funding for
program
operations2
Funding for
administration/
program
management
Funding for
training and
technical
assistance
(T&TA)
Selecting
no will gray
columns to
right
Selecting
yes will drive
skip patterns
in Q12-23
Prepopulated
Include WAP and ARRA funds in this row.
List all funding for weatherization program operations that was passed on to subgrantees, including amounts spent
on measures, labor, health and safety, financial audits, liability insurance, vehicles and equipment.
184
2. Of the Program Year 2010 funds retained by your Agency‘s Weatherization Program for
Program Management (see Column F total in Question 1), how much was used for each function
listed below?
Type of Management Function
Total
Administration*
Agency monitoring
Other (specify)
TOTAL
* Includes planning, finance and accounting, clerical support, outreach, and evaluation.
3. Did your agency classify its expenditures for client intake, audits, and post-weatherization
inspections as program management costs or as allowable costs used in the calculation of
average cost per unit in PY 2010? Please indicate your answer for each type of expenditure by
checking the appropriate cell in the table below.
Type of Expenditure
Expenditures for Client
Intake
Expenditures for Audits
Classified as Program
Management Costs?
Classified as Allowable Costs for
Purpose of Calculating Average
Cost per Unit?
Expenditures for PostWeatherization Quality
Control Inspections
4. Of the TOTAL amount spent by your agency in Program Year 2010 using funds from all
sources (see Column D total in Question 1), please give your best estimate of how much was
spent on Audits and Inspections.
$________
185
5. Please divide your agency‘s Program Year 2010 expenditures on program operations (see
Column E total in Question 1) into in-house expenditures and contractor expenditures, as shown
in the following table.
Type of Expenditure
PY 2010
PY 2010
PY 2010
In-house
Contractor
Total
Expenditur Expenditures Expenditures
es (in $)
(in $)
on program
operations
(in $)
Expenditures for Health and Safety, Repairs, and
Other Non Cost-Effective Measures
All Expenditures Defined as Allowable Costs for
Purpose of Calculating Average Cost per Unit
TOTAL FUNDS
6. Of your agency‘s total Program Year 2010 expenditures on program operations (see Column E
total in Question 1), please give your best estimate of how much was for labor, how much for
materials, and how much for other expenses.
Type of Expenditure
PY 2010 Expenditures on program
operations (in $)
Labor
Materials
Other Expenses (e.g., costs for vehicles,
transportation, maintenance, and storage)
TOTAL FUNDS
186
7. Please provide the following information about ALL low-income dwelling units weatherized
by your agency in Program Year 2010.
Number of Units Weatherized in PY 2010
Non-DOE Units2
Type of Unit Weatherized
Single Family Attached and
Detached
Small Multi-family (2-4
units)
Multifamily (5 or More
Units per Building)
Mobile Home
Shelter
TOTAL UNITS
DOE
Units1
Comprehensive
Weatherization3
Non-comprehensive
Weatherization4
TOTAL
Units
1
These are dwelling units that your agency weatherized and reported to the State as ―DOE Units‖
These are dwelling units that your agency weatherized but did not report as ―DOE Units‖
3
Comprehensive weatherization units are those for which an audit or priority list was used that addressed a large
proportion of potential energy-saving measures.
4
Non-comprehensive weatherization units are those for which a limited set of measures was considered (e.g.,
baseload electric measures only; low cost/no cost measures only), reflecting the needs and priorities of the funding
entity.
2
8. Of all the DOE units weatherized by your agency in Program Year 2010, how many used each
of the following as their main heating fuel (i.e., the fuel providing most of the heat for the
dwelling unit) in the winter prior to weatherization?
Natural gas
_____
Fuel oil
_____
Electricity
_____
Propane/LPG
_____
Kerosene or coal oil
_____
Wood
_____
Other (please specify) _________________________________________ _____
9. Of all the DOE units weatherized by your agency in Program Year 2010, how many housed
members of the following high-priority client populations (leave blank if do not know)?
Children (according to your state‘s definition of that term)
_____
Elderly (age 60 and older)
_____
Disabled
_____
Native American
_____
10. Did your state have official definitions of ―high energy expenditure‖ or ―high energy burden‖
in PY 2010?
Yes
____
No
____
187
11. Of all the DOE units weatherized by your agency in Program Year 2010, how many met your
state‘s definition of having ―high energy expenditures‖ __________ and ―high energy
burden‖___________? (leave blank if do not know or if state did not have that definition)
[Q12 will be activated by ‗yes‘ responses in the associated funding categories of Question 1
and skipped for any ‗no‘ responses in those categories].
12. In the categories below, please specify the name of each funder and the total amount of
weatherization funding received in PY 2010.
Funding entity
(please enter names)
Total weatherization funding
administered in PY 2010
(please enter amounts)
Utilities
Utility 1
Utility 2
Utility 3
All other utilities
In-kind contributions
In-kind 1
In-kind 2
In-kind 3
All other in-kind
Nonprofits
Nonprofit 1
Nonprofit 2
Nonprofit 3
All other nonprofits
Other
Other 1
Other 2
Other 3
All others
188
[Q13 will be activated by ‗yes‘ responses in the LIHEAP category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
13a. When did agency first receive weatherization funds from LIHEAP? _________
13b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
13c. Please describe the change in the leveraging relationship between your agency‘s lowincome weatherization program and LIHEAP during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely negative
13d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
13e. If leveraged funding from LIHEAP decreased from PY 2008, was the change attributable to
increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
13f. Did you require local agencies to follow DOE rules when spending weatherization funds
from LIHEAP in PY 2010?
_____ Yes
_____ No
[13g will appear if respondent answered ‗no‘ in 13f]
13g. What were the major differences in the rules governing the expenditure of LIHEAP
funds in comparison to the rules governing the expenditure of DOE funds?____________
________________________________________________________________________
13h. Did your agency experience any delays or other difficulties in spending funds from
LIHEAP in PY 2010?
_____ Yes
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
189
[Q14 will be activated by ‗yes‘ responses in the PVE category of Question 1 and deactivated
(skipped) for ‗no‘ responses in that category.]
14a. When did agency first receive weatherization funds from Petroleum Violation Escrow
(PVE)? _________
14b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
14c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and PVE during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
14d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
14e. If leveraged funding from PVE decreased from PY 2008, was the change attributable to
increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
14f. Did you require local agencies to follow DOE rules when spending weatherization funds
from PVE in PY 2010?
_____ Yes
_____ No
[14g will appear if respondent answered ‗no‘ in 14f]
14g. What were the major differences in the rules governing the expenditure of PVE
funds in comparison to the rules governing the expenditure of DOE funds?
_____________________________________________________________
________________________________________________________________________
14h. Did your agency experience any delays or other difficulties in spending funds from PVE in
PY 2010?
_____ Yes
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
190
[Q15 will be activated by ‗yes‘ responses in the Other Federal Programs category of
Question 1 and deactivated (skipped) for ‗no‘ responses in that category.]
15a. When did agency first receive weatherization funds from Other Federal Programs?
_________
15b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
15c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and Other Federal Programs during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
15d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
15e. If leveraged funding from Other Federal Programs decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
15f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Other Federal Programs in PY 2010?
_____ Yes
_____ No
[15g will appear if respondent answered ‗no‘ in 15f]
15g. What were the major differences in the rules governing the expenditure of Other
Federal Programs funds in comparison to the rules governing the expenditure of DOE
funds? __________________________________________________________________
________________________________________________________________________
15h. Did your agency experience any delays or other difficulties in spending funds from Other
Federal Programs in PY 2010?
_____ Yes
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
191
_______________________________________________________________________
[Q16 will be activated by ‗yes‘ responses in the State Public Benefit Funds category of
Question 1 and deactivated (skipped) for ‗no‘ responses in that category.]
16a. When did agency first receive weatherization funds from State Public Benefit Funds?
_________
16b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
16c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and State Public Benefit Funds during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
16d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
16e. If leveraged funding from State Public Benefit Funds decreased from PY 2008, was the
change attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
16f. Did you require local agencies to follow DOE rules when spending weatherization funds
from State Public Benefit Funds in PY 2010?
_____ Yes
_____ No
[16g will appear if respondent answered ‗no‘ in 16f]
16g. What were the major differences in the rules governing the expenditure of State
Public Benefit Funds in comparison to the rules governing the expenditure of DOE
funds? __________________________________________________________________
________________________________________________________________________
16h. Did your agency experience any delays or other difficulties in spending funds from State
Public Benefit Funds in PY 2010?
_____ Yes
_____ No
192
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
[Q17 will be activated by ‗yes‘ responses in the Other State Programs category of Question
1 and deactivated (skipped) for ‗no‘ responses in that category.]
17a. When did agency first receive weatherization funds from Other State Programs? _________
17b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
17c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and Other State Programs during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
17d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
17e. If leveraged funding from Other State Programs decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
17f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Other State Programs in PY 2010?
_____ Yes
_____ No
[17g will appear if respondent answered ‗no‘ in 17f]
17g. What were the major differences in the rules governing the expenditure of Other
State Programs funds in comparison to the rules governing the expenditure of DOE
funds?
________________________________________________________________________
________________________________________________________________________
17h. Did your agency experience any delays or other difficulties in spending funds from Other
State Programs in PY 2010?
_____ Yes
193
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
[Q18 will be activated by ‗yes‘ responses in the Utilities category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
18a. When did agency first receive weatherization funds from Utilities? _________
18b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
18c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and Utilities during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
18d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
18e. If leveraged funding from Utilities decreased from PY 2008, was the change attributable to
increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
18f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Utilities in PY 2010?
_____ Yes
_____ No
[18g will appear if respondent answered ‗no‘ in 18f]
18g. What were the major differences in the rules governing the expenditure of Utilities
funds in comparison to the rules governing the expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
18h. Did your agency experience any delays or other difficulties in spending funds from Utilities
in PY 2010?
_____ Yes
194
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
[Q19 will be activated by ‗yes‘ responses in the Program Income category of Question 1
and deactivated (skipped) for ‗no‘ responses in that category.]
19a. When did agency first receive weatherization funds from Program Income? _________
19b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
19c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and Program Income during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
19d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
19e. If leveraged funding from Program Income decreased from PY 20019, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
19f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Program Income in PY 2010?
_____ Yes
_____ No
[19g will appear if respondent answered ‗no‘ in 19f]
19g. What were the major differences in the rules governing the expenditure of Program
Income funds in comparison to the rules governing the expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
19h. Did your agency experience any delays or other difficulties in spending funds from Program
Income in PY 2010?
_____ Yes
195
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
[Q20 will be activated by ‗yes‘ responses in the In-Kind Contributions category of Question
1 and deactivated (skipped) for ‗no‘ responses in that category.]
20a. When did agency first receive weatherization funds from In-Kind Contributions?
_________
20b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
20c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and In-Kind Contributions during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
20d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
20e. If leveraged funding from In-Kind Contributions decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
20f. Did you require local agencies to follow DOE rules when spending weatherization funds
from In-Kind Contributions in PY 2010?
_____ Yes
_____ No
[20g will appear if respondent answered ‗no‘ in 20f]
20g. What were the major differences in the rules governing the expenditure of In-Kind
Contributions in comparison to the rules governing the expenditure of DOE
funds?__________________________________________________________________
________________________________________________________________________
20h. Did your agency experience any delays or other difficulties in spending funds from In-Kind
Contributions in PY 2010?
196
_____ Yes
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
[Q21 will be activated by ‗yes‘ responses in the Non-profits category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
21a. When did agency first receive weatherization funds from Non-profits? _________
21b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
21c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and Non-profits during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
21d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
21e. If leveraged funding from Non-Profits decreased from PY 2008, was the change attributable
to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
21f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Non-Profits in PY 2010?
_____ Yes
_____ No
[21g will appear if respondent answered ‗no‘ in 21f]
21g. What were the major differences in the rules governing the expenditure of NonProfits funds in comparison to the rules governing the expenditure of DOE funds?
__________________________________________________________________
________________________________________________________________________
21h. Did your agency experience any delays or other difficulties in spending funds from Nonprofits in PY 2010?
197
_____ Yes
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
[Q22 will be activated by ‗yes‘ responses in the Third Party category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
22a. When did agency first receive weatherization funds from Third Parties (foundations,
lenders)? _________
22b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
22c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and Third Parties during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
22d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
22e. If leveraged funding from Third Parties decreased from PY 2008, was the change
attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
22f. Did you require local agencies to follow DOE rules when spending weatherization funds
from Third Parties in PY 2010?
_____ Yes
_____ No
[22g will appear if respondent answered ‗no‘ in 22f]
22g. What were the major differences in the rules governing the expenditure of Third
Party funds in comparison to the rules governing the expenditure of DOE funds?
________________________________________________________________________
________________________________________________________________________
198
22h. Did your agency experience any delays or other difficulties in spending funds from Third
Parties in PY 2010?
_____ Yes
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
[Q23 will be activated by ‗yes‘ responses in All Other, Specify category of Question 1 and
deactivated (skipped) for ‗no‘ responses in that category.]
23a. When did agency first receive weatherization funds from [insert text from ‗All Other,
Specify‘]?
23b. Number of years that agency worked to achieve this leveraging relationship (leave blank if
unknown): _________
23c. Please describe the change in leveraging relationships between your agency‘s low-income
weatherization program and [insert text from ‗All Other, Specify‘] during the ARRA period.
____ Extremely positive
____ Positive
____ No change
____ Negative
____ Extremely Negative
23d. How do you expect leveraged funding from this source to change from PY 2010 to the postARRA period in PY 2012?
____ increase
____ decrease
____ stay the same
23e. If leveraged funding from [insert text from ‗All Other, Specify‘] decreased from PY 2008,
was the change attributable to increased ARRA funding?
_____ Yes
_____ No
_____ Not applicable
23f. Did you require local agencies to follow DOE rules when spending weatherization funds
from [insert text from ‗All Other, Specify‘] in PY 2010?
_____ Yes
_____ No
[23g will appear if respondent answered ‗no‘ in 23f]
23g. What were the major differences in the rules governing the expenditure of [insert
text from ‗All Other, Specify‘] funds in comparison to the rules governing the
expenditure of DOE funds?
________________________________________________________________________
199
________________________________________________________________________
23h. Did your agency experience any delays or other difficulties in spending funds from [insert
text from ‗All Other, Specify‘] in PY 2010?
_____ Yes
_____ No
If yes, please provide a short description of the reasons for the delays or other difficulties:
_______________________________________________________________________
24. How important were leveraged funds for your agency‘s Weatherization Program in PY 2010
compared to program years without additional ARRA funding? (Check best answer)
_____Very important
_____ Important
_____ Not very important
_____ Not important at all
25. Did your agency set aside funding to advocate for leveraged resources in PY 2010?
_____ Yes
_____ No
26. How successful would you rate your agency‘s efforts to acquire leveraged funds in PY 2010?
(Check best answer)
_____ Very successful
_____ Successful
_____ Not very successful
_____ Not successful at all
_____ State does not seek leveraged funds
27. What factors limited the success of your agency‘s efforts to acquire leveraged funding in PY
2010? _______________________
28. Did your agency encounter any of the following problems in spending non-DOE funds in
general in PY 2010? (Check all that apply)
_____ Our agency could not easily increase the number of homes weatherized during the
year in order to better spend non-DOE funds
_____ Our agency required the expenditure of DOE weatherization funds before nonDOE funds were expended
_____ We had insufficient staff to manage the receipt and expenditure of non-DOE funds
_____ We had inadequate accounting systems to manage the receipt and expenditure of
non-DOE funds
_____ Guidance received from DOE and/or our state made it difficult to expend nonDOE funds in a timely manner
_____ Other ___________________
200
29. Have you modified your agency‘s weatherization program practices or regulations in the
three years prior to PY 2010 to facilitate spending and reporting on leveraged resources?
_____ Yes
_____ No
30. How has the influx of ARRA funding impacted previously existing leveraging relationships?
a. Extremely positive impact
b. Positive impact
c. No Impact
d. Negative impact
e. Extremely negative impact
31. Can leveraging relationships damaged or lost during ARRA be re-built post-ARRA?
a. Yes, absolutely
b. Yes, probably
c. Uncertain
d. No, probably not
e. No, definitely not
32. What aspects about your agency‘s low-income weatherization program are most
misunderstood by actual and potential leveraging partners? __________________
33. How has your program worked to overcome these misunderstandings? _________________
34. What information would your agency‘s weatherization program like to have that could be
used to overcome these misunderstandings? ____________________________
35. What other information would your agency‘s weatherization program like to have that could
be used to ‗sell‘ leveraging relationships? __________________________
36. What was the quality of the support and assistance on leveraging the Weatherization
Assistance Program with other funding sources and related programs that your agency received
from the state and its contractors in Program Year 2010? (Check best answer)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
36a. If appropriate, why did you rate the quality very low or low? ______________
37. Overall, what is your expectation for total agency leveraged for low-income weatherization
funding post ARRA in PY 2012 compared to pre-ARRA PY 2008
a. Greatly increased
b. Increased
201
c. Same level
d. Decreased
e. Greatly decreased
38. On balance, how beneficial do you think will ARRA funding prove to be over the longerterm on your agency‘s ability to leverage DOE WAP-program funding for low-income
weatherization?
a. Extremely beneficial
b. Beneficial
c. No long-term impact
d. Unbeneficial
e. Extremely unbeneficial
202
PART 3. WAP ARRA Program Information
1. For many years, the national weatherization network and the state-level national
weatherization networks had been relatively stable. During the ARRA period, did the
composition of your agency-level weatherization network change?
a. Yes
b. No (Skip to Question 5)
2. How would you characterize the magnitude of this change?
a. Extreme
b. Great
c. Moderate
d. Small
3. Using the table below, please list the names and contributions of the major new players.
Name
Description of Weatherization Related Contributions (e.g.,
provides training, sells energy efficient products )
203
4. On balance, how beneficial have these new players been to your agency‘s weatherization
program?
a. Extremely beneficial
b. Beneficial
c. Neither beneficial nor unbeneficial
d. Unbeneficial
e. Extremely unbeneficial
Please Explain _____________________
5. During the ARRA period, did organizational responsibilities for your agency‘s weatherization
program (e.g., reporting lines, organizational home) change?
a. Yes
b. No (Skip to Question 9)
6. How did the organizational responsibilities change? Please Explain _______________
7. In your opinion, was this change a result of increased visibility of your agency‘s
weatherization program created by the substantial funding increase?
a. Yes
b. No
8. On balance, how beneficial has this change been to your agency‘s weatherization program?
a. Extremely beneficial
b. Beneficial
c. Neither beneficial nor unbeneficial
d. Unbeneficial
e. Extremely unbeneficial
Please explain ____________________________
9. During the ARRA period, how has state-level oversight of your agency‘s weatherization
assistance program changed?
a. Greatly increased
b. Increased
c. Stayed the same
d. Decreased
e. Greatly decreased
10. During the ARRA period, has your state passed any new laws and/or implemented any new
regulations that have directly impacted your agency‘s weatherization assistance program?
a. Yes
b. No (Skip to Question 14)
11. Please describe the new laws and/or regulations. __________________________
204
12. On balance, how beneficial have these new laws and/or regulations been to your agency‘s
weatherization program?
a. Extremely beneficial
b. Beneficial
c. Neither beneficial nor unbeneficial
d. Unbeneficial
e. Extremely unbeneficial
Please explain ______________________________________
13. Would these laws and/or regulations have been passed with or without the increased attention
on the program attributable to ARRA?
a. Wholely due to ARRA
b. Primarily due to ARRA
c. Somewhat due to ARRA
d. Would have been passed anyway without ARRA
14. During the ARRA period, how has the visibility of your agency‘s weatherization program
changed with respect to elected officials?
a. Greatly increased
b. Increased
c. Stayed the same
d. Decreased
e. Greatly decreased
15. On balance during the ARRA period, how has the public‘s support for your agency‘s
weatherization assistance program changed?
a. Greatly increased
b. Increased
c. Stayed the same
d. Decreased
e. Greatly decreased
Please explain ________________________________
16. How would you rate the public‘s understanding of your agency‘s low-income weatherization
assistance program?
a. Excellent
b. Good
c. Fair
d. Poor
17. Please explain those aspects of the program that are most frequently misunderstood by the
public. ________________________
205
18. During the ARRA period, how has media attention on your agency‘s weatherization program
changed?
a. Greatly increased
b. Increased
c. Stayed the same
d. Decreased
e. Greatly decreased
19. Please rate the quality of the media coverage your agency‘s weatherization program has
received during the ARRA period?
a. very high quality
b. high quality
c. moderate quality
d. low quality
e. very low quality
f. N/A no media coverage (skip to 22)
g. no opinion
20. Please describe the overall media coverage of your agency‘s weatherization program during
the ARRA period:
a. very positive
b. positive
c. neither positive or negative
d. negative
e. very negative
21. What topics did the media mainly focus on? (check all that apply)
a. Jobs created
b. Energy saved
c. Helping low income households
d. Waste, fraud and abuse
e. Lack of energy savings
f. Organizational mis-steps
g. Other ______________________
22. Will weatherization issues identified rightly or wrongly by the media during ARRA have
lasting impacts on weatherization funding for your agency in the future?
a. could substantial reduce funding
b. could reduce funding
c. no impact
d. could increase funding
e. could substantially increase funding
206
23. On balance, how beneficial has been the attention paid to your agency‘s weatherization
program during the ARRA period?
a. Extremely beneficial
b. Beneficial
c. Neither beneficial nor unbeneficial
d. Unbeneficial
e. Extremely unbeneficial
Please explain ____________________________
PART 4. WAP PROGRAM IMPLEMENTATION DURING THE ARRA PERIOD
1. How have relationships changed between your state weatherization office and your local
weatherization agency during ARRA?
a. become much more congenial
b. become more congenial
c. no changes
d. become more strained
e. become much more strained
2. What was the quality of the administrative support and assistance that your agency received
from the state and its contractors in Program Year 2010? (Check best answer)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
2a. If appropriate, why did you rate the quality very low or low? ______________
3. What was the quality of the training that your agency received from the state and its
contractors in Program Year 2010? (Check best answer)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
3a. If appropriate, why did you rate the quality very low or low? ______________
207
4. What was the quality of the support and assistance on client education that your agency
received from the state and its contractors in Program Year 2010? (Check best answer)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
4a. If appropriate, why did you rate the quality very low or low? ______________
5. What was the quality of the support and assistance on leveraging the Weatherization
Assistance Program with other funding sources and related programs that your agency received
from the state and its contractors in Program Year 2010? (Check best answer)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
5a. If appropriate, why did you rate the quality very low or low? ______________
6. What was the quality of the technical support that your agency received from the state and its
contractors in Program Year 2010? (Check best answer)
_____ very high quality
_____ high quality
_____ moderate quality
_____ low quality
_____ very low quality
_____ not applicable
6a. If appropriate, why did you rate the quality very low or low? ______________
7. How flexible did you find the DOE program rules that governed the weatherization program in
Program Year 2010? In other words, did the program rules allow your agency to tailor your
program to your needs (very flexible) or proscribe your program to only one way of operation
(very inflexible (Check best answer)
_____ Very Flexible
_____ Flexible
_____ Inflexible
_____ Very Inflexible
208
7a. Using PY 2010 as the reference point, how should the program rules change?
_____ Become much more flexible
_____ Become more flexible
_____ Stay about the same
_____ Become more inflexible
_____ Become much more inflexible
7b. In what areas should the program rules become more flexible? ________________________
7c. In what areas should the program rules become less flexible? _________________________
8. Please describe any important political issues faced by your agency‘s weatherization program
in Program Year 2010.
______________________________________________________________________________
______________________________________________________________________________
______________________________________________________________________________
__________________________________________________________________
9. Which statement best describes how your agency dealt with program expansion during the
ARRA period?
a. promoted existing weatherization staff to management positions and hired new entry level
workers
b. hired new weatherization managers and hired new entry level workers
c. promoted existing weatherization staff to management positions and hired contractors to ramp
up production
d. simply hired contractors to ramp up production
10. Did expanding your agency‘s level of production result in any economies of scale benefits?
a. No
b. some benefits
c. moderate benefits
d. substantial benefits
11. How did your agency change its in-take procedures during the ARRA-period from the
procedures used pre-ARRA (i.e., PY2008)? (Check Best Answer)
a. No change
b. Eligibility checks more stringent
c. Eligibility checks less stringent
d. Other _______________________
12. Were there any material, equipment or other supply chain bottlenecks during the ARRA
period that negatively impacted weatherization production by your agency?
a. No
b. Yes
If yes, what was in short supply? ______________________
209
13. To what extent have large DOE programs, WAP, State Energy Program, Energy Efficiency
Community Block Grant Program, competed for labor during the ARRA period?
a. Not at all
b. a small extent
c. a moderate extent
d. a large extent
e. a very large extent
14. During the ARRA period, how has state-level oversight of your state‘s weatherization
assistance program changed?
a. Greatly increased
b. Increased
c. Stayed the same
d. Decreased
e. Greatly decreased
15. How did your agency manage increasing workloads and performance expectations?
Please Explain ____________________
16. Has ARRA funding allowed your agency to purchase and implement new information
technologies to improve program administration?
a. Yes
b. No
If yes, please describe these new information technologies. __________________________
17. Has ARRA funding allowed your agency to purchase new field technologies (e.g., Infrared
Thermal Imagers) to improve weatherization audits and measure installation?
a. Yes
b. No
If yes, please describe these purchases.
18. How timely has DOE‘s guidance been during the ARRA period?
a. very timely
b. timely
c. not very timely
d. not timely at all
19. How clear has DOE‘s guidance been during the ARRA period?
a. very clear
b. clear
c. not very clear
d. not clear at all
210
20. During PY 2010, how much of a burden were your agency‘s administrative costs associated
with increased oversight during the ARRA period (e.g., from DOE IG, GAO)?
a. no burden
b. slight burden
c. moderate burden
d. substantial burden
e. extreme burden
21. Did your agency change its audit approach during the ARRA-period from the approach used
pre-ARRA (i.e., PY2008)?
a. Yes, went from priority list to computerized audit
b. Yes, went from computerized audit to priority list
c. Yes, changed from one computerized audit tool to another
d. No
22. To what extent have measure costs used in SIR calculations diverted from actual measure
costs experienced during ARRA in PY 2010? (choose the best description)
a. on balance actual measure costs were much higher than SIR calculations
b. on balance actual measure costs were higher than SIR calculations
c. SIR calculations and actual measure costs were about the same
d. on balance SIR calculations and actual measure costs were about the same but the actual
measure costs fluctuated a great deal up and down
e. on balance actual measure costs were lower than SIR calculations
f. on balance actual measure costs were much lower than SIR calculations
23. Did your agency change it quality assurance procedures during the ARRA-period (e.g., PY
2010) from the procedures used pre-ARRA (i.e., PY2008)?
a. Yes, procedures are much more stringent
b. Yes, procedures are more stringent
c. No
24. Did your agency change its approach to using in-house crews versus contractors during the
ARRA-period (e.g., PY 2010) from the procedures used pre-ARRA (i.e., PY2008)?
a. Contractors are performing a much higher percentage of weatherization jobs
b. Contractors are performing a higher percentage of weatherization jobs
c. In-House crews are performing a higher percentage of weatherization jobs
d. In-house crews are performing a much higher percentage of weatherization jobs
e. No
24a. If there was a change, what prompted the change? (choose the best description)
a. Change was solely made to meet production targets
b. Change was solely made to deal with Davis-Bacon
c. Change was made to meet production targets and deal with Davis-Bacon
d. Other (Please explain) __________________________________________
211
24b. How did your training requirements change with the change in approach?
a. All workers were required to be trained in weatherization work
b. Only in-house crews were required to be trained
c. Only contractors were required to be trained
d. No training was required.
25. Did your agency implement any innovative approaches to weatherization during the ARRAperiod?
a. No
b. Yes
If yes, please describe the innovative approaches ______________________________
26. Did your agency implement any other major programmatic changes during the ARRAperiod?
a. No
b. Yes
If yes, please describe these programmatic changes ______________________________
27. Does your agency have a deferral policy?
a. Yes
b. No
28. Please estimate the number of units where weatherization was deferred during PY 2010?
a. 0%
b. 1-5%
c. 6-10%
d. 10-20%
e. 21-30%
f. more than 30%
28a. What was the leading cause for deferral during PY 2010? __________________________
28b. Please estimate what percentage of units where weatherization was deferred during PY
2010 will eventually be weatherized?
a. 0%
b. 1-25%
c. 26-50%
d. 51-75%
e. 76-90%
f. 91-100%
212
28c. Please describe your agency‘s deferral rates in PY2010 compared to PY2008:
a. very great increase
b. great increase
c. increase
d. no change
e. decrease
f. great decrease
g. very great decrease
29. How much of a priority does your agency place on health and safety when weatherizing
homes?
a. Very high priority
b. High priority
c. Medium priority
d. Low priority
e. Very low priority
f. No priority
30. What would be the ideal amount of money per home your agency would like to spend for
addressing health and safety issues? ________________
31. Of all the homes weatherized by your agency in Program Year 2010, how many did you refer
to non-energy programs for additional services (e.g., nutrition; family counseling)? ___________
32. What approaches did your local agency use to recruit new qualified, reliable and trustworthy
weatherization crew members and how effective were these approaches: ___________________
33. What approaches did your local agency use to train the expanded weatherization workforce?
a. Utilized existing state program-basedtraining resources, i.e. State monitors or other staff
providing training to the network
b. Sent weatherization workers to one of 26 newly funded weatherization training centers
c.Sent weatherization workers to one of 12 legacy weatherization training centers (i.e. COAD,
NRCERT, INCAA, BPC, PG&E Stockton, NYSWDA, FSL, AEA, etc. YY)
d. Utilized private, consultant training providers (i.e. Saturn, BMI, EMC)
e. OTHER Please explain:___________________________________________________
f. No additional training was provided
34. In your opinion, how effective were these approaches?
a. Highly effective
b. Very Effective
c. Somewhat effective
d. No effective
35. Has your agency implemented new certification requirements for weatherization workers
during the ARRA period?
a. No (go to Q28)
b. Yes
213
36. What new certifications are required? (Check all that apply)
a. BPI Building Analyst
b. BPI Envelope
c. BPI Residential Building Envelope Accessible Areas Air Leakage Control Installer
d. Residential Building Envelope Whole House Air Leakage Control Crew Chief
e. BPI Manufactured Housing
f. BPI Heating
g. BPI Air Conditioning and Heat Pump
h. BPI Multifamily
i. HERS
j. LEED
k. Lead Safe Weatherization
l. Lead Certified Renovator
m. NAHB Green Building
n. Other ___________________
37. Did staff turnover change during the ARRA period as compared to the pre-ARRA
period?
a. Turnover greatly increased
b. Turnover increased
c. No, turnover did not change
d. Turnover decreased
e. Turnover greatly decreased
38. Which statement best describes changes in the level of employment in your agency‘s
jurisdiction resulting from Davis-Bacon?
a. large number of jobs created
b. small number of jobs created
c. no change in the number of jobs
d. small number of jobs lost
e. large number of jobs lost
f. don‘t know
39. Which statement best describes the impact of Davis-Bacon on wages paid to
weatherization workers by your agency?
a. large increase in wages
b. small increase in wages
c. no increase in wages
d. small decrease in wages
e. large decrease in wages
40. Did Davis-Bacon paperwork requirements lead some experienced weatherization
contractors who worked for your agency to leave the low-income weatherization field?
a. No
b. Yes, a few
214
c. Yes, many
41. How much of a problem was faced by your agency in coordinating wages stipulated by
Davis-Bacon for weatherization workers whose operations spanned multiple counties?
a. no problem
b. small problem
c. moderate problem
d. large problem
e. very large problem
f. N/A because your agency works exclusively in one county
42. Have changes in weatherization costs associated with Davis-Bacon altered choices of
measures installed in homes?
a. No
b. Yes
If yes, please explain _________________________________________
43. In your agency, on average, how much did Davis-Bacon rules increase the cost of
weatherizing low-income multifamily buildings four stories and higher?
a. no increase
b. 1-10%
c. 11-20%
d. 21-30%
e. 31+%
f. N/A
44. Overall, how did Davis-Bacon impact the costs of weatherization in your agency?
a. greatly increased costs
b. increased costs
c. no change in costs
d. decreased costs
e. greatly decreased costs
45. Does your state allow the weatherization of large low-income multi-family buildings?
a. yes
b. no (go to Q 40)
If not, why not? _______________________________
46. Does your agency weatherize large low-income multi-family buildings?
a. No
b. Yes
215
47. What are the barriers to weatherizing large low-income multi-family buildings? (check
all that apply)
a. lack of trained auditors
b. lack of trained crew
c. too expensive
d. building owners are uncooperative
e. energy savings are not high enough
f. unclear guidance from DOE on owner contributions
g. other __________________
48. Does your state allow the weatherization of public housing, that is, housing owned by a
public housing authority?
a. Yes
b. No (go to Q. 43)
If not, why not? _______________________________
49. Does your agency weatherize public housing?
a. No
b. Yes
50. What are the barriers to weatherizing public housing units?
___________________________________________
51. Does your state allow the weatherization of HUD assisted housing?
a. Yes
b. No (go to Q46)
If not, why not? _______________________________
52. Does your agency weatherize HUD assisted housing?
a. No
b. Yes
53. What are the barriers to weatherizing HUD assisted housing? ______________
54. How many units of each type were weatherized in your state in PY 2010?
Type of Housing
Large Multi-family
Public Housing Multi- HUD Assisted Multi(Not Public Housing or family
family
HUD Assisted
55. Are there any DOE rules that could be changed to make it easier to weatherize large lowincome multi-family buildings?
a. No
b. Yes
If yes, please explain ___________________________________________
216
56. Does your agency‘s weatherization program have an agreement with your State Historic
Preservation Office (SHPO)?
a. Yes
b. No
57. What percentage of units weatherized by your agency in PY 2010 fell under the SHPO
agreement?
a. 0
b. 1-5
c. 6-10
d. 11-15
e. 15+
58. In general, have the historic preservation guidelines changed the types of measures
installed in these homes?
a. No
b. Yes
If yes, please explain ___________________________________________________
59. Has low-income weatherization had an impact in preventing home foreclosures during
the ARRA period in your agency‘s jurisdiction?
a. no
b. little impact
c. moderate impact
d. great impact
e. very great impact
f. don‘t know
59a. Has low-income weatherization had an impact on preventing homelessness during the
ARRA period in your agency‘s jurisdiction?
a. no
b. little impact
c. moderate impact
d. great impact
e. very great impact
60. How many homes were on your wait list for weatherization in PY 2010? _____
61. On average, how long was a home on the wait list before it was weatherized in PY 2010?
_____
217
62. In Program Year 2010, did your agency set targets and actively solicit participation by
dwelling units of the types shown below? (Check all that apply)
Set targets for number
Actively sought participation by
Type of dwelling unit
of dwelling units of this households residing in this type of
type to weatherize
dwelling unit
Single Family Attached
and Detached
Small Multi-family (2-4
units)
Multifamily (5 or More
Units per Building)
Mobile Home
Shelter
63. Which of the following approaches did your agency use in Program Year 2010 to market
your weatherization services to low-income households? (Check all that apply)
Targeted mailings to potential clients
_____
Targeted mailings to landlords of potential clients
_____
Visits to potential clients
_____
Visits to landlords of potential clients
_____
Advertising with other social service agencies
_____
Advertising in local newspapers or magazines
_____
Radio advertising
_____
Television advertising
_____
Posting information on website
_____
Other (please specify) ______________________________________ _____
64. Who was responsible for leading the marketing/outreach efforts described above? (Check
all that apply)
Agency management
_____
In-house outreach coordinator
_____
Contractor outreach coordinator
_____
In-house communications staff
_____
Contractor communications staff
_____
Other in-house staff (please specify) __________________________ _____
Other contractor staff (please specify) _________________________ _____
65. In general, how satisfied were you with the length of time between the client‘s request to
have their home weatherized and when it was actually weatherized in PY 2010? (Check best
answer)
_____ Very satisfied
_____ Satisfied
_____ Not satisfied or dissatisfied
_____ Dissatisfied
_____ Very dissatisfied
218
66. In general, how easy was it to schedule your visits to client homes to perform audits,
weatherization, and inspections in PY 2010? (Check best answer)
_____Very easy
_____ Easy
_____ Not easy or difficult
_____ Difficult
_____ Very difficult
67. What percentage of households whose homes were weatherized by your agency in
Program Year 2010 registered a complaint regarding the quality or nature of the
weatherization job performed on their dwelling unit? ____________
68. Of those households that filed complaints, what percentage of these required some
additional work? __________
219
OMB Control Number: XXXX-XXXX
APPENDIX E. S3: SUBSET OF AGENCIES DETAILED PROGRAM
INFORMATION SURVEY
This data is being collected to conduct a detailed process evaluation of the Weatherization
Assistance Program at the local level. The data you supply will be used to characterize local
agency weatherization activities in Program Year 2010.
Public reporting burden for this collection of information is estimated to average sixteen
hours per response, including the time for reviewing instructions, searching existing data
sources, gathering and maintaining the data needed, and completing and reviewing the
collection of information. Send comments regarding this burden estimate or any other aspect
of this collection of information, including suggestions for reducing this burden, to Office of
the Chief Information Officer, Records Management Division, IM-11, Paperwork Reduction
Project (XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW,
Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA,
Paperwork Reduction Project (XXXX-XXXX), Washington, DC 20503.
All of the information obtained from this survey will be protected and will remain
confidential. The data will be analyzed in such a way that the information provided cannot be
associated back to your state, your agencies, or the housing units and clients that your state
served. Again, please note that the questions refer to PY 2010 unless otherwise noted.
PROGRAM CHARACTERIZATION
1. Please identify your state. ________________________________________
2. Please identify your local agency. _______________________________________
3. Please indicate the number of staff that supported your agency‘s weatherization program
and their work effort in Program Year 2010. In considering the number of staff, please
include everyone who worked full- or part-time or who worked with the weatherization
program as well as other agency programs.
Type of Administrative Function
Management/administration
220
Number of
Agency Staff
(# persons)
Agency Staff
Work Effort
(FTE)
Auditing/inspection
Home weatherization
Other (specify)
4. For the agency staff working on your agency‘s weatherization in each of the following
functional areas in Program Year 2010, please indicate their level of experience with the
weatherization program:
Very
High
High
Average
Low
Very Low
Management/administration
Auditing/inspection
Home weatherization
Other (specify)
5. For the in-house staff working in your agency‘s weatherization program in each of the
functional areas listed below, please indicate the amount of turnover in staff from the
beginning of PY 2009 to the end of PY 2010. (Please check appropriate box representing
the level of turnover for each functional area.)
Management/
administration
Field monitoring/
Auditing
Training and Technical
Assistance
Other (specify)
____________________
No Turnover
Some turnover
(all staff in this functional
area at the beginning of
PY 2009 were in the same
functional area by the end
of PY 2010)
(1-15% of the staff in this
functional area at the
beginning of PY 2009 did
not remain in the same
functional area by the end
of PY 2010)
Substantial
turnover
(more than 15% of the
staff in this functional
area at the beginning of
PY 2009 remained in the
same functional area at
the end of PY 2010)
□
□
□
□
□
□
□
□
□
□
□
□
6. For which of the following functional areas were there certification or licensing
requirements in Program Year 2010 for the in-house or contractor staff serving your state‘s
weatherization program? (Check all that apply)
Certification or
Licensing
Requirement for
In-house Staff
221
Certification or Licensing
Requirement for
Contractor Staff
Management/
administration
Auditing/inspection
Home weatherization
Other (specify)
222
AUDIT
1. What was the primary method that your agency used in Program Year 2010 to select
weatherization measures for clients‘ dwelling units (excluding health, safety, and repair
measures and general heat waste measures)? (Check best answer)
Priority list used for all dwelling units
_____
Calculation procedure (e.g., spreadsheet, computerized audit)
used for all dwelling units
_____
Priority list applied to dwelling units meeting specified guidelines
and calculation procedure used for remaining units
_____
Other (specify) ______________________________________________
_____
2. Including PY 2010, for how many years had your agency used the weatherization
measure selection method indicated above? _____
3. What types of credentials or experience were required of your staff or contractors who
were engaged in measure selection in Program Year 2010? Check all that apply.
Technical certification
_____
Extensive weatherization work experience
_____
Extensive weatherization supervision experience
_____
Construction experience
_____
Other (specify) ______________________________________________
_____
4. Please indicate the level of experience for the agency staff engaged in measure selection
in Program Year 2010 in each of the following functional areas:
Very
High
High
Performing weatherization
work
Supervising weatherization
work
Working in construction
Performing preweatherization audits
223
Average
Low
Very Low
5. On average, approximately how many hours did it take to select weatherization measures
for a typical dwelling unit served by your agency in Program Year 2010, by the major
components listed below?
Preparation/scheduling
_____
Travel
_____
On-site auditing
_____
Post-audit analysis and write-up
_____
Other
_____
TOTAL of all components
_____
6. If your agency used a priority list for at least some dwelling units in Program Year 2010,
how difficult was it for your staff to use that priority list? (Check best answer)
_____Very Difficult
_____ Difficult
_____ Easy
_____ Very Easy
7. If your agency used a priority list in Program Year 2010, how effective was that list?
(Check best answer)
_____ Very Ineffective
_____ Ineffective
_____ Effective
_____ Very Effective
8. If your agency used a calculation procedure for at least some dwelling units in Program
Year 2010, what was the name of the procedure or procedures employed? Check all that
apply.
AK Warm
_____
EA-3
_____
EASY
_____
EA-QUIP
_____
HomeCheck
_____
Meadows
_____
224
REES
_____
REM/Rate
_____
SMOC-ERS
_____
TIPS
_____
TREAT
_____
Weatherization Assistant (NEAT/MHEA)
_____
WXEOR
_____
Other (specify) ______________________________________________
_____
225
9. If your agency used a calculation procedure in Program Year 2010, use the following
scale to describe how difficult it was for your staff to use the applicable procedure(s).
Circle best answer). 1= Very Difficult; 2=Difficult; 3= Easy; 4=Very Easy; 5 =N/A
AK Warm
1
2
3
4
5
EA-3
1
2
3
4
5
EASY
1
2
3
4
5
EA-QUIP
1
2
3
4
5
HomeCheck
1
2
3
4
5
Meadows
1
2
3
4
5
REES
1
2
3
4
5
REM/Rate
1
2
3
4
5
SMOC-ERS
1
2
3
4
5
TIPS
1
2
3
4
5
TREAT
1
2
3
4
5
Weatherization Assistant (NEAT/MHEA)
1
2
3
4
5
WXEOR
1
2
3
4
5
Other (specify) _______________
1
2
3
4
5
10. If your agency used a calculation procedure in Program Year 2010, use the following
scale to describe how effective you found the applicable procedure(s).1= Very
Ineffective; 2=Ineffective; 3= Effective; 4=Very Effective; 5=N/A
a. AK Warm
1
2
3
4
5
b. EA-3
1
2
3
4
5
c. EASY
1
2
3
4
5
d. EA-QUIP
1
2
3
4
5
e. HomeCheck
1
2
3
4
5
f. Meadows
1
2
3
4
5
226
g. REES
1
2
3
4
5
h. REM/Rate
1
2
3
4
5
i. SMOC-ERS
1
2
3
4
5
j. TIPS
1
2
3
4
5
k. TREAT
1
2
3
4
5
l. Weatherization Assistant (NEAT/MHEA)
1
2
3
4
5
m. WXEOR
1
2
3
4
5
n. Other (specify) _______________
1
2
3
4
5
11. If your agency used a calculation procedure for at least some dwelling units in Program
Year 2010, did your state allow under DOE rules the installation of general heat waste
measures (low-cost/no-cost weatherization activities) in those units without the need for
an energy justification?
_____Yes
_____ No (go to Question 13)
12. Please indicate which of the following general heat waste measures your agency was
allowed to install in Program Year 2010. Check all that apply.
Weatherstripping
_____
Caulking
_____
Insulation for plugging air leaks
_____
Low-flow shower heads
_____
Low-flow faucet aerators
_____
Air filters
_____
227
Glass patching
_____
Lighting
_____
Hot water tank insulation (water heater wrap)
_____
Water pipe insulation
_____
Other (specify)
_____
13. What was the primary justification used by your agency in Program Year 2010 for
performing work specifically targeted at reducing air infiltration (i.e., air sealing work)?
Check best answer.
Work should be performed where the air leakage rate as measured by a blower
door test is greater than a minimum number (e.g., minimum ventilation guideline)
calculated for the dwelling unit in question
_____
Work should be performed to address occupant complaints
_____
All significant air leakage sites should be sealed
_____
Air sealing work should be performed on all dwelling units
_____
Other (specify)
_____
14. What other justifications were used by your agency in Program Year 2010 for performing
work specifically targeted at reducing air infiltration (i.e., air sealing work)? Check all
that apply.
Work should be performed where the air leakage rate as measured by a blower
door test is greater than a minimum number (e.g., minimum ventilation guideline)
calculated for the dwelling unit in question
_____
Work should be performed to address occupant complaints
_____
All significant air leakage sites should be sealed
_____
Air sealing work should be performed on all dwelling units
_____
Other (specify)
_____
15. What was the primary method used by your agency in Program Year 2010 to identify air
leakage sites to seal? Check only one.
228
Auditor identified air leakage sites visually and communicated relevant
information
to crew
_____
Auditor identified air leakage sites using a blower door and/or pressure
diagnostics and communicated relevant information to crew
_____
Crew identified air leakage sites visually
_____
Crew identified air leakage sites using a blower door and/or
pressure diagnostics
_____
Other (specify)
_____
16. What other methods were used by your agency in Program Year 2010 to identify air
leakage sites to seal? Check all that apply.
Auditor identified air leakage sites visually and communicated relevant
information
to crew
_____
Auditor identified air leakage sites using a blower door and/or pressure
diagnostics and communicated relevant information to crew
_____
Crew identified air leakage sites visually
_____
Crew identified air leakage sites using a blower door and/or
pressure diagnostics
_____
Other (specify)
_____
17. In Program Year 2010, at what point did your agency stop performing air sealing work on
a given dwelling unit? Check all that apply.
When all identified air leakage sites were sealed
_____
When all significant air leakage sites were sealed
_____
When the air leakage rate as measured by a blower door test dropped below a
minimum number calculated for the dwelling unit in question
_____
When a blower door test indicated that the most recent infiltration reduction
measure installed in the dwelling unit was not cost effective
_____
Other (specify) _________________________________________________
_____
229
18. Did your agency do duct sealing work in Program Year 2010?
_____ Yes
_____ No (go to Question 22)
19. How did your agency determine when duct sealing work was needed for a particular
dwelling unit in PY 2010? Check all that apply.
All houses with ducts received duct sealing measures
_____
All houses with return air ducts get sealed
_____
Ducts were sealed in those cases where leakage sites were visible
_____
Ducts were sealed when a blower door test indicated the presence of leaks
_____
Ducts were sealed when duct diagnostics (blower door subtraction, duct blower,
or pressure pan measurements) indicated that the leakage rate was greater than a
minimum number calculated for the dwelling unit in question
_____
20. What methods were used by your agency in Program Year 2010 to identify duct leakage
sites to seal? Check all that apply.
Auditor identified duct leakage sites visually and communicated relevant
information to crew
_____
Auditor identified duct leakage sites using a blower door and communicated
relevant information to crew
_____
Auditor identified duct leakage sites using duct diagnostics and communicated
relevant information to crew
_____
Crew identified duct leakage sites visually
_____
Crew identified duct leakage sites using a blower door
_____
Crew identified duct leakage sites using duct diagnostics
_____
Other (specify) _________________________________________________
_____
21. In Program Year 2010, at what point did your agency stop performing duct sealing work
on a given dwelling unit? Check all that apply.
When all identified duct leakage sites were sealed
_____
When a blower door test indicated no more flow from the ducts
_____
230
When the duct leakage rate as measured by duct diagnostics dropped below a
minimum number calculated for the dwelling unit in question
_____
Other (specify) _________________________________________________
_____
22. How did you determine when a particular refrigerator should be replaced in PY 2010?
Check all that apply.
Not allowed to replace refrigerators
_____
Energy use of existing refrigerator was metered
_____
Energy use of existing refrigerator was assumed base on rated/nameplate
value_____
Non-energy criteria were used (e.g., age, color, physical appearance)
_____
Refrigerator was replaced if it was no longer running or could not maintain
desired temperature
_____
Other (specify)
_____
22a.How did you determine when a particular air conditioner should be replaced in PY 2010?
Check all that apply.
Not allowed to replace air conditioner
_____
Energy use of existing air conditioner was metered
_____
Energy use of existing air conditioner was assumed base on rated/nameplate value
_____
Non-energy criteria were used (e.g., age, physical appearance)
_____
Air conditioner was replaced if it was no longer running or could not maintain
desired temperature
_____
Other (specify)
_____
Not applicable
_____
23. Which of the following diagnostic procedures did your agency perform in Program Year
2010? Check all that apply.
Pressure diagnostics:
Blower door (house air leakage rate)
_____
231
Zonal pressure measurements
Room-to-room pressure measurements (distribution balancing)
Duct pressure pan measurements
Duct blower measurements (duct air leakage rate)
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)
Heat rise measurements
CO measurements in flues
Draft/spillage (normal operation)
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
HVAC components and cross-cutting diagnostics:
Air handler flow rate
Thermostat anticipator current
Worst case draft/spillage (CAZ)
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)
CO measurements in flues
Draft/spillage (normal operation)
Water flow rates (showerheads and faucets)
Other CO measurements:
CO measurements in equipment rooms
Cooking stove
CO measurements in living areas
Other diagnostics and inspections:
Refrigerator energy use
Exhaust fan air flow rate measurement
Infrared scanning (camera)
Radon testing
Lead testing
Mold and mildew testing
Moisture context testing
Other (please specify) ___________________________________
232
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
_____
24. Which of the diagnostic procedures listed below were initiated by your agency in PY
2010 and the two years prior to PY 2010? If your agency did not use a particular
procedure, leave that item blank.
Pressure diagnostics:
Blower door (house air leakage rate)
_____
Zonal pressure measurements
_____
Room-to-room pressure measurements (distribution balancing)
_____
Duct pressure pan measurements
_____
Duct blower measurements (duct air leakage rate)
_____
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)
_____
Heat rise measurements
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
_____
HVAC components and cross-cutting diagnostics:
Air handler flow rate
_____
Thermostat anticipator current
_____
Worst case draft/spillage (CAZ)
_____
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Water flow rates (showerheads and faucets)
_____
Other CO measurements:
CO measurements in equipment rooms
_____
Cooking stove
_____
CO measurements in living areas
_____
Other diagnostics and inspections:
Refrigerator energy use
_____
Exhaust fan air flow rate measurement
_____
Infrared scanning (camera)
_____
Radon testing
_____
Lead testing
_____
Mold and mildew testing
_____
Moisture context testing
_____
Other (please specify) ___________________________________
_____
233
25. What types of credentials or experience were required of your staff who performed
diagnostic procedures in Program Year 2010? Check all that apply.
Technical certification
_____
Extensive weatherization work experience
_____
Extensive weatherization supervision experience
_____
Construction experience
_____
Other (specify) _________________________________________
_____
26. Approximately how many hours did your agency spend on performing
diagnostic procedures for a typical dwelling unit served by your agency in
Program Year 2010? _____
234
27. Please indicate the cost, amount of training needed, amount of time needed and
effectiveness of the following types of diagnostic procedures relative to each other for PY
2010. Please use the following scale: 1 – very low; 2 -- low; 3 – medium; 4 – high; 5 –
very high.
Cost Training
Time
Effectiveness
Needed
Needed
Pressure diagnostics:
Blower door (house air leakage rate)
Zonal pressure measurements
Room-to-room pressure measurements
(distribution balancing)
Duct pressure pan measurements
Duct blower measurements (duct air leakage rate)
Space-heating system:
Flue gas analysis (steady-state efficiency
measurements)
Heat rise measurements
CO measurements in flues
Draft/spillage (normal operation)
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
HVAC components and cross-cutting diagnostics:
Air handler flow rate
Thermostat anticipator current
Worst case draft/spillage (CAZ)
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency
measurements)
CO measurements in flues
Draft/spillage (normal operation)
Water flow rates (showerheads and faucets)
Other CO measurements:
CO measurements in equipment rooms
Cooking stove
CO measurements in living areas
Other diagnostics and inspections:
Refrigerator energy use
Exhaust fan air flow rate measurement
Infrared scanning (camera)
Radon testing
Lead testing
Mold and mildew testing
Moisture context testing
Other (please specify
235
CLIENT EDUCATION
1. Which of the following client education approaches did your agency use in Program Year
2010? (Check all that apply)
Provide literature at time of client intake
_____
Provide video, CD or DVD at time of client intake
_____
Provide in-person instruction at time of client intake
_____
Provide hardware kit at time of client intake
_____
Provide literature at time of audit
_____
Provide video, CD or DVD at time of audit
_____
Provide in-person instruction at time of audit
_____
Provide hardware kit at time of audit
_____
Provide literature at time of weatherization
_____
Provide video, CD or DVD at time of weatherization
_____
Provide in-person instruction at time of weatherization
_____
Provide hardware kit at time of weatherization
_____
Provide literature at separate client education visit
_____
Provide video, CD or DVD at separate client education visit
_____
Provide in-person instruction at separate client education visit
_____
Provide hardware kit at separate client education visit
_____
Provide literature at time of inspection
_____
Provide video, CD or DVD at time of inspection
_____
Provide in-person instruction at time of inspection
_____
Provide hardware kit at time of inspection
_____
236
Group training class
_____
Other (please specify) _________________________________________
_____
237
2. Which of the following broad topics did your agency cover with clients in Program Year
2010? (Check all that apply)
Thermostat management
_____
HVAC system operation/maintenance
_____
Distribution system adjustment and zoning
_____
Cooling load reduction
_____
Windows
_____
Insulation
_____
Ventilation
_____
Mold
_____
Refrigerator
_____
Hot water use
_____
Water heating system operation/maintenance
_____
Lighting
_____
Laundry
_____
Kitchen appliance operation
_____
Other baseload electric use
_____
Energy Star
_____
Safety monitors (e.g., CO monitors, smoke alarm)
_____
Energy bills
_____
Other (please specify) _________________________________________
_____
3. Which of the following people provided client education for your agency in Program
Year 2010? Check all that apply.
a. In-house manager
_____
238
b. In-house education specialist
_____
c. Contractor education specialist
_____
d. Intake staff person
_____
e. Auditor
_____
f. In-house weatherization crew chief
_____
g. Contractor weatherization crew chief
_____
h. In-house weatherization crew member
_____
i. Contractor weatherization crew member
_____
j. Inspector
_____
k. Other (please specify) _________________________________________
_____
4. If in-person instruction was provided by your agency in Program Year 2010, who was
your preferred target? (Check best answer)
_____ Applicant
_____ Other adult member of household
_____ Child living in household
_____ Adult not living in household
_____ Other (please specify ____________________)
5. If in-person instruction was provided by your agency in Program Year 2010, was it
typically provided to a single person or multiple persons? Check best answer.
_____ single person
_____ multiple persons
6. What types of credentials or experience were required of those who provided client
education for your agency in Program Year 2010? Check all that apply.
College degree
_____
Technical certification
_____
Extensive experience in performing weatherization work
_____
Extensive experience in supervising weatherization work
_____
Educational background
_____
239
Other (please specify) ____________________________________________
_____
7. Which of the client education approaches listed below were initiated by your agency
during the ARRA period? (Check all that apply)
Provide literature at time of client intake
_____
Provide video, CD or DVD at time of client intake
_____
Provide in-person instruction at time of client intake
_____
Provide hardware kit at time of client intake
_____
Provide literature at time of audit
_____
Provide video, CD or DVD at time of audit
_____
Provide in-person instruction at time of audit
_____
Provide hardware kit at time of audit
_____
Provide literature at time of weatherization
_____
Provide video, CD or DVD at time of weatherization
_____
Provide in-person instruction at time of weatherization
_____
Provide hardware kit at time of weatherization
_____
Provide literature at separate client education visit
_____
Provide video, CD or DVD at separate client education visit
_____
Provide in-person instruction at separate client education visit
_____
Provide hardware kit at separate client education visit
_____
Provide literature at time of inspection
_____
Provide video, CD or DVD at time of inspection
_____
Provide in-person instruction at time of inspection
_____
Provide hardware kit at time of inspection
_____
240
Group training class
_____
Other (please specify) _________________________________________
_____
241
8. Please indicate the cost, amount of training needed, amount of time needed and
effectiveness of the following types of client education approaches relative to each other
for PY 2010. Please use the following scale: 1 – very low; 2 –low; 3 – medium; 4 – high;
5 – very high.
Cost
Training
Needed
Provide video, CD or DVD at time of
client intake
Provide in-person instruction at time
of client intake
Provide hardware kit at time of client
intake
Provide literature at time of audit
Provide video, CD or DVD at time of
audit
Provide in-person instruction at time
of audit
Provide hardware kit at time of audit
Provide literature at time of
weatherization
Provide video, CD or DVD at time of
weatherization
Provide in-person instruction at time
of weatherization
Provide hardware kit at time of
weatherization
Provide literature at separate client
education visit
Provide video, CD or DVD at separate
client education visit
Provide in-person instruction at
separate client education visit
Provide hardware kit at separate
client education visit
Provide literature at time of inspection
Provide video, CD or DVD at time of
inspection
Provide in-person instruction at time
of inspection
Provide hardware kit at time of
inspection
Group training class
Other (please specify)
242
Time
Needed
Effectiveness
9. On average, approximately how many minutes were spent in Program Year 2010 on client
education in a typical dwelling? ____________
TRAINING
1. On which of the following weatherization topics did agency staff working on your
agency‘s weatherization efforts receive training in Program Year 2010? Check all that
apply.
(1) Diagnostic procedures
_____
(2) Insulation
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(3) Space heating, ventilation, air conditioning
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Infiltration measures
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(5) Doors and windows
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(6) Hot water heating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(7) Baseloads (e.g., lighting, refrigerators)
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
1a. On which of the following administrative-related topics did agency staff working on your
agency‘s weatherization efforts receive training in Program Year 2010? Check all that apply.
(1) Management
_____
(2) Client education
_____
(3) Auditing/estimating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Monitoring/quality control
_____
(5) Financial topics
_____
(6) Outreach and communications
_____
(7) Other (please specify)
_____
243
1b. On which of the following health and safety topics did agency staff working on your
agency‘s weatherization efforts receive training in Program Year 2010? Check all that apply.
_____ Fire safety
_____ Indoor air quality
_____ Measures to increase security of housing unit
_____ Measures to reduce common household hazards
_____ Mold and mildew
_____ Lead
_____ Asbestos
_____ Vermiculite
_____ General crew safety
_____ Other health and safety
_____ Other (please specify)
244
2. On which of the following diagnostic procedures did agency staff working on your
agency‘s weatherization efforts receive training in Program Year 2010? Check all that
apply.
Pressure diagnostics:
Blower door (house air leakage rate)
_____
Zonal pressure measurements
_____
Room-to-room pressure measurements (distribution balancing)
_____
Duct pressure pan measurements
_____
Duct blower measurements (duct air leakage rate)
_____
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)
_____
Heat rise measurements
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
_____
HVAC components and cross-cutting diagnostics:
Air handler flow rate
_____
Thermostat anticipator current
_____
Worst case draft/spillage (CAZ)
_____
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Water flow rates (showerheads and faucets)
_____
Other CO measurements:
CO measurements in equipment rooms
_____
Cooking stove
_____
CO measurements in living areas
_____
Other diagnostics and inspections:
Refrigerator energy use
_____
Exhaust fan air flow rate measurement
_____
Infrared scanning (camera)
_____
Radon testing
_____
Lead testing
_____
Mold and mildew testing
_____
Moisture context testing
_____
Other (please specify) ___________________________________
_____
245
3. How many of your agency‘s staff were trained at the following events in Program Year
2010?
______ National Weatherization Program Conference
______ Affordable Comfort Conference
______ Other national conference
______ Regional weatherization conference
______ Your state‘s weatherization conference
______ Some other relevant conference in your state
______Weatherization conference given by another state
______ Some other relevant conference given by another state
______ Any state or regional training center class
______ Manufacturer‘s training school class
______ Utility training class
______ Training classes provided by your agency or those agencies you work for
______ One-time state-sponsored class
______ Any other class not sponsored by your state (e.g., another state, trade organization)
______ Visit to an agency you do not work for training
______ Instruction provided by your state to your individual agency or those agencies you
work for
______ In-person expert visit just to your agency (e.g., peer exchange, consultant)
______ Web cast
______ Other (please specify)
246
4. Which of the following weatherization topics listed below were agency staff first trained
on in PY 2010 and two years prior to PY 2010? (Check all that apply)
(1) Diagnostic procedures
_____
(2) Insulation
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(3) Space heating, ventilation, air conditioning
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Infiltration measures
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(5) Doors and windows
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(6) Hot water heating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(7) Baseloads (e.g., lighting, refrigerators)
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
4a. Which of the following administrative-related topics listed below were agency staff first
trained on in PY 2010 and in the two years prior to PY 2010? If your agency did not receive
training on a particular subject, leave that item blank.
(1) Management
_____
(2) Client education
_____
(3) Auditing/estimating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Monitoring/quality control
_____
(5) Financial topics
_____
(6) Outreach and communications
_____
(7) Other (please specify)
_____
247
4b. Which of the following health and safety topics listed below were agency staff first
trained on in PY 2010 and in the two years prior to PY 2010? (Check all that apply)
_____ Fire safety
_____ Indoor air quality
_____ Measures to increase security of housing unit
_____ Measures to reduce common household hazards
_____ Mold and mildew
_____ Lead
_____ Asbestos
_____ Vermiculite
_____ General crew safety
_____ Other health and safety
_____ Other (please specify)
5. On which of the following weatherization topics did your agency provide training to your
own in-house staff in Program Year 2010? (Check all that apply)
(1) Diagnostic procedures
_____
(2) Insulation
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(3) Space heating, ventilation, air conditioning
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Infiltration measures
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(5) Doors and windows
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(6) Hot water heating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(7) Baseloads (e.g., lighting, refrigerators)
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
248
5a. On which of the following administrative-related topics did your agency provide training
to your own in-house staff in Program Year 2010? (Check all that apply)
(1) Management
_____
(2) Client education
_____
(3) Auditing/estimating
_____
-- single family dwellings
_____
-- multifamily dwellings
_____
-- mobile homes
_____
(4) Monitoring/quality control
_____
(5) Financial topics
_____
(6) Outreach and communications
_____
(7) Other (please specify)
_____
5b. On which of the following health and safety topics did your agency provide training to
your own in-house staff in Program Year 2010? (Check all that apply)
_____ Fire safety
_____ Indoor air quality
_____ Measures to increase security of housing unit
_____ Measures to reduce common household hazards
_____ Mold and mildew
_____ Lead
_____ Asbestos
_____ Vermiculite
_____ General crew safety
_____ Other health and safety
_____ Other (please specify)
249
6. On which of the following diagnostic procedures did your agency provide training to
your staff in Program Year 2010? (Check all that apply)
Pressure diagnostics:
Blower door (house air leakage rate)
_____
Zonal pressure measurements
_____
Room-to-room pressure measurements (distribution balancing)
_____
Duct pressure pan measurements
_____
Duct blower measurements (duct air leakage rate)
_____
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)
_____
Heat rise measurements
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
_____
HVAC components and cross-cutting diagnostics:
Air handler flow rate
_____
Thermostat anticipator current
_____
Worst case draft/spillage (CAZ)
_____
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)
_____
CO measurements in flues
_____
Draft/spillage (normal operation)
_____
Water flow rates (showerheads and faucets)
_____
Other CO measurements:
CO measurements in equipment rooms
_____
Cooking stove
_____
CO measurements in living areas
_____
Other diagnostics and inspections:
Refrigerator energy use
_____
Exhaust fan air flow rate measurement
_____
Infrared scanning (camera)
_____
Radon testing
_____
Lead testing
_____
Mold and mildew testing
_____
Moisture context testing
_____
Other (please specify) ___________________________________
_____
250
7. For each broad subject listed in the left-most column of the following table, put a check
mark in the appropriate cell(s) to indicate which training method(s) you believe were
most effective for imparting key skills and information in that area to your agency‘s inhouse or contractor weatherization staff in PY 2010:
Conferences
Primarily
Field
training
Subject
Management
Weatherization skills and
methods
Auditing/
Estimating
Monitoring/
quality control
Financial topics
Outreach and
communications
Health and safety
Diagnostic procedures
Procedures for selecting
weatherization measures
Client education
Other (specify)
251
Primarily
Classroom
training
Agency
visits
Web
casts
Other
(specify)
8. For each broad subject listed in the left-most column of the following table, please
indicate the quality of training received in Program Year 2010 at the training venues
listed in the column headings. Please leave cells blank were your agency did not receive
training during this period of time. Please use the following scale: 1-very low; 2 - low; 3medium; 4- high; 5-very high
National
Weatherization
Program
Conference
Affordable
Comfort
Conference
Regional
Weatherization
Conference
Subject
Management
Weatherization skills
and methods
Auditing/
Estimating
Monitoring/
quality control
Financial topics
Outreach and
communications
Health and safety
Diagnostic
procedures
Procedures for
selecting
weatherization
measures
Client education
Other (specify)
252
State
Weatherization
Conference
State/
Regional
Training
Center
Training
Provided by
Your Own
Agency
9. For those staff working in your agency who needed to have knowledge about the following
list of weatherization topics in PY 2010, how well trained were they in each area in PY
2010? Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 –
moderately well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best
answer.
(1) Diagnostic procedures
1
2
3
4
5 6
(2) Insulation
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(3) Space heating, ventilation, air conditioning
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(4) Infiltration measures
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(5) Doors and windows
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(6) Hot water heating
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(7) Baseloads (e.g., lighting, refrigerators)
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
9a. For those staff working in your agency who needed to have knowledge about the
following list of administrative-related topics, how well trained were they in each area in PY
2010? Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 –
moderately well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best
answer.
(1) Management
1
2
3
4
5 6
(2) Client education
1
2
3
4
5 6
(3) Auditing/estimating
-- single family dwellings
1
2
3
4
5 6
-- multifamily dwellings
1
2
3
4
5 6
-- mobile homes
1
2
3
4
5 6
(4) Monitoring/quality control
1
2
3
4
5 6
(5) Financial topics
1
2
3
4
5 6
(6) Outreach and communications
1
2
3
4
5 6
253
(7) Other (please specify)
1
2
3
4
5
6
9b. For those staff working in your agency who needed to have knowledge about the
following list of health and safety topics, how well trained were they in each area in PY
2010? Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 –
moderately well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best
answer.
(1) Fire safety
(2) Indoor air quality
(3) Measures to increase security of housing unit
(4) Measures to reduce common household hazards
(5) Mold and mildew
(6) Lead
(7) Asbestos
(8) Vermiculite
(9) General crew safety
(10) Other health and safety
(11) Other (please specify
1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
9c. For categories receiving answers of (1)-not at all well trained, or (2)-not well trained to
the above questions, what were the barriers for receiving this training:
a. Funding
b. Time
c. Not a priority
d. Not available
e. Other___________________
254
10. For those staff working in your agency who needed to have knowledge about the
following list of diagnostic topics, how well trained were they in each area in PY 2010?
Please use the following scale: 1– not at all well trained; 2 – not well trained; 3 – moderately
well trained; 4 –well trained; 5 – very well trained; 6 – not applicable Circle best answer.
Pressure diagnostics:
Blower door (house air leakage rate)
1
Zonal pressure measurements
1
Room-to-room pressure measurements
1
Duct pressure pan measurements
1
Duct blower measurements (duct air leakage rate)
1
Space-heating system:
Flue gas analysis (steady-state efficiency measurements)1
Heat rise measurements
1
CO measurements in flues
1
Draft/spillage (normal operation)
1
Air-conditioning system:
Refrigerant charge (e.g., superheat, subcooling)
1
HVAC components and cross-cutting diagnostics:
Air handler flow rate
1
Thermostat anticipator current
1
Worst case draft/spillage (CAZ)
1
Hot-water (water-heating) system:
Flue gas analysis (steady-state efficiency measurements)1
CO measurements in flues
1
Draft/spillage (normal operation)
1
Water flow rates (showerheads and faucets)
1
Other CO measurements:
CO measurements in equipment rooms
1
Cooking stove
1
CO measurements in living areas
1
Other diagnostics and inspections:
Refrigerator energy use
1
Exhaust fan air flow rate measurement
1
Infrared scanning (camera)
1
Radon testing
1
Lead testing
1
Mold and mildew testing
1
Moisture context testing
1
Other (please specify) ____________________
1
2
2
2
2
2
3
3
3
3
3
4
4
4
4
4
5
5
5
5
5
6
6
6
6
6
2
2
2
2
3
3
3
3
4
4
4
4
5
5
5
5
6
6
6
6
2
3
4
5 6
2
2
2
3
3
3
4
4
4
5 6
5 6
5 6
2
2
2
2
3
3
3
3
4
4
4
4
5
5
5
5
2
2
2
3
3
3
4
4
4
5 6
5 6
5 6
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6
6
6
6
10a. For categories receiving answers of (1)-not at all well trained, or (2)-not well trained to
the above question, what were the barriers for receiving this training:
a. Funding
b. Time
255
c. Not a priority
d. Not available
e. Other___________________
11. Overall, how well trained were your agency‘s weatherization crews in PY 2010? (Check
best answer)
_____ Very well trained
_____ Well trained
_____ Neither well nor poorly trained
_____ Poorly trained
_____ Very poorly trained
12. What were the barriers that prevented your crews from receiving all the training they
need? (Check all that apply)
_____ Lack of training funds
_____ Cannot take crews out of the field long enough for training
_____ Training not available at the right times
_____ Training not available at the right places
_____ Available training is poor in quality
INSPECTION
1. Which of the following types of post-weatherization quality control inspection did your
agency perform on your weatherized dwelling units in Program Year 2010? Check all
that apply.
Visual inspection of installed measures
_____
Verification of insulation depths/quantities
_____
Verification of operation of measures installed
_____
Assessment of quality of measures installed
_____
Identification of needed measures that were not installed
_____
Blower door test
_____
Heating system efficiency test (flue gas analysis)
_____
Draft/spillage tests of heating systems
_____
Carbon monoxide (CO) monitoring
_____
Infrared scanning
_____
256
Identification of unresolved health and safety issues
_____
Discussion with occupants
_____
Other (specify) ______________________________________________
_____
2. Please indicate which types of post-weatherization quality control inspection listed below
were initiated since ARRA PY 2009. Check all that apply.
Visual inspection of installed measures
_____
Verification of insulation depths/quantities
_____
Verification of operation of measures installed
_____
Assessment of quality of measures installed
_____
Identification of needed measures that were not installed
_____
Blower door test
_____
Heating system efficiency test (flue gas analysis)
_____
Draft/spillage tests of heating systems
_____
Carbon monoxide (CO) monitoring
_____
Infrared scanning
_____
Identification of unresolved health and safety issues
_____
Discussion with occupants
_____
Other (specify) ______________________________________________
_____
3. Which of the following post-weatherization quality and control inspection topics listed
below were agency staff trained on in PY 2010? Check all that apply.
Visual inspection of installed measures
_____
Verification of insulation depths/quantities
_____
Verification of operation of measures installed
_____
257
Assessment of quality of measures installed
_____
Identification of needed measures that were not installed
_____
Blower door test
_____
Other diagnostic tests
_____
Identification of unresolved health and safety issues
_____
Discussion with occupants
_____
Other (specify) ______________________________________________
_____
4. Please indicate the cost, amount of training needed, amount of time needed and
effectiveness of the following types of post-weatherization quality control inspection
procedures relative to each other for PY 2010. Please use the following scale: 1 – very
low; 2 –low; 3 – medium; 4 – high; 5 – very high.
Cost
Training
Needed
Time
Needed
Effectiveness
Visual inspection of installed measures
Verification of insulation depths/quantities
Verification of operation of measures installed
Assessment of quality of measures installed
Identification of needed measures that were not
installed
Blower door test
Other diagnostic tests
Identification of unresolved health and safety
issues
Discussion with occupants
Other (please specify)
5. Approximately how many hours did it take to perform a typical post-weatherization
quality control inspection in Program Year 2010, by the major components listed below?
Scheduling
_____
Travel
_____
On-site work
_____
Post-inspection analysis and write-up
_____
258
Other
_____
TOTAL of all components
_____
6. Which of the following parties were involved in performing your agency‘s postweatherization quality control inspections in Program Year 2010? Check all that apply.
In-house manager
_____
In-house inspection specialist
_____
Contractor inspection specialist
_____
In-house weatherization crew chief
_____
Contractor weatherization crew chief
_____
In-house weatherization crew member
_____
Contractor weatherization crew member
_____
Other (please specify) ____________________________________________
6a. Which party was primarily responsible for post-weatherization quality control
inspections? Check best answer.
In-house manager
_____
In-house inspection specialist
_____
Contractor inspection specialist
_____
In-house weatherization crew chief
_____
Contractor weatherization crew chief
_____
In-house weatherization crew member
_____
Contractor weatherization crew member
_____
Other (please specify) ____________________________________________
7. About how many agency weatherization office staff went into the field to monitor local
weatherization agencies in your state in Program Year 2010? {Note: do not include people
who do quality assurance at the local agency level for the local agencies.}
259
Agency staff
_____
Agency contractors
_____
Other (please specify) _____________________________________
_____
8. What types of credentials or experience did your agency‘s post-weatherization quality
control inspectors have in Program Year 2010? Check all that apply.
Technical certification
_____
Extensive experience performing pre-weatherization audits
_____
Extensive experience performing weatherization work
_____
Extensive experience supervising weatherization work
_____
Construction experience
_____
Other (please specify) ___________________________________________
_____
9. Please indicate the level of experience for the agency staff engaged post-weatherization
quality control inspections in Program Year 2010 in each of the following functional areas:
Very
High
Average
Low
Very Low
High
Performing weatherization
work
Supervising weatherization
work
Working in construction
Performing preweatherization audits
10. For those dwelling units for which post-weatherization quality control inspections were
performed by your agency in Program Year 2010, typically how many days after
weatherization completion did the initial inspection take place? _____
11. In those cases where a Program Year 2010 post-weatherization quality control inspection
revealed a problem with the job performed, what action was most commonly taken in
response to that finding? Check one.
Sent original crew or contractor back to correct problem
_____
Sent different crew or contractor to correct problem
_____
260
Sent crew supervisor to correct problem
_____
Sent someone from state office to correct problem
_____
No action taken
_____
Other (please specify) ____________________________________________
_____
12. What other actions were taken in Program Year 2010 in response to the discovery of a
problem with the weatherization job performed? Check all that apply.
Sent original crew or contractor back to correct problem
_____
Sent different crew or contractor to correct problem
_____
Sent crew supervisor to correct problem
_____
Sent someone from state office to correct problem
_____
No action taken
_____
Other (please specify) ____________________________________________
_____
13. In Program Year 2010, how many of the dwelling units weatherized by your agency
required some additional work as a result of the findings of your post-weatherization quality
control inspections? _____
13a. Of those requiring some additional work, how many had work done that probably
resulted in more energy savings?_____
14. What were the three most common problems found in the dwelling units inspected by
your agency in Program Year 2010?
1) ___________________________________
2) ___________________________________
3) ___________________________________
15. In Program Year 2010, did your agency use findings from your post-weatherization
quality control inspections to provide feedback to your in-house or contractor crews on
workmanship or related issues? ______
16. To what extent does post-weatherization quality control inspection affect the quality of
future weatherization work?
(1) No extent
(2) Little extent
(3) Moderate extent
261
(4) Substantial extent
(5) Very substantial extent
17. Did the observation of problems with the quality of weatherization work lead to changes
in weatherization training for your staff?
(1) Yes
(2) No
17a. If Yes, what changes were made? ______________
18. Did your agency observe weatherization training sessions to help identify potential
problem areas for inspecting in the field (e.g., with respect to installation of measures that
trainees seem to have trouble understanding)?
(1) Yes
(2) No
18a. If Yes, briefly describe how your in-field inspection activities were affected by your
training session observations.
___________________________________________________________________________
___
262
OMB Control Number: XXXX-XXXX
APPENDIX F. DF2: HOUSING UNIT INFORMATION SURVEY
Thank you for your prompt response to this data request which is part of the ARRAperiod evaluation of the Weatherization Assistance Program. Evaluation results will
provide essential feedback to the weatherization community and inform policymakers
about the program's effects on clients' energy consumption, cost savings, and nonenergy benefits.
This data form collects detailed information about homes weatherized by your agency in
Program Year 2010. The information you supply will be used with billing history data to
better understand energy savings attributable to the Weatherization Assistance Program
under ARRA.
Please use this form (DF2) to provide information about any single family detached and
attached houses, mobile homes, or individual units within multi-family buildings. The
Building Information Survey (DF3) should be used to document information on small or
large multifamily buildings in which the whole building and all units in the building were
weatherized or are waitlisted. Refer to the definitions of each building type provided at the
end of the survey because these definitions are slightly different than those commonly used
within the Weatherization Assistance Program.
All of the information obtained from this survey will be protected and will remain
confidential. The data will be analyzed in such a way that the information provided cannot be
associated back to your state, your agencies, or the housing units and clients that your state
served.
Thank you in advance for completing this survey.
Public reporting burden for this collection of information is estimated to average twenty hours per
weatherization agency, including the time for reviewing instructions, searching existing data sources, gathering
and maintaining the data needed, and completing and reviewing the collection of information. Send comments
regarding this burden estimate or any other aspect of this collection of information, including suggestions for
reducing this burden, to Office of the Chief Information Officer, Records Management Division, IM-11,
Paperwork Reduction Project (___________), U.S. Department of Energy, 1000 Independence Ave SW,
Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork
Reduction Project (_______), Washington, DC 20503.
263
Form completed by: ______________________________ Date: _______________
IDENTIFICATION
[Q1-5 will be pre-completed by the evaluation team]
1. Agency name: ________________________________________
2. State: _______________
3. Agency job number: ____________________
4. Occupant name: ______________________________
5a. Site address: ______________________________ 5b. City:
_________________________
WEATHERIZATION INFORMATION
Weatherization dates (not audit or inspection dates):
6a. Started: __________ __________ __________
6b. Completed:__________ __________ __________
(month)
(day)
(year)
The start date is the first date that weatherization improvements were made to the home.
The weatherization start date is not the date the audit or home assessment was conducted
UNLESS energy efficiency improvements were made at the time of the audit. Client
education and low-cost measures such as light bulbs and showerheads ARE considered
energy efficiency improvements, and if any of those are implemented at the time of the
audit, then the start date is the audit date.
The end date is the last date that weatherization improvements were made to the home,
including any rework required after agency or state-level post-weatherization inspections.
The date of the post-inspection should NOT be used as the weatherization end date unless
the post-inspection was conducted on the last day that improvements were made to the
home and no rework was required.
7. Was this a ―reweatherized‖ unit? (check only one)
Yes
No
Don‘t know
Check “yes” if the home was weatherized prior to PY 2010.
264
8. Did this client file a complaint about the weatherization services you provided? (check
only one)
Yes
No
Don‘t know
HOUSING UNIT
9. Building type – see definitions at the end of the survey: (check only one)
Single-family detached house
Single-family attached house (e.g., side-by-side duplex, townhouse, row house)
Single-family – unknown whether attached or detached
Mobile home
Small multifamily building (2-4 units per building and not a SF attached house)
Large multifamily building (5 or more units per building and not a SF attached
house)
Shelter
Don‘t know
10. Number of stories above grade: (check only one)
1
2
3
or more
Not applicable
Please list the number of stories above ground-level. If there are half-stories, round up to
the nearest whole number. For example, please check “2” for a 1.5-story split-level house.
11. If single-family attached, number of units attached (adjacent) to this unit: (check only
one)
1
2
3
4 or more
Don‘t know
Not applicable
For single family attached homes, please list the number of separate housing units that
share at least one wall with this housing unit. For example, check “2” if housing unit is a
row house with homes on either side. Check “1” for a side-by-side duplex.
265
12. If mobile home, number of rooms that have been added on: (check only one)
None
1
2
3
4 or more
Don‘t know
Not applicable
13. If small or large multifamily building, number of units in the building: (check only one)
2
3
4
5-9
10-19
20-29
30-49
50-99
100 or more
Don‘t know
Not applicable
14. Year house/building originally built: (check only one)
2000 or later
1990 to 1999
1980 to 1989
1970 to 1979
1960 to 1969
1950 to 1959
1940 to 1949
1930 to 1939
1920 to 1929
1910 to 1919
1900 to 1909
Before 1900
Don‘t know
266
Conditioned floor area at the time of weatherization:
15a. Heated floor area: _________ ft²
Don‘t know
15b. Air conditioned floor area: __________ft²
Don‘t know
Include the basement only if it is intentionally conditioned (heated and/or cooled).
If you only know the total square footage of the home, please select “don’t know” rather
than listing the total square footage.
16. Primary fuel used to heat the unit during the winter before weatherization: (check only
one)
Electricity
Natural gas
Propane/LPG
Fuel oil
Wood
Other (specify: ____________________)
Don‘t know
17. Primary fuel used for water heating before weatherization: (check only one)
18. Type of primary space-heating system before weatherization: (check only one)
Central (ducted) warm-air furnace (forced-air or gravity, any fuel including
electricity)
Heat pump
Built-in electric units (e.g., electric baseboards, ceiling heat)
Steam or hot water system (e.g., floor or baseboard radiators, convectors)
Floor, wall, or pipeless (ductless) furnace (e.g., floor or wall furnace)
Room/space heater (nonportable)
Portable space heater
Cooking stove
None
Don‘t know
Select “steam or hot water system” for homes heated with boilers.
267
19. If small or large multifamily building, was the primary space-heating system shared with
other housing units? (check only one)
Yes
No
Don‘t know
Not applicable
20. Supplemental fuel(s) used to heat the unit during the winter before weatherization: (check
all that apply)
Electricity
Natural gas
Propane/LPG
Fuel oil
Wood
Other (specify: ____________________)
Don‘t know
21. Type of operable air conditioning system present before weatherization: (check all that
apply)
Central air conditioner/heat pump
Window/wall units
Evaporative cooling system (―swamp coolers‖)
None
Don‘t know
22. Number of window/wall air conditioning units: (check only one)
None
1
2
3
4 or more
Don‘t know
AUDIT
23. Primary method used to select weatherization measures for this house (excluding health,
safety, and repair measures and general heat waste measures): (check only one)
Priority list
Calculation procedure (e.g., spreadsheet, computerized audit)
Other (specify: ____________________ )
24. If a calculation procedure was used, the name of the procedure(s): (check all that apply)
AK Warm
268
EA-3
EASY
EA-QUIP
HomeCheck
Meadows
REES
REM/Rate
SMOC-ERS
TIPS
TREAT
Weatherization Assistant (NEAT/MHEA)
WXEOR
Other (specify: ____________________ )
Not applicable
DIAGNOSTICS AND INSPECTIONS
If you know when a diagnostic/inspection procedure was performed, please check the
appropriate box(es) in the first three response columns. If a diagnostic/inspection procedure
was performed but you do not know when, please check the box in the ―Performed?‖
column.
Diagnostic measurement or inspection
Diagnostic/inspection performed during:
Audit/house
Measure
Postassessment
installation
inspection
Pressure diagnostics:
25a. House air leakage (blower door measurement)
25b. Zonal pressure
25c. Room-to-room pressures (distribution system
balancing)
25d. Duct pressure pan measurements
25e. Duct blower measurement (duct air leakage rate)
25f. Blower door subtraction meas. (duct air leakage rate)
Space-heating system:
26a. Flue gas analysis (steady-state efficiency
measurement)
26b. Heat rise
26c. CO level in flue
269
Performed?
Diagnostic measurement or inspection
Diagnostic/inspection performed during:
Audit/house
Measure
Postassessment
installation
inspection
26d. CO level of equipment room
26e. Draft/spillage (normal operation)
26f. Worst case draft/spillage (CAZ)
26g. Safety inspection
Air-conditioning system:
27a. Refrigerant charge (e.g., superheat or subcooling)
27b. Safety inspection
HVAC components:
28a. Air handler flow rate
28b. Thermostat anticipator current
Hot-water (water-heating) system:
29a. Flue gas analysis (steady-state efficiency
measurement)
29b. CO level in flue
29c. CO level of equipment room
29d. Draft/spillage (normal operation)
29e. Worst case draft/spillage (CAZ)
29f. Hot water temperature
29g. Shower head flow rate
29h. Faucet flow rate
29i. Safety inspection
Other CO measurements:
30a. Cook stove
30b. Kitchen
30c. Main living area
Other diagnostics and inspections:
270
Performed?
Diagnostic measurement or inspection
Diagnostic/inspection performed during:
Audit/house
Measure
Postassessment
installation
inspection
Performed?
31a. Refrigerator energy use
31b. Exhaust fan air flow rate
31c. Infrared scanning (camera)
31d. Radon testing
31e. Other (specify: _____________________________ )
31f. Other (specify: _____________________________ )
31g. Other (specify: __ __________________________ )
Record the diagnostic measurements taken on THIS housing unit: (fill in all that were
taken)
For diagnostics that were performed multiple times, please provide the measurements that
are closest to the pre-weatherization and post-weatherization conditions of the home.
Diagnostic measurement
Preweatherization
Post
weatherization
House air leakage (blower door measurement):
32a. Air leakage rate
32b. House WRT outside pressure difference 39
cfm
cfm
Pa
Pa
Pa
Pa
Pa
Pa
Duct leakage (pressure pan measurements):
33a. Sum of pressure pan readings40
33b. Number of registers included in sum41
33c. House WRT outside pressure difference 42
Duct leakage (duct blower measurements) 43:
39
Report the pressure differential at which the blower door test was performed. A typical value is 50
Pascals. Do not report baseline pressure (typically less than 5 Pascals).
40
Total all of the individual measurements taken at registers in the home. The value for each register
should be between 0 and 50 Pascals.
41
Total the number of registers at which the test was performed.
42
Report the pressure differential at which the test was performed (from blower door). A typical
value is 50 Pascals.
43
If total duct leakage (inside the home and to the outside) was measured with a Duct BlasterTM or
271
34a. Total duct leakage rate
cfm
cfm
34b. Duct leakage to the outside
cfm
cfm
Pa
Pa
35a. Primary space-heating system
%
%
35b. Secondary space-heating system
%
%
35c. Hot water heater
%
%
34c. Duct WRT outside pressure difference 44
Steady-state efficiency (flue gas analysis):
MEASURES INSTALLED
If you know whether in-house crew or a contractor installed a given measure, please check
the appropriate box in the first two response columns. If a measure was installed but you do
not know whether it was installed by in-house crew or a contractor, please check the box in
the ―Installed?‖ column.
Measure
Installed by
In-house
Contractor
crew
Air sealing work:
36a. General house caulking and weatherstripping (e.g., doors,
windows)
36b. Air sealing emphasizing bypasses (leaks identified by auditor
and/or crew without using a blower door)
36c. Air sealing emphasizing bypasses (leaks identified by auditor
and/or crew with aid of a blower door)
36d. Air distribution system (duct) sealing or repair45
36e. Repairs to broken windows, doors, or other major holes in the
building shell
36f. Other non-window air sealing work (specify: ______________ )
36g. Other non-window air sealing work (specify: ______________ )
Insulation:
37a. Attic insulation
similar equipment, report results in 40a. If duct leakage to the outside was measured, report this
result in 40b. Most agencies will report results in “a” or “b,” but not both.
44
Report the house-to-outside pressure differential (from blower door) at which the leakage-tooutside test was performed. A typical value is 25 Pascals.
45
Check 42d if duct sealing or duct repair was performed. Check 46d if new ductwork was installed. Check 49c
if new vents, grills or registers were installed.
272
Installed?
Measure
Installed by
In-house
Contractor
crew
If attic insulation was installed, please provide quantity:
37b.____________square feet
or
37c.____________pounds
37d. What was the R value of attic insulation prior to weatherization?
_____ (Leave blank if unknown. Enter 0 if there was no existing insulation.)
37e. Wall insulation
If wall insulation was installed, please provide quantity:
37f.____________square feet
or
37g. .____________pounds
37h. Floor insulation46
37i. Rim or band joist insulation (sill box)
37j. Foundation wall insulation
37k. Duct insulation
37l. White roof coat applied to mobile home
37m. Mobile home skirting
37n. Mobile home belly insulation
37o. Other insulation (specify: _____________________________)
37p. Other insulation (specify: _____________________________)
Windows:
38a. New window (justified because cost effective)
38b. New window (justified for reason other than cost effectiveness)
38c. If new windows were installed, please provide quantity: _________
38d. Window glass repair or replacement not included under air
sealing major holes in building shell (42e)
38e. Repair of window sashes or frames
38f. Window screen repair/replacement
38g. Window lock replacement
38h. Storm window installed
38i. Window shading (e.g., awning, film, sun screen)
46
Exclude mobile home belly insulation, which should be listed under 43o.
273
Installed?
Measure
Installed by
In-house
Contractor
crew
38j. Other window treatments (specify: ______________________ )
38k. Other window treatments (specify: ______________________)
Doors:
39a. New door (justified because cost effective)
39b. New door (justified for reason other than cost effectiveness)
39c. Door lock (new or replacement)
39d. Door or door framing repair not included under air sealing major
holes in building shell (42e)
39e. Storm door installed
39f. Other door treatments (specify: ________________________ )
39g. Other door treatments (specify: ________________________ )
Central space heating systems (e.g., furnaces, boilers):47
40a. New heating system (justified because cost effective)
40b. New heating system (justified for reason other than cost
effectiveness)
40c. Space-heating system repair (e.g., controls, safety items, flues)
40d. New ductwork installed
40e. Space-heating system tune-up
40f. Vent damper installed
40g. Intermittent ignition device installed
40h. Other heating system modification (specify: _________ )48
40i. Other heating system modification (specify: __________ )
Air-conditioning systems:
41a. New air conditioner (justified because cost effective)
41b. New air conditioner (justified for reason other than cost
effectiveness)
41c. Air conditioner repair
47
Include central heating systems installed through programs other than WAP, such as emergency heating
system replacements funded by LIHEAP.
48
Check 42d if duct sealing or duct repair was performed. Check 46d if new ductwork was installed. Check 49c
if new vents, grills or registers were installed.
274
Installed?
Measure
Installed by
In-house
Contractor
crew
41d. Air conditioner recharge/tune-up
41e. Ceiling or whole-house fans
41f. Other air-conditioning system modification (specify: ________ )
41g. Other air-conditioning system modification (specify: ________ )
Ventilation:
42a. New bathroom exhaust fan installed
42b. New kitchen exhaust fan installed
42c. Repair to kitchen or bathroom exhaust fan (including ductwork)
42d. Whole-house ventilation system
42f. Other ventilation system improvements (specify: __________)
42g. Other ventilation system improvements (specify: __________)
HVAC accessories:
43a. New programmable (setback) thermostat
43b. New standard thermostat
43c. New duct vents, grills, or registers installed49
43d. Standard air filter installed
43e. High efficiency particulate arresting (HEPA) air filter installed
43f. Other HVAC accessories (specify: ______________________ )
43g. Other HVAC accessories (specify: ______________________ )
Water-heating system:
44a. New water heater (justified because cost effective)
44b. New water heater (justified for reason other than cost
effectiveness)
44c. Water-heating system repair
44d. Water-heater tank insulation wrap
44e. Pipe insulation
49
Check 36d if duct sealing OR duct repair was performed. Check 40d if new ductwork was installed.
275
Installed?
Measure
Installed by
In-house
Contractor
crew
44f. Installed low-flow showerhead
44g. Installed low-flow device on faucet (aerator)
44h. Water heater temperature reduction
44i. Other water heating system measure (specify: ______________ )
44j. Other water heating system measure (specify: ______________ )
Other baseloads:
45a. Indoor lighting (energy efficient bulb or fixture)
45b. Outdoor lighting (energy efficient bulb or fixture)
45c. Lighting (indoor/outdoor location not recorded)
45d. Refrigerator (justified because cost effective)
45e. Refrigerator (justified for reason other than cost effectiveness)
45f. If new refrigerator is installed, how many old refrigerators were
removed? ________________
45g. If new refrigerator is installed, how many stand-alone freezers
were removed? ______________
45h. Other baseload measure (specify: _______________________ )
45i. Other baseload measure (specify: _______________________ )
Health and safety and repair:
46a. Smoke alarm
46b. CO monitor
46c. Attic ventilation
46d. Roof repair
46e. Clothes dryer vent repair or replacement
46f. Ceiling repair
46g. Wall repair
46h. Floor repair
46i. Foundation repair
46j. Ground vapor barrier
46k. Gutter or downspout (installed or repaired)
276
Installed?
Measure
Installed by
In-house
Contractor
crew
46l. Plumbing repair
46m. Sewer repair
46n. Electrical repair
46o. Stair repair
46p. Install/repair non-skid material on stairs
46q. Install/repair safety gate at stairs
46r. Install/repair grab bar in bathroom
46s. Install/repair non-skid material in bathtub
46t. Install/repair metal chimney liner
46u. Lead abatement
46v. Asbestos abatement
46w. Removal or safe storage of household poisons
46x. Other health and safety/repair items (specify: _________ )
46y. Other health and safety/repair items (specify: ___________ )
Client education:
47a. Did the occupants receive an in-home visit in which energy
education was provided?
47b. Did the occupants participate in a classroom training in which
energy education was provided?
SERC AND WIPP MEASURES INSTALLED
48. Please indicate whether any additional measures were installed in this unit that were funded by the
Sustainable Energy Resources for Consumers (SERC) Program and/or Weatherization Innovation Pilot Program
(WIPP).
a. SERC funded measures were installed
b. WIPP funded measures were installed
c. Both SERC and WIPP funded measures were installed
d. The unit was not part of a SERC or WIPP grant (go to Q 61)
277
Installed?
If you know whether in-house crew or a contractor installed a given measure, please check
the appropriate box in the first two response columns. If a measure was installed but you do
not know whether it was installed by in-house crew or a contractor, please check the box in
the ―Installed?‖ column.
Measure
Installed by
In-house
crew
RENEWABLE ENERGY
49a.
S1.1 Solar PV
49b.
S1.2 PV: Shingles
49c. S1.3 Wind: Small-scale Residential
49d. S1.4 Passive Solar Panel
HOT WATER SYSTEMS
50a. S2.1 Solar HW
50b.
S2.2 Tankless/On-demand HW
50c. S2.3 Condensing HW
50d. S2.4 Heat Pump/Hybrid HW
50e. S2.5 Combination HW and Boiler
50f. S2.6 Other hot water
HVAC SYSTEMS
51a. S3.1 Heat Pumps: Geothermal/Ground-Source
51b. S3.2 Heat Pumps: Air
51c. S3.3 Heat Pumps: Mini Split System Ductless
51d. S3.4 Replacement of Improperly Sized HVAC Equipment
51e. S3.5 Solar Thermal (Home Heat)
51f. S3.6 Wood Pellet Stoves
51g. S3.7 Ultra Cooling Systems
51h. S3.8 Central AC Units
51i. S3.9 Window AC Units
51j. S3.10 Micro-combined Heat and Power
51k. S3.11 High-efficiency Furnaces
51l. S3.12 Heat Recovery Ventilators
51m. S3.13 Biomass Thermal Units Installed
51n. S3.14 Evaporative Cooling System
51o. S3.15 Vented Space Heating
51p. S3.16 Solar Powered Attic Ventilation
51q. S3.17 Energy Recovery Ventilator
ROOFING: COOL ROOF
52a. S4.1 Roofing: Cool Roof Technology Installed
278
Contractor
Installed?
APPLIANCES
53a. S5.1 Energy Star Clothes Washer
53b. S5.2 Energy-Efficient Clothes Dryer
53c. S5.3 Energy-Efficient Refrigerator
53d. S5.4 Appliance Energy Meters
INSULATION
54a. S6.1 Insulation: Aerogel/super
54b. S6.2 Insulation: Foam Injection Technology
54c. S6.3 Insulation: Masonry Foam
54d. S6.4 Insulation: Radiant Barrier Attic
54e. S6.5 Insulate: Spray Foam
54f. S6.6 Insulation: Reflective Attic Insulation
WHOLE-HOUSE RETROFIT
55a. S7.1 Centralized Building Controls
55b. S7.2 Deep Energy Retrofits
55c. S7.3 High-Performance Space Conditioning Retrofits
55d. S7.4 High-Performance Building Envelope Retrofits
55e. S7.5 Cold Energy Retrofits
55f. S7.6 Warm Energy Retrofits
55g. S7.7 Foundation Improvements
OUTREACH
56a. S8.1 Home Energy Saver Workshops
56b. S8.2 Households Touched by Behavioral Change Message
EQUIPMENT
57a. S9.1 Monitoring: In-Home Energy Monitors
OTHER
58a. S10.1 Units with Window Upgrades
58b. S10.2 Outdoor Solar Security Lighting
58c. S10.3 Ceiling Fans
58d. S10.4 LED Lights
58e. S10.5 Energy Star Doors
61. If a new space-heating system was installed, indicate the primary fuel used to heat the
unit during the winter after weatherization: (check only one)
Natural gas
Propane/LPG
Kerosene (#1 fuel oil)
Fuel oil (#2 fuel oil)
Electricity
Wood
279
Coal
Other (specify: ____________________)
Don‘t know
Not applicable
62. If a new space-heating system was installed, indicate the type of primary space-heating
system after weatherization: (check only one)
Central (ducted) warm-air furnace (forced-air or gravity, any fuel including
electricity)
Heat pump
Built-in electric units (e.g., electric baseboards, ceiling heat)
Steam or hot water system (e.g., floor or baseboard radiators, convectors)
Floor, wall, or pipeless (ductless) furnace (e.g., floor or wall furnace)
Room/space heater (nonportable)
Portable space heater
Cooking stove
None
__________)
Don‘t know
Not applicable
Select “steam or hot water system” for homes heated with boilers.
63. If a new space-heating system was installed and justified for reasons other than cost
effectiveness, identify the reason it was replaced: (check all that apply)
Cost of repair/retrofit exceeded 50% of replacement cost
Existing heating system was not running
Existing heating system was old (e.g., at end of life, too old to be repaired/adjusted)
To switch fuel
To convert from a steam system to a hot water system
Heat exchanger was cracked
Boiler was leaking
Safety switches/controls were not operational and could not be repaired
To replace unvented space heater(s)
Existing heating system was not safe to run for other reason (specify:
_____________)
Other (specify:
________________________________________________________)
280
64. Please identify any cost-effective energy-efficiency measures (not repair or health and
safety measures) recommended by your energy audit procedures that you were unable to
install in this housing unit because of insufficient funds: (check all that apply)
Air sealing
Duct sealing
Attic insulation
Wall insulation
Floor/foundation insulation
Duct insulation
New window(s)
Storm windows(s)
Door(s)
Storm door(s)
New space-heating system
Space-heating system tune-up
New air conditioner(s)
Air conditioner tune-up(s)
HVAC thermostat
New water heater
Water heater insulation wrap
Water flow devices (e.g., showerheads, faucet aerators)
Lighting
Refrigerator
Other: __________________________________________
None
This question only applies in states where there is a per-home spending limit. If there is
not a per-home spending limit in your state, check “none.”
65. If energy efficiency measures were checked in the previous question, provide a rough
estimate of the cost for installing all the measures checked: $_______________
66. Please identify any repair or health and safety measures recommended by your audit
procedures that you were unable to install in this housing unit because of insufficient funds:
(check all that apply)
New window(s)
Window glazing(s)
Window screen(s)
Window lock(s)
Window repair
New door(s)
Door lock(s)
Door repair
281
New space-heating system
Space-heating system repair
New air conditioner(s)
Air conditioner repair
Ceiling or whole-house fan(s)
Exhaust fan(s) or ventilation system
New water heater
Water-heating system repair
Refrigerator
Smoke alarm
CO monitor
Attic ventilation
Roof, wall, floor, or foundation repair
Plumbing/sewer repair
Electrical repair
Other: __________________________________________
None
This question only applies in states where there is a per-home spending limit. If there is
not a per-home spending limit in your state, check “none.”
67. If repair or health and safety measures were checked in the previous question, provide a
rough estimate of the cost for installing all the measures checked: $_______________
COSTS
68. Provide the total cost of weatherizing this housing unit. Include ALL sources of funding.
Do NOT include program management costs (e.g., intake, audits, final inspections or
program administration) or installation-related overhead costs (e.g., vehicles, equipment and
training).
69. Divide the total costs spent on this housing unit (from Question 68) into the categories
below.
69a. Material costs
69b. Labor costs
69c. Enter total job cost if above categories are not known
69d. Total (should match Q68 total)
[Auto-tally]
282
70. Divide the labor costs (from Question 69b) into the categories below. If labor costs for inhouse crew are not tracked at the job level please leave 70a blank.
70a. In house crew labor1
70b. Contractor labor
70c. Profit/overhead2
70d. Enter total labor costs if above categories are not known
70e. Total (should match Q68b total)
[Auto-tally]
1
Crew-based labor costs should be based on the crew‘s fully loaded hourly rate (rather than the crew‘s
take-home pay rate) which may include costs associated with medical and other insurance, workers
compensation, vacations, and other benefits. These labor costs should include the crew‘s time for
traveling to and from the job site.
2
If contractor profit and overhead are included in the contractor‘s material and labor costs, then leave 70b
blank.
71. Provide estimates of non-monetary contributions to this weatherization job.
71a. Volunteer Hours1
71b. Apprentice Hours2
71c. Estimated Value of Material In-Kind Contributions
71d. Estimated Value of Other In-Kind Contributions
1
An example of a volunteer is an unpaid person working on weatherizing a Habitat for Humanity Home.
An example of an apprentice would be a student whose program of education requires hands-on, real-life
work on weatherization jobs.
2
72. Divide the total costs spent on this housing unit (from Question 68) into the categories
below.
72a. Cost effective energy-related measures (SIR > 1.0)
72b. Health and safety and other non-cost effective measures
72c. Incidental repairs
72d. Enter total job cost if above categories are not known
72e. Total (should match Q68 total)
[Auto-tally]
73. Divide the total costs spent on this housing unit (from Question 68) into these funding
source categories below.
283
73a. DOE-Normal Appropriation/Formula WAP funds1
73b. DOE SERC Funds
73c. DOE WIPP Funds
73d. Non-DOE (leveraged) funds
73e. Total (should match Q68 total)
[Auto-tally]
1
This line includes ARRA funds for standard weatherization
jobs.
Energy Assistance Program (LI-EAP) funding should be considered Non-DOE funds if it
is tracked separately.
74. Provide the amounts spent on the major measure categories below.
74a. HVAC measures
74b. Water heating measures
74c. Replacement windows and doors
74d. All other building shell measures (insulation, air sealing,
etc.)
284
Housing Type Definitions
Single Family Detached – House that provides living space for one family or household, is
contained within walls that go from the basement (or the ground floor, if there is no basement) to
the roof, and has no walls that are shared (or built in contact) with another household. A
manufactured house assembled on site is a single family detached housing unit, not a mobile
home.
Single Family Attached – House that provides living space for one household, is contained within
walls that go from the basement (or the ground floor, if there is no basement) to the roof, has at
least one wall that is shared (or built in contact) with an adjacent household, and has an
independent outside entrance. An attached house does not have any other households living above
or below, and does not share basement or attic space with other housing units. Also, an attached
house does not share a heating or cooling system with any other housing units. Examples include
row houses, townhouses, condominiums and side-by-side duplexes that do not have shared attics,
basements or HVAC equipment.
Small Multi-family (2-4 units) – Building with two to four housing units (i.e., building that is
divided into living quarters for two, three, or four families or households) in which one household
lives above or beside another and does not meet the single family attached house definition.
Includes houses originally intended for occupancy by one family (or for some other use) that have
since been converted to separate dwellings for two to four families. Typical arrangements in these
types of living quarters are separate apartments downstairs and upstairs or one apartment on each
of three or four floors.
Large multifamily (5 or More Units per Building) – Building with five or more housing units (i.e.,
building that contains living quarters for five or more families or households) that does not meet
the single family attached house definition.
Mobile Home – Home that is built on a movable chassis, is moved to the site, and may be placed on
a permanent or temporary foundation. If rooms are added to the structure, it is considered a mobile
home if the added floor area is less than the mobile home’s original floor area; otherwise, it is a
single family detached house. A manufactured house assembled on site is a single family detached
house, not a mobile home.
Shelter - Structure whose principal purpose is to house individuals on a temporary basis who may
or may not be related to one another and who are not living in nursing homes, prisons, or similar
institutional care facilities.
285
OMB Control Number: XXXX-XXXX
APPENDIX G. DF3: BUILDING INFORMATION SURVEY
Thank you for your prompt response to this data request which is part of the ARRAperiod evaluation of the Weatherization Assistance Program. Evaluation results will
provide essential feedback to the weatherization community and inform policymakers
about the program's effects on clients' energy consumption, cost savings, and nonenergy benefits.
This survey collects detailed information about multifamily buildings weatherized by your
agency in Program Year 2010. The information you supply will be used with billing history
data to better understand energy savings attributable to the Weatherization Assistance
Program under ARRA.
Please use this form (DF3) to provide information about small or large multifamily buildings
in which improvements were made to the building shell, common areas, central HVAC or
domestic hot water systems. The Housing Unit Information Survey (DF2) should be used to
document information on weatherized single family detached and attached houses, mobile
homes, or individual units within multifamily buildings. Refer to the definitions of each
building type provided at the end of the survey because these definitions are slightly different
than those commonly used within the Weatherization Assistance Program.
All of the information obtained from this survey will be protected and will remain
confidential. The data will be analyzed in such a way that the information provided cannot be
associated back to your state, your agencies, or the housing units and clients that your state
served.
Thank you in advance for completing this survey.
Public reporting burden for this collection of information is estimated to average twenty hours per
weatherization agency, including the time for reviewing instructions, searching existing data sources, gathering
and maintaining the data needed, and completing and reviewing the collection of information. Send comments
regarding this burden estimate or any other aspect of this collection of information, including suggestions for
reducing this burden, to Office of the Chief Information Officer, Records Management Division, IM-11,
Paperwork Reduction Project (____), U.S. Department of Energy, 1000 Independence Ave SW, Washington,
DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork Reduction Project
(____), Washington, DC 20503.
286
Form completed by: ______________________________ Date: _______________
IDENTIFICATION
[Q1-6 will be pre-completed by the evaluation team]
1. Agency name: ________________________________________
2. State: _______________
3. Building ID number: ____________________
4. Building name: ______________________________
5. Site address: _______________________________
6. City: __________________________
287
WEATHERIZATION INFORMATION
Weatherization dates (not audit or inspection dates):
7a. Started: __________ __________ __________
7b. Completed:__________ __________ __________
(month)
(day)
(year)
The start date is the first date that weatherization improvements were made to the building.
The weatherization start date is not the date the audit or home assessment was conducted
UNLESS energy efficiency improvements were made at the time of the audit. Client
education and low-cost measures such as light bulbs and showerheads ARE considered
energy efficiency improvements, and if any of those are implemented at the time of the
audit, then the start date is the audit date.
The end date is the last date that weatherization improvements were made to the building,
including any rework required after agency or state-level post-weatherization inspections.
The date of the post-inspection should NOT be used as the weatherization end date unless
the post-inspection was conducted on the last day that improvements were made to the
building and no rework was required.
8. Was this a ―reweatherized‖ building? (check only one)
Yes
No
Don‘t know
Check “yes” if the building was previously weatherized in a prior program year.
9. Does the building meet your state‘s definition for being a high residential energy user?
(check only one)
Yes
No
No state definition in place
Don‘t know
10. Did the building owner or any occupants of housing units within the building file a
complaint about the weatherization services you provided? (check only one)
Yes
No
Don‘t know
288
BUILDING INFORMATION
11a. Building type – see definitions at end of the survey: (check only one)
Small multifamily building (2-4 units and not a single family attached house)
Large multifamily building (5 or more units and not a single family attached house)
Don‘t know
11b. If this is a large multi-family building, was HUD‘s list of pre-qualified buildings used
to income qualify the building:
a. Yes
b. No
c. Don‘t know
11.c If this is a large multi-family building, please indicate which description best describes
its ownership:
a. private owner
b. private owner but HUD assisted
c. Publically owned
d. Condominium owned by occupants
e. Other ____________
f. Don‘t know
12. Number of housing units in the building: __________
13. Number of housing units in the building that met WAP eligibility requirements: ___
14. Number of stories above grade: (check only one)
1
2
3
4
5-9
10-19
20 or more
Don‘t know
Please list the number of stories above ground-level. If there are half-stories, round up to
the nearest whole number.
15. Year building originally built: (check only one)
2000 or later
1990 to 1999
1980 to 1989
1970 to 1979
289
1960 to 1969
1950 to 1959
1940 to 1949
1930 to 1939
1920 to 1929
1910 to 1919
1900 to 1909
Before 1900
Don‘t know
290
Conditioned floor area at the time of weatherization:
16a. Heated floor area: _________ ft²
Don‘t know
16b. Air conditioned floor area: __________ft²
Don‘t know
Include the basement or common space only if it is intentionally conditioned (heated
and/or cooled).
If you only know the total square footage of the building, please select “don’t know”
rather than listing the total square footage.
17. Primary fuel used to heat the building during the winter before weatherization: (check
only one)
Natural gas
Propane/LPG
Kerosene (#1 fuel oil)
Fuel oil #2
Fuel oil #4
Fuel oil #6
Electricity
Steam (purchased from a central distribution system)
Hot water (purchased from a central distribution system)
Other (specify: ____________________)
Don‘t know
18. Primary fuel used for water heating before weatherization: (check only one)
Natural gas
Propane/LPG
Electricity
Other (specify: ____________________)
Don‘t know
291
19. Type of primary space-heating system before weatherization: (check only one)
Central (ducted) warm-air furnace (forced-air or gravity, any fuel including
electricity)
Heat pump
Built-in electric units (e.g., electric baseboards, ceiling heat)
Steam or hot water system (e.g., floor or baseboard radiators, convectors)
Floor, wall, or pipeless (ductless) furnace (e.g., floor or wall furnace)
Room/space heater (nonportable)
Portable space heater
Cooking stove
None
Don‘t know
Select “steam or hot water system” for buildings heated with boilers.
20. Was the primary space-heating system a central system? (check only one)
Yes, a central system that supplied heat to all or most of the units in the building
No, each unit had its own heating system
Don‘t know
21. Supplemental fuel(s) used to heat the building during the winter before weatherization:
(check all that apply)
Natural gas
Propane/LPG
Kerosene (#1 fuel oil)
Fuel oil #2
Fuel oil #4
Fuel oil #6
Electricity
Steam (purchased from a central distribution system)
Hot water (purchased from a central distribution system)
Other (specify: ____________________)
Don‘t know
22. Type of operable air conditioning system present before weatherization: (check all that
apply)
Central air conditioner/heat pump
Window/wall units
Evaporative cooling system (―swamp coolers‖)
None
Don‘t know
23. Number of window/wall air conditioning units: (check only one)
292
None
1-4
5-9
10-19
20-49
50 or more
Don‘t know
AUDIT
24. Primary method used to select weatherization measures for this building (excluding
health, safety, and repair measures and general heat waste measures): (check only one)
Priority list
Calculation procedure (e.g., spreadsheet, computerized audit)
Other (specify: ____________________ )
25. If a calculation procedure was used, the name of the procedure(s): (check all that apply)
AK Warm
EA-3
EASY
EA-QUIP
HomeCheck
Meadows
REES
REM/Rate
SMOC-ERS
TIPS
TREAT
Weatherization Assistant (NEAT/MHEA)
WXEOR
Other (specify: ____________________ )
Not applicable
293
DIAGNOSTICS AND INSPECTIONS
If you know when a diagnostic/inspection procedure was performed, please check the
appropriate box(es) in the first three response columns. If a diagnostic/inspection procedure
was performed but you do not know when, please check the box in the ―Performed?‖
column.
If a diagnostic/inspection procedure was performed in ANY of the housing units in the
building please check the appropriate category.
Diagnostic measurement or inspection
Diagnostic/inspection performed during:
Audit/house
Measure
PostPerformed
assessment
installation
inspection
?
Pressure diagnostics:
26a. Unit-level blower door measurement (air leakage
rate for individual dwelling units)
26b. Building-level blower door measurement (total air
leakage rate for the whole building)
26c. Zonal pressure
26d. Room-to-room pressures (distribution system
balancing)
26e. Duct pressure pan measurements
26f. Duct blower measurement (duct air leakage rate)
26g. Blower door subtraction meas. (duct air leakage rate)
Space-heating system:
27a. Flue gas analysis (steady-state efficiency
measurement)
27b. Heat rise
27c. CO level in flue
27d. CO level of equipment room
Space-heating system (continued):
27e. Draft/spillage (normal operation)
27f. Worst case draft/spillage (CAZ)
27g. Safety inspection
Air-conditioning system:
28a. Refrigerant charge (e.g., superheat or subcooling)
294
Diagnostic measurement or inspection
Diagnostic/inspection performed during:
Audit/house
Measure
PostPerformed
assessment
installation
inspection
?
28b. Safety inspection
HVAC components:
29a. Air handler flow rate
29b. Thermostat anticipator current
Hot-water (water-heating) system:
30a. Flue gas analysis (steady-state efficiency
measurement)
30b. CO level in flue
30c. CO level of equipment room
30d. Draft/spillage (normal operation)
30e. Worst case draft/spillage (CAZ)
30f. Hot water temperature
30g. Shower head flow rate
30h. Faucet flow rate
30i. Safety inspection
Other CO measurements:
31a. Cook stove
31b. Kitchen
31c. Main living area
Other diagnostics and inspections:
32a. Refrigerator energy use
32b. Exhaust fan air flow rate
32c. Infrared scanning (camera)
32d. Radon testing
32e. Other (specify: _____________________________ )
32f. Other (specify: _____________________________ )
295
Diagnostic measurement or inspection
Diagnostic/inspection performed during:
Audit/house
Measure
PostPerformed
assessment
installation
inspection
?
32g. Other (specify: ____________________________ )
296
Record the diagnostic measurements taken on THIS building: (fill in all that were taken)
For diagnostics that were performed multiple times, please provide the measurements that
are closest to the pre-weatherization and post-weatherization conditions of the building.
Diagnostic measurement
Preweatherization
Post
weatherization
Building air leakage (blower door measurement): 50
33a. Average air leakage rate per unit based on unit-level
cfm
cfm
33b. Total air leakage rate of the building based on whole
building test
cfm
cfm
Pa
Pa
Pa
Pa
Pa
Pa
35a. Total duct leakage rate
cfm
cfm
35b. Duct leakage to the outside
cfm
cfm
Pa
Pa
testing
33c. House WRT outside pressure difference 51
Duct leakage (pressure pan measurements): 52
34a. Sum of pressure pan readings53
34b. Number of registers included in sum54
34c. House WRT outside pressure difference 55
Duct leakage (duct blower measurements) 56:
35c. Duct WRT outside pressure difference 57
50
Most agencies will report results in ―a‖ or ―b,‖ but not both.
Report the pressure differential at which the blower door test was performed. A typical value is 50 Pascals.
Do not report baseline pressure (typically less than 5 Pascals).
52
If building has more than one duct system, average the results across all systems that were tested.
53
Total all of the individual measurements taken at registers in the building. The value for each register should
be between 0 and 50 Pascals.
54
Total the number of registers at which the test was performed.
55
Report the pressure differential at which the test was performed (from blower door). A typical value is 50
Pascals.
56
If building has more than one duct system, average the results across all systems that were tested. If total duct
leakage (inside the building and to the outside) was measured with a Duct BlasterTM or similar equipment,
report results in 35a. If duct leakage to the outside was measured, report this result in 35b. Most agencies will
report results in ―a‖ or ―b,‖ but not both.
57
Report the house-to-outside pressure differential (from blower door) at which the leakage-to-outside test was
performed. A typical value is 25 Pascals.
51
297
Steady-state efficiency (flue gas analysis):58
36a. Primary space-heating system
%
%
36b. Secondary space-heating system
%
%
36c. Hot water heater
%
%
MEASURES INSTALLED
If you know whether in-house crew or a contractor installed a given measure, please check
the appropriate box in the first two response columns. If a measure was installed but you do
not know whether it was installed by in-house crew or a contractor, please check the box in
the ―Installed?‖ column.
If a measure was installed in ANY of the housing units in the building please check the
appropriate category.
Measure
Installed by
In-house
Contractor
crew
Air sealing work:
37a. General house caulking and weatherstripping (e.g., doors,
windows)
37b. House air sealing emphasizing bypasses (leaks identified by
auditor and/or crew without using a blower door)
37c. House air sealing emphasizing bypasses (leaks identified by
auditor and/or crew with aid of a blower door)
37d. Air distribution system (duct) sealing and repair59
37e. Repairs to broken windows, doors, or other major holes in the
building shell
37f. Other air sealing work (specify: ______________ )
37g. Other air sealing work (specify: ______________ )
58
If test was performed on multiple space- or water-heating systems, provide the average result across all
systems that were tested.
59
Check 37d if duct sealing OR duct repair was performed. Check 41e if NEW ductwork was installed. Check
44c if new vents, grills or registers were installed.
298
Installed?
Measure
Installed by
In-house
Contractor
crew
Insulation:
38a. Attic insulation
If attic insulation was installed, please provide quantity:
38b.____________square feet
or
38c.____________pounds
38d. What was the R value of attic insulation prior to weatherization?
_____ (Leave blank if unknown. Enter 0 if there was no existing insulation.)
38e. Wall insulation
If wall insulation was installed, please provide quantity:
38f.____________square feet
or
37g. .____________pounds
38h. Floor insulation
38i. Rim or band joist insulation (sill box)
38j. Foundation wall insulation
38k. Duct insulation
38l. White roof coat
38m. Other insulation (specify: ____________________________ )
38n. Other insulation (specify: _____________________________ )
Windows:
39a. New window (justified because cost effective)
39b. New window (justified for reason other than cost effectiveness)
39c. If new windows were installed, please provide quantity: _________
39d. Window glass repair or replacement not included under air
sealing major holes in building shell (37e)
39e. Repair of window sashes or frames
39f. Window screen repair/replacement
39g. Window lock replacement
39g. Other window repair (e.g., sashes, frames)
39h. Storm window
39i. Window shading (e.g., awning, film, sun screen)
39j. Other window treatments (specify: _______________________)
299
Installed?
Measure
Installed by
In-house
Contractor
crew
39k. Other window treatments (specify: ______________________)
Doors:
40a. New door (justified because cost effective)
40b. New door (justified for reason other than cost effectiveness)
40c. Door lock (new or replacement)
40d. Door or door framing repair not included under air sealing major
holes in building shell (37e)
40e. Storm door installed
40f. Other door treatments (specify: _________________________)
40g. Other door treatments (specify: _________________________)
Central space heating systems (e.g., furnaces, boilers):60
41a. New heating system (justified because cost effective)
41b. New heating system (justified for reason other than cost
effectiveness)
41c. Heating system repair (e.g., controls, safety items, flues)
41d. Space-heating system tune-up
41e. New ductwork installed
41f. Vent damper
41g. Intermittent ignition device
41h. Other space-heating system modification (specify: _________)61
41i. Other space-heating system modification (specify: __________)
Air-conditioning systems:
42a. New air conditioner (justified because cost effective)
42b. New air conditioner (justified for reason other than cost
effectiveness)
42c. Air conditioner repair
60
Include central heating systems installed through programs other than WAP, such as emergency heating
system replacements funded by LIHEAP.
61
Check 37d if duct sealing OR duct repair was performed. Check 41e if NEW ductwork was installed. Check
44c if new vents, grills or registers were installed.
300
Installed?
Measure
Installed by
In-house
Contractor
crew
42d. Air conditioner recharge/tune-up
42e. Ceiling or whole-house fans
42f. Other air-conditioning system modification (specify: ________ )
42g. Other air-conditioning system modification (specify: ________ )
Ventilation:
43a. New bathroom exhaust fan installed
43b. New kitchen exhaust fan installed
43c. Repair to kitchen or bathroom exhaust fan (including ductwork)
43d. Whole-house ventilation system
43e. Other ventilation system improvements (specify: __________)
43f. Other ventilation system improvements (specify: ___________)
HVAC accessories:
44a. New programmable (setback) thermostat
44b. New standard thermostat
44c. Duct vents, grills, or registers62
44d. Standard air filter
44e. High efficiency particulate arresting (HEPA) air filter
44f. Other HVAC accessories (specify: _______________________)
44g. Other HVAC accessories (specify: ______________________)
Water-heating system:
45a. New water heater (justified because cost effective)
45b. New water heater (justified for reason other than cost
effectiveness)
45c. Water-heating system repair
45d. Water-heater tank insulation wrap
62
Check 37d if duct sealing OR duct repair was performed. Check 41e if new ductwork was installed. Check
44c if new vents, grills or registers were installed.
301
Installed?
Measure
Installed by
In-house
Contractor
crew
45e. Pipe insulation
45f. Installed low-flow showerhead
45g. Installed low-flow device on faucet (aerator)
45h. Water heater temperature reduction
45i. Other water heating system measure (specify: ______________ )
45j. Other water heating system measure (specify: ______________ )
Other baseloads:
46a. Indoor lighting (energy efficient bulb or fixture)
46b. Outdoor lighting (energy efficient bulb or fixture)
46c. Lighting (indoor/outdoor location not recorded)
46d. Refrigerator (justified because cost effective)
46e. Refrigerator (justified for reason other than cost effectiveness)
46f. If new refrigerator is installed, how many old refrigerators were
removed? ________________
46g. If new refrigerator is installed, how many old refrigerators were
removed? ________________
46h. Other baseload measure (specify: ______________________ )
46i. Other baseload measure (specify: _______________________ )
Health and safety and repair:
47a. Smoke alarm
47b. CO monitor
47c. Attic ventilation
47d. Clothes dryer vent repair or replacement
47e. Roof repair
47f. Ceiling repair
47g. Wall repair
47h. Floor repair
47i. Foundation repair
47j. Ground vapor barrier
302
Installed?
Measure
Installed by
In-house
Contractor
crew
47k. Gutter or downspout (installed or repaired)
47l. Plumbing repair
47m. Sewer repair
47n. Electrical repair
47o. Stair repair
47p. Install/repair non-skid material on stairs
47q. Install/repair safety gate at stairs
47r. Install/repair grab bar in bathroom
47s. Install/repair non-skid material in bathtub
47t. Install/repair metal chimney liner
47u. Lead abatement
47v. Asbestos abatement
47w. Removal or safe storage of household poisons
47x. Other health & safety or repair items (specify: _____________)
47y. Other health & safety or repair items (specify: _____________)
Client education:
48a. Did the occupants receive an in-home visit in which energy
education was provided?
48b. Did the occupants participate in a classroom training in which
energy education was provided?
303
Installed?
SERC AND WIPP MEASURES INSTALLED
49. Please indicate whether any additional measures were installed in this building that were
funded by the Sustainable Energy Resources for Consumers (SERC) Program and/or
Weatherization Innovation Pilot Program (WIPP).
SERC funded measures were installed
building was not part of a SERC or WIPP grant (skip to Q60)
If you know whether in-house crew or a contractor installed a given measure, please check
the appropriate box in the first two response columns. If a measure was installed but you do
not know whether it was installed by in-house crew or a contractor, please check the box in
the ―Installed?‖ column.
Measure
Installed by
In-house
crew
RENEWABLE ENERGY
50a.
S1.1 Solar PV
50b.
S1.2 PV: Shingles
50c. S1.3 Wind: Small-scale Residential
50d. S1.4 Passive Solar Panel
HOT WATER SYSTEMS
51a. S2.1 Solar HW
51b.
S2.2 Tankless/On-demand HW
51c. S2.3 Condensing HW
51d. S2.4 Heat Pump/Hybrid HW
51e. S2.5 Combination HW and Boiler
51f. S2.6 Other hot water
HVAC SYSTEMS
52a. S3.1 Heat Pumps: Geothermal/Ground-Source
52b. S3.2 Heat Pumps: Air
52c. S3.3 Heat Pumps: Mini Split System Ductless
52d. S3.4 Replacement of Improperly Sized HVAC Equipment
52e. S3.5 Solar Thermal (Home Heat)
52f. S3.6 Wood Pellet Stoves
52g. S3.7 Ultra Cooling Systems
52h. S3.8 Central AC Units
52i. S3.9 Window AC Units
52j. S3.10 Micro-combined Heat and Power
52k. S3.11 High-efficiency Furnaces
52l. S3.12 Heat Recovery Ventilators
304
Contractor
Installed?
52m. S3.13 Biomass Thermal Units Installed
52n. S3.14 Evaporative Cooling System
52o. S3.15 Vented Space Heating
52p. S3.16 Solar Powered Attic Ventilation
52q. S3.17 Energy Recovery Ventilator
ROOFING: COOL ROOF
53a. S4.1 Roofing: Cool Roof Technology Installed
APPLIANCES
54a. S5.1 Energy Star Clothes Washer
54b. S5.2 Energy-Efficient Clothes Dryer
54c. S5.3 Energy-Efficient Refrigerator
54d. S5.4 Appliance Energy Meters
INSULATION
55a. S6.1 Insulation: Aerogel/super
55b. S6.2 Insulation: Foam Injection Technology
55c. S6.3 Insulation: Masonry Foam
55d. S6.4 Insulation: Radiant Barrier Attic
55e. S6.5 Insulate: Spray Foam
55f. S6.6 Insulation: Reflective Attic Insulation
WHOLE-HOUSE RETROFIT
56a. S7.1 Centralized Building Controls
56b. S7.2 Deep Energy Retrofits
56c. S7.3 High-Performance Space Conditioning Retrofits
56d. S7.4 High-Performance Building Envelope Retrofits
56e. S7.5 Cold Energy Retrofits
56f. S7.6 Warm Energy Retrofits
56g. S7.7 Foundation Improvements
OUTREACH
57a. S8.1 Home Energy Saver Workshops
57b. S8.2 Households Touched by Behavioral Change Message
EQUIPMENT
58a. S9.1 Monitoring: In-Home Energy Monitors
OTHER
59a. S10.1 Units with Window Upgrades
59b. S10.2 Outdoor Solar Security Lighting
59c. S10.3 Ceiling Fans
59d. S10.4 LED Lights
59e. S10.5 Energy Star Doors
305
60. If a new space-heating system was installed, indicate the primary fuel used to heat the
building during the winter after weatherization: (check only one)
Natural gas
Propane/LPG
Kerosene (#1 fuel oil)
Fuel oil #2
Fuel oil #4
Fuel oil #6
Electricity
Steam (purchased from a central distribution system)
Hot water (purchased from a central distribution system)
Other (specify: ____________________)
Don‘t know
61. If a new space-heating system was installed, indicate the type of primary space-heating
system after weatherization: (check only one)
Central (ducted) warm-air furnace (forced-air or gravity, any fuel including
electricity)
Heat pump
Built-in electric units (e.g., electric baseboards, ceiling heat)
Steam or hot water system (e.g., floor or baseboard radiators, convectors)
Floor, wall, or pipeless (ductless) furnace (e.g., floor or wall furnace)
Room/space heater (nonportable)
Portable space heater
Cooking stove
None
Don‘t know
Not applicable
Select “steam or hot water system” for buildings heated with boilers.
62. If a new space-heating system was installed and justified for reasons other than cost
effectiveness, identify the reason it was replaced: (check all that apply)
Cost of repair/retrofit exceeded 50% of replacement cost
Existing heating system was not running
Existing heating system was old (e.g., at end of life, too old to be repaired/adjusted)
To switch fuel
To convert from a steam system to a hot water system
Heat exchanger was cracked
Boiler was leaking
Safety switches/controls were not operational and could not be repaired
To replace unvented space heater(s)
306
Existing heating system was not safe to run for other reason (specify:
_____________)
Other (specify:
________________________________________________________)
63. Please identify any cost-effective energy-efficiency measures (not repair or health and
safety measures) recommended by your audit procedures that you were unable to install in
this housing unit because of insufficient funds: (check all that apply)
Air sealing
Duct sealing
Attic insulation
Wall insulation
Floor/foundation insulation
Duct insulation
New window(s)
Storm windows(s)
Door(s)
Storm door(s)
New space-heating system
Space-heating system tune-up
New air conditioner(s)
Air conditioner tune-up(s)
HVAC thermostat
New water heater
Water heater insulation wrap
Water flow devices (e.g., showerheads, faucet aerators)
Lighting
Refrigerator
Other: __________________________________________
None
This question only applies in states where there is a per-building spending limit. If there is
not a per-building spending limit in your state, check “none.”
64. If energy efficiency measures were checked in the previous question, provide a rough
estimate of the cost for installing all the measures checked: $_______________
65. Please identify any repair or health and safety measures recommended by your audit
procedures that you were unable to install in this building because of insufficient funds:
(check all that apply)
New window(s)
Window glazing(s)
Window screen(s)
Window lock(s)
307
Window repair
New door(s)
Door lock(s)
Door repair
New space-heating system
Space-heating system repair
New air conditioner(s)
Air conditioner repair
Ceiling or whole-house fan(s)
Exhaust fan(s) or ventilation system
New water heater(s)
Water-heating system repair
Refrigerator(s)
Smoke alarm(s)
CO monitor(s)
Attic ventilation
Roof, wall, floor, or foundation repair
Plumbing/sewer repair
Electrical repair
Other: __________________________________________
None
This question only applies in states where there is a per-building spending limit. If there is
not a per-building spending limit in your state, check “none.”
66. If repair or health and safety measures were checked in the previous question, provide a
rough estimate of the cost for installing all the measures checked: $_______________
308
COSTS
67. Provide the total cost of weatherizing this multifamily building. Include ALL sources of
funding. Do NOT include program management costs (e.g., intake, audits, final inspections
or program administration) or installation-related overhead costs (e.g., vehicles, equipment
and training).
68. Divide the total costs spent on this building (from Question 67) into the categories below.
68a. Material costs
68b. Labor costs
68c. Enter total cost if above categories are not known
68d. Total (should match Q67 total)
[Auto-tally]
69. Divide the labor costs (from Question 68b) into the categories below. If labor costs for inhouse crew are not tracked at the building level please leave 69a blank.
69a. In house crew labor1
69b. Contractor labor
69c. Profit/overhead2
69d. Enter total labor costs if above categories are not known
69e. Total (should match Q68b total)
[Auto-tally]
1
Crew-based labor costs should be based on the crew‘s fully loaded hourly rate (rather than the crew‘s
take-home pay rate) which may include costs associated with medical and other insurance, workers
compensation, vacations, and other benefits. These labor costs should include the crew‘s time for
traveling to and from the job site.
2
If contractor profit and overhead are included in the contractor‘s material and labor costs, then leave 69c
blank.
309
70. Provide estimates of non-monetary contributions to this weatherization job.
70a. Volunteer hours1
70b. Apprentice hours2
70c. Estimated value of material in-kind contributions
70d. Estimated value of other in-kind contributions
1
An example of a volunteer is an unpaid person working on weatherizing a Habitat for Humanity Home.
An example of an apprentice would be a student whose program of education requires hands-on, real-life
work on weatherization jobs.
2
71. Divide the total costs spent on this building (from Question 67) into the categories below.
71a. Cost effective energy-related measures (SIR > 1.0)
71b. Health and safety and other non-cost effective measures
71c. Incidental repairs
71d. Enter total job cost if above categories are not known
70e. Total (should match Q67 total)
[Auto-tally]
72. Divide the total costs spent on this housing unit (from Question 67) into these funding
source categories below.
72a. DOE normal appropriation/formula WAP funds1
72b. DOE SERC funds
72c. DOE WIPP funds
72d. Non-DOE (leveraged) funds
72e. Total (should match Q67 total)
1
[Auto-tally]
This line includes ARRA funds for standard weatherization jobs.
Energy Assistance Program (LI-EAP) funding should be considered Non-DOE funds if it
is tracked separately.
310
73. Provide the amounts spent on the major measure categories below.
73a. HVAC measures
73b. Water heating measures
73c. Replacement windows and doors
73d. All other building shell measures (insulation, air sealing,
etc.)
311
Housing Type Definitions
Single Family Detached – House that provides living space for one family or household, is
contained within walls that go from the basement (or the ground floor, if there is no basement) to
the roof, and has no walls that are shared (or built in contact) with another household. A
manufactured house assembled on site is a single family detached housing unit, not a mobile
home.
Single Family Attached – House that provides living space for one household, is contained within
walls that go from the basement (or the ground floor, if there is no basement) to the roof, has at
least one wall that is shared (or built in contact) with an adjacent household, and has an
independent outside entrance. An attached house does not have any other households living above
or below, and does not share basement or attic space with other housing units. Also, an attached
house does not share a heating or cooling system with any other housing units. Examples include
row houses, townhouses, condominiums and side-by-side duplexes that do not have shared attics,
basements or HVAC equipment.
Small Multifamily (2-4 units) – Building with two to four housing units (i.e., building that is
divided into living quarters for two, three, or four families or households) in which one household
lives above or beside another and does not meet the single family attached house definition.
Includes houses originally intended for occupancy by one family (or for some other use) that have
since been converted to separate dwellings for two to four families. Typical arrangements in these
types of living quarters are separate apartments downstairs and upstairs or one apartment on each
of three or four floors.
Large multifamily (5 or More Units per Building) – Building with five or more housing units (i.e.,
building that contains living quarters for five or more families or households) that does not meet
the single family attached house definition.
Mobile Home – Home that is built on a movable chassis, is moved to the site, and may be placed on
a permanent or temporary foundation. If rooms are added to the structure, it is considered a mobile
home if the added floor area is less than the mobile home’s original floor area; otherwise, it is a
single family detached house. A manufactured house assembled on site is a single family detached
house, not a mobile home.
Shelter - Structure whose principal purpose is to house individuals on a temporary basis who may
or may not be related to one another and who are not living in nursing homes, prisons, or similar
institutional care facilities.
312
313
OMB Control Number: XXXX-XXXX
APPENDIX H. DF4: ELECTRIC & NATURAL GAS BILLING INFORMATION
FROM AGENCIES DATA FORM
Utility Information Survey Part A
OMB control number XXXX-XXXX
DF4a
Introduction
Thank you for your prompt response to this data request which is part of the ARRA-period
evaluation of the Weatherization Assistance Program. Evaluation results will provide essential
feedback to the weatherization community and inform policymakers about the program's
effects on clients' energy consumption, cost savings, and non-energy benefits.
This data form collects detailed information about homes weatherized by your agency in Program
Years 2009 and 2010. The data you supply will be used to characterize the program and collect utility
billing histories pre- and post-weatherization to better understand energy savings attributable to the
Weatherization Assistance Program during the ARRA period.
All of the information obtained from this survey and from utilities will be protected and will remain
confidential. The data will be analyzed in such a way that the information provided cannot be
associated back to your state, your agencies, or the housing units and clients that your state served.
Instructions
Part A of the DF4 survey requests a list of the housing units and large multifamily buildings
your agency weatherized in PY 2009* and PY 2010†.
The evaluation team will use the information provided in Part A to randomly select a sample of
housing units and large multifamily buildings weatherized by your agency. After the sample is
determined, you will be asked to complete Part B by providing detailed information about
sampled units: housing type, primary heating fuel, occupant demographics and utility account
information.
*For the purposes of this survey, Program Year 2009 is the WAP/ARRA funding year that includes
the heating season spanning late 2009/early 2010.
†For the purposes of this survey, Program Year 2010 is the WAP/ARRA funding year that includes
the heating season spanning late 2010/early 2011.
Public reporting burden for this collection of information is estimated to average 10 hours per
response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send
comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Office of the Chief Information Officer, Records
Management Division, IM-11, Paperwork Reduction Project (_______), U.S. Department of Energy,
1000 Independence Ave SW, Washington, DC, 20585-1290; and to the Office of Management and
314
Budget (OMB), OIRA, Paperwork Reduction Project (_____), Washington, DC 20503.
Utility Information Survey Part A
OMB control number
DF4a
To be completed by the evaluation team:
Agency name
Agency ID
Contact name
Section 1: List of weatherized housing units
Please list all housing units that were weatherized during PY2009 and PY2010 using funds from
DOE63. Include all housing types. Do not include wait-listed or in-progress jobs.
Program
Housing unit
Building type
Primary heating
year
unique ID
SFA – Singe family attached
fuel
(e.g. job
SFD – Single family detached
EL – electricity
number)
SFU – Single family unknown
NG – natural
attached/detached
gas
MH – Mobile home
LP – propane
SH – Shelter
FO – fuel oil
SMF – Small multifamily (2-4
WO – wood
units)
OT – other
LMF – Large multifamily (5+
units)s
63
Include all units that meet the definition of a ―DOE Unit:‖ A DOE unit is a dwelling on which a DOEapproved energy audit or priority list has been applied and weatherization work has been completed. As funds
allow, the DOE measures installed on this unit have a Savings-to-Investment Ratio (SIR) of 1.0 or greater, but
also may include any necessary energy-related health and safety measures. The use of DOE funds on this unit
may include, but are not limited to auditing, testing, measure installation, inspection, or use of DOE equipment
and/or vehicles, or if DOE provides the training and/or administrative funds. Therefore, a dwelling unit that
meets both the definition of a DOE weatherized unit and has DOE funds used directly on it must be counted as
a DOE unit.
315
Utility Information Survey Part A
OMB control number XXXX-XXXX
DF4a
Section 2: List of large multifamily buildings
[Section 2 will only appear if the respondent selects ―LMF‖ for at least one housing unit in Section 1.]
Please list all large multifamily buildings (5+ housing units) that were weatherized during PY2009
and PY2010 using DOE funds in which improvements were made to the building shell, common
areas, central HVAC or domestic hot water systems. Do NOT include buildings where individual
tenant units were weatherized but no improvements were made to the building shell, common areas,
central HVAC or domestic hot water systems. Do not include wait-listed or in-progress jobs.
Program year
Building unique ID
(building-level
identification number)
Primary heating fuel
EL – electricity
NG – natural gas
LP – propane
FO – fuel oil
WO – wood
OT – other
316
Utility Information Survey Part B
OMB control number XXXX-XXXX
DF4b
Introduction
Thank you for your prompt response to this data request which is part of the ARRA-period
evaluation of the Weatherization Assistance Program. Evaluation results will provide essential
feedback to the weatherization community and inform policymakers about the program's
effects on clients' energy consumption, cost savings, and non-energy benefits.
This data form collects detailed information about homes weatherized by your agency in Program
Years 2009 and 2010. The data you supply will be used to characterize the program and collect utility
billing histories pre- and post-weatherization to better understand energy savings attributable to the
Weatherization Assistance Program during the ARRA period.
All of the information obtained from this survey and from utilities will be protected and will remain
confidential. The data will be analyzed in such a way that the information provided cannot be
associated back to your state, your agencies, or the housing units and clients that your state served.
Instructions
In Part B of the DF4 survey we request detailed information about a randomly-selected sample
of housing units that your agency weatherized in PY 2009* and PY 2010†. Information
requested in Part B includes housing type, primary heating fuel, occupant demographics and
utility account information.
*For the purposes of this survey, Program Year 2009 is the WAP/ARRA funding year that includes
the heating season spanning late 2009/early 2010.
†For the purposes of this survey, Program Year 2010 is the WAP/ARRA funding year that includes
the heating season spanning late 2010/early 2011.
Public reporting burden for this collection of information is estimated to average 20 hours per
response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. Send
comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Office of the Chief Information Officer, Records
Management Division, IM-11, Paperwork Reduction Project (_______), U.S. Department of Energy,
1000 Independence Ave SW, Washington, DC, 20585-1290; and to the Office of Management and
Budget (OMB), OIRA, Paperwork Reduction Project (_____), Washington, DC 20503.
317
Utility Information Survey Part B
OMB control number XXXX-XXXX
DF4b
To be completed by the evaluation team:
Agency name
Agency ID
Contact name
Section 1: Information about sampled housing units
Unique ID for housing unit
(e.g. job number)
Head of household name
(first & last)
Street address
Apt #
Number of
household
occupants
Number of
elderly
(60 or older)
[Pre-populated from Part A data]
City
Occupant demographics
Number of
Number of Native
disabled
American
Zip code
Number of
children (as
defined by state)
Household annual income
Does the housing unit meet your state's
definition of being a high residential energy
user?
□ Yes
□ No
□ Don‘t know
Ownership
□ Owner (paid in full or mortgaged)
□ Renter
□ Occupied without payment
□ Don‘t know
Does the housing unit meet your state's
definition of having a high energy burden?
□ Yes
□ No
□ Don‘t know
Race/ethnicity of head of household
□ American Indian or Alaska Native
□ Asian
□ Black or African American
□ Native Hawaiian or other Pacific Islander
□ White
□ Hispanic or Latino
□ Don‘t know
Is household headed by a single parent?
□ Yes
□ No
□ Don‘t know
318
Utility account information
For units in large multifamily buildings, only list accounts that are paid by the tenant. Do not list utility
accounts that serve multiple dwelling units, common areas, or the entire building. Building-level utility
account information will be requested in Section 2.
[Pre-populated from Part A data]
Housing type
Primary heating fuel
[Pre-populated from Part A data]
Electric utility name
Natural gas utility
name64
Weatherization start
date
Weatherization end date
Electric utility account
number
Natural gas utility
account number65
I certify that the agency has a signed utility billing data release form on
file for this client.
64
□
Check to certify
If primary heating fuel is not natural gas provided by a utility, please leave this cell blank. Do NOT list
propane or fuel oil providers.
65
If primary heating fuel is not natural gas provided by a utility, please leave this cell blank. Do NOT list
propane or fuel oil account numbers.
319
Section 2: Information about sampled large multifamily buildings (5+ units)
[Agencies will see a series of screens, each one requesting the following information about a sampled
large multifamily building.]
[Pre-populate from Section 1]
Building unique ID
Number of housing units in
building
Street Address
Number of weatherized
housing units
City
Does this building have one or more electric or natural gas utility meters that provide
service to multiple dwelling units or common areas in the building? For example,
answer ―yes‖ if the building has a natural gas meter that provides heat for the whole
building or an electric meter that serves hallways and common areas.
Utility account information for building-level meters
Electric account number 1
Electric account number 2
Electric account number 3
Electric account number 4
Electric account number 5
Natural gas account number 1
Natural gas account number 2
Natural gas account number 3
Natural gas account number 4
Natural gas account number 5
Do you have signed utility billing data release forms on file for any of these mastermetered accounts?
320
Zip code
□
□
□
Yes
No
Don‘t know
Check if this meter
serves multiple
buildings
□
□
□
□
□
□
□
□
□
□
□ Yes
□ No
□ Don‘t know
OMB Control Number: XXXX-XXXX
APPENDIX I: DF5A - NATIONAL WEATHERIZATION ASSISTANCE PROGRAM
EVALUATION HOUSEHOLD ELECTRICITY USAGE FORM
This data is being collected to assist in the evaluation of energy savings attributable to the U.S.
Department of Energy‘s Weatherization Assistance Program.
Public reporting burden for this collection of information is estimated to average twenty-four hours
per response, including the time for reviewing instructions, searching existing data sources, gathering
and maintaining the data needed, and completing and reviewing the collection of information. Send
comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Office of the Chief Information Officer, Records
Management Division, IM-11, Paperwork Reduction Project (XXXX-XXXX), U.S. Department of
Energy, 1000 Independence Ave SW, Washington, DC, 20585-1290; and to the Office of
Management and Budget (OMB), OIRA, Paperwork Reduction Project (XXXX-XXXX),
Washington, DC 20503.
All of the information obtained from this data form will be protected and will remain confidential.
The data will be analyzed in such a way that the information provided cannot be associated back to
your utility or the housing units and buildings that your utility served.
STEP 1 – Review List of Customer Accounts
This form is accompanied by a list of households that have voluntarily participated in the
Weatherization Assistance Program (WAP). These program participants have indicated that your
company furnishes electric services to their household and have signed an Authorization Form that
allows you to release data to ORNL for purposes of program evaluation. The list of customer
accounts includes the following information:
Field
Purpose
Notes
Utility Company ID
Internal reference number
Developed by Evaluation Team
WAP Project ID
Internal reference number
Developed by Evaluation Team
Customer Account Number
Identifies record to be
extracted
Collected by local agency from
client at the time of service delivery
Customer Last Name
Last name of program
participant
Customer First Name
First name of program
participant
Service Address (Line 1)
Service address
Service Address (Line 2)
Supplemental address
information
City
City
State
State
ZIP
ZIP
321
Individual who signed the utility
data Authorization Form
Address at which the weatherization
services were delivered
Please review this list of customers and confirm that the listed account numbers are consistent with
your records.
STEP 2 – Prepare a Data File with Customer Usage and Charge Information
Please furnish an electronic data file with monthly electric usage and charge information for the
period from XXXX through XXXX. The extract should furnish one record for each month for each
customer. The required information includes:
Field
Purpose
Notes
Utility Company ID
Internal reference number
Developed by Evaluation Team
WAP Project ID
Internal reference number
Developed by Evaluation Team
Customer Account
Number
Identifies record to be
extracted
Collected by local agency from
client at the time of service delivery
Customer Last Name
Last name of program
participant
Individual who signed the utility
data Authorization Form
Customer First Name
First name of program
participant
Service Address (Line 1)
Service address
Service Address (Line 2)
Supplemental address
information
City
City
State
State
ZIP
ZIP
Meter Read Date
Date of the read
MM/DD/YYYY or MM/DD/YY
Days in Billing Period
Allows computation of
start date
XXX
Meter Reading Code
Allows assessment of
data quality
A = Actual
E = Estimated
P = Phone / postcard customer read
C = Corrected
F = Final
Usage Amount (kWh)
Furnishes consumption
amount
Reported in kWh
Usage Charge ($$$.cc)
Furnishes charge amount
Report usage charge only (exclude
service charge and charges for other
services)
Address at which the weatherization
services were delivered
We are interested in the monthly electricity consumption for each housing unit, even if a change of
occupancy has occurred. We will review the monthly customer name field to identify a change of
occupancy
322
STEP 3 – Return Data File via CD or FTP
Your case manager will furnish instructions for delivery of the data file. The data can be
delivered on a password protected CD or via a secure FTP site.
HOUSEHOLD ELECTRIC USAGE FORM EXAMPLE
Utility
Data
Request
ID
Project
Housing
Unit ID
Account
Number
Last
Name
First
Name
Service
Address
Service
Address
Line 2
323
City
State
ZIP
Code
Meter Read
Date
(mm/dd/yy)
Days
in
Billing
Cycle
MeterRead
Code
A/E/P/C/F
Usage
(kWh)
Charge
($$$.cc)
NATIONAL WEATHERIZATION ASSISTANCE PROGRAM EVALUATION
HOUSEHOLD ELECTRICITY USAGE FORM
FREQUENTLY ASKED QUESTIONS
What is the purpose of the National WAP Evaluation?
We are evaluating the performance of the U.S. Department of Energy‘s Weatherization Assistance
Program, which is a program that installs energy efficiency measures in the homes of low-income
clients. A primary component of the evaluation is to determine the electricity savings achieved by the
Program in housing units heated primarily by electricity or natural gas.
What is the purpose of this form?
These data are being collected to estimate change in electricity use in homes weatherized by the
Weatherization Assistance Program. The data you supply will be used in statistical procedures to
estimate electricity savings.
How do I know this is a valid U.S. Government survey?
All U.S. Government surveys are required to be reviewed by the U.S. Office of Management and
Budget (OMB). An OMB approved survey will have a valid OMB number and expiration date on the
data collection form. You will find the OMB approval number and expiration date at the top righthand corner of this form. In addition, if you wish to contact someone at the Oak Ridge National
Laboratory to verify that this is a valid survey, call Bruce Tonn at 865-574-4041 or you can email him
at [email protected].
What data are to be reported and in what format?
The monthly billing data should be provided to us in electronic format. At a minimum for each
month, the billing information should identify the account number, the customer name, the service
address, the date the meter was read, the meter reading or the consumption, any code associated with
the reading (e.g., estimated value), and the kWh charge. See the attached form for an example of how
we would like the data formatted. We are interested in the monthly energy consumptions of each
housing unit, even if a change of occupancy has occurred.
How do I know that this information request does not violate my customer‘s privacy?
We have obtained fuel release forms for all the account numbers shown above. Please contact us if
you need to see copies of these signed forms.
In addition, the information that you provide will be protected and will remain confidential. When
analysis results are reported, energy use and savings will not be associated with the housing units,
account numbers, or the names of the clients in any way.
324
Who is conducting the survey?
The sponsor of this survey, the Energy Efficiency and Renewable Energy Branch of the U.S.
Department of Energy, has contracted with Oak Ridge National Laboratory (ORNL) to conduct this
evaluation. ORNL has subcontracted with APPRISE Incorporated and their partner the Energy
Center of Wisconsin (ECW) to collect the information for this evaluation. You will return your
completed forms on CD to ECW or will upload your data file to the secure website identified by
ECW.
How long will it take to complete this form?
Public reporting burden for this collection of information is estimated to average 24 hours per
response, including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information.
How may I report these data? What format can I use?
We are furnishing a data file with account number, name, and service address. By merging these data
will your records, you will be able to develop an electronic data file with the required information.
You should send this data file by mail to ECW on a password protected CD or upload this data file to
the secure FTP site furnished by ECW. Under special circumstances, you can furnish data in other
formats. Please consult with your case manager at ECW to discuss other options.
325
326
OMB Control Number: XXXX-XXXX
APPENDIX J. DF5B: NATIONAL WEATHERIZATION ASSISTANCE PROGRAM
EVALUATION HOUSEHOLD NATURAL GAS USAGE FORM
This data is being collected to assist in the evaluation of energy savings attributable to the U.S.
Department of Energy‘s Weatherization Assistance Program.
Public reporting burden for this collection of information is estimated to average twenty-four hours
per response, including the time for reviewing instructions, searching existing data sources, gathering
and maintaining the data needed, and completing and reviewing the collection of information. Send
comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to Office of the Chief Information Officer, Records
Management Division, IM-11, Paperwork Reduction Project (XXXX-XXXX), U.S. Department of
Energy, 1000 Independence Ave SW, Washington, DC, 20585-1290; and to the Office of
Management and Budget (OMB), OIRA, Paperwork Reduction Project (XXXX-XXXX),
Washington, DC 20503.
All of the information obtained from this data form will be protected and will remain confidential.
The data will be analyzed in such a way that the information provided cannot be associated back to
your utility or the housing units and buildings that your utility served.
STEP 1 – Review List of Customer Accounts
This form is accompanied by a list of households that have voluntarily participated in the
Weatherization Assistance Program (WAP). These program participants have indicated that your
company furnishes natural gas services to their household and have signed an Authorization Form
that allows you to release data to ORNL for purposes of program evaluation. The list of customer
accounts includes the following information:
Field
Purpose
Notes
Utility Company ID
Internal reference number
Developed by Evaluation Team
WAP Project ID
Internal reference number
Developed by Evaluation Team
Customer Account Number
Identifies record to be extracted
Collected by local agency from client at the time
of service delivery
Customer Last Name
Last name of program participant
Customer First Name
First name of program participant
Individual who signed the utility data
Authorization Form
Service Address (Line 1)
Service address
Service Address (Line 2)
Supplemental address information
City
City
State
State
ZIP
ZIP
Address at which the weatherization services
were delivered
Please review this list of customers and confirm that the listed account numbers are consistent with
your records.
STEP 2 – Prepare a Data File with Customer Usage and Charge Information
327
Please furnish an electronic data file with monthly natural gas usage and charge information for the
period from XXXX through XXXX. The extract should furnish one record for each month for each
customer. The required information includes:
Field
Purpose
Notes
Utility Company ID
Internal reference number
Developed by Evaluation Team
WAP Project ID
Internal reference number
Developed by Evaluation Team
Customer Account Number
Identifies record to be extracted
Collected by local agency from client at the time
of service delivery
Customer Last Name
Last name of program participant
Customer First Name
First name of program participant
Individual who signed the utility data
Authorization Form
Service Address (Line 1)
Service address
Service Address (Line 2)
Supplemental address information
City
City
State
State
ZIP
ZIP
Meter Read Date
Date of the read
MM/DD/YYYY or MM/DD/YY
Days in Billing Period
Allows computation of start date
XXX
Meter Reading Code
Allows assessment of data quality
A = Actual
Address at which the weatherization services
were delivered
E = Estimated
P = Phone / postcard customer read
C = Corrected
F = Final
Usage Amount
Furnishes consumption amount
Report units in next column
Units
Furnishes units
C = CCF
M = MCF
T = Therms
D = dTherms
O = Other
Usage Charge ($$$.cc)
Furnishes charge amount
Report usage charge only (exclude service charge
and charges for other services)
We are interested in the monthly natural gas consumption of each housing unit, even if a change of
occupancy has occurred. We will review the monthly customer name data to identify a change of
occupancy
STEP 3 – Return Data File via CD or FTP
Your case manager will furnish instructions for delivery of the data file. The data can be delivered on
a password protected CD or via a secure FTP site.
328
NATURAL GAS USAGE DATA FORM EXAMPLE
Utility
Data
Request
ID
Project
Housing
Unit ID
Account
Number
Last
Name
First
Name
Service
Address
Service
Address
Line 2
City
329
State
ZIP
Code
Meter
Read
Date
(mm/dd/
yy)
Days
in
Billing
Cycle
MeterRead
Code
A/P/E/
C/F
Usage
(report
units
in next
colum
n)
Units
C/M/
T/D/
O
Charge
($$$.cc)
NATIONAL WEATHERIZATION ASSISTANCE PROGRAM EVALUATION
HOUSEHOLD NATURAL GAS USAGE FORM
FREQUENTLY ASKED QUESTIONS
What is the purpose of the National WAP Evaluation?
We are evaluating the performance of the U.S. Department of Energy‘s Weatherization Assistance
Program, which is a program that installs energy efficiency measures in the homes of low-income clients.
A primary component of the evaluation is to determine the natural gas savings achieved by the Program
in housing units heated primarily by natural gas.
What is the purpose of this form?
These data are being collected to estimate change in natural gas use in homes weatherized by the
Weatherization Assistance Program. The data you supply will be used in statistical procedures to
estimate natural gas savings.
How do I know this is a valid U.S. Government survey?
All U.S. Government surveys are required to be reviewed by the U.S. Office of Management and Budget
(OMB). An OMB approved survey will have a valid OMB number and expiration date on the data
collection form. You will find the OMB approval number and expiration date at the top right-hand corner
of this form. In addition, if you wish to contact someone at the Oak Ridge National Laboratory to verify
that this is a valid survey, call Bruce Tonn at 865-574-4041 or you can email him at [email protected].
What data are to be reported and in what format?
The monthly billing data should be provided to us in electronic format. At a minimum for each month, the
billing information should identify the account number, the customer name, the service address, the date
the meter was read, the meter reading or the consumption, the consumption units, any code associated
with the reading (e.g., estimated value), and the usage charge. See the attached form for an example of
how we would like the data formatted. We are interested in the monthly energy consumptions of each
housing unit, even if a change of occupancy has occurred.
How do I know that this information request does not violate my customer‘s privacy?
We have obtained fuel release forms for all the account numbers shown above. Please contact us if you
need to see copies of these signed forms.
In addition, the information that you provide will be protected and will remain confidential. When
analysis results are reported, energy use and savings will not be associated with the housing units, account
numbers, or the names of the clients in any way.
Who is conducting the survey?
The sponsor of this survey, the Energy Efficiency and Renewable Energy Branch of the U.S. Department
of Energy, has contracted with Oak Ridge National Laboratory (ORNL) to conduct this evaluation.
ORNL has subcontracted with APPRISE Incorporated and their partner the Energy Center of Wisconsin
(ECW) to collect the information for this evaluation. You will return your completed forms on CD to
ECW or will upload your data file to the secure website identified by ECW.
330
How long will it take to complete this form?
Public reporting burden for this collection of information is estimated to average 24 hours per response,
including the time for reviewing instructions, searching existing data sources, gathering and maintaining
the data needed, and completing and reviewing the collection of information.
How may I report these data? What format can I use?
We are furnishing a data file with account number, name, and service address. By merging these data
will your records, you will be able to develop an electronic data file with the required information. You
should send this data file by mail to ECW on a password protected CD or upload this data file to the
secure FTP site furnished by ECW. Under special circumstances, you can furnish data in other formats.
Please consult with your case manager at ECW to discuss other options.
331
332
OMB Control Number: XXXX-XXXX
APPENDIX K. S4: OCCUPANT SURVEY
This data is being collected to conduct a survey of occupants about their experiences with the
Weatherization Assistance Program and benefits they believe they have received from the program. The
data you supply will be used to describe occupant satisfaction with the program as well as changes in
energy education and non-energy benefits provided by the program.
Public reporting burden for this collection of information is estimated to average two hours per response,
including the time for reviewing instructions, searching existing data sources, gathering and maintaining
the data needed, and completing and reviewing the collection of information. Send comments regarding
this burden estimate or any other aspect of this collection of information, including suggestions for
reducing this burden, to Office of the Chief Information Officer, Records Management Division, IM-11,
Paperwork Reduction Project (XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW,
Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork
Reduction Project (XXXX-XXXX), Washington, DC 20503.
Lastly, all of the information obtained from this survey will be protected and will remain confidential.
The data will be analyzed in such a way that the information provided cannot be associated back to you or
your household. Your answers will not be shared with or reported back to anyone within the agency that
served you or your state.
PRE-WEATHERIZATION SURVEY
{INTERVIEWER: STATE PRIOR TO PRE-WEATHERIZATION SURVEY ONLY}I will need to ask
these questions of the adult in the household most involved with the weatherization of the home or the
head of the household. Am I speaking to the right person?
POST-WEATHERIZATION SURVEY
{INTERVIEWER: VERIFY RESPONDENT IS THE SAME RESPONDENT FROM THE PREWEATHERIZATION SURVEY}Before we begin, I need to verify that this is the same person who
completed this survey before your home received weatherization services. Am I speaking to the same
person?
PLEASE RECORD RESPONDENT‘S NAME, GENDER and AGE
Name
Gender
Main Respondent:
1. How long have you lived in your current home?
{If less than one year} Enter: _________ months
Enter: _________ years
333
Age
1a. {IF RESPONDENT HAS LIVED IN THE HOME < ONE YEAR} Has ANY OTHER ADULT living
in the household lived in the home for more than one year?
(1) Yes
(2) No
{IF YES} Name ______________________
{IF NO: STOP SURVEY}
PART I. Energy Consumption, Non-Energy Impacts, Health and Demographics
2. Are you currently…?
(1) Married
(2) Divorced
(3) Widowed
(4) Separated
(5) Never married
(6) A member of an unmarried couple
(7) Refused
3. What is the highest degree or level of school you have completed?
(1) No Schooling Completed
(2) Kindergarten to grade 12 (No Diploma)
(3) High school diploma or GED
(4) Some college, no degree
(5) Associate‘s degree (for example: AA, AS)
(6) Bachelor‘s degree (for example: BA, BS)
(7) Master‘s degree (for example: MA, MS, MBA)
(8) Professional degree (for example: MD, JD)
(9) Doctorate degree (for example: PhD, EdD)
(10) Refused
4. Do you consider yourself to be of Hispanic or Latino
origin, such as Mexican, Puerto Rican, Cuban, or other Spanish background?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
5. Which describes your race? You can select one or more categories.
(1) White
(2) Black or African-American
(3) American Indian or Alaska Native
(4) Asian
(5) Native Hawaiian or Other Pacific Islander
(6) Other (if volunteered)
(7) Hispanic or Latino (if volunteered)
(8) Refused
334
5a. {IF MORE THAN ONE} Which ONE of these groups best represents your
race? You can select one or more categories.
(1) White
(2) Black or African-American
(3) American Indian or Alaska Native
(4) Asian
(5) Native Hawaiian or Other Pacific Islander
(6) Other (if volunteered)
(7) Hispanic or Latino (if volunteered)
(8) Refused
6. Were you born a citizen of the United States or did you become a citizen of the United States through
naturalization?
(1) Born
(2) Naturalized
(3) Neither
(4) Refused
In this next section, I will be asking you about your home and your use of energy.
7. Do you rent or own your current residence?
(1) Rent
(2) Own
(3) Neither (Please describe the housing agreement).
________________________________________________________________________
8. Which of the following best describes the location of your home? Do you live in a city, a
town, the suburbs, or in a rural area?
(1) City
(2) Town
(3) Suburbs
(4) Rural
(5) Don‘t Know/Not Sure
(6) Refused
9. In the past 12 months has anyone in your household owned or had the regular use of any cars,
trucks, vans, sports-utility-vehicles or similar vehicles? DO NOT INCLUDE MOTORCYCLES
OR MOPEDS.
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
INTERVIEWER: ―REGULAR USE‖ MEANS THE VEHCILE IS KEPT AT HOME AND IS
AVAILABLE FOR SOME PERSONAL USE.
335
10. Thinking of the area where you live, in the past 12 months, have members of your household had
regular access to public transportation? {PROBE: buses, trolley buses, trains, trams, rapid transit
(metro/subway/underground), water taxi/ferries, free transportation offered by community services
agencies, Medicaid covered transportation…)
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
11. Which of the following do you believe best describes your current home? Is it a… (INTERVIEWER:
DEFINE EACH HOUSING TYPE IF NEEDED)
(1) Single-family detached house
(2) a Single-family attached house,
(3) an Apartment building with 2-4 units,
(4) an Apartment building with 5 or more units, or
(5) a Mobile home?
12. How many bedrooms do you have in your home? [Include bedrooms in finished attics or
finished basements.] ______________
13. Now think about other rooms in your home besides bedrooms and bathrooms. Not including
unfinished areas, hallways, and closets, how many other rooms are there in your home?
__________________
16. Is your home heated during the winter?
(1) Yes (SKIP to Q17)
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
16a. {IF NO}You have just told me that you don‘t heat your home during the winter. Just to clarify, is it
that you have heating equipment but don‘t use it, or does your home just not have any heating equipment?
(1) Have equipment, but don‘t use it (SKIP to Q8)
(2) Don‘t have any heating equipment
(3) Don‘t Know/Not Sure
(4) Refused
17. Last winter, did you heat all # (sum from Q12 ad Q13) rooms?
(1) Yes (SKIP to Q18)
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
17a. How many of those rooms were not heated last winter?
Enter the number ___________
18. Is any air conditioning equipment used in your home?
(5) Yes (SKIP to Q19)
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
336
18a. (IF NO) Just to clarify, do you have air conditioning equipment but don‘t use it, or does your home
just not have any air conditioning equipment?
(1) Have equipment, but don‘t use it (SKIP to Q 20)
(2) Don‘t have any air conditioning equipment
(3) Don‘t Know/Not Sure
(4) Refused
19. Last summer, did you air condition all # (sum Q12 and Q13) rooms?
(1) Yes (SKIP to Q20)
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
19a.How many of those rooms were not cooled last summer?
Enter the number ___________
20. Is any part of your home over a crawl space with exposed dirt as the floor?
(1) Yes
(2) No
(3) Don‘t Know/Not sure
(4) Refused
21. Is any part of your home over a basement?
(1) Yes
(2) No (SKIP TO 22)
(3) Don‘t Know/Not Sure
(4) Refused
21a. Do you use your basement for living space? That is, do you use it for work, play or sleep?
(1) Yes
(2) No (SKIP to Q22)
(3) Don‘t Know/Not Sure
(4) Refused
21b. Is the basement warm enough to be used as a living space in the winter?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
21c. Is the basement cool enough to be used as a living space in the summer?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
337
22. An attic is an area directly below the roof, accessible by stairs, with space for you to stand
upright and easily move about. Does your home have an attic?
(1) Yes
(2) No (SKIP to Q23)
(3) Don‘t Know/Not Sure
(4) Refused
22a. Do you use the attic for living space? That is, do you use it for work, play or sleep?
(1) Yes
(2) No (SKIP to Q23)
(3) Don‘t Know/Not Sure
(4) Refused
22b. Is the attic warm enough to be used as a living space in the winter?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
(5)
22c. Is the attic cool enough to be used as a living space in the summer?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
23. Does your home have a garage that is attached to or part of your home?
(1) Yes
(2) No (SKIP to 24)
(3) Don‘t Know/Not Sure
(4) Refused
23a. Do you warm up your vehicle in your garage?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
23b. Have you or anyone else living in your home observed the smell of vehicle exhaust inside your
home?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
I have some questions about heating your home.
338
24.{IF NO TO HEAT FROM PREVIOUS SECTION SKIP TO COOLING}Let‘s start with the
main source of heating in your home. Please tell me which type of heating equipment
provides most of the heat for your home. Remember to include portable heaters, fireplaces,
heating stoves and cooking stoves.
(1) Heat pump
(2) Central furnace with ducts to individual rooms
(3) Steam/Hot water system with radiators or pipes in each room
(4) Built-in electric units in each room installed in walls, ceilings, baseboards, or floors
(5) Built-in floor/wall pipeless furnace
(6) Built-in room heater burning gas, oil, or kerosene
(7) Heating stove burning wood, coal, or coke
(8) Portable heaters
(9) Fireplace
(10) Cooking stove used to heat your home as well as to cook
(11) Some other equipment (Specify __________________)
25. Does the main heating equipment for your home also heat any other apartments, condos, households,
businesses, or farm buildings?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
26. What is the main fuel used for heating your home? That is, which fuel is the one that
provides the most heat for your home?
(1) Electricity
(2) Natural gas from underground pipes
(3) Propane (bottled gas)
(4) Fuel oil
(5) Kerosene
(6) Wood
(7) Biomass
(8) Solar or Wind
(9) Geothermal
(10) District steam
(11) Some other fuel (Specify __________)
27. You told me that [EQUIPM] is the main source of heat in your home. In the past 12 months,
did you use any other types of heating equipment? Remember to include portable heaters,
fireplaces, heating stoves and cooking stoves. CHECK ALL THAT APPLY
(1) No other equipment
(2) Heat pump
(3) Central furnace with ducts to individual rooms
(4) Steam/Hot water system with radiators or pipes in each room
(5) Built-in electric units in each room installed in walls, ceilings, baseboards, or floors
(6) Built-in floor/wall pipeless furnace
(7) Built-in room heater burning gas, oil, or kerosene
(8) Heating stove burning wood, coal, coke, or biomass (such as pellets or corn)
(9) Portable heaters
(10) Fireplace
(11) Cooking stove used to heat your home as well as to cook
(12) Some other equipment (Specify __________________)
339
28. What fuel does the [FILL: EQUIPAUX] use?
(1) Electricity
(2) Natural gas from underground pipes
(3) Propane (bottled gas)
(4) Fuel oil
(5) Kerosene
(6) Wood
(7) Biomass (wood pellets or corn)
(8) Solar or Wind
(9) Geothermal
(10) District steam
(11) Some other fuel (Specify __________)
29. {IF YES WIND OR SOLAR FOR EITHER MAIN OR OTHER FUELS USED}
Do you have any on-site system that generates electricity such as a solar system or a small
wind turbine?
(1) Yes
(2) No (SKIP to Q30)
(3) Don‘t Know/Not Sure
(4) Refused
29a. What type of on-site system do you have?
(1) Solar or Photovoltaic system
(2) Small wind turbine
(3) Combined Heat and Power system
(4) Other . Please specify____________
29b. Is your on-site system connected to the grid?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
30. {IF YES HEATING STOVE}Which statement best describes your heating stove?
(1) Manufactured before 1992
(2) Energy Star
(3) Neither
(4) Don‘t Know/Not Sure
(5) Refused
31. {IF YES FIREPLACE} Does this fireplace have a flue to the outside or is it entirely self-contained?
(1) Flue to the outside
(2) Flueless (self-contained)
(3) Don‘t Know/Not Sure
(4) Refused
340
32. {IF YES HEATING STOVE OR FIREPLACE}In the past 12 months how often did you have to burn
garbage, cardboard, plastics, foam, colored ink, magazines, boxes, or wrappers to keep warm?
(1) Never
(2) Once
(3) Sometimes
(4) Often
(5) Most of the winter
(6) Don‘t Know/Not Sure
(7) Refused
33. {IF YES HEATING STOVE OR FIREPLACE}In the past 12 months how often did you have to burn
coated, painted, or pressure-treated wood, driftwood, plywood, particle board, or any wood with glue in it
to keep warm?
(1) Never
(2) Once
(3) Sometimes
(4) Often
(5) Most of the winter
(6) Don‘t Know/Not Sure
(7) Refused
34. {IF YES HEATING STOVE OR FIREPLACE}In the past 12 months how often did you have to burn
wet, rotted, diseased, or moldy wood
to keep warm?
(1) Never
(2) Once
(3) Sometimes
(4) Often
(5) Most of the winter
(6) Don‘t Know/Not Sure
(7) Refused
35. What fuel does the cooking stove and/or oven use? CHECK ALL THAT APPLY
(1) Electricity
(2) Natural gas from underground pipes
(3) Propane (bottled gas)
(4) Fuel oil
(5) Kerosene
(6) Wood
(7) Some other fuel (Specify __________)
(8) No working stove or oven in the home
36. Is an exhaust fan that vents to the outside used regularly when cooking in your kitchen?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
341
37. In the past 12 months how often have you used your oven to heat your house?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
38. Does your household use a microwave oven?
(1) Yes
(2) No (SKIP to 39)
(3) Don‘t Know/Not Sure
(4) Refused
38a. Which answer best describes how frequently your household uses the microwave to prepare hot
meals and snacks in a typical week?
(1) Used to cook or reheat most meals and snacks
(2) Used to cook or reheat about half of meals and snacks
(3) Used to cook or reheat a few meals and snacks
(4) Used very little
(5) Don‘t Know/Not Sure
(6) Refused
39. Does your heating system have an air filter?
(1) Yes
(2) No (SKIP to 40)
(3) Don‘t know/Not Sure
(4) Refused
39a. Is the air filter in your heating system a High Efficiency Particulate Arresting (HEPA) filter?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
39b. Approximately, how often does someone in your household change (or clean) the air filter in your
heating system?
(1) Monthly
(2) Every three months
(3) Every six months
(4) Once a year
(5) Once every two years
(6) Don‘t change (or clean) it
(7) Air filter is changed by service company
(8) Don‘t Know/Not Sure
(9) Refused
342
40. {IF YES FURNACE}Do you know when was the last time your furnace received maintenance
service by a furnace contractor to ensure optimum and safe operation?
(1) Yes
(2) No (SKIP to Q41)
(3) Refused
40a. How many years and months ago did this occur? _______________
41. Do you have a CO (or carbon monoxide) monitor in your house?
(1) Yes
(2) No (SKIP to Q 42)
(3) Don‘t know/Not Sure
(4) Refused
41a. Is your CO monitor currently working?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
42. Do you have one or more smoke detectors in your house?
(1) Yes
(2) No (SKIP to Q43)
(3) Don‘t know/Not Sure
(4) Refused
42a. How many smoke detectors are there in your house?
Enter Number __________
(1) Don‘t Know/Not Sure
(2) Refused
42b. How many of these smoke detectors are currently working?
Enter Number __________
(1) Don‘t Know/Not Sure
(2) Refused
43. In the past 12 months how many times has the fire department been called to put out a fire in your
home during the past year? _________
44. In the past 12 months did any fire start in your home as a result of using an
alternate heating source, such as space heaters, electric blankets, your kitchen stove or oven, heating
stove, furnace, or your fireplace?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
45. In the past 12 months, how many individuals needed medical attention because of fire?
Enter Number_______
343
Now I have some questions about cooling your home.
46. (IF NO TO AIR CONDITIONING FROM PREVIOUS SECTION SKIP TO CEILING FAN USE
Q47) Central air conditioning requires that the system have ducts to carry the cooled air to the individual
rooms. These ducts may also carry warm air for space heating. Does your home have ducts like these?
(1) Yes
(2) No (SKIP to 47)
(3) Don‘t Know/Not Sure
(4) Refused
46a. Does the central air conditioning equipment that cools your home also cool any other
apartments, condos, households, businesses, or farm buildings?
(1) Yes
(2) No
46b. Which of the statements shown best describes the way your central air conditioning system
was used last summer?
(1) Not used at all (if volunteered)
(2) Turned on only a few days or nights when really needed
(3) Turned on quite a bit
(4) Turned on just about all summer
47. Which of the following statements best describes the way your household used the
most used window/wall air conditioning unit last summer?
(1) Turned on only a few days or nights when really needed
(2) Turned on quite a bit
(3) Turned on just about all summer
(4) No working window/wall units in home
48. Which of the following statements best describes the way your household used a
Swamp or Evaporative Air Cooler last summer?
(1) Turned on only a few days or nights when really needed
(2) Turned on quite a bit
(3) Turned on just about all summer
(4) No swamp/evaporative air cooler in home.
49. How many ceiling fans does your household have?
Enter Number_________
(1) None (SKIP to Q 50)
(2) Don‘t Know/Not Sure
(3) Refused
49a. How many ceiling fans does your household use?
Enter Number of ceiling fans_______
(1) None (SKIP to 50)
(2) Don‘t Know/Not Sure
(3) Refused
344
49b. Thinking about the ceiling fan that you use the most, how often was this fan used last summer?
Is it . . .
(1) Used only a few days or nights, when it‘s really needed,
(2) Used quite a bit, or
(3) Used just about all summer?
(4) Not used at all
(5) Don‘t Know/Not Sure
(6) Refused
49c. Thinking about the ceiling fan that you use the most, how often was this fan used last winter?
Is it . . .
(1) Used only a few days or nights,
(2) Used quite a bit, or
(3) Used just about all winter?
(4) Not used at all
(5) Don‘t Know/Not Sure
(6) Refused
50. Opening windows on opposite sides of the house to cool the indoor temperature is called natural cross
ventilation. In the past 12 months, has your household used window fans to assist with natural cross
ventilation in the warmer months?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
51. How often are your windows open in the summer?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
52. How often are your windows open in the winter?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
53. Do any large trees shade your home from the afternoon summer sun?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
345
54. Do you close the drapes, curtains, shades, and/or blinds during the day to block out the sun during the
summer?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
INTERVIEWER INSTRUCTION: RECORD ANY INFORMATION HERE ABOUT THE
AIRCONDITIONING EQUIPMENT IN THIS HOUSING UNIT AND ITS‘ USAGE THAT
MIGHT PROVIDE CLARIFICATION TO THE RESPONDENT‘S ANSWERS.
_____________________________________________________________________________________
_____________________________________________________________________________________
_____________________________________________________________________________________
Now I have some questions on the indoor air temperature of your home.
55. Does your home have a thermostat that controls the heating and/or cooling in your home?
(1) Yes
(2) No (SKIP to Q 64)
(3) Don‘t Know
(4) Refused
55a. What equipment does your thermostat control?
(1) Central heating only
(2) Central cooling only
(3) Central heating and cooling
(4) Don‘t know
56. Some thermostats can be programmed so that the temperature changes automatically at different times
of the day; for example, the heat can be automatically turned down or lowered at night when you go to
bed, then automatically adjusted up again in the morning. Is the thermostat that controls your main
[heating and/or cooling] equipment programmable?
(1) Yes
(2) No (SKIP to Q57)
(3) Don‘t know/Not Sure
(4) Refused
56a. Do you or someone else in your household know how to use the programmable thermostat?
(1) Yes
(2) No (SKIP to Q57)
(3) No, someone who does not live in my home programs the thermostat for use
(4) Don‘t know/Not Sure
(5) Refused
346
56b. Is your thermostat programmed to change the temperature at different times of the day?
(1) Yes
(2) No (SKIP to Q56d)
(3) Don‘t know/Not Sure
(4) Refused
56c. Please indicate how the programmable thermostat is used. (Select all that apply.)
(1) Thermostat is automatically adjusted to a lower temperature at night during the winter
(2) Thermostat is automatically adjusted to a lower temperature during the day when no one is
home during the winter
(3) Thermostat is automatically adjusted to a higher temperature at night during the summer
(4) Thermostat is automatically adjusted to a higher temperature during the day when no one is
home during the summer
(5) Other __________________
(6) Don‘t know/Not Sure
(7) Refused
56d. Which statement best describes your programmable thermostat…
(1) It is very easy to use
(2) It is somewhat easy to use
(3) It is neither easy nor difficult to use
(4) It is somewhat difficult to use
(5) It is very difficult to use
(6) Refused
56e. Typically, how often is your programmable thermostat reprogrammed, that is, the time schedule and
desired indoor temperature setting changed permanently?
(1) Daily
(2) Weekly
(3) Monthly
(4) Every three months of so
(5) Once a year
(6) Less than once a year
(7) Never
(8) Don‘t know/Not Sure
(9) Refused
56f. How often is the current temperature setting ―overridden‖ temporarily and why?
(1) Daily
(2) Weekly
(3) Monthly
(4) Every three months of so
(5) Once a year
(6) Less than once a year
{IF ANSWERED ANY OF THE ABOVE} Please explain why_________
(7) Never
(8) Don‘t know/Not Sure
(9) Refused
347
56g. How often is the ―hold‖ mode used and why?
Daily
(2) Weekly
(3) Monthly
(4) Every three months of so
(5) Once a year
(6) Less than once a year
{IF ANSWERED ANY OF THE ABOVE} Please explain why._________
(7) Never
(8) Don‘t know/Not Sure
(9) Refused
56h. Please indicate what features need improvement in your programmable thermostat.
(1) Size of words and numbers?
(2) Ease of programming?
(3) Additional features, such as the energy use in my home?
(4) Being able to change its settings using a cell phone, Internet?
(5) Other features. Please Specify. _________________
(6) Don‘t Know/Not Sure
(7) Refused
56i. Is there anyone in your home who doesn‘t operate the programmable thermostat? Why? (select all
that apply)
(1) It‘s too complicated/too difficult
(2) We do not have an instruction manual / Nobody showed us how
(3) We haven‘t needed to
(4) Other (Specify)________
(5) Don‘t Know/Not Sure
(6) Refused
56j. {IF NO TO 56b. or ANSWERED (7)NEVERTO 56e}Why is your programmable thermostat not
programmed to automatically change the temperature? (select all that apply)
(1) We change the temperature manually/don‘t need to program
(2) It takes too much effort to use
(3) Household members cannot agree on what temperatures at which to set the thermostat
(4) It is better to always keep the temperature setting always the same. (Why?)
(5) There is always somebody at home
(6) Other (Specify)________
(7) Don‘t Know/Not Sure
(8) Refused
57. During the winter, what is the temperature when someone is inside your home during the day? [IF NO
ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET? PROBE 2:
CAN I JUST HAVE YOUR BEST ESTIMATE?]
Enter degrees Fahrenheit________
348
58. During the winter, what is the temperature when no one is inside your home during the day? [IF NO
ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET? PROBE 2:
CAN I JUST HAVE YOUR BEST ESTIMATE?]
Enter degrees Fahrenheit________
59. During the winter, what is the temperature inside your home at night? [IF NO ANSWER, PROBE 1:
THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET? PROBE 2: CAN I JUST HAVE
YOUR BEST ESTIMATE?]
Enter degrees Fahrenheit________
Now I would like you to think about the temperature inside your home when using your central air
conditioning equipment last summer. [If NUMTHERM>1: Earlier you reported having [FILL:
NUMTHERM] thermostats. For the next questions, if the thermostats are set at different temperatures,
only report for the thermostat that affects the rooms where most of the people are.]
60. During the summer, what is the temperature when someone is inside your home during the day? IF
NO ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET? PROBE
2: WHAT‘S YOUR BEST ESTIMATE?
Enter degrees Fahrenheit ________
Air-conditioner Turned Off
61. During the summer, what is the temperature when no one is inside your home during the day? IF NO
ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET? PROBE 2:
WHAT‘S YOUR BEST ESTIMATE?
Enter degrees Fahrenheit ________
Air-conditioner Turned Off
62. During the summer, what is the temperature inside your home at night? IF NO ANSWER, PROBE 1:
THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET? PROBE 2: WHAT‘S YOUR BEST
ESTIMATE?
Enter degrees Fahrenheit ________
Air-conditioner Turned Off
63. Answer the following statements -- true or false:
a. If the thermostat is turned up very high in the winter, my home will get warmer faster.
(1) True
(2) False
b. The thermostat controls the temperature of the air coming from the heating/cooling unit into my home.
(1) True
(2) False
349
c. The thermostat only senses the temperature in the air in the room where the thermostat is located. It
turns the heating unit off when the temperature in the room reaches the temperature on the thermostat
setting.
(1) True
(2) False
d. If the thermostat is turned down at night or when no one is home, then more energy is used than saved
when your home is heated up again.
(1) True
(2) False
64. In the past 12 months, was your household unable to use any of the following equipment
because it was broken? CHECK ALL THAT APPLY
(1) Main Heating Equipment
(2) Central Air Conditioner
(3) Room Air Conditioner
(4) Don‘t Know/Not Sure
(5) Refused
65. Which of the following statements best describes the indoor temperature of your home during the
winter:
(1) Very cold
(2) Cold
(3) Comfortable (SKIP)
(4) Hot
(5) Very hot
(6) Other ________________
(7) Refused
66. Which of the following statements best describes the indoor temperature of your home during the
summer:
(1) Very cold
(2) Cold
(3) Comfortable (SKIP TO NEXT Q….)
(4) Hot
(5) Very hot
(6) Other ________________
(7) Refused
67. In the past 12 months, has a landlord controlled the temperature inside your home?
(1) Yes
(2) No
(3) Do not have landlord
(4) Don‘t Know/Not Sure
(5) Refused
350
68. In the past 12 months how often did your household keep your home at a temperature that you
felt was unsafe or unhealthy?
(1) Almost every month
(2) Some months
(3) 1 or 2 months
(4) Never
(5) Don‘t Know/Not Sure
(6) Refused
69. In the past 12 months, has anyone in the household needed medical attention because your
home was too cold?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
70. In the past 12 months did anyone in your household need medical attention
because your home was too hot?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
71. During the past 12 months, how often have you or other members of your household found
your home too drafty? Would you say it is. . .
(1) All the time,
(2) Most of the time,
(3) Some of the time, or
(4) Never
(5) Don‘t Know/Not Sure
(6) Refused
The next group of questions is about laundry appliances and water use in your home.
72. Is a clothes washing machine used in your home? Do not include community clothes washers
that are located in the basement or laundry room of your apartment building.
(1) Yes
(2) No (SKIP to Q73)
(3) Refused
72a. In an average week, how many loads of laundry are washed in your clothes washer?
(1) 1 load or less each week
(2) 2 to 4 loads each week
(3) 5 to 9 loads each week
(4) 10 to 15 loads each week
(5) More than 15 loads each week
(6) Don‘t Know/Not Sure
(7) Refused
351
72b. Does your household wash only full loads of laundry?
(1) Always
(2) Most of the time
(3) Some of the time
(4) Never
(5) Don‘t know/Not Sure
(6) Refused
72c. What water temperature setting is usually used for the wash cycle of your clothes washer? Is it hot,
warm, or cold water?
(1) Hot
(2) Warm
(3) Cold
(4) Don‘t Know/Not Sure
(5) Refused
72d. What water temperature setting is usually used for the rinse cycle of your clothes washer? Is it hot,
warm, or cold water?
(1) Hot
(2) Warm
(3) Cold
(4) Don‘t Know/Not Sure
(5) Refused
73. Do you use a clothes dryer in your home? Do not include community clothes dryers that are
located in the basement or laundry room of your apartment building.
(1) Yes
(2) No (SKIP TO Q74)
(3) Don‘t Know/Not Sure
(4) Refused
73a. Does your household dry only full loads of laundry….?
(1) Always
(2) Most of the time
(3) Some of the time
(4) Never
(5) Don‘t Know/Not Sure
(6) Refused
73b. Does your clothes dryer vent directly to the outdoors?
(1) Yes
(2) No
(3) Dryer is ventless
(4) Don‘t Know/Not Sure
(5) Refused
73c. Do you clean your clothes dryer‘s lint filter after every use?
(1) Yes
(2) No
(3) Dryer has not lint filter
(4) Don‘t Know/Not Sure
(5) Refused
352
74. How frequently does your household hang clothes to dry?
(1) Very frequently
(2) Frequently
(3) Infrequently
(4) Very infrequently
(5) Never
(6) Don‘t Know/Not Sure
(7) Refused
75. In the last 12 months, has the temperature of your hot water heater been adjusted?
(1) Yes, the temperature is much warmer
(2) Yes, the temperature is warmer
(3) No adjustment has been made to the temperature
(4) Yes, the temperature is cooler
(5) Yes, the temperature is much cooler
(6) Hot water heater was not in working order for the last 12 months
(7) No water heater
(8) Don‘t Know/Not Sure
(9) Refused
76. Over the past 12 months, has the duration of the showers taken by household members changed?
(1) Increased a lot
(2) Increased some
(3) No change
(4) Decreased some
(5) Decreased a lot
(6) No Shower
(7) Don‘t Know/Not Sure
(8) Refused
77. Does your main bathroom have a ventilation fan in it that works?
(1) Yes
(2) No (SKIP to Q78)
(3) Don‘t know/Not Sure (SKIP to Q50)
(4) Refused
77a. How often do you or members of your household operate the fan while showering?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
353
77b. How long after showering do you or members of your household operate the fan?
(1) Don‘t turn the fan on for showers
(2) The fan is turned off when leaving the shower area
(3) A few minutes
(4) Several minutes
(5) Until the steam in the shower area is gone
(6) Don‘t know/Not Sure
(7) Refused
78. Electric dehumidifiers remove moisture from the air and are often used in the summer. Is a
dehumidifier used in your home?
(1) Yes
(2) No (SKIP to Q79)
(3) Don‘t Know/Not Sure
(4) Refused
78a. In the past 12 months, how many months was the dehumidifier used?
(1) 1 to 3 months, 4 to 6 months,
(2) 7 to 9 months,
(3) 10 to 11 months, but not all year, or is it
(4) Turned on all year long?
(5) Don‘t Know/Not Sure
(6) Refused
Now I have some questions about lights inside your home and energy efficiency.
79. How often do you find lights left on in rooms that are not occupied?
(1) Never
(2) Almost never
(3) Sometimes
(4) Most of the time
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
80. Do members of your household purchase or intentionally seek out and install compact fluorescent
bulbs in your home?
(1) Yes
(2) No (SKIP to Q81)
(3) I do not know what compact fluorescent bulbs are (SKIP to Q81)
(4) Don‘t know/Not Sure
(5) Refused
80a. How do you dispose of compact fluorescent light bulbs that are broken or no longer working?_
(1) Directly in household garbage
(2) Doubled bagged in plastic in household garbage
(3) Transport to local recycling center
(4) Other (Please Specify)________________
(5) Don‘t Know/Not Sure
(6) Refused
354
81. Are you familiar with the Energy Star label?
(1) Yes
(2) No (SKIP to 82)
(3) Refused
81a. Has your household bought or intentionally installed appliances or consumer electronics that have an
Energy Star label?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
82. Do you unplug any appliances like TVs, VCRs, stereos, radios, clocks, or computers to save energy
when they are turned off?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
In this next set of questions, I will ask you about other conditions of your home.
83. How much outdoor noise do you hear indoors when the windows are closed?
(1) A great deal
(2) Some
(3) Hardly any
(4) None at all
(5) Don‘t know/Not Sure
(6) Refused
84. Please rate the outside appearance of your home:
(1) Very attractive
(2) Attractive
(3) Neither attractive nor unattractive
(4) Unattractive
(5) Very unattractive
(6) Refused
85. Over the past 12 months, how has the property value of your home changed?
(1) Very much higher
(2) Higher
(3) No change
(4) Lower
(5) Very much lower
(6) Not applicable, don‘t own the home or live in an apartment
(7) Don‘t know/Not Sure
(8) Refused
355
86. How infested is your home with cockroaches or other insects or spiders?
(1) Extremely infested
(2) Very infested
(3) Somewhat infested
(4) Hardly infested
(5) Not infested at all (SKIP to Q87)
(6) Don‘t know/Not Sure
(7) Refused
86a. What have you done about the cockroaches, other insects or spiders?
(1) Nothing
(2) Used insecticides, bug sprays, or poison
(3) Hired an exterminator or other professional
(4) Other. Please specify._________
(5) Don‘t Know/Not Sure
(6) Refused
87. How infested is your home with rats or mice?
(1) Extremely infested
(2) Very infested
(3) Somewhat infested
(4) Hardly infested
(5) Not infested at all (SKIP to Q88)
(6) Don‘t know/Not Sure
(7) Refused
87a. What have you done about the pests?
(1) Nothing
(2) Used bait or poison
(3) Hired an exterminator or other professional
(4) Other. Please specify._________
(5) Don‘t Know/Not Sure
(6) Refused
88. Does your home frequently have a mildew odor or musty smell?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
89. How often do you observe standing water anywhere in your home?
(1) Never
(2) Rarely
(3) Sometimes
(4) Often
(5) Always
(6) Don‘t Know/Not Sure
(7) Refused
356
90. Have you seen mold in your home in the past 12 months?
(1) Yes
(2) No {SKIP to Q91}
(3) Don‘t know/Not Sure
(4) Refused
90a. {If YES MOLD}What have you done about the mold?
(1) Nothing
(2) Cleaned with bleach
(3) Cleaned with other chemical mold remover
(4) Cleaned with natural mold remover (vinegar or natural product)
(5) Air Conditioned
(6) Ventilation (fans)
(7) Used a dehumidifier
(8) Contacted a Professional
(9) Other. Please Specify _________
(10) Don‘t know/Not Sure
(11) Refused
Now I would like to ask you a few questions about your energy bills.
Some households may have faced challenges in paying home energy bills. The following
questions ask about challenges your household may have had paying home energy bills or
maintaining heating and cooling equipment. When thinking about these questions, include all of
your experiences in the past 12 months.
91. Some energy utilities and suppliers offer budget payment plans that allow a household to pay
the same amount on the home energy bill each month. In the past 12 months, did your household
use a budget plan for any home energy bill?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
INTERVIEWER INSTRUCTION: BUDGET PLANS ARE NOT USUALLY RELATED TO LATE OR
DISCONNECT NOTICES AND ARE NOT A WAY TO PAY DOWN PAST DUE BALANCES.
92. How well do you understand the information on your energy bill other than the amount owed (e.g.,
information about how much energy your household used during the billing period compared to the same
billing period one year ago)?
(1) Very well
(2) Well
(3) Neither well nor not well
(4) Not well
(5) Not well at all
(6) Don‘t know/Not Sure
(7) Refused
357
93. How hard is it to pay your energy bills?
(1) Very hard
(2) Hard
(3) Neither hard or not hard
(4) Not hard
(5) Not hard at all
(6) Don‘t know/Not Sure
(7) Refused
94. Has your household ever had to move in the past 5 years because your household could not pay the
energy bills?
(1) Yes
(2) No (go to Q20)
(3) Don‘t know (go to Q 20)
95. Over the past 12 months, how often has your household not paid energy bills in order to pay other
utility bills (PROBE: water/sewage/telephone/Secondary energy fuel type)?
(1) Every month
(2) Every other month
(3) Every few months
(4) Every six months
(5) Once in twelve months
(6) Never
(7) Don‘t Know/Not Sure
(8) Refused
96. Over the past 12 months, how often has your household not paid other utilities in order to pay the
primary energy bill (PROBE: water/sewage/telephone/Secondary energy fuel type)?
(1) Every month
(2) Every other month
(3) Every few months
(4) Every six months
(5) Once in twelve months
(6) Never (SKIP to Q97)
(7) Don‘t Know/Not Sure
(8) Refused
96 a. What utilities were not paid for in order to pay an energy bill? (Check all that apply)
(1) Water
(2) Sewage
(3) Telephone
(4) Secondary energy fuel type
(5) Other__________
(6) Don‘t Know/Not Sure
(7) Refused
358
97. Over the past 12 months, how often has your household not purchased food in order to pay an energy
bill?
(1) Every month
(2) Every other month
(3) Every few months
(4) Every six months
(5) Once in twelve months
(6) Never
(7) Don‘t Know/Not Sure
(8) Refused
98. Over the past 12 months, how often has your household not paid energy bills in order to purchase
food?
(1) Every month
(2) Every other month
(3) Every few months
(4) Every six months
(5) Once in twelve months
(6) Never
(7) Don‘t Know/Not Sure
(8) Refused
99. In the past four weeks, did you or any household member go a whole day and night without eating
anything because there was not enough food?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
100. In the past four weeks, did you worry that your household members would not have nutritious food?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
101. Some households receive additional assistance to help pay for food. In the past 12 months did you or
any members of your household receive food stamps or WIC assistance (Women, Infants, and Children
nutrition program)? (ASK ONCE/DO NOT REPEAT QUESTION FOR EACH INDIVIDUAL)
(1) Yes
(2) No; Did not apply
(3) No; Applied for, but rejected
(4) Don‘t Know/Not Sure
(5) Refused
359
102. In the past 5 years have you or anyone in the household experienced any of the following as a result
of energy bills? (Check all that apply)
(1) Eviction from home
(2) Foreclosure on mortgage
(3) Moved in with friends or family
(4) Moved into a shelter or been homeless
(5) Family Separation
(6) Refused
102a. {If YES FAMILY SEPARATION}In what way or ways was the family separated?
(1) Adult partners only separated
(2) One adult partner separated from partner and children
(3) One parent separated from children only
(4) Both parents separated from children
(5) Elder parent or relative separated from family
(6) Refused
103. In the past 12 months how often did your household pay an amount less than what you owed on your
home energy bill, because you were unable to afford the whole home energy bill?
(1) Almost every month
(2) Some months
(3) 1 or 2 months
(4) Never
(5) Don‘t Know/Not Sure
(6) Refused
104. In the past year, have you used any of the following to assist with paying your energy bill?
(1) Payday loan
(2) Tax Refund Anticipation Loan
(3) Car Title loan
(4) Other type of short term, high-interest loan
(5) Pawn shop
(6) Don‘t Know/Not Sure
(7) Refused
104a. {If YES to any of the above}In the past year, in order to pay your home energy bill, how often did
you need to use a payday loan, a Tax Refund Anticipation Loan, a car title loan, another type of shortterm, high-interest loan, or pawn shop?
(1) Almost every month
(2) Some months
(3) 1 or 2 months
(4) Never
(5) Don‘t Know/Not Sure
(6) Refused
360
105. When home energy bills are not paid on time, it is common for energy utilities and suppliers to send
late notices. If the bill is very late, they will send a disconnect, shut-off, or non-delivery notice. How
often did you receive a disconnect, shut-off, or non-delivery notice?
(1) Almost every month
(2) Some months
(3) 1 or 2 months
(4) Never (SKIP to Q106)
(5) Don‘t Know/Not Sure
(6) Refused
105a. Did you enter into a payment arrangement with your energy utility or supplier in response to the
disconnect shut-off, or non-delivery notice?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
106. In the past 12 months was your electricity or natural gas ever disconnected because you
were unable to pay your home energy bill?
(1) Yes
(2) No (SKIP to Q107)
(3) Don‘t Know/Not Sure
(4) Refused
106a. While your electricity or natural gas was disconnected, was there a time when you wanted
to use your main source of heat but were unable to?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
106b. While your electricity was disconnected, was there a time when you wanted to use
your air conditioner but were unable to?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
107. In the past 12 months did your fuel oil, kerosene, propane, or wood ever run out because you were
unable to pay for a home energy delivery?
(1) Yes
(2) No (SKIP to Q108)
(3) Don‘t Know/Not Sure
(4) Refused
107a. When you ran out of your fuel oil, kerosene, propane, or wood was there a time when you
wanted to use your main source of heat but were unable to?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
361
Next, I will be asking about health care and coverage.
108. In the past 12 months have you had any kind of health care coverage, including health insurance,
prepaid plans such as HMOs, or government plans such as Medicare?
(1) Yes (SKIP to Q108b)
(2) No
(3) Refused
108a. {IF NO COVERAGE} According to the information given, you do NOT have health care coverage
of any kind. Do you have health insurance or coverage through a plan I might have missed?
(INTERVIEWER: REVIEW PLANS IF INFORMANT IS UNSURE.)
(1) NO/NOT COVERED BY ANY PLAN (SKIP to Q109)
(2) HEALTH INSURANCE PLAN FROM A
(3) CURRENT OR PAST EMPLOYER/
(4) UNION/SCHOOL
(5) A HEALTH INSURANCE PLAN BOUGHT ON
HIS/HER OWN/PROF. ASSN
(6) A PLAN BOUGHT BY SOMEONE WHO
DOES NOT LIVE IN THIS HOUSEHOLD
(7) MEDICARE
(8) MEDICAID/STATE NAME
(9) CHAMPUS/CHAMP-VA, TRICARE, VA,
(10) OTHER MILITARY
(11) INDIAN HEALTH SERVICE
(12) [FILL STATE PLAN]
(13) OTHER PLAN [SPECIFY]
(14) DON‘T KNOW/NOT SURE
(15) REFUSED
108b. {IF YES COVERAGE} During the past 12 months was there any time that you did not have any
health insurance coverage?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
108c. Does your health plan pay for at least some of the cost of prescription medicines
prescribed by a doctor?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
109. During the past 12 months, was there any time your household members needed prescription
medicines but didn‘t get them because you couldn‘t afford it?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
362
110. Over the past 12 months, how often did members of your household not fill a prescription or took
less than the full dose of a prescribed medicine in order to pay the utility bill?
(1) Every month
(2) Every other month
(3) Every few months
(4) Every six months
(5) Once in twelve months
(6) Never
(7) Don‘t Know/Not Sure
(8) Refused
111. Over the past 12 months, how frequently has your household not paid energy bills in order to
purchase prescription medicines?
(1) Every month
(2) Every other month
(3) Every few months
(4) Every six months
(5) Once in twelve months
(6) Never
(7) Don‘t Know/Not Sure
(8) Refused
112. Was there a time in the past 12 months when you needed to see a doctor but could not
because of cost?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
113. During the past 12 months, have you or other adults in your household had any problems paying
medical bills?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{Note: Ask these questions of the adult in the household most involved with all the other members of the
household because proxy responses are required.}
114. Including yourself, how many people normally live in this household? Do not include
anyone who is just visiting, those away in the military, or children who are away at college.
Enter Number _______________
363
115. Can you please tell me their first names, gender and age, and your relationship to the person?
First Name
Gender
Age
Relationship
In school (Y/N)
Person 1.
Person 2.
Person 3.
Person 4.
Person 5.
Person 6.
Person 7.
Person 8.
Person 9.
Person 10.
116. On a typical week day is there someone at home most or all of the day?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
For this section, I will be asking health related questions.
117. Now thinking about physical health, which includes physical illness and injury, for how many days
during the past 30 days was your physical health not good?
(1) Number of days______
(2) None
(3) Don‘t know / Not sure
(4) Refused
118. Now thinking about your mental health, which includes stress, depression, and problems with
emotions, for how many days during the past 30 days was your mental health not good?
(1) Number of days_______
(2) None
(3) Don‘t know / Not sure
(4) Refused
119. During the past 30 days, for about how many days have you felt you did not get enough rest
or sleep?
(1) Number of days______
(2) None
(3) Don‘t know / Not sure
(4) Refused
120. During the past 30 days, for about how many days have you felt very healthy
and full of energy?
(1) Number of days______
(2) None
(3) Don‘t know / Not sure
(4) Refused
364
121. During the past 30 days, for about how many days did poor physical or mental health
keep you from doing your usual activities, such as self-care, work, or recreation?
(1) Number of days _______
(2) None
(3) Don‘t know / Not sure
(4) Refused
Next, I am going to ask you whether you have had some particular health
problems in the last 3 months. In the past 3 months, have you had . . .
122. Shortness of breath when lying down, waking up,
or with light work or light
exercise?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
123. Headaches that are either new or more
frequent or severe than ones you have
had before?
(1) Yes
(2) No
(3) Don‘t know
(4) Refused
In the past 12 months were you or anyone else in the household ever told by a doctor or health
professional that you or they have… (Check all that apply)
124. Lead poisoning
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{IF YES}Please list all individuals, including yourself:
____________________
____________________
____________________
365
125. Three or more ear infections per year
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
126. Any kind of respiratory allergy
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
127. Flu
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
128. Persistent Cold symptoms lasting more than 14 days (SYMPTOMS INCLUDE COUGHING, SORE
THROAT, SNEEZING, SINUS PAIN, CONGESTION, FEVER, FATIGUE, AND HEADACHE)
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
366
129. Sinus infection or Sinusitis
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
130. Bronchitis
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
131. Have you ever been told by a doctor or other health professional that you have asthma?
(1) Yes
(2) No (SKIP to Q132)
(3) Don‘t Know/not sure
(4) Refused
131a. Do you still have asthma?
(1) Yes
(2) No
(3) Don‘t Know/Not sure
(4) Refused
131b. During the past 12 months, how many times did you see a doctor or health professional for a
routine checkup for your asthma? ______________
READ: Symptoms of asthma include coughing, wheezing, shortness of breath, chest tightness or phlegm
production when you have a cold or respiratory infection.
367
131c. How long has it been since you last had any symptoms of asthma?
(1) Never
(2) Less than one day ago
(3) 1-6 Days ago
(4) 1 week to less than 3 months ago
(5) 3 months to less than 1 year ago
(6) 1 year to less than 3 years ago
(7) 3 years to 5 years ago
(8) More than 5 years ago
(9) Don‘t Know/Not sure
(10) Refused
131d. During the past 12 months did you have to stay overnight in the hospital because of asthma?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
131e. Not counting hospitalizations, during the past 12 months, did you go to an emergency room because
of asthma?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
These next questions are about cigarette smoking.
132. Which one of the following statements best describes the rules about smoking in your home…
(1) No one is allowed to smoke anywhere inside your home
(2) Smoking is allowed at some places or at sometimes
(3) Smoking is permitted anywhere
(4) Don‘t know/Not sure
(5) Refused
133. Have you smoked at least 100 cigarettes in your entire life?
(1) Yes
(2) No
(3) Don‘t know/Not sure
(4) Refused
134. Do you now smoke cigarettes every day, some days or not at all?
(1) Everyday
(2) Some days
(3) Not at all
(4) Don‘t Know/Not sure
(5) Refused
368
135. In the past 12 months has anyone in the household been food poisoned from eating food inside your
home and therefore went to see a medical professional?
(1) Yes
(2) No
(3) Don‘t Know/Not sure
(4) Refused
136. In the past 12 months, has anyone in the household been poisoned by breathing in carbon monoxide,
and therefore went to see a medical professional?
(1) Yes
(2) No
(3) Don‘t Know/Not sure
(4) Refused
137. In the past 12 months, has anyone in the home been burned from scalding hot water coming out of a
faucet or showerhead in your home?
(1) Yes
(2) No (SKIP to Q138)
(3) Don‘t Know/Not Sure
(4) Refused
137a. {IF YES BURN}Did you talk to or see a medical professional about this injury?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
138. I am going to read some statements about health and medical care.
Usually, you go to the doctor as soon as you start to feel bad. Is that:
(1) definitely true,
(2) mostly true,
(3) mostly false, or
(4) definitely false?
(5) Don‘t Know/Not Sure
(6) Refused
139. You will do just about anything to avoid going to the doctor. Is that . . .
(1) definitely true,
(2) mostly true,
(3) mostly false, or
(4) definitely false?
(5) Don‘t Know/Not Sure
(6) Refused
369
In this last section I will be asking employment and school related questions.
140. Are you or primary wage earner in the household currently…?
(1) Employed for wages
(2) Self-employed
(3) Out of work for more than 1 year (SKIP to Q140c)
(4) Out of work for less than 1 year (SKIP to Q140c)
(5) A Homemaker (SKIP to Q141)
(6) A Student
(7) Retired (SKIP to Q141)
(8) Unable to work (SKIP to Q141)
(9) Refused
140a. Are you or the primary wage earner in the household employed full-time or part-time?
(1) Full-time
(2) Part-time
(3) Don‘t Know/Not Sure
(4) Refused
140b. How many hours per week do you or the primary wage earner usually work at all of your jobs?
Enter ______ hrs
140c. {IF ANSWERED (3) or (4) to Q140} Have you/Has the primary wage earner looked for work
during the last 4 weeks?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
140d. {IF ANSWERED (3) or (4) to Q140}What is the main reason you were/the primary wage earner
was not looking for work during the LAST 4 WEEKS? CHECK ALL THAT APPLY (DO NOT READ
LIST)
(1) Believes no work available in line of work or area
(2) Couldn't find any work
(3) Lacks necessary schooling, training, skills or experience
(4) Employers think too young or too old
(5) Other types of discrimination
(6) Can't arrange child care
(7) Family responsibilities
(8) In school or other training
(9) Ill health, physical disability
(10) Transportation problems
(11) Other
(12) Don‘t Know/Not Sure
(13) Refused
370
141. Does a physical, mental or emotional problem NOW keep you or the primary wage earner from
working at a job or business?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
142. In the past 12 months did anyone in the household receive income from any of the
following sources? (Check all that apply)
(1) Supplemental Security Income (SSI)
(2) Welfare payments or case assistance
(3) Veteran‘s payments (VA Benefits)
(4) Unemployment Compensation
(5) Don‘t Know/Not Sure
(6) Refused
143. During the past 12 months have you or the primary wage earner had more than one job (or business),
including part time, evening, or weekend work?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
144. {IF ANSWERED (1), (2), (4), or (6) to Q140}Thinking about the last 12 months, is your or the
primary wage earner‘s main job, where you work(ed) the most amount of hours, considered seasonal?
(1) Yes
(2) No (SKIP to Q145)
(3) Don‘t Know/Not Sure
(4) Refused
144a. In what season do you/does the primary wage earner work the least amount of hours?
(1) Winter
(2) Spring
(3) Summer
(4) Fall
145. {IF ASWERED (1) or (2) TO Q 140}In the past 12 months, about how many days of work did you
or the primary wage earer miss work at a job or business because of illness or injury (DO NOT
INCLUDE MATERITY LEAVE}.
Enter Number________
(1) None
(2) Don‘t Know/Not Sure
(3) Refused
371
146. {IF ASWERED (1) or (2) TO Q 140}In the past 12 months, about how many days of work did you
or the primary wage earner miss because of illness or injury of another household member?
Enter Number________
(1) None
(2) Don‘t Know/Not Sure
(3) Refused
147. In the past 12 months, about how many days of school have you, the primary wage earner, and/or
those in the household enrolled in school, missed because of illness or injury? {IF PRE-SCHOOLER IN
HOME}Please tell us about the preschooler who has missed the most number of days. {IF SCHOOL
AGED CHILD IN HOME} Please tell us about the school aged child who has missed the most number of
days.
Enter Number:
Main Respondent _________ days
(1) Not in School
(2) Don‘t Know/Not Sure
(3) Refused
Primary Wage Earner _________ days
(1) Not in School
(2) Don‘t Know/Not Sure
(3) Refused
Pres-School Child who has missed the most amount of school_________ days
(1) None
(2) Don‘t Know/Not sure
(3) Refused
School Aged child who has missed the most amount of school_________ days
(1) None
(2) Don‘t Know/Not sure
(3) Refused
148. {IF STUDENT}In the past 12 months, how frequently did you find it hard to study in your home
because of excessive heat or cold?
(1) Very frequently
(2) Frequently
(3) Not frequently or infrequently
(4) Infrequently
(5) Very infrequently
(6) Never
(7) Does not study at home
(8) Don‘t Know/Not sure
(9) Refused
372
149. {IF SCHOOL AGED CHILDREN IN THE HOME} In the past 12 months, how frequently does any
school aged child in the home find it hard to study because of excessive heat or cold?
(1) Very frequently
(2) Frequently
(3) Not frequently or infrequently
(4) Infrequently
(5) Very infrequently
(6) Never
(7) Does not study at home
(8) Don‘t Know/Not sure
(9) Refused
That is the end of the survey. Thank you for your participation! You will receive your $25 gift card in the
mail to compensate you for your time. Could you please verify your mailing address:
Address: __________________________
__________________________
__________________________
__________________________
{INTERVIEWER: ADMINISTER AS A SEPARATE SURVEY POST-WEATHERIZATION}
PART II. Client Satisfaction
{Note: Ask these questions of the adult in the household most involved with the weatherization of the
home or the head of the household}
Approximate Date of Weatherization Job: _____________________
1. How long have you known about your local weatherization program?
ENTER THE RESPONSE IN YEARS ___________________
2. How did you find out about your local weatherization program? CHECK ALL THAT APPLY
(1) A call from the weatherization agency
(2) Information received in the mail from the weatherization agency
(3) Local newspaper
(4) Found the program on the Internet
(5) Relative or friend mentioned the weatherization program
(6) Neighbor who had their home weatherized
(7) Agency providing utility assistance such as LIHEAP
(8) Email from an organization with which you are a member
(9) Church
(10) Other (Specify_______________)
3. How long ago did you request that your home be weatherized?
ENTER THE RESPONSE IN YEARS ____________________
373
4. How satisfied are you with the length of time between your request to have your home weatherized and
when it actually was weatherized?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
5. How easy was it to request that your house be weatherized?
(1) Very easy
(2) Easy
(3) Not easy or difficult
(4) Difficult
(5) Very difficult
6. How easy was it to schedule the initial audit of your home?
(1) Very easy
(2) Easy
(3) Not easy or difficult
(4) Difficult
(5) Very difficult
7. How timely were those who did the initial audit of your home?
(1) Early or On Time
(2) <30 Minutes Late
(3) 30-60 Minutes Late
(4) 1 to 3 Hours Late
(5)More than 4 Hours Late
(6) Did not show up on scheduled day
8. How courteous were those who did the initial audit of your home?
(1) Very Courteous
(2) Courteous
(3) Not Courteous or Rude
(4) Rude
(5) Very Rude
9. How easy was it to schedule the time for the weatherization crew to come to your home?
(1) Very easy
(2) Easy
(3) Not easy or difficult
(4) Difficult
(5) Very difficult
10. How timely was the weatherization crew?
(1) Early or On Time
(2) <30 Minutes Late
(3) 30-60 Minutes Late
(4) 1 to 3 Hours Late
(5)More than 4 Hours Late
(6) Did not show up on scheduled day
374
11. How courteous was the weatherization crew?
(1) Very Courteous
(2) Courteous
(3) Not Courteous or Rude
(4) Rude
(5) Very Rude
12. How careful of your home and belongings was the weatherization crew?
(1) Very careful
(2) Careful
(3) Neither careful or careless
(4) Careless
(5) Very careless
13. Overall, how clean did the weatherization crew leave the inside of your home?
(1) Very clean
(2) Clean
(3) Neither clean nor dirty
(4) Dirty
(5) Very dirty
13a. Overall, how clean did the weatherization crew leave the outside of your home?
(1) Very clean
(2) Clean
(3) Neither clean nor dirty
(4) Dirty
(5) Very dirty
14. Overall, how satisfied are you with final condition the inside of your home was left in?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
14a. Overall, how satisfied are you with final condition the outside of your home was left in?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
15. How easy was it to schedule the final inspection of your home?
(1) Very easy
(2) Easy
(3) Not easy or difficult
(4) Difficult
(5) Very difficult
(6) Final inspection was not done (go to Q18)
375
16. How timely were those who did the final inspection of your home?
(1) Early or On Time
(2) <30 Minutes Late
(3) 30-60 Minutes Late
(4) 1 to 3 Hours Late
(5)More than 4 Hours Late
(6) Did not show up on scheduled day
17. How courteous were those who did the final inspection of your home?
(1) Very Courteous
(2) Courteous
(3) Not Courteous or Rude
(4) Rude
(5) Very Rude
18. How satisfied are you with the work performed in your home?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
19. How satisfied are you with any new equipment installed in house?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
20. Do you feel that other things should have been installed in your home to help you save energy?
(1) Yes
(2) No (go to Q21)
20a. What other things? ______________________
21. How satisfied are you with the energy savings achieved after having your home weatherized?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
(6) Too soon to tell
(7) Don‘t know
22. Did the weatherization agency staff check your home for major repairs (e.g., fixing roof)?
(1) Yes
(2) No (go to Q23)
22a. Were major repairs needed in your home?
(1) Yes
(2) No (go to Q23)
376
22b. Were major repairs done to your home?
(1) Yes
(2) No
(3) Not yet but expecting repairs to be done
23. Did the weatherization staff ask you about the health of the member(s) of your household?
(1) Yes (go to Q24a)
(2) No
24. Without the weatherization staff asking, did you provide to them any information about the health of
the member(s) of your household?
(1) Yes
(2) No (SKIP to Q25)
24a. Were any of the members of your household in need of care that they were not receiving at the time?
(1) Yes
(2) No (go to Q25)
24b. Did the weatherization staff help you to obtain the needed care?
(1) Yes
(2) No
25. Did your weatherization agency refer you to any other housing and/or social service programs?
(1) Yes
(2) No (go to Q26)
25a. What program or programs? ____________________________
26. Did you file a complaint about the weatherization services provided?
(1) Yes
(2) No (go to Q27)
26a. What was the complaint about?____________________
26b. How satisfied are you with the resolution of the situation you complained about?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
26c. How might the agency have done a better job of resolving your complaint?_______
27. Did you get any information on ways to save energy in your home from the people who weatherized
your home?
(1) Yes
(2) No (go to Q33)
377
28. How much time did the weatherization agency staff talk to you about ways to save energy?
(1) Less than 5 minutes
(2) 5 to 14 minutes
(3)15 to 29 minutes
(4) 30 to 60 minutes
(5) More than one hour
29. How well did you understand what the weatherization agency staff said to you about saving energy?
(1) Very well (go to Q30)
(2) Well (go to Q30)
(3) Neither well or not well (go to Q30)
(4) Not well
(5) Not well at all
29a. Why did you not understand what the weatherization agency staff said?
CHECK ALL THAT APPLY
(1) The staff person did not speak my primary language
(2) The staff person was confusing
(3) The staff person did not speak well
(4) The staff person was hurried
(5) The staff person was boring
(6) I did not get along with the staff person
(7) Other_________________
30. What materials about saving energy did the weatherization agency staff give you? (Check all that
apply)
(1) One or more brochures, booklets, or manuals
(2) One or more compact discs (CDs), videos, or DVDs
(3) Hardware kit of weatherization materials
(4) No materials were provided (go to Q31)
(5) Weatherization staff spent time demonstrating how to save energy (go to Q31)
30a. How much time have you spent reading/reviewing the materials about saving energy that the
weatherization agency staff gave you?
(1) No time (go to Q31)
(2) Less than 5 minutes
(3) 5 to 14 minutes
(4) 15 to 29 minutes
(5) 30 to 59 minutes
(6) More than one hour
30b. How well did you understand the energy savings materials that the weatherization agency staff gave
you?
(1) Very well
(2) Well
(3) Neither well or not well
(4) Not well
(5) Not well at all
378
30c. How useful have the energy savings materials been to you?
(1) Very useful
(2) Useful
(3) Neither useful or not useful
(4) Not useful
(5) Not very useful
30d. What about the materials were particularly useful? ___________________
30e. How could the materials have been improved for your use? ____________________
31. How satisfied are you with the ways that the weatherization agency provided you with information
about saving energy?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
32. Did you get any information on ways to improve health and safety in your home from the people who
weatherized your home?
(1) Yes
(2) No (go to Q38)
33. How much time did the weatherization agency staff talk to you about ways to improve health and
safety?
(1) Less than 5 minutes
(2) 5 to 14 minutes
(3) 15 to 29 minutes
(4) 30 to 59 minutes
(5) More than one hour
34. How well did you understand what the weatherization agency staff said to you about improving health
and safety?
(1) Very well (go to Q36)
(2) Well (go to Q36)
(3) Neither well or not well (go to Q36)
(4) Not well
(5) Not well at all
34a. Why did you not understand what the weatherization agency staff said?
(Check all that apply)
(1) The staff person did not speak my primary language
(2) The staff person was confusing
(3) The staff person did not speak well
(4) The staff person was hurried
(5) The staff person was boring
(6) I did not get along with the staff person
(7) Other_________________
379
35. What materials about improving health and safety did the weatherization agency staff give you?
(Check all that apply)
(1) One or more brochures, booklets and manuals
(2) One or more compact discs
(3) One or more videos (including DVD‘s)
(4) No materials were provided (go to Q36)
35a. How much time have you spent reading/reviewing the materials about improving health and safety
that the weatherization agency staff gave you?
(1) No time (go to Q36)
(2) Less than 5 minutes
(3) 5 to 14 minutes
(4) 15 to 29 minutes
(5) 30 to 59 minutes
(6) More than one hour
35b. How well did you understand the improving health and safety materials that the weatherization
agency staff gave you?
(1) Very well
(2) Well
(3) Neither well or not well
(4) Not well
(5) Not well at all
35c. How useful have the improving health and safety materials been to you?
(1) Very useful
(2) Useful
(3) Neither useful or not useful
(4) Not useful
(5) Not very useful
35d. What about the materials were particularly useful? ___________________
35e. How could the materials have been improved for your use? ____________________
36. How satisfied are you with the ways that the weatherization agency provided you with information
about improving health and safety?
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
37. How could the agency improve the ways that it provides households with information about
improving health and safety? ___________________________________
38. What are some of the greatest benefits your household received by participating in the weatherization
program? ___________________
39. What suggestions do you have for how the weatherization program can be improved?
________________
380
40. In the last 12 months, have you informed other people who might be interested in receiving
weatherization services of the program?
(1) Yes
(2) No
41. Do you know if these people have had their homes weatherized or are scheduled to have their home
weatherized, as a result of your suggestion?
(1) Yes
(2) No
42. Why did you apply for the Weatherization Assistance Program?66
(1) Reduce energy bills
(2) Support environmental efforts to conserve energy
(3) Make home more comfortable
(4) Receive free services
(5) Other
(6) Don‘t know
43. Prior to receiving weatherization services, in what ways did your household attempt to weatherize
your home?
Please
Explain______________________________________________________________________________
_____________________________________________________________________________________
________________________________________________________________
44. Would you say your household is now less likely to move from your current home as a result of
weatherization?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
45. Please rate the chances of your household‘s moving during the next 12 months for any reason or
combination of reasons:
(1) Very high
(2) High
(3) Medium
(4) Low
(5) Very low
(6) No chance
46. Were you asked about preexisting health conditions that you or your family members living in the
home may have?
a. Yes
b. No
66
Ohio REACH
381
47. Did the crew take precautions so that those conditions were not made worse?
a. Yes
b. No
c. Don‘t Know
d. N/A
48. How do you feel as far as the health and safety of your home since weatherization was conducted?
a. Much better
b. Better
c. Neither better or worse
d. Worse
e. Much Worse
49. Finally, please rate your overall satisfaction with the weatherization program.
(1) Very satisfied
(2) Satisfied
(3) Not satisfied or dissatisfied
(4) Dissatisfied
(5) Very dissatisfied
QUESTIONS TO BE ADDED TO CLIENT SATISFACTION PART
Financial Programs – Household Participants
1. Do you understand the terms of the loan/on-bill payment agreement?
a. Yes
b. No
If not, what do you not understand? Please explain _________________
2. Have you noticed a change in your utility bills since your home was weatherized?
a. Yes, bills have gone down substantially
b. Yes, bills have gone down somewhat
c. No, no change
d. Yes, bills have gone up somewhat
e. Yes, bills have gone up substantially
f. Don‘t know
3. Have you been able to afford the loan/ on-bill payments?
a. No
b. Yes, but just barely
c. Yes
4. In retrospect, would you make the same decision to participate in this loan/on-bill payment program?
a. Yes
b. No
If No, please explain __________________________________
382
5. How did you hear about this program?
a. Advertisement
b. Directly contacted by program
c. From relatives/friends
d. Browsing the Internet
e. Other ____________________
Financial Programs --Multi-family renting residents
1. Do you know that your building owner is participating in a program that subsidizes weatherization of
your building?
a. No
b. Yes
2. Did the owner consult with you and other tenants about the program before signing up for it?
a. No
b. Yes
3. Did your building association consult with you about the program?
a. No
b. Yes
c. N/A, there is no building association
4. Has your rent increased since XXXX?
a. Yes, rent has gone down substantially
b. Yes, rent has gone down somewhat
c. No, no change
d. Yes, rent has gone up somewhat
e. Yes, rent has gone up substantially
f. Don‘t know
5. Have your utility bills changed since XXXX?
a. Yes, bills have gone down substantially
b. Yes, bills have gone down somewhat
c. No, no change
d. Yes, bills have gone up somewhat
e. Yes, bills have gone up substantially
f. Don‘t know
g. I do not pay utility bills
383
6. Over the past XXXX months, please rate the changes in the following in your building and/or your
unit:
Much Better Better
No Change
Worse
Much Worse
Not
Applicable
Hallway lighting
Stairway lighting
Entrance lighting
Comfort of your
unit in winter
(If survey is
administered
prior to their
experiencing
a winter since
the retrofit)
Same as
above.
Comfort of your
unit in the
summer
7. Have you received any education about how to reduce energy consumption in your unit during the past
12 months?
a. No
b. Yes
To those who received new technology (active):
Are you aware that you received ???? in conjunction with the Weatherization Program you applied for?
a. Yes
b. No
Did you receive ???? at the same time as the organization weatherized your home?
a. Yes
b. No
c. Don‘t Know
Did the agency mention anything different about ???? as compared to the other improvements they made
in your home?
a. Yes
b. No
1. Do you understand how ???? works?
a. No
b. Yes
2. Did you receive any education and/or training about how to operate ?????
a. No
b. Yes
c. N/A
384
3. Did you receive any education and/or training about how to maintain ????
a. No
b. Yes
c. N/A
4. Does ???? seem to work?
a. Yes (go Q7)
b. No
c. Don‘t Know
5. In your opinion, did ???? ever work?
a. No (go to Q7)
b. Yes
6. How long ago do you think that ???? stopped working?
a. 1-2 months
b. 2-4 months
c. 5-6 months
d. 7-8 months
e. more than 8 months ago
7. Have your utility bills changed since ???? was installed?
a. Yes, bills have gone down substantially
b. Yes, bills have gone down somewhat
c. No, no change
d. Yes, bills have gone up somewhat
e. Yes, bills have gone up substantially
f. Don‘t know
8. Have you experienced any other benefits to yourself or your household since ???? was installed?
a. No
b. Yes
If yes, please explain ___________________________________________
9. Do you have any worries about ????
a. No
b. Yes
If yes, please explain _____________________________________________
10. Are you glad, in retrospect, to have ????? installed?
a. No
b. Yes
11. Have any of your friends and/or relatives asked you about your new ?????
a. No
b. Yes
12. If so, does it seem as though they would also be interested in having ???? installed in their homes?
a. Yes, most are interested
b. Yes, some are interested
c. No, none seem interested
385
In-home monitors in addition
1. Do you understand the information that the in-home monitor provides to you?
a. Yes
b. No
If no, please explain ___________________________________________
2. How often do you read the information in the in-home monitor?
a. Several times a day
b. About once a day
c. Once every few days
d. Once a week
e. Once every couple of weeks
f. Once a month
g. Fewer than once a month
h. Never
3. How easy is the in-home monitor to read?
a. Very easy
b. Easy
c. Neither easy or hard
d. Hard
e. Very hard
4. Did you receive education about how to read the in-home monitor?
a. No
b. Yes
5. From your perspective, how could the in-home monitor be improved? ________________
6. Has the in-home monitor influenced how your household uses energy?
a. No
b. Yes
If yes, please explain ________________________________________________
7. Do you change thermostat settings in the winter based on information from the in-home monitor?
a. No
b. Sometimes
c. Frequently
8. Do you change thermostat settings in the summer based on information from the in-home monitor?
a. No
b. Sometimes
c. Frequently
386
Financial Programs – Household Participants
1. How did you hear about this program?
a. Local weatherization agency
b. From relatives or friends
c. Advertisement on TV, radio or in newspaper
d. Other _____________________
e. Don‘t know
2. Have you noticed a change in your utility bills since your home was weatherized?
a. Yes, bills have gone down substantially
b. Yes, bills have gone down somewhat
c. No, no change
d. Yes, bills have gone up somewhat
e. Yes, bills have gone up substantially
f. Don‘t know
3. Have you been able to afford the loan/ on-bill payments?
a. No
b. Yes, but just barely
c. Yes
4. Do you understand the terms of the loan/on-bill payment agreement?
a. Yes
b. No
If not, what do you not understand? Please explain _________________
5. In retrospect, would you make the same decision to participate in this loan/on-bill payment program?
a. Yes
b. No
If No, please explain __________________________________
Financial Programs --Multi-family renting residents
1. Do you know that your building owner is participating in a program that subsidizes weatherization of
your building?
a. No
b. Yes
2. Did the owner consult with you and other tenants about the program before signing up for it?
a. No
b. Yes
3. Did your building association consult with you about the program?
a. No
b. Yes
c. N/A, there is no building association
387
4. Has your rent increased since XXXX?
a. Yes, rent has gone down substantially
b. Yes, rent has gone down somewhat
c. No, no change
d. Yes, rent has gone up somewhat
e. Yes, rent has gone up substantially
f. Don‘t know
5. Have your utility bills changed since XXXX?
a. Yes, bills have gone down substantially
b. Yes, bills have gone down somewhat
c. No, no change
d. Yes, bills have gone up somewhat
e. Yes, bills have gone up substantially
f. Don‘t know
g. I do not pay utility bills
6. Over the past XXXX months, please rate the changes in the following in your building and/or your
unit:
Much Better Better
No Change
Worse
Much Worse
Not
Applicable
Hallway lighting
Stairway lighting
Entrance lighting
Comfort of your
unit in winter
(If survey is
administered
prior to their
experiencing a
winter since
the retrofit)
Same as
above.
Comfort of your
unit in the
summer
7. Have you received any education about how to reduce energy consumption in your unit during the past
12 months?
a. No
b. Yes
388
To those who received new technology (active):
1. Are you aware that you received ???? in conjunction with the Weatherization Program you applied for?
c. Yes
d. No
2. Did you receive ???? at the same time as the organization weatherized your home?
d. Yes
e. No
f. Don‘t Know
3. Did the agency mention anything different about ???? as compared to the other improvements they
made in your home?
c. Yes
d. No
4. Do you understand how ???? works?
a. No
b. Yes
5. Did you receive any education and/or training about how to operate ?????
a. No
b. Yes
c. N/A
6. Did you receive any education and/or training about how to maintain ????
a. No
b. Yes
c. N/A
7. Does ???? seem to work?
a. Yes (go Q7)
b. No
c. Don‘t Know
8. In your opinion, did ???? ever work?
a. No (go to Q7)
b. Yes
9. How long ago do you think that ???? stopped working?
a. 1-2 months
b. 2-4 months
c. 5-6 months
d. 7-8 months
e. more than 8 months ago
389
10. Have your utility bills changed since ???? was installed?
a. Yes, bills have gone down substantially
b. Yes, bills have gone down somewhat
c. No, no change
d. Yes, bills have gone up somewhat
e. Yes, bills have gone up substantially
f. Don‘t know
11. Have you experienced any other benefits to yourself or your household since ???? was installed?
a. No
b. Yes
If yes, please explain ___________________________________________
12. Do you have any worries about ????
a. No
b. Yes
If yes, please explain _____________________________________________
13. Are you glad, in retrospect, to have ????? installed?
a. No
b. Yes
14. Have any of your friends and/or relatives asked you about your new ?????
a. No
b. Yes
15. If so, does it seem as though they would also be interested in having ???? installed in their homes?
a. Yes, most are interested
b. Yes, some are interested
c. No, none seem interested
In-home monitors installed in homes
1. Do you understand the information that the in-home monitor provides to you?
a. Yes
b. No
If no, please explain ___________________________________________
2. How often do you read the information in the in-home monitor?
a. Several times a day
b. About once a day
c. Once every few days
d. Once a week
e. Once every couple of weeks
f. Once a month
g. Fewer than once a month
h. Never
390
3. How easy is the in-home monitor to read?
a. Very easy
b. Easy
c. Neither easy or hard
d. Hard
e. Very hard
4. Did you receive education about how to read the in-home monitor?
a. No
b. Yes
5. From your perspective, how could the in-home monitor be improved? ________________
6. Has the in-home monitor influenced how your household uses energy?
a. No
b. Yes
If yes, please explain ________________________________________________
7. Do you change thermostat settings in the winter based on information from the in-home monitor?
a. No
b. Sometimes
c. Frequently
8. Do you change thermostat settings in the summer based on information from the in-home monitor?
a. No
b. Sometimes
c. Frequently
That is the end of the survey. Thank you for your participation! You will receive your $10 gift card in the
mail to compensate you for your time. Could you please verify your mailing address:
Address: __________________________
__________________________
__________________________
__________________________
391
392
OMB Control Number: 1910-5151
APPENDIX L – DF9 OCCUPANT SURVEY INFORMATION DATA FORM
DF9: Contact Information from Agencies for Occupant Survey
This information is being collected to choose a sample of homes that are expected to receive
weatherization in order to administer a survey to the occupants. The information will be used to build a
sampling frame to use to randomly select homes for this survey.
Public reporting burden for this collection of information is estimated to average one hour per response,
including the time for reviewing instructions, searching existing data sources, gathering and maintaining
the data needed, and completing and reviewing the collection of information. Send comments regarding
this burden estimate or any other aspect of this collection of information, including suggestions for
reducing this burden, to Office of the Chief Information Officer, Records Management Division, IM-11,
Paperwork Reduction Project (1910-5151), U.S. Department of Energy, 1000 Independence Ave SW,
Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork
Reduction Project (1910-5151), Washington, DC 20503.
Introduction: As part of the national evaluation of the Weatherization Assistance Program, occupants of
housing units weatherized in Program Year 2010 will be surveyed before and after weatherization to
obtain information on their health and demographics, determine their energy consumption behavior and
knowledge, evaluate non-energy benefits, and establish their satisfaction with the Program. In addition,
occupants of households that have received (or will receive) assistance from the Low-Income Home
Energy Assistance Program (LIHEAP) but whose houses have not been (or will not be) weatherized will
be surveyed to serve as a control group for the occupant survey.
We would like to survey several such occupants served by your agency. In order for us to randomly select
several weatherized households, please provide us with the names and contact information for clients
currently in the queue (i.e., in line) for receiving audits using the first attached form. Similarly, please use
the second attached form to provide us with the names and contact information for clients your agency
has provided, or will provide, LIHEAP assistance in Program Year 2010 so we can select several control
households (provide a maximum of 20 names). [NOTE: If weatherization services and LIHEAP
assistance are provided by two separate organizations, then similar instructions and forms will be used
that only request information on weatherization clients or LIHEAP recipients.]
The information that you provide and that we obtain from the occupants will be protected and will remain
confidential. When results of the survey are reported, information collected from the occupants will not
be associated with their names in any way. In order to maintain the integrity of the survey, we will not be
able to tell you which clients have been selected for the survey. We will be paying the occupants a small
incentive to participate in the survey, so an undo burden will not be placed on your clients.
393
Clients in Queue for Audits
Agency name: _______________________________
State: ______________________________________
Date: ______________________________________
Please provide the following information for clients currently in the queue to receive audits by your
agency.
Name
Address
City
State
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
394
Zip Code
Phone
Number
Expected
Audit
Date
OMB Control Number: _ _ _ _ - _ _ _ _
S4 - OCCUPANT SURVEY: MULTI-FAMILY UNIT VERSION
This data is being collected to conduct a survey of occupants about their experiences with the
Weatherization Assistance Program and benefits they believe they have received from the
program. The data you supply will be used to describe occupant satisfaction with the program as
well as changes in energy education and non-energy benefits provided by the program.
Public reporting burden for this collection of information is estimated to average ____ minutes
per response, including the time for reviewing instructions, searching existing data sources,
gathering and maintaining the data needed, and completing and reviewing the collection of
information. Send comments regarding this burden estimate or any other aspect of this
collection of information, including suggestions for reducing this burden, to Office of the Chief
Information Officer, Records Management Division, IM-11, Paperwork Reduction Project
(XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW, Washington, DC,
20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork Reduction
Project (XXXX-XXXX), Washington, DC 20503.
Lastly, all of the information obtained from this survey will be protected and will remain
confidential. The data will be analyzed in such a way that the information provided cannot be
associated back to you or your household. Your answers will not be shared with or reported back
to anyone within the agency that served you or your state.
POST-WEATHERIZATION SURVEY
{INTERVIEWER: VERIFY RESPONDENT IS THE TENANT ON THE APARTMENT
LEASE}Before we begin, I need to verify that this is one of the people on the apartment lease. Is
this your apartment?
PLEASE RECORD RESPONDENT‘S NAME, GENDER and AGE
Name
Gender
Age
Main Respondent:
1. How long have you lived in your current apartment?
{If less than one year} Enter: _________ months
Enter: _________ years
1a. {IF RESPONDENT HAS LIVED IN THE APARTMENT < ONE YEAR} Has ANY
OTHER ADULT living in the household lived in the apartment for more than one year?
395
(3) Yes
(4) No
{IF YES} Name ______________________
{IF RESPONDENT DID NOT LIVE IN THE APARTMENT PRIOR TO WEATHEIZATION:
STOP SURVEY. IF OTHER ADULT LIVED IN THE APARTMENT PRIOR TO
WEATHERIZATION: REQUEST TO INTEVIEW THAT INDIVIDUAL OR CALL BACK IF
NOT AVAILABLE}
1b. Did you know that your building owner participated in a program that subsidized the
weatherization of your building?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
{ IF NOT SURE, IF LIVED IN BUILDING FOR N MONTHS INFORM THEM THAT
THE WORK COMPLETED ON THE BUIDLING WAS PART OF A WEATHERIZATION
PROGRAM AND THAT YOU WILL BE REFERRING TO THIS WORK AS
WEATHERIZATION THROUGHOUT THE SURVEY.}
PART I. Energy Consumption, Non-Energy Impacts, Health and Demographics
In this next section, I will be asking you about your apartment and your use of energy.
2. How many bedrooms do you have in your apartment? [Include bedrooms in finished attics or
finished basements.] ______________
3. How many bathrooms do you have in your apartment?____________
4. Now think about other rooms in your apartment besides bedrooms and bathrooms. Not
including unfinished areas, hallways, and closets, how many other rooms are there in your
apartment?
Number of Rooms__________________
I have some questions about heating your apartment.
5. Before the weatherization of the building, did you use any of the following
types of heating equipment in your apartment?
CHECK ALL THAT APPLY
(13) Portable heaters
(14) Fireplace
(15) Cooking stove used to heat your apartment as well as to cook
(16) Some other equipment (Specify __________________)
(17) No other equipment
6. During the past N months (SINCE WEATHEIZATION WORK WAS COMPLETED), have
you used any of the following types of heating equipment in your apartment in the winter?
396
CHECK ALL THAT APPLY
(1) Portable heaters
(2) Fireplace
(3) Cooking stove used to heat your apartment as well as to cook
(4) Some other equipment (Specify __________________)
(5) No other equipment
7. What fuel does your cooking stove and/or oven use? CHECK ALL THAT APPLY
(9) Electricity
(10) Natural gas from underground pipes
(11) Propane (bottled gas)
(12) Fuel oil
(13) Kerosene
(14) Wood
(15) Some other fuel (Specify __________)
(16) No working stove or oven in the apartment
8. Which of the categories shown best describes how often the most used oven is used?
(1) Three or more times a day
(2) Two times a day
(3) Once a day
(4) A few times each week
(5) About once a week
(6) Less than once a week
(7) Not used (if volunteered)
9. During the past N months, how many times has the fire department been called to put out a
fire in your apartment? _________
10. During the past N months, has any fire start in your apartment as a result of using an alternate
heating source, such as space heaters, electric blankets, your kitchen stove or oven, or your
fireplace?
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
Now I have some questions about cooling your apartment.
(IF NO TO AIR CONDITIONING EQUIPMENT SKIP TO CEILING FAN USE Q15)
11. Is any air conditioning equipment used in your apartment?
(9) Yes (SKIP to Q13)
(10) No
(11) Don‘t Know/Not Sure
(12) Refused
397
12. (IF NO) Just to clarify, do you have air conditioning equipment but don‘t use it, or does your
apartment just not have any air conditioning equipment?
(5) Have equipment, but don‘t use it (SKIP to Q 20)
(6) Don‘t have any air conditioning equipment
(7) Don‘t Know/Not Sure
(8) Refused
13. Which of the statements shown best describes the way your central air conditioning system
was used last summer?
(5) Not used at all (if volunteered)
(6) Turned on only a few days or nights when really needed
(7) Turned on quite a bit
(8) Turned on just about all summer
14. Which of the following statements best describes the way your household used the
most used window/wall air conditioning unit last summer?
(5) Turned on only a few days or nights when really needed
(6) Turned on quite a bit
(7) Turned on just about all summer
(8) No working window/wall units in apartment
15. How many ceiling fans does your apartment have?
Enter Number_________
(4) None (SKIP to Q 29)
(5) Don‘t Know/Not Sure
(6) Refused
16. How many ceiling fans does your household use?
Enter Number of ceiling fans_______
(4) None (SKIP to 50)
(5) Don‘t Know/Not Sure
(6) Refused
27. Thinking about the ceiling fan that you use the most, how often was this fan used last
summer? Is it . . .
(7) Used only a few days or nights, when it‘s really needed,
(8) Used quite a bit, or
(9) Used just about all summer?
(10) Not used at all
(11) Don‘t Know/Not Sure
(12) Refused
28. Thinking about the ceiling fan that you use the most, how often was this fan used last winter?
Is it . . .
(7) Used only a few days or nights,
(8) Used quite a bit, or
(9) Used just about all winter?
398
(10) Not used at all
(11) Don‘t Know/Not Sure
(12) Refused
29. How often are your windows open in the summer?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
29. How often are your windows open in the winter?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
30. Do any large trees shade your apartment from the afternoon summer sun?
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
31. Do you close the drapes, curtains, shades, and/or blinds during the day to block out the sun
during the summer?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
INTERVIEWER INSTRUCTION: RECORD ANY INFORMATION HERE ABOUT THE
AIRCONDITIONING EQUIPMENT IN THIS HOUSING UNIT AND ITS‘ USAGE THAT
MIGHT PROVIDE CLARIFICATION TO THE RESPONDENT‘S ANSWERS.
______________________________________________________________________________
________________________________________________________________________
________________________________________________________________________
Now I have some questions on the indoor air temperature of your apartment.
399
32. Does your apartment have a thermostat that controls the heating and/or cooling in your
apartment?
(5) Yes
(6) No (SKIP to Q 33)
(7) Don‘t Know
(8) Refused
32a. What equipment does your thermostat control?
(1) Heating only
(2) Cooling only
(3) Heating and cooling
(4) Don‘t know
32b. Some thermostats can be programmed so that the temperature changes automatically at
different times of the day; for example, the heat can be automatically turned down or lowered at
night when you go to bed, then automatically adjusted up again in the morning. Is the thermostat
that controls your main [heating and/or cooling] equipment programmable?
(5) Yes
(6) No (SKIP to Q32d)
(7) Don‘t know/Not Sure
(8) Refused
32c. Do you or someone else in your household know how to use the programmable thermostat?
(6) Yes
(7) No
(8) No, someone who does not live in my apartment programs the thermostat for use
(9) Don‘t know/Not Sure
(10) Refused
32d. During the winter, what is the temperature when someone is inside your apartment during
the day? [IF NO ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE
THERMOSTAT SET? PROBE 2: CAN I JUST HAVE YOUR BEST ESTIMATE?]
Enter degrees Fahrenheit________
32e. During the winter, what is the temperature when no one is inside your apartment during the
day? [IF NO ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE
THERMOSTAT SET? PROBE 2: CAN I JUST HAVE YOUR BEST ESTIMATE?]
Enter degrees Fahrenheit________
32f. During the winter, what is the temperature inside your apartment at night? [IF NO
ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET?
PROBE 2: CAN I JUST HAVE YOUR BEST ESTIMATE?]
Enter degrees Fahrenheit________
400
Now I would like you to think about the temperature inside your apartment when using your
central air conditioning equipment last summer. [If NUMTHERM>1: Earlier you reported
having [FILL: NUMTHERM] thermostats. For the next questions, if the thermostats are set at
different temperatures, only report for the thermostat that affects the rooms where most of the
people are.]
32g. During the summer, what is the temperature when someone is inside your apartment during
the day? IF NO ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE
THERMOSTAT SET? PROBE 2: WHAT‘S YOUR BEST ESTIMATE?
Enter degrees Fahrenheit ________
Air-conditioner Turned Off
32h. During the summer, what is the temperature when no one is inside your apartment during
the day? IF NO ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE
THERMOSTAT SET? PROBE 2: WHAT‘S YOUR BEST ESTIMATE?
Enter degrees Fahrenheit ________
Air-conditioner Turned Off
32i. During the summer, what is the temperature inside your apartment at night? IF NO
ANSWER, PROBE 1: THEN AT WHAT TEMPERATURE IS THE THERMOSTAT SET?
PROBE 2: WHAT‘S YOUR BEST ESTIMATE?
Enter degrees Fahrenheit ________
Air-conditioner Turned Off
We‘re interested in knowing how tenants in apartment buildings control the temperature inside
their apartments especially when they feel too hot or too cold. Remember, your answers are
confidential and they will not be collected or reported in a way that ties them back to you.
33. During the past N months, has your thermostat kept your apartment at a comfortable
temperature?
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) NA: No Thermostat
(9) Refused
33a. (IF NO) What have you done about it?____________________________________
34. Before the building was weatherized, how would you describe the temperature in your
apartment in the winter?
(8) Very cold
(9) Cold
401
(10)
(11)
(12)
(13)
(14)
Comfortable (SKIP)
Hot
Very hot
Other ________________
Refused
35. During that time, other than adjustments to a thermostat, how did you control the indoor
temperature of your apartment in the winter? ____________________________________
35a. Do you continue to do those things now that the weatherization work is complete?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
36. Before the building was weatherized, how would you describe the temperature in your
apartment in the summer?
(1) Very cold
(2) Cold
(3) Comfortable (SKIP)
(4) Hot
(5) Very hot
(6) Other ________________
(7) Refused
37. During that time other than adjustments to a thermostat, how did you control the indoor
temperature of your apartment in the summer?
37a. Do you continue to do those things now that the weatherization work is complete?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
38. During the past N months, was your household unable to use any of the following equipment
because it was broken?
CHECK ALL THAT APPLY
(6) Main Heating Equipment
(7) Central Air Conditioner
(8) Room Air Conditioner
(9) Don‘t Know/Not Sure
(10)
Refused
39. Thinking about the past N months, which of the following statements best describes the
indoor temperature of your apartment during the winter:
402
(1)
(2)
(3)
(4)
(5)
(6)
(7)
Very cold
Cold
Comfortable
Hot
Very hot
Other ________________
Refused
40. Thinking about the past N months, which of the following statements best describes the
indoor temperature of your apartment during the summer:
(8) Very cold
(9) Cold
(10)
Comfortable (SKIP TO NEXT Q….)
(11)
Hot
(12)
Very hot
(13)
Other ________________
(14)
Refused
41. During the past N months, has a landlord or superintendant controlled the temperature inside
your apartment?
(1) Yes
(2) No
(3) Do not have landlord
(4) Don‘t Know/Not Sure
(5) Refused
42. During the past N months, how often was your apartment at a temperature that you felt was
unsafe or unhealthy?
(7) Almost every month
(8) Some months
(9) 1 or 2 months
(10) Never
(11) Don‘t Know/Not Sure
(12) Refused
43. During the past N months, has anyone in the household needed medical
attention because your apartment was too cold?
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
44. During the past N months, did anyone in your household need medical attention because
your apartment was too hot?
(5) Yes
(6) No
403
(7) Don‘t Know/Not Sure
(8) Refused
The next group of questions is about laundry appliances and water use in your apartment.
45. Is a clothes washing machine used in your apartment? Do not include community clothes
washers that are located in the basement or laundry room of your apartment building.
(4) Yes
(5) No
(6) Refused
46. Do you use community clothes washers in the basement or laundry room of your apartment
building?
(1) Yes
(2) No
(3) Refused
47. In an average week, how many loads of laundry are washed in your clothes washer?
(8) 1 load or less each week
(9) 2 to 4 loads each week
(10) 5 to 9 loads each week
(11) 10 to 15 loads each week
(12) More than 15 loads each week
(13) Don‘t Know/Not Sure
(14) Refused
48. What water temperature setting is usually used for the wash cycle of your clothes washer? Is
it hot, warm, or cold water?
(6) Hot
(7) Warm
(8) Cold
(9) Don‘t Know/Not Sure
(10) Refused
49. What water temperature setting is usually used for the rinse cycle of your clothes washer? Is
it hot, warm, or cold water?
(6) Hot
(7) Warm
(8) Cold
(9) Don‘t Know/Not Sure
(10) Refused
50. Do you use a clothes dryer in your apartment? Do not include community clothes dryers that
are located in the basement or laundry room of your apartment building.
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
404
51. Do you use community clothes washers in the basement or laundry room of your apartment
building?
(1) Yes
(2) No
(3) Don‘t Know/Not Sure
(4) Refused
52. How frequently does your household hang clothes to dry?
(1) Very frequently
(2) Frequently
(3) Infrequently
(4) Very infrequently
(5) Never
(6) Don‘t Know/Not Sure
(7) Refused
53. Does your main bathroom have a ventilation fan in it that works?
(1) Yes
(2) No (SKIP to Q56)
(3) Don‘t know/Not Sure (SKIP to Q56)
(4) Refused
54. How often do you or members of your household operate the fan while showering?
(1) Never
(2) Rarely
(3) Sometimes
(4) Frequently
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
55. How long after showering do you or members of your household operate the fan?
(1) Don‘t turn the fan on for showers
(2) The fan is turned off when leaving the shower area
(3) A few minutes
(4) Several minutes
(5) Until the steam in the shower area is gone
(6) Don‘t know/Not Sure
(7) Refused
56. Do any of your faucets or bath or shower fixtures leak?
57. Electric dehumidifiers remove moisture from the air and are often used in the summer. Is a
dehumidifier used in your apartment?
(5) Yes
405
(6) No (SKIP to Q59)
(7) Don‘t Know/Not Sure
(8) Refused
58. Before the building was weatherized, was a dehumidifier used in your apartment?
(1) Yes
(2) No
Now I have a couple of questions about lights inside your apartment and energy efficiency.
59. How often do you find lights left on in rooms that are not occupied?
(1) Never
(2) Almost never
(3) Sometimes
(4) Most of the time
(5) All the time
(6) Don‘t Know/Not Sure
(7) Refused
60. About how may CFL bulbs are in lighting fixtures in your apartment right now?
Number of bulbs:____________
Now I have some questions about other appliances used in the apartment.
61. Do you have a computer in your unit?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
(IF YES)
61a. How many computers do you have?______________
62. Do you unplug any appliances like TVs, VCRs, stereos, radios, clocks, or computers to save
energy when they are turned off?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
63. Do you have an extra refrigerator or freezer in the unit?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
64. Does your household use a microwave oven?
406
(5)
(6)
(7)
(8)
Yes
No (SKIP to 66)
Don‘t Know/Not Sure
Refused
65. Which answer best describes how frequently your household uses the microwave to prepare
hot meals and snacks in a typical week?
(7) Used to cook or reheat most meals and snacks
(8) Used to cook or reheat about half of meals and snacks
(9) Used to cook or reheat a few meals and snacks
(10) ...................................................................................................................... Used
very little
(11) ...................................................................................................................... Don‘t
Know/Not Sure
(12) ...................................................................................................................... Refus
ed
Now I would like to ask you a few questions about your energy bills.
Some households may have faced challenges in paying apartment energy bills. The following
questions ask about challenges your household may have had paying apartment energy. When
thinking about these questions, include all of your experiences since the work on the building
was completed.
66. Are your energy bills for your apartment paid by you separate from your rent?
(1) Yes
(2) No {SKIP to Q69}
(3) Don‘t Know/Not Sure
(4) Refused
67. How well do you understand the information on your energy bill other than the amount owed
(e.g., information about how much energy your household used during the billing period
compared to the same billing period one year ago)?
(1) Very well
(2) Well
(3) Neither well nor not well
(4) Not well
(5) Not well at all
(6) Don‘t know/Not Sure
(7) Refused
68. How hard is it to pay your energy bills?
(8) Very hard
(9) Hard
(10) Neither hard or not hard
(11) Not hard
407
(12) Not hard at all
(13) Don‘t know/Not Sure
(14) Refused
69. Has your household ever had to move in the past 5 years because your household could not
pay the energy bills?
(4) Yes
(5) No
(6) Don‘t know
70. In the past 5 years have you or anyone in the household experienced any of the following as a
result of energy bills? CHECK ALL THAT APPLY.
(7) Eviction from apartment
(8) Foreclosure on mortgage
(9) Moved in with friends or family
(10)
Moved into a shelter or been apartment less
(11)
Family Separation as a result of moving
(12)
Refused
71. Including yourself, how many people normally live in this household? Do not include
anyone who is just visiting, those away in the military, or children who are away at college.
Enter Number _______________
72. Can you please tell me their first names, gender and age, and your relationship to the person?
First Name
Gender
Age
Relationship
In school (Y/N)
Person 1.
Person 2.
Person 3.
Person 4.
Person 5.
Person 6.
Person 7.
Person 8.
Person 9.
Person 10.
73. On a typical week day is there someone at the apartment most or all of the day?
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
For this section, I will be asking health related questions.
408
In the past 12 months were you or anyone else in the household ever told by a doctor or health
professional that you or they have. CHECK ALL THAT APPLY
74. Lead poisoning
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
{IF YES}Please list all individuals, including yourself:
____________________
____________________
____________________
75. Any kind of respiratory allergy
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
____________________
76. Have you or anyone else in the household ever been told by a doctor or other health
professional that you have asthma?
(5) Yes
(6) No (SKIP to Q79)
(7) Don‘t Know/not sure
(8) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
____________________
77. Do you or they still have asthma?
(5) Yes
(6) No
(7) Don‘t Know/Not sure
409
(8) Refused
{IF YES} Please list all individuals, including yourself:
____________________
____________________
____________________
____________________
READ: Symptoms of asthma include coughing, wheezing, shortness of breath, chest tightness or
phlegm production when you have a cold or respiratory infection.
78. How long has it been since you last had any symptoms of asthma?
(11) Never
(12) Less than one day ago
(13) 1-6 Days ago
(14) 1 week to less than 3 months ago
(15) 3 months to less than 1 year ago
(16) 1 year to less than 3 years ago
(17) 3 years to 5 years ago
(18) More than 5 years ago
(19) Don‘t Know/Not sure
(20) Refused
79. Which one of the following statements best describes the rules about smoking in your
apartment…
(6) No one is allowed to smoke anywhere inside your apartment
(7) Smoking is allowed at some places or at sometimes
(8) Smoking is permitted anywhere
(9) Don‘t know/Not sure
(10)
Refused
80. During the past N months, has anyone in the household been poisoned by breathing in carbon
monoxide, and therefore went to see a medical professional?
(5) Yes
(6) No
(7) Don‘t Know/Not sure
(8) Refused
81. During the past N months, has anyone in the apartment been burned from scalding hot water
coming out of a faucet or showerhead in your apartment?
(5) Yes
(6) No (SKIP to Q83)
(7) Don‘t Know/Not Sure
(8) Refused
410
82. {IF YES BURN}Did you talk to or see a medical professional about this injury?
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
In this last section I will be asking employment and school related questions.
83. Does a physical, mental or emotional problem NOW keep you or the primary wage earner
from working at a job or business?
(5) Yes
(6) No
(7) Don‘t know/Not Sure
(8) Refused
84. {IF STUDENT} During the past N months, how frequently have you found it hard to study
in your apartment because of excessive heat or cold?
(10)
Very frequently
(11)
Frequently
(12)
Not frequently or infrequently
(13)
Infrequently
(14)
Very infrequently
(15)
Never
(16)
Does not study at apartment
(17)
Don‘t Know/Not sure
(18)
Refused
85. {IF SCHOOL AGED CHILDREN IN THE APARTMENT} During the past N months, how
frequently has any school aged child in the apartment found it hard to study because of excessive
heat or cold?
(10)
Very frequently
(11)
Frequently
(12)
Not frequently or infrequently
(13)
Infrequently
(14)
Very infrequently
(15)
Never
(16)
Does not study at apartment
(17)
Don‘t Know/Not sure
(18)
Refused
411
{INTERVIEWER: ADMINISTER AS A SEPARATE SURVEY POST-WEATHERIZATION}
PART II. Client Satisfaction
1. How did you first learn that your building was going to have weatherization work done on it?
(1) Informed during a tenant meeting
(2) By written notice
(3) Phone call from building management
(4) Observation of work when it started
(5) Other tenant(s)
(6) By workers/crew when they introduced themselves inside your apartment
(7) Don‘t Know/Not Sure
(8) Refused
2. Did the owner consult with you and other tenants about the program before they signed up for
it?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
(4) Refused
3. Did your building association consult with you about the program?
(1) Yes
(2) No
(3) N/A: There is no Building Association
(4) Don‘t know/Not Sure
(5) Refused
4. Was an energy audit conducted in your apartment?
(1) Yes
(2) No (SKIP to Q6)
(3) Don‘t Know/Not Sure (SKIP to Q6)
(4) Refused (SKIP to Q6)
5. How courteous were those who did the initial audit of your apartment?
(1) Very Courteous
(2) Courteous
(3) Not Courteous or Rude
(4) Rude
(5) Very Rude
6. Were energy conservation measures installed in your apartment? (GIVE EXAMPLES IF
NEEDED)?
(1) Yes
(2) No (SKIP to Q14)
412
(3) Don‘t Know/Not Sure (SKIP to Q14)
(4) Refused (SKIP to Q14)
7. How courteous was the weatherization crew when installing these measures?
(1) Very Courteous
(2) Courteous
(3) Not Courteous or Rude
(4) Rude
(5) Very Rude
8. How careful of your apartment and belongings was the weatherization crew?
(6) Very careful
(7) Careful
(8) Neither careful or careless
(9) Careless
(10)
Very careless
9. Overall, how clean did the weatherization crew leave the inside of your apartment?
(6) Very clean
(7) Clean
(8) Neither clean nor dirty
(9) Dirty
(10)
Very dirty
10. Overall, how satisfied are you with final condition the inside of your apartment was left in?
(5) Very satisfied
(6) Satisfied
(7) Not satisfied or dissatisfied
(8) Dissatisfied
(5) Very dissatisfied
11. How courteous were those who did the final inspection of your apartment?
(1) Very Courteous
(2) Courteous
(3) Not Courteous or Rude
(4) Rude
(5) Very Rude
12. How satisfied are you with the work performed in your apartment?
(5) Very satisfied
(6) Satisfied
(7) Not satisfied or dissatisfied
(8) Dissatisfied
(5) Very dissatisfied
413
13. How satisfied are you with any new equipment installed in apartment?
(5) Very satisfied
(6) Satisfied
(7) Not satisfied or dissatisfied
(8) Dissatisfied
(5) Very dissatisfied
14. Other than the measures installed in your building and/or apartment, do you feel that other
things should have been installed in your apartment to help you save energy?
(3) Yes
(4) No (go to Q15)
14a. What other things? ______________________
15. If you pay a utility bill for your apartment how satisfied are you with the energy savings
achieved after having your apartment weatherized?
(8) Very satisfied
(9) Satisfied
(10)
Not satisfied or dissatisfied
(11)
Dissatisfied
(12)
Very dissatisfied
(13)
Too soon to tell
(14)
Don‘t Know/Not Sure
(15)
I do not pay utility bill (SKIP to Q17)
16. Have your utility bills changed since your building was worked on?
a. Yes, bills have gone down substantially
b. Yes, bills have gone down somewhat
c. No, no change
d. Yes, bills have gone up somewhat
e. Yes, bills have gone up substantially
f. Don‘t know/Not Sure
g. I do not pay utility bills
17. Has your rent changed since your building was worked on?
a. Yes, rent has gone down substantially
b. Yes, rent has gone down somewhat
c. No, no change
d. Yes, rent has gone up somewhat
e. Yes, rent has gone up substantially
f. Don‘t know
18. Are you worried that your rent will increase within the next 5 years?
(1) Yes
(2) No
(3) Don‘t know/Not Sure
414
(4) Refused
19. Have you received any education about how to reduce energy consumption in your unit as
part of actions taken to lower the energy use in the building?
(1) Yes
(2) No (SKIP to Q24)
(3) Don‘t know/Not Sure
(4) Refused
20. How much time did the weatherization team talk to you about ways to save energy?
(1) Less than 5 minutes
(2) 5 to 14 minutes
(3)15 to 29 minutes
(4) 30 to 60 minutes
(5) More than one hour
21. How well did you understand what the weatherization team said to you about saving energy?
(1) Very well (go to Q22)
(2) Well (go to Q22)
(3) Neither well or not well (go to Q22)
(4) Not well
(5) Not well at all
21a. Why did you not understand what the weatherization team said?
CHECK ALL THAT APPLY
(1) The team member did not speak my primary language
(2) The team member was confusing
(3) The team member did not speak well
(4) The team member was hurried
(5) The team member was boring
(6) I did not get along with the team member
(7) Other_________________
22. What materials about saving energy did the weatherization team give you? CHECK ALL
THAT APPLY
(1) One or more brochures, booklets, or manuals
(2) One or more compact discs (CDs), videos, or DVDs
(3) Hardware kit of weatherization materials
(4) No materials were provided (go to Q23)
(5) Weatherization staff spent time demonstrating how to save energy (go to Q23)
22a. How much time have you spent reading/reviewing the materials about saving energy that
the weatherization team gave you?
(1) No time (go to Q31)
(2) Less than 5 minutes
(3) 5 to 14 minutes
415
(4) 15 to 29 minutes
(5) 30 to 59 minutes
(6) More than one hour
22b. How well did you understand the energy savings materials that the weatherization team
gave you?
(6) Very well
(7) Well
(8) Neither well or not well
(9) Not well
(10)
Not well at all
22c. How useful have the energy savings materials been to you?
(1) Very useful
(2) Useful
(3) Neither useful or not useful
(4) Not useful
(5) Not very useful
22d. What about the materials were particularly useful? ___________________
22e. How could the materials have been improved for your use? ____________________
23. How satisfied are you with the ways that the weatherization team provided you with
information about saving energy?
(6) Very satisfied
(7) Satisfied
(8) Not satisfied or dissatisfied
(9) Dissatisfied
(5) Very dissatisfied
24. What is the greatest benefit your household received from the weatherization of your
apartment building? ___________________
25. Would you say your household is now less likely to move from your current apartment as a
result of weatherization?
(5) Yes
(6) No
(7) Don‘t Know/Not Sure
(8) Refused
26. Since the building improvement please rate the changes in the following in your building
and/or your unit:
416
Much Better
Better
No Change
Worse
Much Worse
Entrance
lighting
Stairway
lighting
Hallway
lighting
Lighting in
your apartment
Comfort of
your unit in
winter
Comfort of
your unit in the
summer
27. Please rate your overall satisfaction with the weatherization work completed on your
building.
(6) Very satisfied
(7) Satisfied
(8) Not satisfied or dissatisfied
(9) Dissatisfied
(10)
Very dissatisfied
Demographics
1. Are you currently…?
(8) Married
(9) Divorced
(10)
Widowed
(11)
Separated
(12)
Never married
(13)
A member of an unmarried couple
(14)
Refused
2. What is the highest degree or level of school you have completed?
(11) No Schooling Completed
(12) Kindergarten to grade 12 (No Diploma)
(13) High school diploma or GED
(14) Some college, no degree
(15) Associate‘s degree (for example: AA, AS)
(16) Bachelor‘s degree (for example: BA, BS)
417
Not
Applicable
(17) Master‘s degree (for example: MA, MS, MBA)
(18) Professional degree (for example: MD, JD)
(19) Doctorate degree (for example: PhD, EdD)
(20) Refused
3. Do you consider yourself to be of Hispanic or Latino
origin, such as Mexican, Puerto Rican, Cuban, or other Spanish background?
(5) Yes
(6) No
(7) Don‘t know/Not Sure
(8) Refused
4. Which describes your race? You can select one or more categories.
(9) White
(10)
Black or African-American
(11)
American Indian or Alaska Native
(12)
Asian
(13)
Native Hawaiian or Other Pacific Islander
(14)
Other (if volunteered)
(15)
Hispanic or Latino (if volunteered)
(16)
Refused
4a. {IF MORE THAN ONE} Which ONE of these groups best represents your
race? You can select one or more categories.
(9) White
(10)
Black or African-American
(11)
American Indian or Alaska Native
(12)
Asian
(13)
Native Hawaiian or Other Pacific Islander
(14)
Other (if volunteered)
(15)
Hispanic or Latino (if volunteered)
(16)
Refused
That is the end of the survey. Thank you for your participation! You will receive your $----- gift
card in the mail to compensate you for your time. Could you please verify your mailing address:
Address: __________________________
__________________________
__________________________
__________________________
418
OMB Control Number: XXXX-XXXX
APPENDIX M: WEATHERIZATION STAFF SURVEY
This data is being collected to evaluate weatherization staff training and other workforce issues.
Public reporting burden for this collection of information is estimated to average thirty minutes
per response, including the time for reviewing instructions, searching existing data sources,
gathering and maintaining the data needed, and completing and reviewing the collection of
information. Send comments regarding this burden estimate or any other aspect of this
collection of information, including suggestions for reducing this burden, to Office of the Chief
Information Officer, Records Management Division, IM-11, Paperwork Reduction Project
(XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW, Washington, DC,
20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork Reduction
Project (XXXX-XXXX), Washington, DC 20503.
Part I. National Weatherization Staff
Introduction: Thank you for agreeing again to participate in the Weatherization Staff Survey
being conducted as part of the national evaluation of the Weatherization Assistance Program. All
of the information that we obtain from this survey will remain confidential and will be analyzed
in such a way that your answers cannot be associated with your name. Your answers will not be
shared with or reported back to anyone within your agency or state.
Part I – For National Weatherization Staff Survey Respondents
1. The first time we interviewed you, you reported you were working for ???????. Do you still
work for ?????
a. Yes
b. No (go to Q13)
2. The first time we interviewed you, you reported your primary weatherization job title as ?????.
Is this still your primary weatherization job title?
a. Yes (go to Q4)
b. No
3a. What is your primary weatherization job title?
a. Administrator
b. Auditor
c. Inspector
d. Crew leader/foreman
e. Crew member
f. Other ___________
g. I work for the same employer but not in weatherization any more. (go to Q17)
3b. Please indicate your current certifications:
419
a. BPI
b. HERS
c.Weatherization Installer
e. Weatherization Crew Chief
f. Weatherization Auditor
g. Weatherization Monitor
h. Other
3c. Do you feel that you would like training that is not currently offered? If yes, what?
4. How satisfactory are these aspects of your job weatherizing low-income homes?
Neither
satisfactory
Very
nor
Very
satisfactory Satisfactory unsatisfactory Unsatisfactory unsatisfactory
a. Pay
□
□
□
□
□
b. Health benefits
□
□
□
□
□
c. Steady work
□
□
□
□
□
d. Boss/supervisor(s)
□
□
□
□
□
e. Co-workers
□
□
□
□
□
f. Interactions with
□
□
□
□
□
clients
g. Flexibility of work
□
□
□
□
□
schedule
h. Dress code
□
□
□
□
□
i. Paid time off policy
□
□
□
□
□
j. Retirement benefits
□
□
□
□
□
k. Job safety
□
□
□
□
□
5. Including your weatherization employer, how many employers do you have?
□ One (go to Q7)
□ Two
□ More than two
6. Is working for your current weatherization employer your main job?
□ Yes
□ No
7. Considering all your employers, do you work full-time or part-time?
□ Full-time
□ Part-time
□ More than full-time
420
8. How many hours per week do you usually work at all of your jobs?
Enter ______ hrs
9. Do you have health insurance?
□ Yes
□ No (go to Question 11)
10. Who provides your health insurance?
a.
Your current weatherization employer
b.
A non-weatherization employer
c.
Your state
d.
You purchase your own insurance
e.
You have insurance through a family member
f.
Other ____________________
11. What is your annual income from your weatherization job?
a. $0-$10,000
b. $10,001 - $15,000
c. $15,001 - $20,000
d. $20,001 - $25,000
e. $25,001 - $30,000
f. $30,001 - $40,000
g. $40,001 - $50,000
h. $50,001 - $75,000
i. $75,001 and over
12. How likely would it be that you would be unemployed if you did not have a job with your
current weatherization employer?
a. Very likely
b. Likely
c. Neither likely or unlikely
d. Unlikely
e. Very unlikely
END SURVEY
13. Do you still work in low-income weatherization?
a. Yes
b. No (go to Q15)
14. For whom do you work?
o Local weatherization agency
o Private weatherization contractor
o Other _______________
14a. What is your primary weatherization job title?
421
□ Administrator
□ Auditor
□ Inspector
□ Crew leader/foreman
□ Crew member
□ Other ___________
GO TO QUESTION 4
15. Are you currently…?
(1) Employed for wages (go to Q16)
(2) Self-employed (go to Q16a)
(3) Out of work for more than 1 year (SKIP to Q21)
(4) Out of work for less than 1 year (SKIP to Q21)
(5) A Homemaker (END SURVEY)
(6) A Student (SKIP to Q23)
(7) Retired (END SURVEY)
(8) Unable to work (END SURVEY)
(9) Refused (END SURVEY )
16. Please choose the description that best describes your current primary employer:
a. Private sector contractor (Choose if you are self-employed in this field)
b. Local government
c. State government
d. Federal government
e. Non-profit organization
f. Other type of for-profit firm
g. Other ______________
16a. Please choose the description that best describes what industry you work in:
a. home retrofit
b. construction
c. clean energy
d. other _________________________________________
17. Do you have health insurance?
o Yes
o No (go to Question 19)
18. Who provides your health insurance?
a.
Your current primary employer
b.
Your state
c.
You purchase your own insurance
d.
You have insurance through a family member
e.
Other ____________________
19. What is your annual income from your primary job?
422
□
□
□
□
□
□
□
□
□
$0-$10,000
$10,001 - $15,000
$15,001 - $20,000
$20,001 - $25,000
$25,001 - $30,000
$30,001 - $40,000
$40,001 - $50,000
$50,001 - $75,000
$75,001 and over
20. How satisfactory are these aspects of your job primary job?
a. Pay
b. Health benefits
c. Steady work
d.
Boss/supervisor(s)
e. Co-workers
f. Interactions with
clients
g. Flexibility of
work schedule
h. Dress code
i. Paid time off
policy
j. Retirement
benefits
k. Job safety
Neither
satisfactory
Very
nor
Very
satisfactory Satisfactory unsatisfactory Unsatisfactory unsatisfactory
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
□
END SURVEY
21. Have you looked for work during the last 4 weeks?
(1) Yes (END SURVEY)
(2) No
□ Don‘t Know/Not Sure
□ Refused (END SURVEY)
22. What is the main reason you were not looking for work during the LAST 4 WEEKS?
CHECK ALL THAT APPLY
423
(1) Believes no work available in line of work or area
(2) Couldn't find any work
(3) Lacks necessary schooling, training, skills or experience
(4) Employers think too young or too old
(5) Other types of discrimination
(6) Can't arrange child care
(7) Family responsibilities
(8) In school or other training
(9) Ill health, physical disability
(10) Transportation problems
(11) Other
(12)
Don‘t Know/Not Sure
(13) Refused
END SURVEY
23. Please choose the best description of your current educational status.
a. Enrolled in high school
b. Enrolled in community college
c. Enrolled in 4-year university or college
d. Enrolled in graduate school
e. Other ________________
END SURVEY
Part II – For Weatherization Training Center Trainees
24a. If you are currently employed do you feel that your training has increased your ability to:
a. Perform your current job (Go to Q24c)
b. Get a better job (Go to Q24c)
c. No benefit (Go to Q24c)
d. No currently employed
24b. If you are not currently employed, do you feel that your training has increased your ability
to get a job?
a. Yes
b. No
c. Maybe
d. Don‘t know
(Go to Q15)
24c. Do you currently work for a local agency that conducts low-income weatherization or a
private contractor that performs low-income weatherization?
424
a. Yes
b. No (Go to Q15)
25. For whom do you work?
o Local weatherization agency
o Private weatherization contractor
o Other _______________
GO TO Q3
425
OMB Control Number: XXXX-XXXX
APPENDIX N: DF11 - WEATHERIZATION STAFF SURVEY DATA FORM
This information is being collected to help implement the Weatherization Staff Survey. Public reporting
burden for this collection of information is estimated to average fifteen minutes per response, including
the time for reviewing instructions, searching existing data sources, gathering and maintaining the data
needed, and completing and reviewing the collection of information. Send comments regarding this
burden estimate or any other aspect of this collection of information, including suggestions for reducing
this burden, to Office of the Chief Information Officer, Records Management Division, IM-11,
Paperwork Reduction Project (XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW,
Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork
Reduction Project (XXXX-XXXX), Washington, DC 20503.
Thank you in advance for completing this data form.
1. Please identify your local weatherization agency. _____________________________
2. Please provide the following information about each individual who has conducted energy audits for
you agency so far in XXXX:
NAME
TELEPHONE NUMBER
426
EMAIL ADDRESS
427
APPENDIX O. SAMPLE SIZE JUSTIFICATION
This appendix addresses sample sizes needed for the various WAP evaluation component surveys.
Populations to be sampled are discussed in Section O.1. In addition to dependencies on weather,
weatherization data tends to be noisy because of complex and frequently changing behavior of occupants.
Wide confidence intervals often occur for estimates computed even from relatively large samples.
Equivalently, relatively large samples are often required to achieve specified levels of accuracy. The
WAP evaluation will require statistical sampling of weatherization agencies is discussed in Section O.2.
Statistical sampling associated with the Weatherization Innovation Pilot Project, Sustainable Energy
Resources for Consumers, Deferral, and Social Network Studies are found in Sections O.3-O.6,
respectively.
The usual approach taken in justifying a survey sample size is to identify a main quantity to be estimated,
to justify a sample size necessary for estimating it, and to argue that additional information to be collected
in the survey will be obtained at little or no additional burden to either subjects or analysts. The sample
size calculation ordinarily requires (1) a preliminary assessment of the variability of the main quantity to
be estimated (for example, a preliminary estimate of its coefficient of variation, and (2) a specification of
the accuracy required of the main quantity to be estimated—for example ―to within 10% of its true value
with 90% confidence.‖ Also, in cases of complex (multi-stage, stratified, probability-sampled, and
control-adjusted) designs, as is proposed for the WAP evaluation, sample size formulas for simpler (e.g.,
simple random sampling) designs are often used as an approximation. The sample design in
preliminary/pilot studies is usually much simpler anyway. These approaches are taken below.
Much of the material in these sections will be used directly for the WAP evaluation OMB Paperwork
Reduction Act (PRA) submission and Supporting Statement required for surveys employing statistical
methods (www.whitehouse.gov/omb/inforeg/83i-fill.pdf). Some of the discussion is therefore written in
that context. Section O.7 contains additional notes about topics such as nonresponse that will also have to
be addressed in the OMB PRA submission.
O.1 POPULATIONS SAMPLED
The WAP evaluation will require statistical sampling of weatherization agencies. Agency weatherization
staff and weatherization and control homes/occupants will be subsampled. WAP management will also be
interviewed, and 100% of State weatherization programs will be surveyed. WAP management and State
weatherization programs will be sampled completely (i.e., 100%) rather than statistically, because these
populations are small and it is expedient to sample them that way. (For PRA purposes, DOE program
management interviews are not considered a ―collection of information,‖ because the information
acquired is to be used for a specific purpose (i.e., the WAP evaluation) rather than general statistical
purposes—―‗Collection of information‘ includes questions posed to…employees of the United States, if
the results are to be used for general statistical purposes…‖ [5CFR1320.3].)
A justification for taking a complete sample rather than a statistical subsample of State agencies, which is
required by OMB, is given by OMB (2006): ―When the target population is small and each unit is unique,
a census is likely to be preferred over a sample survey. For example, when an agency evaluates a Federal
program that is implemented by the states (each one perhaps somewhat differently), a census of state
program directors may provide higher quality information with little cost difference from a sample survey
of a slightly smaller number of states. In this case, there may also be concerns about missing practices of
some states that were not included in the sample if a census were not conducted.‖
428
Several case studies will also be conducted as components of the WAP evaluation. The case studies will
not involve statistical sampling or inference, however, or they will be below the PRA threshold of ten or
more observations. They do not require OMB approval because either the sample size will be less than ten
or they will be conducted by direct observations by social scientists or weatherization experts of homes or
workers during the execution of normal weatherization activities, and such information collections are not
subject to the PRA: ―‗Information‘ does not generally include items in the following categories...Facts or
opinions obtained through direct observation by an employee or agent of the sponsoring agency or
through nonstandardized oral communication in connection with such direct
observations;‖ [5CFR1320.3]. Sample sizes for the case-studies will be based on judgment rather than
formal statistical calculations.
O.2 AGENCY SAMPLING
O.2.1 Agency sampling for energy benefits (billing data analysis)
As in the 1990 evaluation (Brown et al, 1993), occupant-level energy consumption and dwelling
characteristic data will be obtained by first sampling agencies and then acquiring occupant-level data
from the agencies. In the 1990 evaluation, 400 agencies were sampled, 361 agencies responded, and one
third of the records for the weatherized units and one third of the records for the control units were subsampled for each agency that responded (ibid.). This rate applied to single-family detached dwellings as
well as mobile homes. To properly determine weatherization savings in multifamily buildings, all units in
each selected building must be analyzed. Therefore, multifamily buildings were selected at the rate of one
third of buildings.
The 1990 evaluation (ibid.) lead to the following mean standard error control-adjusted natural gas and
electricity savings estimates per weatherized unit per year:
Primary Heating
Fuel
Natural gas
Electricity
Average Savings
per Weatherized
Unit
17.8 MMBtu/year
1,830 kWh/year
Standard Error of
Average
1.8 MMBtu/year
358 kWh/year
Precision
10%
20%
The precision obtained in the 1990 evaluation was subsequently found to adequate for that evaluation and
is assumed adequate for the proposed evaluation as well. Thus the basic objective of the proposed study is
to update the 1990 evaluation by repeating it today. However, the proposed evaluation will incorporate
several minor refinements and additions.
The 1990 evaluation was stratified by agency size and geographic region. The agency size strata were
sampled at the same rate except for the largest-size stratum which was certainty (100%) sampled. Thus,
with the exception of the very largest agencies, large agencies had no greater chance of selection than
small ones. Yet, in general, the larger the agency, the greater its contribution to total energy savings. In
the proposed evaluation probability proportional to size (PPS) sampling with size measured as agency
funding will provide a refinement of the 1990 stratification by agency size by allowing agency size to be
continuously reflected in the sampling probabilities, with the effect that all agencies will be statistically
represented, but larger agencies will be sampled preferentially.67
67
An alternative measure of an agency‘s size is the number of units it weatherizes. The number of units
weatherized is not as good a measure of size, however, because of an accounting feature in the WinSAGA
database (see following discussion) in which units weatherized with only one dollar of DOE funding can
429
The geographic stratification in the proposed evaluation will also be a refinement of the 1990 study‘s. The
earlier study employed ten climate subregions, which were approximations of standard climate regions
based on state boundaries. For the proposed study, in addition to representing all climate zones, it was
considered politically advantageous to guarantee representation of all states. (Furthermore some states
may commission add-on survey components of their own.) Therefore stratification for the proposed study
will be by state.
Although the sampling for the proposed study is thus a refinement of the 1990 sampling, the two studies
will still be substantially similar, and the 1990 study is by far the best available source of prior
information for the proposed one. For the purpose of sample size calculations, it is reasonable to regard
the proposed evaluation as emulating the 1990 study. Therefore 400 agencies will be sampled in the
proposed study, as in the 1990 evaluation.
For each sampled agency, houses (or buildings for multifamily buildings) will be randomly sampled. In
general, the rate of sampling within agencies will be fixed for all agencies. That rate will be
approximately one-in-three and will thus achieve the same overall sample size and precision achieved in
the 1990 evaluation. However, the exact sampling rate for each agency will depend on the following
factors that must be determined during the course of the study: (1) The target population of weatherized
and control units must be restricted to ―DOE‖ units, that is, units weatherized primarily with DOE funds.
Agencies weatherize homes using various funding sources (states and utilities also provide funding), and
use different bookkeeping methods for counting weatherized units, and in particular, DOE units. The
target population of DOE weatherized and control units has to be determined for each agency on a caseby-case basis. Target population sizes are expected be smaller than in the 1990 evaluation, and sampling
proportions will be larger to achieve a comparable sample overall size. (2) In the proposed study, more so
than in 1990, some agencies will be able to deliver complete sets of data in electronic formats. However,
the extent to which this happens and the ease of compliance of agencies with requests for data in general
is unclear. Recent interaction with a few agencies has shown that many agencies still rely on paper record
keeping and that delivery by an agency of a complete electronic database is still likely to be more the
exception than the rule. Nevertheless, agencies that can just as easily deliver all of their data as sample
part of it will be asked to do so. (3) As in the 1990 evaluation, some utilities will not comply with
requests for data. Utility nonresponse is considered independent of agency weatherization performance,
however, and is thus nonbiasing. Utility nonresponse is also likely to be lower than in the in the 1990
evaluation, because of advances in electronic bookkeeping.
A listing of weatherization agencies along with their planned dollar allocation and units sampled can be
obtained from the WinSAGA (Systems Approach to Grants Administration for Windows) data base.68 As
of 2005, there were 927 agencies. A PPS sample of 400 agencies with the 2005 funding allocation as the
measure of PPS size and stratified by state was selected in order to see what the PPS sampling of agencies
and one-third subsampling of units would likely translate to in terms of allocation dollars and weatherized
units in the actual sample. A ten percent nonresponse rate was assumed. Because of constraints on PPS
sampling, the very largest agency had to be sampled with certainty. Thus 361 agencies were actually
sampled (as in the 1990 evaluation). Results of the sample are summarized in Table 1.
After deducting the nonresponders, 361 agencies were sampled, which is 39% of the 927. Of course the
sample of 361 agencies represents all 927 agencies in a statistical sense. Because size is agency funding
allocation, although the sample accounts for 39% of agencies, it accounts for 65% of funding. The sample
be counted as DOE weatherized units.
68
WinSAGA data kindly provided by Christine Askew, Office of the Weatherization and
Intergovernmental Program, Energy Efficiency and Renewable Energy, U.S. DOE.
430
also accounts for 63% of weatherized units. (The reason for the differences in the funding percentage
(65%) and the units percentage (63%) may be related to the way units are counted in the WinSAGA
database, with only one dollar of DOE funding necessarily required for a unit to be counted as a DOE
unit.)
Table 1 also shows the units actually sampled (17,232) in a one-in-three subsample of units, and,
assuming (as in the 1990 evaluation) that 60% of units are gas or electric, the number of units (10,339)
potentially available for billing the analyses. Utility nonresponse will depreciate this number. However,
the final column in the table shows that if data is obtained for 46% of the units potentially available for
billing the analysis, then the number of units sampled will equal the number sampled in the 1990
evaluation. The 46% acquisition rate is reasonable and similar to the corresponding rate in the 1990
evaluation (ibid.).
O.2.2 Agency sampling for program characterization and process assessment
In addition to energy use and savings data, information about the weatherization process and program,
also necessary for the evaluation, will be obtained from the 400 sampled agencies. The justification for
collecting this additional information is based on the above for the metered fuel (natural gas and electric)
savings studies and that the additional burden in collecting/delivering this information from agencies from
which energy use and savings data is already being collected is small.
Table O.1 WAP 2005 National and Approximate PPS Sample Totals
Size = 2005 Dollar Allocation and 10% Nonresponse
National
Agencies
(2005
Listing)
Agencies
Sampled
929
400
2005
National
Units
Planned
82,701
Agencies
Responding
Units
Represented
by
Responders
Percent
of
National
Agencies
Responding
361
39% $258,231,144 $168,535,723
Percent
of Units
Represented
by
Responders
51,695
2005
National
Allocation
Allocation
Represented
by
Responders
Units
Subsampled
(at 33%)
63%
17,232
Gas/Elec
Units
Subsampled
(60% Approx)
10,339
Percent
Allocation
Represented
by
Responders
65%
Capture Rate
To Achieve
1990
Number
Usable
(4,796)
46%
O.2.3 Agency staff subsampling
A subsample of agency staff members will be taken from a list compiled from the agency staff contact
information. To ensure adequate representation, the sample will be stratified by staff functional
classification (crew, supervisor, auditor/inspector) with equal-size strata. A computer assisted telephone
interviewing (CATI) survey will be conducted of the sampled staff. Several technical questions will be
posed to the staff members to characterize current staff understanding and awareness of weatherization
methods and technologies. The primary endpoint of interest is the combined proportion of correct
431
responses and how it relates to staff training experience. The proportion of correct responses is expected
also to serve as a baseline for future studies.
Sampling will be implemented by random sampling from staff lists identified by the sampled agencies.
As an approximation in reckoning sample sizes, we ignore the agency sampling weights, though they will
be accounted for in the data analysis. The proportions of correct responses in each sampling strata (crew,
supervisor, auditor/inspector) will be estimated to within five percentage points with 90% confidence.
The standard error of the combined proportion of correct responses can be no greater than the standard
error of an individual (correct/incorrect) response, which cannot exceed .5/n1/2 (maximum standard error
of binomial proportion). This will be achieved if 1.645.5/n1/2 = .05, that is, if n = 271, where n is the
sample size in each stratum. Nonresponse in this survey is expected to be negligible, because the survey
will be of agency employees whose contact information has been provided by the agencies. The total
sample size will be 2713 = 813.
O.3 WEATHERIZATION INNOVATION PILOT PROGRAM SAMPLING
The WIPP currently comprises sixteen projects, all of which have been selected for evaluation. Various
project components will be evaluated, not all of which necessarily require OMB approval or even
statistical planning. For example, because each project is unique and there is only one grantee per project,
it seems unlikely that substantially the same question will be asked to all sixteen grantees, in which case
the PRA would not apply to the grantee evaluation. Similarly, case studies probably also will not involve
asking substantially the same question to ten or more individuals. However other WIPP evaluation
components will most likely involve asking the same questions to ten or more individuals. These include
(1) occupant surveys, (2) home inspections, (3) utility bill analyses, and (4) weatherization staff surveys.
These four evaluation components are discussed below.
432
Occupant Surveys and Home Inspections
Table O.2 below lists the sixteen WIPP projects, the numbers of homes slated for weatherization,
and the federal requests for the projects.
Table O.2. WIPP Projects, Numbers of Units to be Weatherized, and Federal
Funding
Number Federal Percent of
WIPP Project
of Units Request ($) Total Request
-----------------------------------------------------------------------------------------------------Green and Healthy Homes Initiative
220 2,400,000
8.00
In Home Monitoring
2,500 2,400,000
8.00
Performance-based Revolving Loan Pilot...
450
850,000
2.83
Energy Pioneer Solutions Weatherization...
250 2,400,000
8.00
SAHF Energy Performance Contracting...
2,500
810,000
2.70
Connecticut Green and Healthy Homes...
2,285 3,000,000
10.00
Streamlined Weatherization Improvements... 800 2,000,000
6.67
Leveraging Smart Grid Technology to...
550
720,000
2.40
YouthBuild USA Weatherization...
998 1,400,000
4.67
Habitat for Humanity Weatherization...
1,770 3,000,000
10.00
Community Environmental Center...
1,200 3,000,000
10.00
Building Deep Efficiency...
425
600,000
2.00
Project with the City matching federal...
300 1,015,746
3.39
Tackling the Problem of Weatherizing...
1,700 1,898,938
6.33
Replicable, Innovative, Sustainable...
2,240 3,000,000
10.00
People Working Cooperatively...
340 1,500,000
5.00
------------------------------------------------------------------------------------------------------All Projects
18,528 29,994,684
100.00
Although the units in Table O.2 cannot yet be listed in a population frame, they are well-defined.
A table such as Table O.3 can be used to suggest a sample size for sampling from all 18,528 of
the units. The table gives the sample sizes needed to estimate a proportion (e.g., proportion of
customers who say they are satisfied) to within the specified margin of error with the specified
statistical confidence.
433
Table O.3. Suggested Sample Sizes for Sampling Units (for Occupant Survey/Home
Inspections)
Corrected
Sample Size
Margin
Infinite
(For Finite
of
Population Population
Confidence Error
Sample Size of 18,528)
----------------------------------------------------------------------90
0.10
68
68
0.05
271
267
0.03
752
723
95
0.10
97
96
0.05
385
377
0.03
1,068
1,009
----------------------------------------------------------------------Occupant surveys will be conducted by telephone interviewing. The 3% margin of error and 95%
statistical confidence (03-95 criteria, last row of Table O.3) are standard for this kind of telephone
survey. For home inspections, because of the much higher expense per subject, a 10% margin of error
and 90% statistical confidence (10-90 criteria, first row of Table O.3) are more appropriate. For these
criteria, sample sizes of 1,009 and 68 are needed for the occupant survey and home inspections
respectively. However, Table 2 is for simple random sampling, and we will stratify the sampling by the
sixteen WIPP projects. As the stratification is expected to improve the survey's precision, however, the
figures in Table 2 are slightly higher than actually necessary.
We use the projects' percentage of the total federal request for the sixteen projects to allocate the sample
across the sixteen WIPP projects. As shown in Table O.4, for each project, a sample of the specified size
will be sampled from a frame of the corresponding number of units. Because the necessary number in
each stratum was rounded up to the next whole number, the numbers in Table O.4 add to slightly more
than the targets of 68 units for inspection and 1,009 occupants.
434
Table O.4. Housing Unit Sample Sizes Allocated Across the Sixteen WIPP Projects
Sample
Sample
Number
Size for
Size for
WIPP Project
of Units
10-90
03-95
------------------------------------------------------------------------------------------------------Green and Healthy Homes Initiative
220
6
81
In Home Monitoring
2,500
6
81
Performance-based Revolving Loan Pilot... 450
2
29
Energy Pioneer Solutions Weatherization... 250
6
81
SAHF Energy Performance Contracting... 2,500
2
28
Connecticut Green and Healthy Homes... 2,285
7
101
Streamlined Weatherization Improvements... 800
5
68
Leveraging Smart Grid Technology to...
550
2
25
YouthBuild USA Weatherization...
998
4
48
Habitat for Humanity Weatherization...
1,770
7
101
Community Environmental Center...
1,200
7
101
Building Deep Efficiency...
425
2
21
Project with the City matching federal...
300
3
35
Tackling the Problem of Weatherizing...
1,700
5
64
Replicable, Innovative, Sustainable...
2,240
7
101
People Working Cooperatively...
340
4
51
------------------------------------------------------------------------------------------------------All Projects
18,528
75
1,016
Utility Bill Analysis
In each of the sixteen WIPP projects, billing data or equivalent or better energy use data will be collected
for each weatherized unit. The billing data--for all weatherized units--can be obtained in conjunction with
the DF4 surveys (lists of units and buildings weatherized). Comparison data will be collected from an
equal or greater number of units from the comparison (i.e., not weatherized) group data collected in the
retrospective evaluation. Units will be matched as closely as possible by city, heating and cooing degree
days (when not matched on city), size, construction type, and other characteristics. The billing data for all
sixteen projects will be analyzed by the evaluation team.
Staff Survey
A survey of staff members, analogous to the occupant survey, will be conducted for the sixteen WIPP
projects. As in the occupant survey, a sampling frame (in this case a list of project staff members) has not
yet been compiled but will be compiled for each project, and the overall sample will be allocated to the
individual projects in proportion to federal funding. Table O.4, for staff members, is analogous to Table
O.2 for weatherized units:
435
Table O.5. WIPP Projects, Jobs Created or Retained, and Federal Funding
Number
Federal
Percent of
WIPP Project
of Jobs
Request ($) Total Request
--------------------------------------------------------------------------------------------------------------Green and Healthy Homes Initiative
96
2,400,000
12.64
In Home Monitoring
120
2,400,000
12.64
Performance-based Revolving Loan Pilot...
38
850,000
4.48
Energy Pioneer Solutions Weatherization...
25
2,400,000
12.64
SAHF Energy Performance Contracting...
123
810,000
4.27
Connecticut Green and Healthy Homes...
593
3,000,000
15.81
Streamlined Weatherization Improvements... 25
2,000,000
10.54
Leveraging Smart Grid Technology to...
16
720,000
3.79
YouthBuild USA Weatherization...
74
1,400,000
7.38
Habitat for Humanity Weatherization...
168
3,000,000
15.81
Community Environmental Center...
63
3,000,000
10.00
Building Deep Efficiency...
28
600,000
2.00
Project with the City matching federal...
10
1,015,746
3.39
Tackling the Problem of Weatherizing...
85
1,898,938
6.33
Replicable, Innovative, Sustainable...
169
3,000,000
10.00
People Working Cooperatively...
673
1,500,000
5.00
--------------------------------------------------------------------------------------------------------------All Projects
2,306
18,980,000
100.00
A table such as Table O.6 (analogous to Table O.4) can be used to suggest a sample size for sampling
from the (approximately) 2,306 staff members. The table gives the sample sizes needed to estimate a
proportion (e.g., proportion of customers who say they are satisfied) to within the specified margin of
error with the specified statistical confidence.
Table O.6. Suggested Sample Sizes for Sampling Staff Members
Corrected
Sample Size
Margin
Infinite
(For Finite
Statistical
of
Population
Population
Confidence Error
Sample Size of 2,306)
-----------------------------------------------------------------------90
0.10
68
66
0.05
271
243
0.03
752
567
95
0.10
97
93
0.05
385
330
0.03
1,068
730
------------------------------------------------------------------------
436
Staff survey interviews are not as critical to the evaluation as occupant surveys. (Presumably the staff
will also be studied in assessing the affects of the ARRA.) Therefore, for the staff survey, we use the
90% statistical confidence and .05 margin of error criteria, which suggests a sample size of 243. Again
the overall sample is allocated across the sixteen WIPP programs in proportion to funding. Table O.7
shows the allocated sample sizes (rounded up to the next whole number) for the individual projects.
Table O.7. Staff Sample Sizes Allocated Across the Sixteen WIPP Projects
Sample
Number
Size for
WIPP Project
of Jobs
05-90
--------------------------------------------------------------------------------------Green and Healthy Homes Initiative
96
20
In Home Monitoring
120
20
Performance-based Revolving Loan Pilot... 38
7
Energy Pioneer Solutions Weatherization... 25
20
SAHF Energy Performance Contracting... 123
7
Connecticut Green and Healthy Homes...
593
25
Streamlined Weatherization Improvements... 25
17
Leveraging Smart Grid Technology to...
16
6
YouthBuild USA Weatherization...
74
12
Habitat for Humanity Weatherization...
168
25
Community Environmental Center...
63
25
Building Deep Efficiency...
28
5
Project with the City matching federal...
10
9
Tackling the Problem of Weatherizing...
85
16
Replicable, Innovative, Sustainable...
169
25
People Working Cooperatively...
673
13
---------------------------------------------------------------------------------------All Projects
2,306
252
O.4 SUSTAINABLE ENERGY RESOURCES FOR CONSUMERS SAMPLING
As discussed above, all 92 SERC grantees will be included in the sub-sample of 450 sub-grantees that
will be asked to provide information re weatherized homes and utility account information.
O.5 DEFERRAL STUDY SAMPLING
Ten states and ten agencies will be sampled for this a deferral study, and the deferral incidence and
process (e.g., quality assurance) will be examined for a random sample of weatherized units from each
sampled agency. The number of units to be sampled is (at least tentatively) specified in the draft
Evaluation Plan as "200 occupants engaged in the deferral process." Deferral rates encountered by some
agencies are reckoned by agency staff to be as high as twenty percent.
Agency selection will be purposive for agencies for which deferrals are understood to be troublesome.
Inferences will therefore be restricted to the ten sampled agencies. However the ten sampled agencies
437
will serve collectively as anecdotal evidence about the extent to which deferrals can be a problem, and the
overall deferral rate for the ten sampled agencies will thus be a parameter of primary interest in the
analysis. Since we'll ask substantially the same question to each of the agencies, we need to develop a
formal OMB 83-i submission for this. Note that if (1) we sampled only nine agencies instead of ten, and
(2) the information about the deferral could be obtained directly from agency staff, without posing
additional questions to home owners, we might be able to circumvent the OMB submission. Whether or
not we pursue such an approach, however, we still need to decide about how many of each agency's units
to sample for the deferral analysis. That statistical question is considered in the next paragraph.
The specifications can be tweaked, of course, but suppose for the moment that we would like to estimate
the overall deferral rate for the ten agencies to within five percentage points with 90% confidence. Given
that deferral rates encountered by some agencies are reckoned by agency staff to be as high as 20%,
suppose that the deferral rate for any sampled agency is no higher than 25%. This implies that the
variance of the overall deferral rate estimate is maximum when the deferral rate is 25% for each sampled
agency. That is, the worst-case (i.e., maximum) variance of the overall deferral rate estimate is .25(1.25)/(10*n), where n is the number of units sampled per agency. Hence, letting Z = 1.645 (95th percentile
of the standard normal distribution), the worst-case required sample size for the 5-percentage-point-90%confidence specification is
n = Z*Z*.25(1-.25)/(10*(.05)**2) = 20.3 units per agency.
So the sample size of 200 units seems reasonable.
O.6 PERSISTENCE STUDY SAMPLING
Sample Size for the Persistence Study
The methodological approach is to base the Persistence Study on a comparison of a treatment group of
homes weatherized circa 1995 with a control group of homes selected from recent WAP-applicant homes
to match the treatment group on age and other characteristics. The primary comparison measurements
will be blower-door test cfm50 values, and we'll base sample size calculations on those measurements.
As no circa-1995 blower-door tests were conducted for the controls, the comparison will be based on
blower-door tests conducted currently (i.e., circa 2011-2012).
We now have pre and post-weatherization blower-door measurements from the Retrospective
Evaluation's IAQ study homes, which can be used as pilot data for the sample size calculations. The IAQ
study blower-door data is summarized in the following table:
Table O.8. IAQ Study Pre and Post-Weatherization CFM50's
Variable
N
Mean
Std Dev
--------------------------------------------------Pre-Wx cfm50 2,324
3,466.7
1,829.28
Post-Wx cfm50 2,251
2,362.5
1,003.41
Difference
2,220
-1,093.9
1,177.64
--------------------------------------------------The treatment-control group matching for the Persistence Study could be conducted in two ways: (i) The
treatment and control homes could be matched on a case-by-case basis, with the data then analyzed with a
paired t-test, or (ii) the treatment and control homes could be matched more loosely, with age and other
distributions of the two groups kept the same, but without case-by-case matching or even the same sample
sizes in the two groups, with the data then analyzed with a two-sample t-test. If case-by-case matching
438
can be done precisely, which will be the initial goal of this study, and with many characteristics, then
approach (i) may lead to more precise comparisons. However, the looser approach (ii) is more likely to
be reasonable. Furthermore we have pilot study data to support approach (ii) only. Therefore we assume
approach (ii) for reckoning sample sizes. If approach (i) is ultimately used instead, the sample size
suggested here may be a little bigger than necessary.
The current, residual effect of the 1995 weatherizations will thus be measured as the difference between
the averages of the current blower-door measurements for the 1995-weatherized (treatment) homes and
the pre-weatherization control homes. Persistence can then be estimated by comparing that difference to
the average of either the 1995 post-pre blower-door differences for the treatment homes or the current
post-pre blower-door difference for the control homes. The average of the current differences for the
control homes today is probably a better reference, however, as blower-door tests conducted in 1995 are
considered to be less accurate than those tests conducted today. Presently, in lieu of control home
differences, we can us the Retrospective Evaluation's IAQ study for pilot data for calculating sample
sizes. IAQ Study blower-door cfm50 flow rates were reduced by 1093.9 cfm on average after
weatherizations (Table O.8). Fifteen years after weatherization we'd expect the effect of the
weatherization to be reduced from its initial effect. Thus we'd like to detect a mean difference delta
between the treatment and control blower-door cfm50 measurements on the order of, say, delta=250, 500,
or 750 cfm.
To determine a sample size, we consider a statistical test of the null hypothesis Ho: delta=0 vs the
alternative hypothesis H1: delta > 0, with the requirements that (1) if delta=0, then the probability alpha
that the test rejects (i.e., finds a difference) is low, say alpha=.10 or .05, and (2) if delta=250, 500, or 750
cfm, then the probability beta that the test rejects is high, say .90 or .95. We use a two-sample z-test as an
approximation to the two-sample t-test.
Let Nw and Nc denote the numbers of treatment and control homes (possibly but not necessarily the
same), and let rho=Nw/Nc. Let Vw and Vc denote the variance of the treatment and control cfm50's
respectively. Then by straightforward calculation the smallest Nw that satisfies conditions (1) and (2) is
Nw = (Vw + rho*Vc)*((z_beta - z_alpha)/delta)**2.
where z_alpha and z_beta are the alpha and beta quantiles of the normal distribution. Substituting the
squares of the Pre-Wx cfm50 and Post-Wx cfm50 standard deviations in Table O.8 for Vc and Vw
respectively then leads to the approximate necessary and sufficient sample sizes in Table 2 below. The
only values of rho considered in the table are 0.5, 1, and 2, although other values might also be
reasonable. The total number of homes, N=Nw+Nc, is also shown, in the last column of the table.
The WAP ARRA-Period Evaluation Plan calls for a Persistence Study treatment group of approximately
Nw=125 single family and mobile homes, and a corresponding control group. A total sample size of 250
corresponds to about the middle of the table: delta=500 cfm, alpha=.10, and beta=.90. For this delta,
alpha, and beta, the required total (for both treatment and control) sample sizes are 211 (Nw=70,
Nc=141), 228 (Nw=114, Nc=114), and 303 (Nw=202, Nc=101) for rho=0.5, 1, and 2, respectively. For
delta=500 cfm and alpha and beta either .05 and .90 or .10 and .95, a sample size of approximately 276
(Nw=92, Nc=184) would be needed for rho=0.5.
439
Table O.9. Approximate Necessary and Sufficient Sample Sizes for Persistence Study
delta
(cfm) alpha beta
rho
Nw
Nc
N=Nw+Nc
==========================================================================
250
0.05 0.90
0.5
367
734
1,101
1.0
596
596
1,192
2.0
1,055
527
1,582
250
0.05
0.95
0.5
1.0
2.0
464
754
1,333
928
754
667
1,392
1,508
2,000
250
0.10
0.90
0.5
1.0
2.0
282
458
809
563
458
405
845
916
1,214
250
0.10
0.95
500
0.05
0.95
0.5
1.0
2.0
116
188
333
232
188
167
348
376
500
500
0.10
0.90
0.5
1.0
2.0
70
114
202
141
114
101
211
228
303
500
0.10
0.95
750
0.05
0.95
0.5
1.0
2.0
52
84
148
103
84
74
155
168
222
750
0.10
0.90
0.5
1.0
2.0
31
51
90
63
51
45
94
102
135
750
0.10
0.95
0.5
367
734
1,101
1.0
596
596
1,192
2.0
1,055
527
1,582
-------------------------------------------------------------------------500
0.05 0.90
0.5
92
184
276
1.0
149
149
298
2.0
264
132
396
0.5
92
184
276
1.0
149
149
298
2.0
264
132
396
-------------------------------------------------------------------------750
0.05 0.90
0.5
41
82
123
1.0
66
66
132
2.0
117
59
176
0.5
41
82
123
1.0
66
66
132
2.0
117
59
176
==========================================================================
440
O.7 ADDITIONAL NOTES
The following notes taken from OMB guidance (OMB 2006) identify several additional issues of
key importance in developing the evaluation plan.
O.7.1 Pretest/Pilot Studies
According to OMB (2006), ―Agencies should always consider conducting pretests (small trials of
the measurement process) or pilot studies (larger trials yielding statistical information) when planning for
a new information collection or changing methods and procedures for an ongoing survey. These kinds of
tests may provide critical information necessary to ensure the quality of the data and smoothness of
operations needed in the full-scale information collection. They can provide essential information to the
agency and result in higher data quality than would have been achieved without them and may be the only
vehicle for measuring the effects of different changes an agency is considering implementing. Thus,
agencies will need to weigh the importance and use of pretests against the time and resources needed to
conduct them.‖
The proposed evaluation uses the 1990 evaluation, which it is intended to emulate, and several
other studies (Blasnik, 2006; Cavallo and Mapp, 2000; Levins and Ternes, 1994; Ternes and Levins,
1992) as a pilot studies.
The OMB (2006) guidance continues, ―It is important that agencies test their survey
questionnaires in all modes that they plan to use to collect information for the full-scale survey (see
section on Questionnaire Design). Usability testing of computer survey instruments should also be
included as part of questionnaire pretesting to identify problems either interviewers or respondents may
have with the instrument....‖
The survey instruments for the proposed evaluation are being pre-tested with internal and (fewer than ten)
external personnel. They are also being peer-reviewed by subject matter experts.
O.7.2 Influential information
According to OMB (2006, Question 18), information is considered ―influential‖ if ―an agency can
reasonably determine that dissemination of the information will have or does have a clear and substantial
impact on important public policies or important private sector decisions,‖ and agencies should ―hold the
information they designate as ‗influential‘ to a higher standard of reproducibility and transparency than
information that is not defined as influential.‖
The information collected for the proposed WAP evaluation is thus ―influential.‖
O.7.3 Response rates
According to OMB (2006), ―An agency‘s justification for a survey response rate should reflect, at least in
part, the intended use of the data. For example, surveys collecting influential information or information
that will otherwise have a substantial impact on an agency‘s programs or policies should be designed to
minimize all sources of survey error (see question #20), including nonresponse bias...Agencies need to
document in their ICRs the importance and use of the information and the methods they will use to
achieve acceptable response rates for their collections… ICRs for surveys with expected response rates of
80 percent or higher need complete descriptions of the basis of the estimated response rate and a detailed
description of steps that will be taken to achieve the expected response rate…‖
441
In general, response rates in the proposed evaluation are expected to be high (much higher than 80%),
because weatherization recipient contact information will be provided by weatherization agencies, and
because recipients have received weatherization services. For various reasons including legal issues,
utilities in the past have not always provided customer billing data requested by agencies. Although this is
a form nonresponse, because it is independent of performance by the agencies in weatherizations, it is
reasonable to treat it as nonbiasing and therefore not requiring adjustment in the analysis. In general,
nonresponse is not expected to be a problem in the analysis of the weatherization data.
The OMB 83-i (www.whitehouse.gov/omb/inforeg/83i-fill.pdf) form requires a description of ―methods
to maximize response rates and to deal with issues of non-response.‖ Methods that will be employed to
accomplish this include
contacting an appropriate person at each utility to identify and smooth out the data collection
process
planning the billing data requests so that data for multiple housing units and buildings are
requested from each utility at the same time
requesting billing data at regular intervals to reduce the chance that the utilities will not be able to
provide data because it has been archived and no longer readily accessible, but limit the number
of such requests so that utilities have to provide data just several times during the course of the
evaluation
soliciting assistance from utility regulatory commissions and similar organizations as needed.
O.8 REFERENCES
Brown, Marilyn A., Berry, Linda G., Balzer, Richard A., and Faby, Ellen (1993), ―National Impacts of
the Weatherization Assistance Program in Single-Family Dwellings,‖ ORNL/CON-326, Oak Ridge
National Laboratory, Oak Ridge, Tennessee, May, 1993 (http://weatherization.ornl.gov/pdf/ORNL_CON326.pdf).
OMB (2006) ―Questions and Answers When Designing Surveys for Information Collections,‖ Office of
Information and Regulatory Affairs, Office of Management and Budget,
http://www.whitehouse.gov/omb/inforeg/pmc_survey_guidance_2006.pdf, January 2006.
Iowa Bureau of Weatherization (2005), ―Statistics,‖ Innovative Outstanding Weatherization Assistance
News, Volume 3, Issue 2. (http://www.dcaa.iowa.gov/bureau_weath/April2005.pdf)
Levins, William. P., and Ternes, Mark P. (1994). ―Impacts of the Weatherization Assistance Program in
Fuel-Oil Heated Houses,‖ ORNL/CON-327, October 1994
(http://weatherization.ornl.gov/pdf/0RNL_CON_327.pdf).
Sharp, T. (1994), ―The North Carolina Field Test: Field Performance of the Preliminary Version of an
Advanced Weatherization Audit for the Department of Energy's Weatherization Assistance Program.,‖
ORNL/CON-362, June 1994 (http://weatherization.ornl.gov/pdf/ORNL_CON-362.pdf).
442
443
APPENDIX P. BUILDING TYPE DEFINITIONS
The energy analyses methods are dependent on the building/housing type. Throughout the evaluation,
housing units will be categorized into four building types as defined below:
•
Single-family—A single-family housing unit, detached or attached, provides living space for one
family or household and is contained within walls extending from the basement (or the ground
floor, if there is no basement) to the roof. An attached house, such as a townhouse, row house,
and duplex, is considered a single-family housing unit as long as it (a) is not divided into more
than one housing unit, (b) there is no household living above another one within the walls
extending from the basement to the roof to separate the units, and (c) it has an independent
outside entrance.
•
Mobile home—A mobile home is built on a movable chassis, moved to the site, and typically
placed on a permanent or temporary foundation. If rooms are added to the structure, it is
considered a mobile home if the added floor area is less than the mobile home‘s original floor
area; otherwise, it is a single-family housing unit. A manufactured house assembled on site is a
single-family housing unit, not a mobile home.
•
Small multifamily—A small multifamily housing unit is in a building with two to four housing
units (i.e., in a building structure that is divided into living quarters for two, three, or four families
or households in which one household lives above or beside another). This category includes
houses originally intended for occupancy by one family (or for some other use) that have since
been converted to separate dwellings for two to four families. Typical arrangements in these types
of living quarters are separate apartments downstairs and upstairs or one apartment on each of
three or four floors.
•
Large multifamily—A large multifamily housing unit is in a building with five or more housing
units (i.e., in a building structure that contains living quarters for five or more families or
households and in which one household lives above or beside another).
These definitions are consistent with those used in the Residential Energy Consumption Survey
conducted by the DOE Energy Information Agency with one exception. Mobile homes with rooms added
to the structure are considered to be single-family housing units in RECS in all cases. Mobile homes
weatherized by the Program often have small rooms added to them (e.g., small rooms added to the front
or back doors, rooms that connect to and extend the living room). Mobile homes with room additions will
still be classified as mobile homes in this evaluation as long as the floor area of the additions is less than
the floor area of the original mobile home for two reasons. First, these mobile homes are still treated in
the Program using the diagnostic approaches, weatherization measures, and installation techniques unique
to mobile homes. Second, the majority of Program expenditures are typically directed at improvements to
the mobile home structure rather than the room additions. If room additions exceed the floor area of the
original mobile home, then the house should be classified as a single-family housing unit.
These definitions are also generally consistent with those used in the 1990 National evaluation. The one
uncertainty is with mobile homes because it is not known how mobile homes were defined in the 1990
National evaluation. However, it is believed that the categorization of mobile homes by weatherization
agencies is more consistent with the definition developed for this evaluation than the RECS definition.
Also, the 1990 National evaluation made a distinction between attached and detached single-family
housing units that will not be made in this evaluation.
444
The Program uses three rather than four categories of housing unit types in its quarterly reporting and
other functions. Mobile homes and large multifamily categories are conceptually the same as those
defined above for this evaluation. However, the single-family category as defined by the Program
includes the small multifamily category as defined for this evaluation. It is important to have a clear
definition of small multifamily housing units and a separate category for these units for two reasons:
1.
Small multifamily housing units were often mistakenly identified as belonging to the large
multifamily housing category by agencies in the 1990 National evaluation. This led to
considerable sampling and analysis problems. Providing a separate category for small multifamily
housing units should help avoid this identification problem.
2.
The energy analysis methods and approaches to be employed to study single family homes are
different and/or will be applied differently than those for large multifamily units. The analyses
methods and approaches for small multifamily housing units may also be unique because they
have characteristics of both of these other two groups of houses.
445
APPENDIX Q. UNIT/BUILDING LEVEL ENERGY ANALYSIS
Q.1 SINGLE-FAMILY HOUSES AND MOBILE HOMES
Natural gas billing data will be collected on homes heated by natural gas as part of the comprehensive
billing data sample. These data will be analyzed using the Princeton Scorekeeping Method (PRISM, Fels
et al., 1995) to calculate annual, weather-normalized pre- and post-weatherization energy consumptions
and energy savings for each individual home. Since natural gas can be used in these homes for water
heating, cooking, and possibly clothes drying in addition to space heating, these energy consumptions and
savings represent both space heating and baseload uses combined (i.e., natural gas consumption and
savings for space heating will not be calculated separately from baseload).
The Network Planning Committee felt that the energy consumption and savings of houses heated by fuel
oil and propane are critical to the evaluation and need to be measured. Billing/delivery data are usually
insufficient for a PRISM type analysis because fuel oil and propane are typically delivered just several
times a year to a house at infrequent intervals, and because household storage tanks are not always filled
at each delivery (so that the amount of fuel delivered is not necessarily equal to consumption). Therefore,
energy use in these homes must be specially monitored by submetering the space heating system or
collecting more accurate and frequent delivery data. If submetered fuel oil and propane use along with
indoor and outdoor temperatures are collected on houses heated by these fuels in the fuel-oil and propane
monitored sample, then an energy use model will be developed for each house by regressing weekly or
daily consumptions (the dependent variable) vs. the temperature difference between the indoors and
outdoors (the independent variables) for each consumption period. Annual, weather-normalized pre- and
post-weatherization energy consumptions and energy savings will be calculated using the regression
models, historical weather for each home location, and a standard indoor temperature (e.g., 68 or 70°F) or
the actual indoor temperature for each house. Uncertainty statistics and indicators of model reliability
comparable to those calculated by PRISM will also be calculated. Since fuel oil and propane are not
typically used in single-family or mobile homes for water heating or cooling, the annual energy
consumptions and savings based on the analysis of submetered fuel oil and propane use typically
represent just space-heating use. If more accurate and frequent delivery data are collected, then these data
will be analyzed using PRISM or a simply calculating the ratio of energy use to heating degree days for
the monitoring period.
Electricity billing data will be collected on all the homes sampled (not just those heated by electricity) as
part of the billing data sample, fuel-oil and propane monitored sample, and hot climate submetered
sample. These data will be analyzed using PRISM to calculate annual pre- and post-weatherization
electricity consumptions and energy savings for each individual home. PRISM‘s model selection feature
will be used to select the best PRISM model for each house (i.e., heating-only, cooling-only, or heating
and cooling model). In homes heated by electricity, these electricity consumptions and savings include
both space heating and baseload uses in addition to space cooling if employed. In non-electrically heated
homes, these energy consumptions and savings include baseload uses, space cooling if employed, and any
supplemental space heating that is done using electricity. Electricity consumptions and savings will not be
broken down into their separate heating, cooling, and baseload components.
Although the Network Planning Committee felt that submetering and subsequent analysis of a sample of
natural gas and electrically heated houses could improve accuracy and add context to the PRISM billing
data analysis results, such sampling and analysis is not currently planned because of the costs that would
be required.
446
Q.2 LARGE MULTIFAMILY BUILDINGS
The calculation of energy use and savings is complicated in large multifamily buildings for many reasons,
including:
•
some buildings have central building heating and/or hot water systems while in others
each unit has its own heating (central apartment system or in-space heaters) and/or hot
water system,
•
the whole building is weatherized in some cases while in others only individual units are
weatherized,
•
some buildings have significant common areas (including recreation rooms, offices, and
kitchens) while others have little or none,
•
operating ventilation systems are present in some buildings and not in others,
•
there are many billing meters in the building (especially those with individual heating
systems) so rarely is there a complete and consistent set of data for all the meters (i.e.,
there is usually some gap in data for at least a few meters),
•
occupancy turnover and vacancies can be prevalent, and
•
many buildings use fuel oil which has the same problems with the delivery data as with
single-family homes (i.e., fuel is delivered at infrequent and/or random intervals and the
building storage tanks are not always filled at each delivery).
These complications can manifest themselves at different times during the analysis process, such as in
estimating annual consumption, normalizing consumption to a per unit level, and/or aggregating the
consumptions of various buildings or units together.
For the reasons outlined above, the process of calculating the energy use and savings in each multifamily
building and determining how to include these values in the totals or averages with other multifamily
buildings must be developed individually for each building. Although some basic analysis approaches are
outlined below for the two primary building types that will be commonly encountered, experienced
analysts will need to follow and analyze the data collected on each building individually since cookiecutter approaches will not work.
Q.2.1 Buildings with central building heating systems
In buildings with central building heating systems, a whole building energy analysis is usually required
because the weatherization activity in such buildings usually focuses on the central heating system rather
than just selected apartment units. A whole building energy analysis will be performed as follows:
1.
Depending on which fuel is used by the central building heating system, natural gas billing data
for the building‘s master meter or fuel-oil delivery data will be collected as part of the large
multifamily billing data sample. These data will be analyzed using PRISM to calculate annual
pre- and post-weatherization energy consumptions and energy savings for the building. These
energy consumptions and savings represent either just space heating, space heating and hot water,
or space heating and baseload (e.g., hot water, cooking) depending on the fuel and what other
447
systems are connected to the master meter or fuel tank (e.g., central building hot water system,
apartment stoves).
Although fuel-oil delivery data are usually insufficient for a PRISM type analysis in single family
homes, there is a greater chance that such data can be analyzed for multifamily buildings,
especially if several years of data can be collected, because deliveries may be more frequent (ratio
of consumption to tank size may be greater) and/or fills may be more common.
As part of the monitored sample, fuel oil consumption will be specially monitored in some
buildings either through submetering or collecting more accurate and frequent delivery data.
Annual pre- and post-weatherization energy consumptions and energy savings for the building
will be calculated using these data and PRISM if possible (if data during some non-heating
periods can be collected) or using regression models and historical weather for each building
(models developed by regressing weekly or daily consumptions vs. outdoor temperature or
heating degree days).
2.
Electricity billing data will be collected from all electric meters (building and apartment level)
installed in the sampled buildings (not just those heated by electricity) as part of the billing data
sample or fuel-oil and propane monitored sample. Apartment level data will be aggregated for
each billing period and then analyzed using PRISM to calculate annual pre- and postweatherization electricity consumptions and energy savings for each building. Some refinements
will be needed to account for missing data and/or apartments for which no billing data can be
collected. The building electricity data will be analyzed similarly either separately or by
aggregating it with the apartment level data. PRISM‘s model selection feature will be used to
select the best PRISM model for each building (i.e., heating-only, cooling-only, or heating and
cooling model). In buildings with central building heating systems, these electricity consumptions
and savings include apartment-level baseload uses (possibly including hot water if the building
has individual hot water systems in each apartment fueled by electricity), common area electricity
consumption, space cooling if employed, and any supplemental space heating that is done using
electricity. The electricity consumptions and savings will not be broken down into their separate
heating, cooling, and baseload components.
3.
Natural gas billing data will be collected from any other building and apartment level natural gas
meters installed in the sampled buildings (other than the master meter supplying the central
building heating system) as part of the billing data sample or fuel-oil and propane monitored
sample. These data will be analyzed using PRISM in a manner similar to that for electric billing
data. In buildings with central building heating systems, these natural gas consumptions and
savings include apartment-level baseload uses (cooking and possibly hot water if the building has
individual hot water systems in each apartment fueled by natural gas) and any common area
consumption.
4.
Building-level energy consumptions and savings calculated above will be divided by the number
of units in the building to calculate unit-level values which will facilitate comparison and
aggregation with other buildings. In addition, natural gas and fuel oil consumptions and savings
will be added into one value.
The Inverse Modeling Toolkit (IMT) is a software program available from the American Society
of Heating, Refrigerating, and Air Conditioning Engineers (ASHRAE) for calculating linear, changepoint linear, variable-based degree-day, multilinear, and combined regression models (Kissock, Haberl,
and Claridge, 2004). As such, it offers an alternative approach to PRISM in the analysis presented above.
The use of IMT in this evaluation will be investigated. Depending on analytical difficulties encountered
448
with PRISM in the above analyses and the results of the IMT investigation, IMT will be used to
supplement and/or supersede the PRISM analyses as warranted.
Q.2.2 Buildings with apartment-level heating systems
In buildings with apartment-level heating systems (e.g., central systems in the apartment, baseboard
electric heaters), a whole building energy analysis will be performed if the whole building was
weatherized. If just individual units in a multifamily building were weatherized, then the energy analysis
will be performed on an apartment or unit basis. These two approaches are described below:
1.
In buildings in which the whole building was weatherized, billing data will be collected as part of
the large multifamily billing data sample for all building-level and apartment-level electricity
meters and natural gas (if present) installed in the building (fuel oil is not usually used in such
buildings). For each building, apartment-level and building-level electricity data will be
aggregated for each billing period and then analyzed using PRISM to calculate annual buildinglevel pre- and post-weatherization electricity consumptions and energy savings. PRISM‘s model
selection feature will be used to select the best PRISM model for each building (i.e., heating-only,
cooling-only, or heating and cooling model). If natural gas is used in the building, the same
procedure as used for electricity will be followed to calculate annual building level pre- and postweatherization natural gas consumptions and energy savings. Some refinements will be needed to
account for missing data and/or apartments for which no billing data can be collected.
The electricity consumptions and savings calculated above represent space heating, space
cooling, and baseload use (for the apartments and common areas combined) depending on the
specific end uses that electricity is used for. Likewise, the natural gas consumptions and savings
calculated above can represent space heating and/or baseload use. The energy consumptions and
savings will not be broken down into their separate heating, cooling, and baseload components.
Building-level energy consumptions and savings calculated above will be divided by the number
of units in the building to calculate unit-level values which will facilitate comparison and
aggregation with other buildings. The use of IMT may be used to supplement and/or supersede
the PRISM analyses as warranted.
2.
In buildings in which just individual units were weatherized, annual energy consumptions and
savings will be calculated as if they were single-family dwelling units (see Sect.3.2.1). Electricity
billing data and natural gas billing data (if present) will be collected on such units as part of the
large multifamily billing sample. These data will be analyzed using PRISM to calculate annual
pre- and post-weatherization energy consumptions and energy savings for the individual dwelling
(apartment) units. PRISM‘s model selection feature will be used to select the best PRISM model
for each house when analyzing the electricity data (i.e., heating-only, cooling-only, or heating and
cooling model).
Since natural gas (if present) can be used in these homes for water heating, cooking, and possibly
clothes drying in addition to space heating, the natural gas consumptions and savings represent
both space heating and baseload uses combined. In units heated by electricity, the electricity
consumptions and savings include both space heating and baseload uses in addition to space
cooling if employed. In non-electrically heated units, these energy consumptions and savings
include baseload uses, space cooling if employed, and any supplemental space heating that is
done using electricity. The electricity and natural gas consumptions and savings will not be
broken down into their separate heating, cooling, and baseload components.
449
Q.3 SMALL MULTIFAMILY BUILDINGS
Energy consumptions and savings in small multifamily buildings will be analyzed in a manner similar to
that for large multifamily buildings (see Section D.2):
•
Small multifamily buildings with central building heating systems or with apartment-level
heating systems that were weatherized as a building—A whole building analysis will be
performed because weatherization costs will likely be known at the building rather than unit
level. Building level consumptions and savings will then be normalized to a per unit basis by
dividing by the number of units in the building.
•
Buildings in which just individual apartments are weatherized—A unit-level analysis will be
performed.
This analysis approach will be applied primarily to electricity and natural gas billing data collected from
the comprehensive billing sample. If fuel-oil consumption data are collected on small multifamily
buildings as part of the monitored sample, then these data will be analyzed as described for single-family
homes (see Sect. 3.2.1).
450
451
APPENDIX R. ORNL AGGREGATE MODEL
ORNL‘s aggregate model applies the basic logic of the PRISM approach to billing and weather databases
aggregated over many houses to determine an overall program effect. It was developed and will be used
in the national evaluation to support and supplement the PRISM analysis when needed.
Many factors that often occur in the low-income homes that the Program serves can mask weather-related
correlations with fuel usage. These include:
•
periods of zero or below normal consumption because of service shutoff due to
nonpayment, household vacancies, and equipment breakdowns;
•
erratic use of heating and/or cooling equipment by occupants;
•
undersized heating and/or cooling equipment that runs at full capacity after a threshold
outdoor temperature is reached;
•
behavioral and occupancy changes; and
•
availability of just few months of data for a house before and/or after weatherization
Use of PRISM, which relies on a linear model of the relationship between weather and energy
consumption, can lead to high model failure rates at the individual household level (e.g., low R², high
coefficient of variance on the normalized annual consumption estimate, unrealistic balance point
temperature) because of these factors. Excluding large numbers of homes from the statistical analysis due
to model failures likely introduces potential sample bias, making the measurement of representative
Program impacts difficult because so many homes are eliminated. Although every effort will be made to
reduce model failures (e.g., use of PRISM‘s flatness index, data collection design to ensure a year‘s worth
of data before and after weatherization), an alternative approach is still needed.
The ORNL aggregate model examines and performs weather-normalizations on a group of houses to
smooth out the confounding effects that occur at the individual house level. It is a simpler method than
PRISM in that it does not assume that a linear model has to fit every house and does not attempt to
produce house-specific savings estimates. Its advantages are that it can use ―noisy‖ data and analyze data
for a more complete and representative sample of houses. As a result, the ORNL aggregate model focuses
on the overall Program (i.e., group) effect, rather than individual household savings, and reduces sample
bias due to excessive exclusion of households from the analysis.
452
PRISM and the ORNL aggregate model can be compared and contrasted as follows:
PRISM
ORNL Aggregate Model
•
Assumes a linear model of the
relationship between energy
consumption and heating degree days
for each individual house.
•
Assumes a linear model of the
relationship between energy
consumption and heating degree days
for a group of houses.
•
For each house, energy consumption
and heating degree days for multiple
billing periods are used to determine the
model coefficients for the house (i.e.,
and ).
•
For the group of houses, energy
consumption and heating degree days
over one time period for each house are
used to determine the model coefficients
for the group of houses (i.e., and ).
•
•
Regression performed for ―all possible‖
reference temperatures, and reference
temperature and subsequent model
coefficients chosen that gives the
highest model R².
A fixed reference temperature (e.g.,
65°F) is used for all houses to calculate
heating degree days.
•
For group of houses, total or average
consumption or savings calculating by
adding or average values for individual
houses.
•
For group of houses, total or average
consumption or savings calculated
directly from model.
453
R.1 PRISM
The PRISM model can be written as:
ei = + hddi(t)
(1)
Where:
ei = average daily energy use (e.g., Btu/day) for billing period i,
= average daily baseload (non-heating) energy use (e.g., Btu/day),
= heating slope (e.g., Btu/HDD), and
hddi(t) = heating degree days per day for billing period i calculated at balance point
temperature t (°F).
Typically, energy consumption data and daily outdoor temperatures corresponding to each billing period
are obtained for a house (preferably about 12 monthly billing periods over a year). Knowing the number
of days in each billing period, PRISM converts these data into an average daily energy use (ei) and
average daily heating degree days [hddi(t)] for each billing period and then uses these average daily
values to estimate three parameters for the house using linear least-squared regression techniques: the
average daily baseload energy use (), heating slope (), and balance point temperature (t) that produces
the highest correlation coefficient (R²). The normalized annual energy consumption (NAC) for the house
can then be calculated using these estimated parameters as follows:
NAC = ( 365.25) + ( HDDo(to))
(2)
Where:
NAC = normalized annual energy consumption (e.g., Btu), and
HDDo(to) = annual heating degree days calculated at the optimum balance point
temperature to (°F) estimated by PRISM.
When data are available before and after a house is weatherized, the NAC before weatherization (NACpre)
and after weatherization (NACpost) can be calculated, and the normalized annual energy savings (NAS) for
the house can be determined by:
NAS = NACpre NACpost
(3)
Where:
NAS = normalized annual energy savings,
NACpre = normalized annual energy consumption before weatherization, and
NACpost = normalized annual energy consumption after weatherization.
454
The average per household savings for a group of houses can be estimated using the NAS calculated for
each house individually by:
AGS = (NASi)/n
(4)
Where:
AGS = average group savings,
NASi = normalized annual energy savings for house i, and
n = number of houses used in the summation.
If data are available for a group of control homes, the net savings of a weatherized group can be
calculated by:
Net savings = AGSweatherized AGScontrol
(5)
Where:
AGSweatherized = average group savings of the weatherized houses, and
AGScontrol = average group savings of the control houses.
In Eqs. 1-5, statistical uncertainties associated with estimated parameters and calculated values can be
determined using normal statistical procedures.
R.2 AGGREGATE PRISM
An aggregate version of PRISM exists in which a linear model of energy use vs. heating degree days is fit
to a group of houses rather than to houses individually. In this case, each house must have the same
number of billing periods and the billing periods must coincide for each house (e.g., bills were read on the
15th of each month for 12 months for each house), and all the houses must be in the same geographic area
so that the heating degree days for the houses are the same. The energy consumption data for each billing
period are averaged and used with Eq. 1 so that the and that are estimated represent an average house
in the group. The NAC and the NAS for an average house can be calculated using Eqs. 2 and 3; in this
case, the NAS is equivalent to the average group savings calculated using Eq. 4 for houses that were
analyzed individually.
R.3 ORNL AGGREGATE MODEL
The PRISM model described by Eq. 1 above could be rewritten slightly as:
Ei = Di + HDDi(t)
(6)
Where:
Ei = energy use (e.g., Btu) for billing period i,
= average daily baseload (non-heating) energy use (e.g., Btu/day),
Di = number of days in billing period i,
= heating slope (e.g., Btu/HDD), and
HDDi(t) = heating degree days for billing period i calculated at balance point temperature
t (°F).
455
Note that Ei and HDDi(t) are energy use and heating degree days for billing period i, whereas ei and hddi
in Eq. 1 are average daily values for billing period i (ei = Ei/Di and hddi(t) = HDDi(t)/Di).
Data for an individual house could be applied to this model as before.
The ORNL aggregate model modifies Eq. 6 so that it can be applied to a group of houses simultaneously
rather than to houses individually. The model is:
Ei = Di + HDDi(65)
(7)
Where:
Ei = energy use (e.g., Btu) for house i over some time period,
= average daily baseload (non-heating) energy use (e.g., Btu/day),
Di = number of days in the time period for house i,
= heating slope (e.g., Btu/HDD), and
HDDi(65) = heating degree days over the time period for house i calculated at a balance
point temperature of 65°F.
Note that the subscript i in Eq. 7 identifies different houses, whereas the subscript i in Eq. 6 identifies
different billing periods for the same house. Also note that the ORNL aggregate model uses heating
degree days calculated at a fixed balance point temperature of 65°F for all houses, whereas PRISM
calculates a house-specific balance point temperature.
To apply the ORNL model, the following three values must be known for each house: the energy use over
a given time period, the number of days in the time period, and the number of heating degree days in the
time period (calculated at the same balance point temperature of 65°F for each house). Typically, monthly
billing data for each house are aggregated to obtain a single set of values for the house (e.g., 11 monthly
bills for a house are combined so that the energy use, the number of days, and the number of heating
degree days in the 11-month period are known for that house). The time periods and the length of the time
periods do not need to be the same for each house. Houses do not need to be geographically constrained;
houses from one area can be combined with houses from another area as long as the location appropriate
heating degree days are calculated for the time period associated with each house and at the same base
temperature.
These three values for many houses are used to estimate the average daily baseload energy use () and
heating slope () for an average house in the group using least-squared regression techniques. A NAC for
an average house in the group at a selected geographical location can then be calculated by:
NAC = 365.25 + HDDo(65)
(8)
Where:
NAC = normalized annual energy consumption (e.g., Btu), and
HDDo(65) = annual heating degree days for the chosen location calculated at a balance
point temperature of 65°F.
By aggregating/totaling billing data for each house over time, some of the variability in the energy data is
smoothed out and the influence of weather variations on consumption can be modeled more easily. By
456
analyzing a group of houses in an aggregate manner, unexplained variability seen at the individual house
level is removed and a model of an average house that represents the group of houses is produced.
If data are available for the group of houses before and after each one is weatherized, the NACpre and
NACpost for an average house in the group at a selected geographical location can be calculated, and the
NAS for such a house can be determine using Eq. 3.
If an average NAC and/or NAS were desired that was representative of all the houses weatherized,
covering multiple locations, weighted annual heating degree days could be calculated and used in Eq. 8,
or the annual heating degree days for each house could be used in Eq. 8 and the subsequent NACs
averaged.
The ORNL aggregate model was presented in simplistic form above. In reality, pre- and postweatherization differences in energy use can be modeled to avoid the need to model pre- and postweatherization consumption separately and then subtracting to obtain savings. Energy savings can be
modeled as follows:
Ei = Ei, pre Ei, post = [pre Di, pre + pre HDDi, pre(65)] [post Di, post + post HDDi, post(65)] (9)
Where:
Ei = energy use difference (e.g., Btu) between pre and post periods for house i,
Ei, pre = energy use (e.g., Btu) for house i for pre-weatherization period,
Ei, post = energy use (e.g., Btu) for house i for post-weatherization period,
pre = average daily baseload energy use (e.g., Btu/day) for pre period,
post = average daily baseload energy use (e.g., Btu/day) for post period,
Di, pre = number of days for house i in pre period,
Di, post = number of days for house i in post period,
pre = heating slope (e.g., Btu/HDD) for pre period,
post = heating slope (e.g., Btu/HDD) for post period,
HDDi, pre(65) = heating degree days for house i calculated at a balance point temperature
of 65°F for pre period, and
HDDi, post(65) = heating degree days for house i calculated at a balance point temperature
of 65°F for post period.
By modeling the difference in energy use, the correlation between pre- and post-weatherization
consumption for the same household is dealt with much like in a paired t-test.
The model can be simplified and the statistical variability of the remaining parameters reduced if there is
reason to assume that pre and post are equal to each other (because weatherization primarily impacts the
degree-day parameters) and/or if the bias in making the parameters equal to each other is acceptable. The
model can also be expanded to include control adjustments directly and, like PRISM, can include weather
adjustment for cooling energy consumption using cooling degree days simultaneously with adjustment for
heating energy consumption using heating degree days (although there is no provision for automatically
selecting the model as in PRISM). Furthermore, additional terms can be added to the model (e.g., for
climate region), with fitting occurring in one stage.
As with PRISM, the statistical uncertainties associated with estimated parameters and calculated values in
Eqs. 7-9 can be determined using normal statistical procedures.
457
APPENDIX S. ASSESSMENT OF A POTENTIAL SPLIT-WINTER RCT
CONFIRMATORY PROJECT
Despite the barriers to a classical RCT approach to WAP evaluation, we also consider here a hypothetical
―split-winter‖ RCT that could be conducted in conjunction with the QE WAP evaluation study. By ―splitwinter,‖ we mean that weatherization would be performed during one particular winter, and the total
duration of the study would generally be less than in a full evaluation. Sample size calculations for the
split-winter RCT suggest that, even if the legislative and cultural barriers could be circumvented, this
alternative RCT approach would still not be a good idea because of very large (and therefore expensive)
sample size requirements. This further supports the assumption that an RCT is not feasible in the WAPARRA context and that a carefully conducted QE study is a better approach.
Section S.I of this section presents additional background and some notation, and suggests another,
control-only, validation study design, as a potential alternative to a split-winter RCT approach. Sample
sizes calculations for the split-winter study are presented in Section S.II. Sample size calculations for the
alternative control-only validation study are in Section S.III. Concluding remarks are in Section S.IV.
S.I Background. One objective (among many) of the WAP evaluation study will be to estimate the
normalized annual savings (NAS) due to weatherization performed during a target time period, for
example, a program year. In the QE approach, treated subjects are selected randomly from the population
of subjects who received weatherization during the target time period. Control subjects are selected
randomly from (i) WAP-eligible subjects who applied during the target time period but did not receive
weatherization during that period, and (ii) eligible subjects who applied during a subsequent time period
(and may or may have yet received weatherization). In either cases (i) or (ii), only pre-weatherization
control-group consumption data is used to characterize the controls; post-weatherization control-group
data is inadmissible.
Treated subjects in the QE approach are selected as they would be in a hypothetical RCT in which a
random subset of subjects initially selected for treatment (per all priority constraints) are shunted off to
serve as controls. However, control subjects in this RCT would differ statistically from the controls in the
QE study. In the QE approach, the overall time periods (both beginning and ending dates) for the controls
are shifted toward times later than the target time period for subjects who receive weatherization. In the
RCT, by definition of RCT, the overall time periods must have exactly the same distribution for both the
treatment and control groups. Therefore, either (1) the weatherization eventually performed for the
controls must be delayed fairly long, or (2) the post-weatherization periods for the weatherized subjects
and must be fairly short.
In both the QE and RCT approaches, pseudo-weatherization dates are assigned to control subjects so that
the distribution of pseudo-weatherization dates for the controls is the same as the distribution of actual
weatherization dates for the treated subjects. The pseudo-weatherization dates are used in the analysis of
the control energy consumption data in the same way that the actual weatherization dates are used in the
analysis of the weatherized subject data.
However, in the QE approach, unlike the RCT approach, if the WAP applicant population changes during
the overall time-period (either for subjects who receive weatherization or controls), then the QE NAS
estimates could be biased. Therefore it is reasonable to consider (at least hypothetically) an RCT-based
check on potential bias in the QE NAS estimates. Although an across-the-board RCT approach would not
be feasible, it might be feasible to implement a confirmatory RCT on a much smaller scale.
The following notation will be needed in discussing NAS estimates. We are trying to estimate the
control-adjusted NAS (CNAS):
458
CNAS = NASW - NASC
where
NASW = Normalized Annual Savings, Weatherized = Cwb - Cwa
NASC = Normalized Annual Savings, Controls NASC = Ccb - Cca
where in turn
Cwb = weather-adjusted average annual consumption before weatherization for subjects who
receive weatherization
Cwa = weather-adjusted average annual consumption after weatherization for subjects who
receive weatherization
Ccb = weather-adjusted average annual consumption before pseudo-weatherization for controls
Cwa = weather-adjusted average annual consumption after pseudo-weatherization for controls
Note that the QE weatherized group sample is actually selected the same way as the weatherized group
sample would be selected in an RCT in which selected subjects are randomly assigned to either the
treatment or control groups. For this reason, in theory, a full RCT (e.g., a split-winter study) would not
actually be needed to validate the QE CNAS estimate. Instead, a validation check on the QE estimate
could be performed by (1) selecting an independent sample of controls (as in the RCT approach) from the
same population (i.e., target time period) that the weatherized group is selected from, and then (2)
estimating the NASC for the independent sample and comparing it to the NASC estimate from the QE
study. A conclusion of no-difference between the two estimates supports the QE approach. This
validation check would require only supplemental control subjects, not an entire supplemental RCT.
However, as with the split-winter approach, either the control subjects would have to be denied treatment
for a long time (and compensated accordingly), or the post-pseudo-weatherization study periods would
have to be relatively short.
The longer weatherization is delayed, the less feasible a study becomes. Therefore, a control-only
validation study with a randomly selected control group that matches the study period of the main QE
study weatherization group may not be feasible. On the other hand, if the study period is shorter than the
study period for the main study, then this kind of validation would be compromised. In view of this
tradeoff, we will consider two studies supplemental to a main WAP QE evaluation study: (1) a splitwinter RCT with post-weatherization periods shorter than the post-weatherization periods in the main QE
study, and (2) a comparison study of the main QE control group with a set of controls randomly selected
from subjects initially selected for weatherization in the QE study, and with weatherization delayed as
needed for consistent comparisons with the main study weatherization group. In case (1) the main study
CNAS estimate would be compared with the split-winter validation study CNAS. Agreement supports
the main study CNAS estimates. In case (2) the main study control group would be compared with the
validation study control reference group. Agreement supports the use of the main study control group and
thus the main study CNAS estimates. Sample sizes for a split-winter RCT are considered in Section S.II.
Sample sizes for the alternative control-only validation are discussed in Section S.III.
S.II Sample Size Requirements for a Split-Winter RCT Study To implement a split winter study requires
installing submeters in homes to directly measure energy use. These studies are more expensive than
billing history studies because of the cost of the submeters and staff to install and retrieve the submeters.
To reckon sample sizes necessary for the split-winter study, we estimate the variability of both the RCT
459
and QE studies using estimates from the 1990 WAP evaluation.69 We use the approximation that the
RCT and QE designs with the same number of subgrantees and housing units (and assuming the same
subsampling of housing units for both studies) have the same precision. This approximation may actually
lead to underestimates of necessary sample size, because, in fact, the split-winter studies control
estimates, although more accurate, are likely to be more variable than corresponding QE estimates. This
is true for the reason discussed above, that for the RCT study, the control data period ends at the end of
the split winter study, whereas the control data period is much longer for the QE study. This potential
variability underestimation is discussed further below.
Consider the CNAS for the main national QE study and corresponding CNAS‘ for the RCT study, where
because of differences in the study time periods, CNAS and CNAS' are not necessarily equal. As a
check, we will test the hypothesis that they are. Let = CNAS - CNAS'. We will test the null
hypothesis Ho: = 0 with a hypothesis test constructed so that:
(1) the level of the test (i.e., probability of rejecting under the null hypothesis) is (e.g., .05 or 0.1)
(2) for specified , a hypothetical difference between the main CNAS and the RCT CNAS, the
power (i.e., probability of rejecting Ho) when = is at least a specified probability (e.g., .8 or .9)
Property (1) is to control the probability of false rejection, and property (2) is to control the probability of
false acceptance, in particular at = . (The probability of false acceptance is 1 minus the power.)
Given various values for , , and the power, we can compute sample sizes for various validation RCTs.
Here we compute the RCT sample size as a percentage of the main-study sample of 400 subgrantees.
Appropriate values for are suggested by the 1990 study results: Tables 5.1 and 5.3 in the 1990 study
report list the following CNAS estimates and standard errors:
CNAS
Standard
Fuel
Estimate
Units
Error
————————————————————————
Electricity
1830
kwh/year
358
Natural Gas 173
ccf/year
18
69
Brown, M., Berry, L., Balzer, R., Faby, E., National Impacts of the Weatherization Assistance Program in SingleFamily Dwellings, ORNL/CON-326, Oak Ridge National Laboratory, Oak Ridge, Tennessee, May, 1993. Available
at http://weatherization.ornl.gov/pdf/ORNL_CON-326.pdf.
460
We take to be either .5, 1, or 2 times the CNAS estimate from the 1990 evaluation. For either .05 or
.1, and for power of either .8 or .9, we then get the following tables of values for the necessary sample
size for the corresponding RCT validation study, expressed as percentage of the main (1990 or 2010)
study:
Fuel=Electricity, CNAS Estimate=1,830, CNAS Estimate StdErr=358, Units=kwh/year
Percentage
Test
Power of
of Main
Level
Test at
Study
Factor
()
=
Needed
———————————————————————————————————————————————————————
915
0.5
.05
.8
240.3
.9
321.7
.10
.8
189.3
.9
262.2
1,830
1.0
.05
.8
60.1
.9
80.4
.10
.8
47.3
.9
65.5
3,660
2.0
.05
.8
15.0
.9
20.1
.10
.8
11.8
.9
16.4
Fuel=Natural Gas, CNAS Estimate=173, CNAS Estimate StdErr=18, Units=ccf/year
Percentage
Test
Power of
of Main
Level
Test at
Study
Factor
()
=
Needed
———————————————————————————————————————————————————————
87
0.5
.05
.8
68.0
.9
91.0
.10
.8
53.5
.9
74.2
173
1.0
.05
.8
17.0
.9
22.7
.10
.8
13.4
.9
18.5
346
2.0
.05
.8
4.2
.9
5.7
.10
.8
3.3
.9
4.6
So, roughly speaking, to detect in the main national CNAS and the validation RCT CNAS' a difference
the size of 100% of the main national CNAS (Factor = 1.0 in the tables) would require the supplemental
validation RCT to be about 60-80% of the size of the national evaluation for electricity, and about 1520% of the size of the national evaluation for natural gas. These percentages for even this limited RCT
study are high enough to suggest skipping the split-winter RCT study completely. Note too that the value
1 (and .5 and 2) considered here for is rather large. Smaller values of entail even larger sample sizes.
As discussed above, the use here of the approximation that the RCT and QE designs have the same
precision may also lead to understated sampled sizes.
461
The large (e.g., 100% of CNAS) values for and the large necessary sample sizes (expressed as
percentages of the national evaluation) reflect the high variability inherent in weatherization billing data
and, ultimately, in the national CNAS estimates. The coefficients of variation (CV) of the 1990-study
CNAS estimates are
CV = 100 × 358 / 1,830 = 19.6% for Electricity
CV = 100 × 18 / 173 = 10.4% for Natural Gas
which are quite high for overall estimates for a national study of that size. High variability seems to be a
fact of life for domestic energy consumption and billing data. The estimated sample sizes, and the
difference between the split-wither RCT and the QE study's time spans suggests that the RCT approach
may not be worth pursuing at all in the context of WAP evaluation.
S.III Sample Size Requirements for an Alternative, Control-Only Validation Study Next consider the
RCT validation study, with validation-study controls only, where a supplemental validation control group
is selected for which weatherization is delayed for as long as the main-study weatherized group is
observed. Except for the weatherization, such a validation control group would be statistically the same
as the weatherized group from the main QE study. Only supplemental control subjects (no supplemental
weatherized subjects) would be needed with this approach, and so the approach would require
substantially less resources than the split-winter RCT. On the other hand, it would require longer delays
in the weatherizations for the control subjects.
The hypothesis of interest in the validation is that the control group NASC from the main study and the
validation control group NASC' are the same. Redefining above so that = NASC - NASC', we are
again interested in testing the null hypothesis Ho: = 0, and in the power of the test when assumes a
particular value . For this control-only validation approach, however, the appropriate values for are
still relative to the CNAS estimate, which is the ultimate objective of the study. Again from Tables 5.1
and 5.3 in the 1990 study:
NASC
CNAS
NASC Estimate
Fuel
Estimate
Estimate
Units
Standard Error
———————————————————————————————
Electricity
-963
1830
kwh/year
238
Natural Gas -37
173
ccf/year
14
462
So again taking to be either .5, 1, or 2 times the CNAS estimate from the 1990 evaluation, either .05
or .1, and the power to be either .8 or .9, we get the following tables of values for the necessary sample
size for the alternative control-only validation study, again expressed as a percentage of the control group
from the 1990 (or 2010) evaluation
Fuel=Electricity, CNAS Estimate=1,830, NASC Estimate StdErr=238, Units=kwh/year
Percentage
of Main
Test
Power of
Control
Level
Test at
Group
Factor
()
=
Needed
———————————————————————————————————————————————————————
915
0.5
.05
.8
106.2
.9
142.2
.10
.8
83.7
.9
115.9
1,830
1.0
.05
.8
26.6
.9
35.5
.10
.8
20.9
.9
29.0
3,660
2.0
.05
.8
6.6
.9
8.9
.10
.8
5.2
.9
7.2
Fuel=Natural Gas, CNAS Estimate=173, NASC Estimate StdErr=14, Units=ccf/year
Percentage
of Main
Test
Power of
Control
Level
Test at
Group
Factor
()
=
Needed
———————————————————————————————————————————————————————
87
0.5
.05
.8
41.1
.9
55.0
.10
.8
32.4
.9
44.9
173
1.0
.05
.8
10.3
.9
13.8
.10
.8
8.1
.9
11.2
346
2.0
.05
.8
2.6
.9
3.4
.10
.8
2.0
.9
2.8
So, roughly speaking, to detect in the main national NASC and validation NASC' a difference the size of
100% of the national CNAS (Factor = 1.0 in the tables) would require the supplemental validation control
group to be about 20-30% of the size of main control group for electricity and 10-12% of the size of the
main control group for natural gas. Although these sample sizes are much smaller than what is required
for the split-winter study, they are still substantially, particularly in the context of (1) the 100% (or 50%
or 200%) of the national CNAS that the study would be designed to detect, and (2) the procedural
difficulty of extended delays in weatherization of the supplemental control group.
463
S.IV Observations
The sample-size requirements for a split-winter RCT are so substantial they suggest that RCTs are not
feasible for a WAP evaluation. The sample-size requirements for an alternative validation approach
involving only control subjects are more reasonable, but they are still substantial, particularly in view of
the large differences they are designed to detect and the procedural constraints that would have to be
circumvented with that approach. The sample sizes that would be needed for either of the split-winter
RCT or control-only approaches suggest that these approaches are not be worth pursuing even as
supplemental validation or partial-validation studies, and the RCT approach certainly does not seem
worth pursuing as a design for a main WAP evaluation.
S.V. Conclusions
If a classical RCT approach is inappropriate, the encouragement design is limited in its scope, and a splitwinter design is infeasible, the QE approach becomes a natural contender to consider as an alternative.
The weatherization group in the QE approach is selected as it would be in an RCT. The issue in the QE
approach, then, is whether because of application/weatherization time differences between the
weatherized and comparison groups, differences between the comparison group and the ideal control
group that would be selected in an RCT might be substantial enough to render the comparison group so
flawed that the QE approach (and in that case most likely any other approach) should not be pursued at
all.
The comparison group in the QE approach is an approximation. However, in a QE study, the comparison
and weatherization groups are generally compared with respect to potentially influential characteristics
(e.g., dwelling size, primary fuel, participant income), and if necessary the analysis can be stratified
according to those characteristics. Of course, in weatherization studies, weather-adjustments are made for
both weatherized and comparison dwellings. Thus the comparison group is examined and controlled in
every feasible way to ensure its adequacy as an approximation. Analysts know if the approximation is
inadequate.
The QE approach cannot be justified to the absolute standard that an RCT (were it feasible) might be
justified to. As the GAO report points out, however, insistence on an absolute standard will ―exclude
many potentially effective and worthwhile practices.‖ Thus the QE approach provides a viable alternative
to an RTC as the design basis for a WAP evaluation.
464
465
OMB Control Number: XXXX-XXXX
APPENDIX T. S6: ALL STATES POST-ARRA SURVEY
The U.S. Department of Energy‘s (DOE) Weatherization Assistance Program received an unprecedented
level of support from the American Recovery and Reinvestment Act (ARRA). The Program‘s grantees
and subgrantees have successfully ramped up their efforts to meet the weatherization goals created for the
ARRA-period. The next challenge facing the Program and its grantees and subgrantees is to transition to
post-ARRA world. This survey is being administered to all of the Program‘s ARRA-period grantees. The
data will assist the Program and its grantees and subgrantees to better manage this transition.
Public reporting burden for this collection of information is estimated to average one hour per response,
including the time for reviewing instructions, searching existing data sources, gathering and maintaining
the data needed, and completing and reviewing the collection of information. Send comments regarding
this burden estimate or any other aspect of this collection of information, including suggestions for
reducing this burden, to Office of the Chief Information Officer, Records Management Division, IM-11,
Paperwork Reduction Project (XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW,
Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork
Reduction Project (XXXX-XXXX), Washington, DC 20503.
All of the information obtained from this survey will be protected and will remain confidential. The data
will be analyzed in such a way that the information provided cannot be associated back to your state, your
agencies, or the housing units and clients that your state served.
466
1. Please indicate how leveraged funding changed from PY 2008 to PY2010 and you expect leveraged
funding to change from PY2010 to PY2012.
Provide the amount of leveraged funding received in PY 2010 by source. Please forecast leveraging
relationship for the post-ARRA period (i.e., during PY 2012).
Source of Leveraged
Weatherization Funding
Administered by State
How did leveraged funding
from this source to change
from PY 2008 to PY 2010?
1 = increased greatly
2 = increased
3 = no change
4 = decreased
5 = greatly decreased
6 = don‘t know
How do you expect leveraged funding
from this source to change from PY 2010
to the post-ARRA period in PY 2012?
1 = increase greatly
2 = increase
3 = no change
4 = decrease
5 = greatly decrease
6 = uncertainty is too great to answer this
question
LIHEAP
Petroleum Violation Escrow
(PVE)
Other Federal Programs
State Public Benefit Funds
Other State
Utilities
Program Income
In-Kind
Non-Profits
Other
2. Under ARRA, the allowable average amount of investment was increased from $2500 to $6500.
Moving forward, what level of average investment in homes does your state prefer?
a. $2500
b. $6500
c. Other ___________
d. No preference
3. Under ARRA, the eligibility threshold was increased from 150% of the poverty level to 200%. Moving
forward, what income eligibility threshold does your state prefer?
a. 150% of poverty threshold
b. 200% of poverty threshold
c. Other ____________
d. No preference
4. Should the policy on re-weatherization of homes to those weatherized before 1994 be re-considered in
light of program changes and new technology ‗bumps‘?
a. No
b. Yes
If yes, how should the policy be changed? __________________________
467
5. Should WAP endeavor to go ‗deeper‘ into homes?
a. No
b. Yes
If yes, please explain ________________________________________________
6. Should WAP formally incorporate water conservation into its program?
a. No
b. Yes
7. Does your state plan on increasing large multi-family weatherization post-ARRA?
a. Yes, increase substantially
b. Yes, increase somewhat
c. No Change
d. No, decrease somewhat
e. No, decrease substantially
8. Should DOE set goals for MF weatherization (i.e., to equal the percentage of multi-family units in the
state)?
a. No
b. Yes
If yes, what should the goals be? ______________________________
9. Does your state expect a pre-mature degradation in production as the ARRA period draws to a close in
March, 2012?
a. No
b. Yes
If yes, please explain ______________________________________________
10a. What specific types of training assistance can DOE provide to support post-ARRA ramp down?
(check all that apply)
a. Multi-family
b. Health and Safety
c. Other __________________
10b. What other types of assistance can DOE provide to support post-ARRA ramp down? (check all that
apply)
a. Up-dating and expanding the national curriculum
b. Up-dating and expanding other national training resources
c. Support and expand a certification scheme for the weatherization workforce
d. Other ______________________
11. Will your state act to consolidate subgrantees post-ARRA?
a. No
b. Yes, limited consolidation
c. Yes, substantial consolidation
12. How many new staff have been added to your state weatherization office during the ARRA period?
_____
13. How many staff from your state weatherization office will be laid-off post-ARRA? _____
468
14. What strategies are being used to retain new, young weatherization staff hired during ARRA by your
state office? Please describe ______________________
15. Is the training capacity created by your state‘s program and agencies, community colleges, etc.
sustainable after ARRA?
a. Yes
b. No
c. No new training capacity was added
16a. Does your state training plan include utilizing weatherization training centers for worker training?
a. Yes (go to 16c)
b. No
16b. If no, how does your state intend to provide worker training?
a. Staff resources- i.e. State monitors to provide worker training
b. Private, consultant trainers- i.e, Saturn, BMI, ECM etc.
c. Other ____________________
d. No future training is planned.
16c. Does your state training plan include funding a weatherization training center either existing or new?
a. Yes
b. No
16d. Will your state require that training resources be accredited weatherization training providers?
a. Yes
b. No
16e. In the future, will your state require weatherization workers to be certified in their job category
(assuming that a national Weatherization certification is available)?
a. Yes
b. No
17. Is your state doing anything to transition workers to other jobs post-ARRA?
a. No
b. Yes
If yes, what?
18. Does your state have plans to use alternative workforces for weatherization (e.g., volunteer labor)?
a. No
b. Yes
19. How many weatherization jobs created during ARRA may be lost after ARRA in your state?
____________
20. How substantial will be the costs that are associated with workforce reductions (e.g., workers
compensation) to local weatherization agencies in your state?
a. very substantial
b. substantial
c. not very substantial
d. no additional costs
469
21. Does your state see health and safety audits as a growing business for your subgrantees over the next
five years?
a. No
b. Yes, somewhat
c. Yes, substantially
22. Which alternative financing mechanisms to support weatherization post-ARRA might your state
pursue post-ARRA? (check all that apply)
a. revolving loan funds
b. on-bill payments
c. PACE programs
d. ESCO financing
e. carbon offsets
f. none of the above
23a. How can DOE provide support for expansion of the use of alternative third party financing
mechanisms (e.g., interest in WAP funding in the form of a (e.g. revolving loans, loan loss reserves,
rather than a grant ? One target property type for a revolving loan product are Low-Income Housing Tax
Credit Properties)?
23b. What discussions have you had with third parties regarding financing? What has the response been?
_____________________________
23c. Do you have any existing relationships with lenders?
a. Yes
b. No, but working on developing relationships
c. No
24. Over the next five years, what percentage of your state‘s subgrantees‘ service areas may become
saturated with respect to low-income weatherization?
a. 0
b. 1-10%
c. 11-20%
d. 21-30%
e. 30+%
25. Where do you see the market for non-low income home retrofit in your state over the next five years?
a. greatly increasing
b. increasing
c. no change
d. decreasing
e. greatly decreasing
26. Have you heard of the Home Energy Score system?
a. No
b. Yes
470
27. If so, how might this initiative impact low-income weatherization in your state?
a. no impact
b. might lead to some funding increases
c. might lead to moderate funding increases
d. might lead to substantial funding increases
28. Post-ARRA, compared to pre-ARRA funding levels, how do you foresee the size of your state‘s
weatherization assistance program?
a. Greatly expanded
b. Expanded
c. About the same
d. Reduced
e. Greatly reduced
29. On balance, how beneficial do you think that the ARRA funding will eventually be to the longer-term
prospects for your state‘s WAP-program?
a. Extremely beneficial
b. Beneficial
c. No long-term impact
d. Unbeneficial
e. Extremely unbeneficial
471
OMB Control Number: XXXX-XXXX
APPENDIX U: S7: ALL AGENCIES POST-ARRA SURVEY
The U.S. Department of Energy‘s (DOE) Weatherization Assistance Program received an unprecedented
level of support from the American Recovery and Reinvestment Act (ARRA). The Program‘s grantees
and subgrantees have successfully ramped up their efforts to meet the weatherization goals created for the
ARRA-period. The next challenge facing the Program and its grantees and subgrantees is to transition to
post-ARRA world. This survey is being administered to all of the Program‘s ARRA-period subgrantees.
The data will assist the Program and its grantees and subgrantees to better manage this transition.
Public reporting burden for this collection of information is estimated to average one hour per response,
including the time for reviewing instructions, searching existing data sources, gathering and maintaining
the data needed, and completing and reviewing the collection of information. Send comments regarding
this burden estimate or any other aspect of this collection of information, including suggestions for
reducing this burden, to Office of the Chief Information Officer, Records Management Division, IM-11,
Paperwork Reduction Project (XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW,
Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork
Reduction Project (XXXX-XXXX), Washington, DC 20503.
All of the information obtained from this survey will be protected and will remain confidential. The data
will be analyzed in such a way that the information provided cannot be associated back to your state, your
agencies, or the housing units and clients that your state served.
472
1. Please provide the amount of leveraged funding received in PY 2010 by source. Please forecast
leveraging relationship for the post-ARRA period (i.e., during PY 2012).
Source of PY 2010 Leveraged
Weatherization Funding
Administered by Agency
How did leveraged
funding from this
source to change from
PY 2008 to PY 2010?
1 = increased greatly
2 = increased
3 = no change
4 = decreased
5 = greatly decreased
6 = don‘t know
How do you expect leveraged funding from
this source to change from PY 2010 to the
post-ARRA period in PY 2012?
1 = increase greatly
2 = increase
3 = no change
4 = decrease
5 = greatly decrease
6 = uncertainty is too great to answer this
question
LIHEAP
Petroleum Violation Escrow
(PVE)
Other Federal Programs
State Public Benefit Funds
Other State
Utilities
Program Income
In-Kind
Non-Profits
Other
2. Under ARRA, the allowable average amount of investment was increased from $2500 to $6500.
Moving forward, what level of average investment in homes does your agency prefer?
a. $2500
b. $6500
c. Other ___________
d. No preference
3. Under ARRA, the eligibility threshold was increased from 150% of the poverty level to 200%. Moving
forward, what income eligibility threshold does your agency prefer?
a. 150% of poverty threshold
b. 200% of poverty threshold
c. Other ____________
d. No preference
4. Should the policy on re-weatherization of homes to those weatherized before 1994 be re-considered in
light of program changes and new technology ‗bumps‘?
a. No
b. Yes
If yes, how should the policy be changed? _______________________
473
5. Should WAP endeavor to go ‗deeper‘ into homes?
a. No
b. Yes
If yes, please explain ______________________________________________
6. Should WAP formally incorporate water conservation into its program?
a. No
b. Yes
7. Does your agency plan on increasing large multi-family weatherization post-ARRA?
a. Yes, increase substantially
b. Yes, increase somewhat
c. No Change
d. No, decrease somewhat
e. No, decrease substantially
8. Should DOE set goals for multi-family weatherization (i.e., to equal the percentage of multi-family
units in the state)?
a. No
b. Yes
If yes, what should the goals be? ______________________________
9. Does your agency expect a pre-mature degradation in production as the ARRA period draws to a close
in March, 2012?
a. No
b. Yes
If yes, please explain _________________________________________
10. What assistance can DOE provide to support post-ARRA ramp down?
a. Health and safety audit training
b Support and expansion of a national weatherization certification scheme
c. Training assistance to prepare workers for certification
d. Other _________________
11. How many new staff have been added to your agency‘s weatherization program during the ARRA
period? _____
12. How many staff from your agency‘s weatherization program might be laid-off post-ARRA? _____
13. What strategies are being used to retain new, young weatherization or trained staff hired during
ARRA by your agency? Please describe ______________________
14. Are expenses for new equipment, software, etc. purchased by your agency during ARRA sustainable
after ARRA?
a. Yes
b. No
c. No new equipment, software, etc. was purchased
474
15. Is the training capacity created by your agency sustainable after ARRA?
a. Yes
b. No
c. No new training capacity was added
16. Is your agency doing anything to transition workers to other jobs post-ARRA?
a. No
b. Yes
If yes, what?
17. Does your agency have plans to dispose of extra trucks post-ARRA?
a. No, we do not have extra trucks
b. No, we may have extra trucks but we do not have a disposal plan yet
c. Yes, we have extra trucks and we have a disposal plan
If yes, please describe the disposal plan ______________________________
18. Does your agency have plans to dispose of extra blower doors post-ARRA?
a. No, we do not have extra blower doors
b. No, we may have extra blower doors but we do not have a disposal plan yet
c. Yes, we have extra blower doors and we have a disposal plan
If yes, please describe the disposal plan ______________________________
19. Does your agency have plans to dispose of extra infra-red and/or cameras post-ARRA?
a. No, we do not have extra cameras
b. No, we may have extra cameras but we do not have a disposal plan yet
c. Yes, we have extra cameras and we have a disposal plan
If yes, please describe the disposal plan ______________________________
20. How will your agency cost-out unused equipment post-ARRA? ________________________
21. Does your agency have plans to use alternative workforces for weatherization (e.g., volunteer labor)
post-ARRA?
a. No
b. Yes
If yes, please explain __________________________________________
22. How many weatherization jobs created during ARRA may be lost post-ARRA in your agency‘s
jurisdiction?
____________
23. If your agency uses contractors to perform weatherization, how much will your contractor pool
decrease post-ARRA?
a. 0%
b. 1-10%
c. 11-20%
d. 21-30%
e. 31-40%
f. more than 40%
475
24. Will some of your agency‘s contractors go out of business?
a. No
b. Yes
If yes, how many? _________________
25. How substantial be will the costs that are associated with workforce reductions (e.g., workers
compensation) to your agency?
a. very substantial
b. substantial
c. not very substantial
d. no additional costs
26. Will your agency abandon Davis-Bacon wages post-ARRA?
a. No
b. Yes
27. Does your agency see its weatherization clientele changing demographically over the next five years?
a. No
b. Yes
If yes, please explain _____________________________________________
28. Does your agency see the types of homes in your weatherization program changing over the next five
years?
a. No
b. Yes
If yes, please explain ______________________________________________
29. As of the end of PY 2010, what is the demand for low-income weatherization services in your
agency‘s jurisdiction?
a. extremely high
b. very high
c. high
d. moderate
e. low
f. very low
30. Does your agency see the demand for weatherization services changing over the next five years?
a. No
b. Yes
If yes, please explain ________________________________________________
31. Will your agency explore fee for service projects post-ARRA?
a. No
b. Yes
32. Does your agency see health and safety audits as a growing business for your subgrantees over the
next five years?
a. No
b. Yes, somewhat
c. Yes, substantially
476
33. Which new measures would your agency encourage DOE approving to be installed in homes over the
next five years? ___________________________
34. Which alternative financing mechanisms to support weatherization post-ARRA might your agency
pursue post-ARRA? (Check all that apply)
a. revolving loan funds
b. on-bill payments
c. PACE programs
d. ESCO financing
e. carbon offsets
f. none of the above
35. What help can DOE provide to support expansion of the use of alternative third-party financing
mechanisms? _____________________________
36. Does your agency see the demographics of its weatherization workforce changing over the next five
years?
a. No
b. Yes
If yes, please explain __________________________________________________
37. What are your agency‘s major weatherization workforce challenges over following five years?
_______________________________________________________
38. Where do you see the market for non-low income home retrofit in your agency‘s jurisdiction over the
next five years?
a. greatly increasing
b. increasing
c. no change
d. decreasing
e. greatly decreasing
39. Have you heard of the Home Energy Score system?
a. No (go to Q41)
b. Yes
40. If so, how might this initiative impact low-income weatherization conducted by your agency?
a. no impact
b. might lead to some funding increases
c. might lead to moderate funding increases
d. might lead to substantial funding increases
41. Post-ARRA, compared to pre-ARRA funding levels, how do you foresee the size of your agency‘s
weatherization assistance program?
a. Greatly expanded
b. Expanded
c. About the same
d. Reduced
e. Greatly reduced
477
42. Is your agency resigned to going back to business as usual post-ARRA?
a. No
b. Yes
43. On balance, how beneficial do you think that the ARRA funding will eventually be to the longer-term
prospects for your agency‘s WAP-program?
a. Extremely beneficial
b. Beneficial
c. No long-term impact
d. Unbeneficial
e. Extremely unbeneficial
478
APPENDIX V. S8: WEATHERIZATION TRAINING CENTERS POST-ARRA SURVEY
OMB Control Number: _ _ _ _ - _ _ _ _
S8 – WEATHERIZATION TRAINING CENTERS POST-ARRA SURVEY
This data is being collected to conduct a survey of the directors of DOE funded Weatherization
Training Centers to gain insights into weatherization training strategies and needs anticipated
post-ARRA. The ARRA period is defined to run from April 2009 to March 2012.
Public reporting burden for this collection of information is estimated to average 60 minutes per
response, including the time for reviewing instructions, searching existing data sources,
gathering and maintaining the data needed, and completing and reviewing the collection of
information. Send comments regarding this burden estimate or any other aspect of this
collection of information, including suggestions for reducing this burden, to Office of the Chief
Information Officer, Records Management Division, IM-11, Paperwork Reduction Project
(XXXX-XXXX), U.S. Department of Energy, 1000 Independence Ave SW, Washington, DC,
20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork Reduction
Project (XXXX-XXXX), Washington, DC 20503.
Lastly, all of the information obtained from this survey will be protected and will remain
confidential. The data will be analyzed in such a way that the information provided cannot be
associated back to you or your training center.
1. What is the name of your weatherization training center? _____________________
2. Please describe the weatherization training offered by your Center? {Inquire about training
topics, certifications given and supported} _____________
3. What percentage of your weatherization training program was supported by DOE funds preARRA, during the ARRA period, and expected to be post-ARRA? __________________
4. What was your Center‘s budget pre-ARRA, during ARRA, and expected to be post-ARRA?
______
5. How many staff did your Center employ pre-ARRA, during ARRA, and expected to employ
post-ARRA? _____________________________
6. How did your Center ramp up its weatherization training program during the ARRA period?
{Inquire about increases in number of courses, hiring of staff, non-traditional/innovative training,
marketing techniques, and equipment/technology purchases.} ___________
479
7. Did you find the support for your Center from DOE useful in implementing your ARRA
period training and planning? What worked well? What didn‘t? _________________________
8. Is DOE providing adequate support for post-ARRA planning? What is working? What else do
you need? ________________________
9. What assistance ought DOE provide to assist in your center‘s post-ARRA transition?
_____________________
10. How is or will your center ramp down its weatherization training program post-ARRA?
{Inquire about laying off staff, transitioning into non-low income programs, finding new
leveraging partners, selling equipment, changing fee structures for classes, dropping
weatherization classes.} ________________
11. Is there a risk that your Center may have to close down post-ARRA? If so, what would be
required to prevent that? _____________________
12. At the peak during the ARRA period, how many weatherization trainees were served by your
center per year and how many do you expect to serve post-ARRA? __________
13. How will weatherization training offered by your center change post-ARRA? {e.g., less
hands-on} ______________
14. Will your relationship with the grantee(s) continue post-ARRA? ___________________
15. Please describe your thoughts about the demand for weatherization training in your area and
nationally over the next five years. _____________________________
16. What types of trainees will be attracted to this field over the next five years and what types of
jobs will they be able to land? {Inquire about potential problems in attracting new trainees}
___________________________
480
| File Type | application/pdf |
| Author | Bruce Edward Tonn |
| File Modified | 2011-05-20 |
| File Created | 2011-05-20 |