Synopsis of WAP ARRA Period Evaluation

SYNOPSIS OF WAP ARRA PERIOD EVALUATION

Draft April 12, 2011

1. INTRODUCTION

The U.S. Department of Energy/Office of Weatherization and Intergovernmental Program (DOE/OWIP) has tasked Oak Ridge National Laboratory with evaluating its Weatherization Assistance Program (WAP) as it operated during the American Recovery and Reinvestment Act period. The WAP ARRA period evaluation is a companion project to the WAP retrospective evaluation. This synopsis first describes the similarities and differences between the retrospective and ARRA period evaluations. Section 3 presents the ARRA period evaluation surveys and dataforms, noting how they are similar or different from those administered by the retrospective evaluation. Section 4 presents the timelines for administering the ARRA period surveys and dataforms.

2. COMPARISON OF WAP RETROSPECTIVE AND ARRA PERIOD EVALUATIONS

There are several important points to make regarding the similarities, differences, and overlaps between the retrospective evaluation of WAP as described in Ternes et al. (2007) and the WAP-ARRA evaluation described in Tonn et al. (2011).¹ First, as Table 1 shows, the two evaluations generally encompass the same research tasks. For example, the central component of each evaluation is the collection of billing histories for homes heated with electricity and natural gas; these data provide the basis for the national estimate of energy savings attributable to the program as well as associated cost-effectiveness analyses. Non-energy impacts are assessed by both evaluations. Additionally, both evaluations administer a core set of surveys and data forms (e.g., S1: All States Program Information Survey). All of these tasks focus on the Program Years (PYs) indicated in the Table 1.

The third column of Table 1 shows that the retrospective evaluation overlaps with the ARRA period in five research areas. When the retrospective evaluation was conceived in 2005 and when the plan was written in 2006 and 2007, it was assumed that the Program would not undergo any major changes that could change the evaluation results. Thus, the retrospective evaluation was designed to estimate national energy savings and to constitute most of the process assessment for the immediately past Program Year (PY2008) while simultaneously implementing several research tasks during the Program Year in which the evaluation was to take place (i.e., PY 2009). When the retrospective evaluation began, it was decided to look back to pre-ARRA Program Years 2007 and 2008 though the ARRA period had already begun.

Thus, these five research tasks that are being funded by the retrospective evaluation are actually assessing weatherization activities that took place during the ARRA period: analysis of sub-metered data for homes heated with bulk fuels; a major indoor air-quality study; case studies of high-performing agencies and exemplary client-education programs; and the administration of two major surveys, S4 (Occupant Survey) and S5 (Weatherization Staff Survey).

Table 1 also indicates that three major tasks undertaken by the retrospective evaluation will not be duplicated by the WAP-ARRA period evaluation: Bulk Fuels Study, Indoor Air Quality Study, and Field Process Study. Additionally, the retrospective evaluation will fund case studies of high-performing agencies and exemplary client-education programs, but the WAP-ARRA evaluation will not. Conversely, the WAP-ARRA period evaluation will fund case studies of underperforming weatherized units and one U.S. territory that received new, substantial ARRA funding (e.g., Puerto Rico). In summary, Table 1 indicates that twelve areas of major tasking are similar between the two evaluations, three major tasks being conducted by the retrospective evaluation will not be reprised by the ARRA period evaluation, and seven new tasking areas are being added to the WAP ARRA period evaluation.

Table 1. Comparison of WAP Retrospective and WAP-ARRA Period Evaluations

	Retrospective Evaluation on Weatherization Assistance Program	Retrospective Evaluation Research Taking Place during (Overlapping with) ARRA Period	WAP-ARRA Period Evaluation Research
Analysis of Billing Histories: Homes heated with electricity and natural gas	Program Years 2007 and 2008		Program Years 2009, 2010, and 2011
Analysis of Submetered Data: Homes heated with propane and fuel oil		Winter 2010-2011 and Winter 2011-2012
Analysis of Persistence of Energy Savings in Weatherized Homes			Program Years in the 1990s
Cost-effectiveness Analyses	Program Year 2008		Program Year 2010
Non-Energy Impacts	Program Year 2008		Program Year 2010
Social Network Study			Program Year 2011
GHG Emissions Study			Program Years 2007-2010
S1: All States Program Information Survey	Program Year 2008		Program Year 2010
S2: All Agencies Program Information Survey	Program Year 2008		Program Year 2010
S3: Subset of Agencies Detailed Program Information Survey	Program Year 2008		Program Year 2010
S4: Occupant Survey		CY’s 2011 and 2012	CY 2012
S5: Weatherization Staff Survey		CY 2011	CY 2012
Program Characterization	Program Year 2008		Program Year 2010
Case Studies		Program Year 2008 – Six High Performing Agencies and Six Exemplary Client Ed Programs	Program Year 2010 – One U.S. Territory
Weatherization Deferral Study			Program Year 2010
Under-Performers Study			Program Years 2007-2009
DF2/3: Housing and Building Information Data Forms	Program Year 2008		Program Year 2010
DF4: Utility Information from Agencies Data Form	Program Years 2007 and 2008		Program Years 2009, 2010, 2011
DF5: Utility Billing History Data Forms	Program Years 2007 and 2008		Program Years 2009, 2010, 2011
Indoor Air Quality		Winter 2010-2011, Summer 2011, Winter 2011-2012
Field Process Study		CY 2011
Special Studies			WIPP and SERC Program Years 2010-2011; Encouragement Design – Program Year 2011
Post-ARRA Surveys (S6, S7 and S8)			CY 2011

3. WAP ARRA PERIOD EVALUATION SURVEYS AND DATAFORMS

Table 2 lists the surveys and dataforms that will be administered as part of the WAP ARRA period evaluation. Overall, the ARRA period evaluation will utilize surveys and dataforms developed under the retrospective evaluation that have been reviewed and approved by OMB. Specifically, five surveys and six dataforms will be re-used. Table 2 indicates the revisions, if any, made to the surveys and dataforms for their use in the ARRA period evaluation. Revisions to the dataforms are minor and reflect a shuffling of questions between DF2/3 and DF4 to make it easier for respondents to fill them out. A significant number of questions were added to S1 and S2 to capture ARRA period specific issues. Three new surveys have been added; S6, S7, and S8. These collect information from states, agencies, and DOE Weatherization Training Centers, respectively, on their post-ARRA issues, challenges and strategies that will be useful management information for DOE/OWIP. It should be noted that DF1, DF10, DF6, DF7 and DF8 have been dropped.

Table 2. WAP ARRA Period Evaluation Surveys and Dataforms

Identifier	Name	Changes from WAP Retrospective Version
S1	All States Program Information Survey	Questions pertaining to ARRA period added; ~ 10% of original questions deleted
S2	All Agencies Program Information Survey	Questions pertaining to ARRA period added; ~ 10% of original questions deleted
S3	Subset of Agencies Detailed Program Information Survey	Several questions moved to S2
S4	Occupant Survey	Client satisfaction questions pertaining to WIPP and SERC Programs added; Multi-family section added
S5	Weatherization Staff Survey	Questions designed to follow-up from WAP Retrospective were added
S6	All States Post-ARRA Survey	New
S7	All Agencies Post-ARRA Survey	New
S8	DOE Weatherization Training Centers Post-ARRA Survey	New
DF2	Housing Unit Information Data Form	Minor revisions
DF3	Building Information Form	Minor revisions
DF4	Electric & Natural Gas Bills Information from Agencies	Minor revisions
DF5	Electric & Natural Gas Bills from Utilities	No Change
DF9	Contact Information from Agencies for Occupant Survey Data Form	No Change
DF11	Contact Information from Agencies for Weatherization Staff Survey Data Form	No Change

All the surveys and dataforms will be administered in the same way as they were in the retrospective evaluation. Specifically, they will all be web-based, except S4 Occupant Survey, which will be administered by phone using a CATI system and S8, which will also be conducted by phone using open-ended questions. The same packages of introductory letters and non-response follow-up protocols will be followed.

It is anticipated that response burdens upon respondents will be less than for the retrospective evaluation for several reasons. First, states, agencies and utilities will benefit from having completed these tasks once before for the retrospective evaluation. Prior experiences with utilities especially suggest that much time is saved from routinized response to requests for billing histories. Second, the ORNL team has learned how to extract much information from state weatherization databases that can pre-populate surveys and dataforms and in some cases eliminate the need for the team to collect billing histories on its own. Third, a shortened version of DF2/3 will be given to large agencies in order to cut down on their burdens in completing these forms.

4.0 Timelines

Table 3 presents timelines for when survey and dataforms for the WAP ARRA period evaluation will be administered, given an anticipated approval from OMB to begin administration by the third quarter of CY2011. It should be noted that six of the eight surveys will only be administered once, all in CY2011. Conversely, the project will administer dataforms on three separate occasions to ultimately collect billing histories to estimate national program energy savings for Program Years 2009, 2010, 2011.

Table 3. Survey and Dataform Administration Timelines

Instrument/Administration	CY2011		CY2012				CY2013
	Q3	Q4	Q1	Q2	Q3	Q4	Q1	Q2	Q3	Q4
S1 – All States Prog. Info.		X
S2 – All Agencies Prog. Info.		X
S3 – Subset of Agencies Prog. Info.		X
S4 – Occupant Survey: WIPP	X				X
S4 – Occupant Survey: SERC					X
S4 – Occupant Survey :Under Performers Study				X
S5 – WX Staff Survey: Follow-up						X
S5 – WX Staff Survey: WIPP					X
S6 – All States Post-ARRA	X
S7 – All Agencies Post-ARRA	X
S8 – WX Training Centers Post-ARRA	X
DF2/3 – Housing/Building Info.		X	X		X	X			X	X
DF4 – Billings Info. From Agencies		X			X				X
DF5 – Billings Data from Utilities		X	X		X	X			X	X
DF9 – Occupant Contact Info.				X	X
DF11 – WX Staff Contact Info.					X

1 Both reports can be found at http://weatherization.ornl.gov