Synopsis of WAP ARRA Period Eval Revised 5-2012

SYNOPSIS OF WAP ARRA PERIOD EVALUATION.docx

Weatherization Assistance Program ARRA-Period Evaluation

Synopsis of WAP ARRA Period Eval Revised 5-2012

OMB: 1910-5168

Document [docx]
Download: docx | pdf

SYNOPSIS OF WAP ARRA PERIOD EVALUATION

May 5, 2012


1. INTRODUCTION


The U.S. Department of Energy/Office of Weatherization and Intergovernmental Program (DOE/OWIP) has tasked Oak Ridge National Laboratory with evaluating its Weatherization Assistance Program (WAP) as it operated during the American Recovery and Reinvestment Act period. The WAP ARRA period evaluation is a companion project to the WAP retrospective evaluation. This synopsis first describes the similarities and differences between the retrospective and ARRA period evaluations. Section 3 presents the ARRA period evaluation surveys and dataforms, noting how they are similar or different from those administered by the retrospective evaluation. Section 4 presents the timelines for administering the ARRA period surveys and dataforms.


2. COMPARISON OF WAP RETROSPECTIVE AND ARRA PERIOD EVALUATIONS


There are several important points to make regarding the similarities, differences, and overlaps between the retrospective evaluation of WAP as described in Ternes et al. (2007) and the WAP-ARRA evaluation described in Tonn et al. (2011).1 First, as Table 1 shows, the two evaluations generally encompass the same research tasks. For example, the central component of each evaluation is the collection of billing histories for homes heated with electricity and natural gas; these data provide the basis for the national estimate of energy savings attributable to the program as well as associated cost-effectiveness analyses. Non-energy impacts are assessed by both evaluations. Additionally, both evaluations administer a core set of surveys and data forms (e.g., S1: All States Program Information Survey). All of these tasks focus on the Program Years (PYs) indicated in the Table 1.


The third column of Table 1 shows that the retrospective evaluation overlaps with the ARRA period in five research areas. When the retrospective evaluation was conceived in 2005 and when the plan was written in 2006 and 2007, it was assumed that the Program would not undergo any major changes that could change the evaluation results. Thus, the retrospective evaluation was designed to estimate national energy savings and to constitute most of the process assessment for the immediately past Program Year (PY2008) while simultaneously implementing several research tasks during the Program Year in which the evaluation was to take place (i.e., PY 2009). When the retrospective evaluation began, it was decided to look back to pre-ARRA Program Years 2007 and 2008 though the ARRA period had already begun.


Thus, these five research tasks that are being funded by the retrospective evaluation are actually assessing weatherization activities that took place during the ARRA period: analysis of sub-metered data for homes heated with bulk fuels; a major indoor air-quality study; case studies of high-performing agencies and exemplary client-education programs; and the administration of two major surveys, S4 (Occupant Survey) and S5 (Weatherization Staff Survey).


Table 1 also indicates that three major tasks undertaken by the retrospective evaluation will not be duplicated by the WAP-ARRA period evaluation: Bulk Fuels Study, Indoor Air Quality Study, and Field Process Study. Additionally, the retrospective evaluation will fund case studies of high-performing agencies and exemplary client-education programs, but the WAP-ARRA evaluation will not. Conversely, the WAP-ARRA period evaluation will fund case studies of underperforming weatherized units and one U.S. territory that received new, substantial ARRA funding (e.g., Puerto Rico). In summary, Table 1 indicates that twelve areas of major tasking are similar between the two evaluations, three major tasks being conducted by the retrospective evaluation will not be reprised by the ARRA period evaluation, and six new tasking areas are being added to the WAP ARRA period evaluation.


Table 1. Comparison of WAP Retrospective and WAP-ARRA Period Evaluations


Retrospective Evaluation on Weatherization Assistance Program

Retrospective Evaluation Research Taking Place during (Overlapping with) ARRA Period


WAP-ARRA Period Evaluation Research

Analysis of Billing Histories: Homes heated with electricity and natural gas

Program Years 2007 and 2008


Program Years 2009, 2010, and 2011

Analysis of Submetered Data: Homes heated with propane and fuel oil


Winter 2010-2011 and Winter 2011-2012


Analysis of Persistence of Energy Savings in Weatherized Homes



Program Years in the 1990s

Cost-effectiveness Analyses

Program Year 2008


Program Year 2010

Non-Energy Impacts

Program Year 2008


Program Year 2010

Social Network Study



Program Year 2011

S1: All States Program Information Survey

Program Year 2008


Program Year 2010

S2: All Agencies Program Information Survey

Program Year 2008


Program Year 2010

S3: Subset of Agencies Detailed Program Information Survey

Program Year 2008


Program Year 2010

S4: Occupant Survey


CY’s 2011 and 2012

CY 2012

S5: Weatherization Staff Survey


CY 2011

CY 2012

Program Characterization

Program Year 2008


Program Year 2010

Case Studies


Program Year 2008 – Six High Performing Agencies and Six Exemplary Client Ed Programs

Program Year 2010 – One U.S. Territory

Weatherization Deferral Study



Program Year 2010

Under-Performers Study



Program Years 2007-2009

DF2/3: Housing and Building Information Data Forms

Program Year 2008


Program Year 2010

DF4: Utility Information from Agencies Data Form

Program Years 2007 and 2008


Program Years 2009, 2010, 2011

DF5: Utility Billing History Data Forms

Program Years 2007 and 2008


Program Years 2009, 2010, 2011

Indoor Air Quality


Winter 2010-2011, Summer 2011, Winter 2011-2012


Field Process Study


CY 2011


Special Studies



WIPP and SERC Program Years 2010-2011;

Encouragement Design – Program Year 2011

DF12: Weatherization Deferral Data Form



Program Year 2010


3. WAP ARRA PERIOD EVALUATION SURVEYS AND DATAFORMS


Table 2 lists the surveys and dataforms that will be administered as part of the WAP ARRA period evaluation. Overall, the ARRA period evaluation will utilize surveys and dataforms developed under the retrospective evaluation that have been reviewed and approved by OMB. Specifically, five surveys and six dataforms will be re-used. Table 2 indicates the revisions, if any, made to the surveys and dataforms for their use in the ARRA period evaluation. Revisions to the dataforms are minor and reflect a shuffling of questions between DF2/3 and DF4 to make it easier for respondents to fill them out. A significant number of questions were added to S1 and S2 to capture ARRA period specific issues. One new survey has been added, S8, to collect information from DOE Weatherization Training Centers, respectively, on their post-ARRA issues, challenges and strategies that will be useful management information for DOE/OWIP. One new dataform has been added, DF12, which will identify households that had their weatherization service deferred. The evaluation is assessing how often and through what processes deferred homes do or do not eventually received weatherization services. It should be noted that DF1, DF10, DF6, DF7 and DF8 have been dropped.



Table 2. WAP ARRA Period Evaluation Surveys and Dataforms

Identifier

Name

Changes from WAP Retrospective Version

S1

All States Program Information Survey

Questions pertaining to ARRA period added; ~ 10% of original questions deleted

S2

All Agencies Program Information Survey

Questions pertaining to ARRA period added; ~ 10% of original questions deleted

S3

Subset of Agencies Detailed Program Information Survey

Several questions moved to S2

S4

Occupant Survey

Client satisfaction questions pertaining to WIPP and SERC Programs added; Multi-family section added

S5

Weatherization Staff Survey

Questions designed to follow-up from WAP Retrospective were added

S8

DOE Weatherization Training Centers Post-ARRA Survey

New

DF2

Housing Unit Information Data Form

Minor revisions

DF3

Building Information Form

Minor revisions

DF4

Electric & Natural Gas Bills Information from Agencies

Minor revisions

DF5

Electric & Natural Gas Bills from Utilities

No Change

DF9

Contact Information from Agencies for Occupant Survey Data Form

Minor revisions

DF11

Contact Information from Agencies for Weatherization Staff Survey Data Form

Minor revisions

DF12

Weatherization Deferral Data Form

New


All the surveys and dataforms will be administered in the same way as they were in the retrospective evaluation. Specifically, they will all be web-based, except S4 Occupant Survey, which will be administered by phone using a CATI system and S8, which will also be conducted by phone using open-ended questions. The same packages of introductory letters and non-response follow-up protocols will be followed.


It is anticipated that response burdens upon respondents will be less than for the retrospective evaluation for several reasons. First, states, agencies and utilities will benefit from having completed these tasks once before for the retrospective evaluation. Prior experiences with utilities especially suggest that much time is saved from routinized response to requests for billing histories. Second, the ORNL team has learned how to extract much information from state weatherization databases that can pre-populate surveys and dataforms and in some cases eliminate the need for the team to collect billing histories on its own. Third, a shortened version of DF2/3 will be given to large agencies in order to cut down on their burdens in completing these forms.


4.0 Timelines


Table 3 presents timeslines for when survey and dataforms for the WAP ARRA period evaluation will be administered, given an anticipated approval from OMB to begin administration by the second quarter of CY2012. It should be noted that all of the surveys and data forms will only be administered once, all in CY2012 and CY2013.


Table 3. Survey and Dataform Administration Timelines

Instrument/Administration

CY2012

CY2013


Q3

Q4

Q1

Q2

Q3

Q4

S1 – All States Prog. Info.


X





S2 – All Agencies Prog. Info.


X





S3 – Subset of Agencies Prog. Info.


X





S4 – Occupant Survey: WIPP

X


X




S4 – Occupant Survey: SERC



X




S5 – WX Staff Survey: Follow-up



X




S5 – WX Staff Survey: WIPP




X



S8 – WX Training Centers Post-ARRA

X






DF2/3 – Housing/Building Info.

X

X

X




DF4 – Billings Info. From Agencies

X

X

X




DF5 – Billings Data from Utilities


X

X

X



DF9 – Occupant Contact Info.



X




DF11 – WX Staff Contact Info.




X



DF12 – WX Deferral

X







1 Both reports can be found at http://weatherization.ornl.gov

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorBruce Edward Tonn
File Modified0000-00-00
File Created2021-02-01

© 2024 OMB.report | Privacy Policy