Supporting Statement (Part B)
Collection of Information Employing Statistical Methods
U.S. Department of Housing and Urban Development
Office of Policy Development and Research
Information Collection for the
Energy Evaluation of
Public Housing Capital Fund (PHCF),
Category 4, Option 2 Grantees
Note: This submission is presented as an application for a new collection of information.
As part of the American Recovery and Reinvestment Act (ARRA), the Department of Housing and Urban Development (HUD) was allocated funds to invest in energy efficiency and green building programs. This legislation included a $4 billion appropriation to the Public Housing Capital Fund (PHCF) for the modernization and renovation of the nation’s public housing stock, and a $250 million appropriation to establish the Green Retrofit Program for Multi-Family Housing (GRP), which provides loans and grants for green building retrofits of privately-owned rental housing receiving project based rental assistance.
Most of the Capital Fund Recovery funding was appropriated to Public Housing Agencies (PHAs) by formula. However, a small part of the funding was awarded to grantees through a competitive process. In September 2009, the Office of Public and Indian Housing (PIH) awarded $995 million to Public Housing Agencies (PHAs) through 396 Capital Fund Recovery Competitive grants. The grant program types include the following:
Category 1: Improvements Addressing the Needs
of the Elderly and/or Persons with
Disabilities.
Category 2: Public Housing Transformation.
Category 3: Gap Financing for Projects that are Stalled Due to Financing Issues.
Category 4: Creation of Energy Efficient, Green Communities:
Option 1, Substantial Rehabilitation or New Construction.
Option 2, Moderate Rehabilitation.
The U.S. Department of Housing and Urban Development (HUD) Office of Policy Development and Research developed an evaluation to assess the effectiveness of the distributed funds. The Green and Energy Retrofit Assessment (GERA) study spans four years and seeks to evaluate the short- and long-term performance of the investments in energy conservation measures. Specifically, the overall purpose of this study is to estimate the energy savings of energy retrofits funded through ARRA. The GERA effort encompasses several specific evaluation objectives (see Part A, Justification Statement).
The main objective of the Green and Energy Retrofit Assessment (GERA) is to understand the savings in utility usage that occurred within the set of public housing retrofit programs. The survey for which this PRA package is being submitted seeks to gather the pre- and post-retrofit energy utility data maintained by Category 4, Option 2 grantees. These grantees were provided funding to conduct their choice of proposed energy conservation and other “greening” retrofit at properties operated by the PHA, based on the particular needs of those properties. Grantees were required to conduct an energy audit of their property prior to applying for a grant, and then, within the grant application, each PHA proposed to utilize one or more of a set of 37 potential energy conservation measures (ECMs) that could be performed with grant funding.
The data collected under this data collection request will be used for the evaluation of investments made in Capital Fund Competitive Grants the funded energy-efficient retrofit of existing housing units (Category 4, Option 2), one of the objectives of this study. As a condition of grant funding per the NOFA, PHAs are required to maintain records of one year of pre-retrofit, and one year of post-retrofit utility costs for the entire AMP under study. PHA personnel will be able to access this information through utility bills kept at properties, or in the PHA accounting system. The survey is expected to be administered in July-August 2014.
The survey will be in an excel file format, and will be sent to all public housing authorities (PHAs) who received PIH-competitive Category 4, Option 2 grants. The PHAs will be asked to provide monthly data from one year of pre-retrofit and one year post-retrofit energy utility bills for a particular Asset Management Project (AMP)1 that the PHA has identified as receiving the energy efficiency retrofit funded by the grant. The timeframe for the pre-retrofit utility bills will fall within Calendar Year (CY) 2009 and CY 2013, depending on the time of retrofit completion for each AMP.
The unit of observation for the analysis of changes in utility usage pre- and post-retrofit will be an AMP. The AMP has been chosen as the unit of analysis because the AMP is the major unit of asset management for a PHA, and many PHA data are collected at the AMP level. An AMP is often one property, but can include multiple properties that are managed together. A particular grant may be used to finance retrofit at one AMP, or at multiple AMPs. In many cases, an AMP that received a competitive grant may have also received some retrofit financed by a formula grant as well. PHAs are required to report on the retrofit conducted as part of each grant. For the purposes of this study, it may be difficult to isolate the impact of the competitive grant in particular. For example, a property may receive new insulation paid for with formula funding, and a new HVAC system paid for with competitive funds. It would be difficult to empirically identify the portion of the energy savings resulting from each individual ECM, and therefore from each grant. The study team will attempt to “discount” the measured impact of the competitive grants based on the assumed energy savings of the formula grants also utilized for retrofit at the property; however, such an exercise has significant reliability limitations.
As part of the GERA effort to achieve its objectives, this package requests clearance for a one-time data collection effort to be conducted on the universe of competitive Category 4, Option 2 recipients (229 AMPs distributed across 127 PHAs)
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
This survey will be conducted on the universe of Public Housing Agencies (PHAs) that implemented Capital Fund Recovery Competitive Grants. As a result, no sampling method is necessary.
Potential respondent universe: The universe of respondents is all PHAs that received funding from Category 4-Option 2. Each PHA will receive a spreadsheet with a page for each AMP that they reported receiving grant-funded retrofit. Overall, the universe size is 229 AMPs.
Respondents |
Sample/Universe Size |
Sampling Method |
AMPs/PHAs administrators |
229 AMPs |
None |
Expected response rate: A minimum response rate of 80% is expected. As a condition of the funding received, Category 4, Option 2 grantees are required to maintain monthly or quarterly pre-retrofit and post-retrofit utility records for up to three years and with a minimum of one year. Grantees are also required to maintain a list of the Energy Conservation measures (ECMs) implemented. As a result, all grantees are expected to have the necessary survey data readily available, minimizing the cost and maximizing the likelihood of response.
The response rate will be enhanced via the utilization of a pre-survey notification sent from HUD PIH explaining the importance of this data collection effort. In addition, there will be follow up e-mails and phone calls to PHAs who are late with their responses.
Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection,
Because the survey will go to the entire population of PHAs who received Category 4 Option 2 competitive grants, there is no stratification or sampling for this data collection effort.
Estimation procedure
The goal is to isolate the impact on utility usage of the retrofits conducted by competitive grantees. Several different factors can affect utility usage at a property including weather changes, occupancy changes, major property configuration changes, or other retrofits that occurred outside the program. Therefore, any analysis of usage savings must control for these factors. To do this, “avoided costs” are measured. Measuring avoided costs involves determining the usage savings in the post-retrofit period, in comparison to an estimate of what costs or usage would have been in the post-retrofit period if the retrofits had not taken place. The study team will measure avoided energy usage in terms of electricity usage and natural gas usage.2
The most widely accepted standard for measuring avoided costs is the method outlined by American Society of Heating, Refrigerating and Air-Conditioning Engineers (ASHRAE)’s Guideline 14-20023, as well as by the International Performance Measurement and Verification Protocol (IPMVP)4. The IPMVP standard is the basis for performance-based contracts under the Federal Energy Management Program (FEMP). There are four measurement and verification protocols for measuring avoided costs recognized in the IPMVP. HUD proposes using one of these protocols, the “Whole Facility” Protocol (Protocol C), which is designed to measure whole-facility changes in utility usage between a pre-retrofit period and a post-retrofit period.
The IPMVP Protocol C involves development of a series of regression models that model different types of fuel usage at the property. This protocol is designed to measure a whole facility’s changes in utility costs between a pre-retrofit period and a post-retrofit period. The data required to use this protocol involves utility, vacancy and weather from the twelve-month pre-retrofit period as well twelve months of continuous utility, vacancy and weather that pertain to the post-retrofit utility monitoring period.
The IPMVP methodology has three basic steps:
Determine the pre-retrofit usage profile, using the pre-retrofit period monthly data, based on weather and occupancy in the time period being studied.
Estimate what usage would have been in the post-retrofit period had the retrofits not taken place, using the pre-retrofit usage profile and controlling for the factors in Step 1 above.
Compare the estimated energy usage based on pre-retrofit conditions to actual energy usage post-retrofit.
To account for differences in weather patterns between the pre- and post-retrofit periods, the usage for both properties has to be normalized. This normalization is done via a linear ordinary least squares (OLS) model where the dependent variable is electric or gas usage, and the independent variables are the total number of heating degree days (HDD) and/or cooling degree days (CDD) in the period under study.
HDD and CDD are environmental measurements designed to reflect the demand for energy needed to cool or heat a building, derived from measurements of outside air temperature. The cooling or heating requirements for a home at a specific location are considered to be directly proportional to the number of HDD at that location. HDD and CDD are defined relative to a base temperature—the outside temperature above which a building needs no heating or cooling. In order to weather-normalize our results, HUD will collect data on HDD and CDD by zip code available from a paid service, http://www.degreedays.net/, and from the National Oceanic and Atmospheric Administration’s (NOAA’s) National Climatic Data Center.5, 6 These data will be used to ensure that retrofit savings are not an artifact of cooler or warmer temperatures, but are in fact caused by the ARRA-funded retrofit.
The heating requirements for a building at a specific location are directly proportional to the number of heating degree days (HDD) at that location. A similar measurement, cooling degree days (CDD), reflects the amount of energy used to cool a home or business. Both are measured by the difference in the average temperature from a certain baseline temperature above/below which a building needs no heating/cooling.
The resulting equation can be used to predict the energy usage of the property at the post-retrofit condition in any occupancy and weather conditions. By comparing this usage to that in the pre-retrofit period, one can find the difference in usage attributable directly to retrofits, and not to differences in weather or occupancy.
Using this method, the study team can separately estimate the electricity and natural gas usage savings from a single property that were a direct result of green retrofit, each occurring over a slightly different period of time, based on when the year of pre-retrofit and post-retrofit energy usage occurred.
For each property, the study team will then have an estimated percentage change in energy usage one year after retrofit. If there is a 100% response rate to the survey, one can simply apply each AMP’s estimated percentage savings function to the most recent full calendar year of utility usage data from the HUD Utility Expense Level (UEL) CY2013 dataset. The aggregated usage savings estimates will provide an overall, annual portfolio-wide usage savings estimate for the entire program.
However, some level of non-response to the survey is expected. Because each grant comprises a very different scope of retrofit, it is not possible to extrapolate the average energy-savings results of the sample of grantees to the entire population.
An alternative is to utilize data from the HUD IMS-PIC, UEL, and RAMPS datasets. These datasets contain a rich set of covariates at the AMP level from a full census of grantees regarding the ECMs undertaken, key property characteristics and geographic conditions.
Therefore, to address potential non-respondents energy savings, the study team proposes using a regression-based approach to estimate a percentage usage savings function for each non-respondent AMP. This estimated percentage savings function can then be similarly applied to the HUD UEL CY2013 dataset to provide an estimate of electricity and natural gas saved at each AMP over one year. The dependent variable for each regression will be the expected percentage decrease in overall AMP-level energy usage.
With this purpose, separate regressions will be developed for:
Heating-related savings for natural-gas-heated properties
Heating-related savings for electricity-heated properties
Cooling-related savings (electricity)
Non-heating-related natural gas savings
Non-cooling/heating related electricity savings
The exact model specification will be determined upon empirical review of the data received. However, the following set of covariates is available in the survey data frame, and can be utilized for the study:
Percentage of units implementing the following ECMs with an expected impact on energy consumption;
Replacement of Inefficient Heating Plants
Replace Central AC
Replace Inefficient. DHW
Solar Thermal DHW
Boiler Temp Controls
Radiator Controls installed
Replacement Windows
Window AC Replaced
Units affected by installing Green Roofs
Refrigerators replaced
Constant Air Regulating Dampers installations
Energy Efficient Storm Windows
Years since last retrofit
Building type
Elevator Structure
Multifamily/Walkup Apts
Row or Townhouse
Semi Detached
Single Family/Detached
Climate zone
Average Unit size
Elderly/Family occupancy
Property size (in units)
Vacancy
Degree of accuracy needed for the purpose described in the justification,
The information requested from AMPs will be collected in the same format as grantees maintain it, and these records are directly received form the energy utility companies. Thus, no estimation error is expected from these data. Any measurement error that may exist will result directly from inherent meter inaccuracy levels, which is expected to be minimal.
However, modeled estimates of energy usage changes at the building level assigned to a particular ECM are subject to error from various sources, including differences in microclimate around a property that cannot be measured through publically available data, other changes in property condition that may not be recorded, and changes in behavior over time with changing residents. The IPMVP estimates that avoided cost estimates should generally be assumed to have no better than a +/-10% accuracy rate. No specific degree of accuracy is needed for the purposes stated in the justification; however, the team will utilize best practices per the IPMVP procedures noted above to ensure that estimates are as accurate as possible when estimating electricity and natural gas usage savings. Due to small sample size issues, the study team will not attempt to extrapolate total savings to properties utilizing fuel oil, propane, and solar photovoltaic cells.
Unusual problems requiring specialized sampling procedures, and
Not Applicable
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
None. This is a one-time data collection
Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
Response rates maximization and non-response issues: The survey will be preceded by an email from HUD PIH notifying respondents of the upcoming survey. A request to complete the Excel-based survey will be delivered in an email that will explain the importance of the project. The excel spreadsheet will be provided to respondents, making the survey easy to complete. Because it is an excel spreadsheet, respondents will be able to enter and exit it if they wish to complete it in more than one sitting. The spreadsheet will be pre-populated from the IMS-PIC database and the RAMPS database with available information on the AMPs receiving the retrofit, the ECMs utilized, and the characteristics of the buildings comprising the AMP.
After two weeks, all non-respondents will receive weekly follow-up emails reminding them of the survey. If they have not completed the excel survey within the 30 days period, they will receive a phone call and be asked to complete the survey over the phone.
As a condition of the funding received, Category 4, Option 2 grantees are required to maintain pre-retrofit and post-retrofit utility records for a minimum of one year, all grantees are expected to have the necessary survey data readily available. This minimizes grantees’ costs of and maximizes the likelihood of responding to the survey.
Adequacy of the information collected: Through combining Excel spreadsheet and telephone follow-up surveys, along with survey pre-notification and follow-up emails, a response rate of 80% or higher is expected.
Due to the regression-based method to be used for extrapolation of results to the population in case of nonresponse, non-random patterns of response in terms of types of grantee characteristics including ECMs implemented, building type, or climate zone will be controlled for in the extrapolation procedure. So, for example, if grantees that conducted a large amount of rehabilitation are more likely to respond to the survey, this will be controlled for through the regression that relates the change in energy usage to the major types of ECMs utilized.
However, the estimates of interest might be subject to nonresponse error if PHAs who have energy usage savings that were lower than initially forecasted are less likely to complete the survey. To examine whether this might be the case, the study team will examine respondents’ reported utility usage savings by comparing respondent energy savings performance to that of grantees in another grant program for which we have a full census of data.
The Green Retrofit Program (GRP) is a similar, ARRA-funded voluntary grant program for which HUD-assisted properties were eligible to participate. All 221 GRP recipients were required to provide the same utility data as are being requested in the survey. In addition, energy audits are available for both competitive grantees, and Green Retrofit Program (GRP) recipients. Energy audits provide an estimate of projected energy savings for each scope of retrofit proposed. By comparing the audit estimates with the actual usage savings, we can categorize grantees in each program according to their performance compared to the audit estimate of savings. To compare energy audit predicted savings with actual savings, the following procedures will be used:
Review provided bid materials/receipts to determine what retrofits were actually undertaken.
For each retrofit undertaken, review the energy audit to determine predicted savings.
Aggregate predicted savings to determine an expected range of energy usage savings predicted by the audit for the AMP.
Compare weather-normalized actual usage savings to the expected range of savings predicted by the audit for each utility type.
Categorize usage savings as “Comparable to Audit” if overall savings were at least 90% of the lower bound of the audit-predicted range. Categorize usage savings as “Less than Audit-Predicted” if usage savings were less than that, and “More than Audit-Predicted” if savings exceed audit-predicted savings by at least 10%.
If the nonrespondents in the survey to Capital Fund Recovery Competitive Grantees are randomly distributed in terms of expected ECM performance, then one would expect the distribution of AMPs reporting “Comparable to”, “Less than” and ‘More Than” Audit-predicted savings to be similar to that of the full census of GRP program grantees. If there is a significantly larger share of Capital Fund Recovery Competitive respondent grantees with “More Than Audit Predicted” savings than is found among GRP grantees, this may indicate a biased pattern of higher response rates from better-performing AMPs, which may require mitigation or consideration in the final extrapolation of results.
Another potential and complementary approach to address nonrespondents energy savings is to use data from PIH UEL Utility dataset. These data provide yearly aggregate energy consumption for all Capital Fund Recovery Competitive Grantees in pre- and post-retrofit periods. These data can provide a relation between the energy consumption pre- and post- retrofit between respondents and non-respondents and relative to the energy audit each grantee conducted as part of the NOFA application. This information, along with data on characteristics of each AMP can also indicate whether there is a bias, and the direction of this bias between respondents and nonrespondents in terms of expected ECMs performance. A potential issue of utilizing this approach is that, because the UEL data spans from July 1st to June 30th of each year, appropriate pre- and post- retrofit utility consumption periods may not be available for the analysis.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.
The survey spreadsheet is being reviewed and will be pilot tested by up to 9 PHA representatives. The initial review by these individuals will provide important insight into how questions will be interpreted by potential respondents and whether the information requested and the manner in which it is requested is clear and relevant. As a result of this review, some of the data fields and relevant information provided may be modified. The test respondents will be contacted by HUD and individual meetings will be scheduled to complete the spreadsheet and gather feedback on their comprehension of data fields and their ability to provide reliable answers. Additional changes based on this pre-test will be incorporated into the final survey spreadsheet instrument before it is submitted to the universe of AMPs.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
Summit Consulting, LLC
718 7th St NW, Suite 314, Washington, DC 20001:
Albert Lee Principal 202.407.8302
Amy Deora Senior Manager 202.407.8304
Claudia Gonzalez Martinez Manager 202.407.8324
LMI Consulting
Stuart Funk Program Manager 703-917-7277
Michael Canes Distinguished Fellow 703-917-7201
U.S. Department of Housing
and Urban Development (HUD) PIH
451 7th St SW,
Washington, DC 20410
Mike Early Engineer 202-402-2566
Robert Dalzell Senior Program Adviser 202-402-4216
Jeff Riddel Director 202-402-7378
Candace Simms Program Liaison Specialist 410-209-6589
Chad Ruppel Program Analyst 414-935-6735
See Attachment.
1An AMP is either one building, or a collection of buildings that is managed as a single unit by a PHA. For this study we are defining “properties” as the development components according to the pre-asset management reform definition that make up each new AMP. For this document and throughout the study we will use three terms to reflect three concepts. Development is defined as it was in the pre-asset management reform world. Asset Management Project (AMP) is the grouping of units and buildings that form a new “development.” In many cases old developments have been merged into a single AMP, and in some cases developments have been split into a number of AMPs. Property is the component of an old development that moved into a new AMP. For a single development that moved into a single new AMP the development, property and AMP are all the same. For an AMP that is comprised of a number of old developments, the properties in the new AMP are the old developments. In cases where old developments were split into a number of AMPs, the property is the portion of the old development that moved into the AMP. In this report, we use "property" and "development" interchangeably to refer to the entities used for sampling and described by analysis. (http://portal.hud.gov/hudportal/documents/huddoc?id=PH_Capital_Needs.pdf)
2 Approximately 2% of all PIH AMPs are heated with fuel oil. Less than 1% of all PIH AMPs record usage of propane. The study team does not plan to estimate savings for fuel oil or propane usage, the sample size will be very small (or potentially 0). We will, however, capture data on fuel oil and propane usage in the survey that will be used in the evaluation for descriptive purposes.
3 This standard can be downloaded here for a fee: http://resourcecenter.ashrae.org/store/ashrae/newstore.cgi?itemid=9012&view=item
4 The IPMVP Protocol can be downloaded here for free: http://www.evo-world.org/index.php?option=com_content&view=article&id=272&Itemid=279&lang=en
5 The data provided by http://www/degreedays.net are obtained from Weather Underground (www.wunderground.com), which in turn obtains its data from almost 2,000 Automated Surface Observation Systems (ASOS), over 16,000 Personal Weather Stations (PWSs), and over 26,000 weather stations managed by the National Oceanic and Atmospheric Administration (NOAA).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Michael J. Early |
| File Modified | 0000-00-00 |
| File Created | 2021-01-26 |