State Energy Program Evaluation

OMB Control No. XXXXXXXX

ID-11A-NR: Retrofits (Program Managers) Non-Residential

OMB Control No. XXXXXXXX

This interview guide is designed to provide direction to the interviewer to ensure that all relevant topics are explored to the extent possible and appropriate with the respondent. Note that our interviews are meant to be somewhat informal and open ended – not all topics will be covered in all interviews and we expect that some interviews will lead to the exploration of topics not included in this guide.

Information in [BRACKETS] will be customized to reflect the unique PA program [PROGRAM] and year of the PA offering [PROGRAM YEAR].

BACKGROUND INFORMATION (to be filled in prior to interview):

Program Administrator Name:
Year:
BPAC Area
2008 Budget:
2008 Market Title Sampled
2009-2010 ARRA budget
2009-2010 ARRA Market Title Sampled
Known programmatic activities prior to interview:
Description of target markets:
Understood structure of SEP/ARRA funded activities from informal discussions with SEP representative from first round discussions (from database)
Contact Name:
Contact Company:
Contact Phone:
Contact Disposition:

PART 1. INTRODUCTION

My name is [INTERVIEWER NAME] from [INTERVIEWER ORGANIZATION]. The US Department of Energy’s State Energy Program has hired us to talk with organizations involved with the management of your Non-Residential-retrofit programs that received SEP/ARRA funding in [PROGRAM YEAR]. This interview is being conducted as part of an evaluation of the State Energy Program being conducted by Oak Ridge National Laboratory on behalf of the U. S. Department of Energy.

The U.S. Department of Energy (DOE) would like to inform each individual that the information requested here is being solicited under the statutory authority of Title III of the Energy Policy and Conservation Act of 1975, as amended, which authorizes DOE to administer the State Energy Program (SEP). This information is being sought as part of a national evaluation of SEP, the purpose of which is to reliably quantify Program accomplishments and help inform decisions on future operations. The sole use of the information collected will be for an analysis of national-level Program impacts. Disclosure of this information is voluntary and there will be no adverse effects associated with not providing all or any part of the requested information.

PART 2. IDENTIFY THE APPROPRIATE RESPONDENT(S)

Our understanding is that you are the program manager for Non-Residential retrofit programs that received SEP/ARRA funding in [YEAR]? Is this correct:

Yes – Correct person and year for SEP/ARRA funded Non-Residential retrofit programs	[Continue]
Yes – Correct person, wrong year for SEP/ARRA funded Non-Residential retrofit programs	[Correct year and Continue]
No – Incorrect person for SEP/ARRA funded Non-Residential retrofit programs	[Ask for correct person for Non-Residential programs]
No – There were no SEP/ARRA funded Non-Residential retrofit programs	[Thank you and terminate]

Your participation in this survey is voluntary, and your answers are confidential. The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. The length of the interview varies from person to person, but most interviews last about 50 minutes. For quality control purposes, this call will be recorded and monitored. If you have any questions about this study, you can contact [ORNL CONTACT INFORMATION].

I would like to ask you some questions about the Non-Residential retrofit activities that received SEP/ARRA funding in [PROGRAM YEAR].

1. Were you involved in the design and/or management of retrofit programs that received SEP/ARRA funding in [PROGRAM YEAR]?

1 Yes

2 No [IF NOT, COLLECT CONTACT NAME AND TELEPHONE NUMBER. IF NO ONE AT ORGANIZATION IS KNOWLEDGEABLE ABOUT THE PROGRAM, STATE DETAILED REASON]

Yes	[GO TO Q3]
No	[ASK Q2]

2. Who else at your organization is most knowledgeable about the overall allocation of funding to Non-Residential retrofit programs that received SEP/ARRA funding?

Name/contact info/role: ________________________________________

Name/contact info/role: ________________________________________

Name/contact info/role: ________________________________________

3. [IF OTHERS MENTIONED IN Q2] Among you and [INSERT QUESTION 2 RESPONSE], who is most familiar with the activities conducted by your organization for [PROGRAM] in [PROGRAM YEAR]?

[IF NOT RESPONDENT, THANK AND TERMINATE AND CONTACT MOST KNOWLEGEABLE PERSON]

4. What were your responsibilities in regard to retrofit programs that received SEP/ARRA funding in [PROGRAM YEAR]?

5. In what year did you first become involved with the retrofit programs that received SEP ARRA funding?

6. Are you still involved with managing this program? If not, when did your involvement end?

7. How long have you been involved in administering programs funded by SEP?

8. Based on information provided by DOE, you received $__________ in SEP/ARRA funding for Non-Residential retrofit programs. Our understanding is that these funds may have been applied to programmatic activities supporting non-residential retrofits through either direct payment to targeted retrofit projects or through loans, grant or incentive based retrofit programs. Is this correct?

Listed Amount	Corrected Amount
[AMOUNT ]_______	[AMOUNT ]
	[REMAINING AMOUNT]

[IF LESS FUNDS WERE APPLIED TO NON-RESIDENTIAL RETROFITS THAN INDICATED IN THE TRACKING DATA ASK 8a-8c]

8a. To which of the following other types of programmatic activities was the remaining SEP/ARRA funding applied?

• Residential Retrofit Programs – Provides financial incentives for building retrofit and equipment replacement projects in residential buildings. Incentives allocated according to an open application process for eligible customer groups. Financial incentives are the principal program offering, but may be combined with others such as audits.

• Renewable Energy Projects – Provides financial incentives and/or technical assistance to support the development of renewable energy facilities including: solar, wind, biomass, small hydro. Projects directly result in increased generation of energy through the adoption of new solar, wind, biogas, biomass, or other renewable sources. Facilities may be customer sited or utility-oriented. Specific projects or customers may be identified in the SEP funding application or the program may be open to applications from eligible customers. Includes renewable projects installed on residential buildings.

• Renewable Energy Manufacturing – PAs develop or expand existing manufacturing capacity for renewable energy equipment or components. At least some portion of the output of the new or expanded capacity is intended for domestic installation.

• Alternative Fuels, Ride Share and Traffic Optimization – Provides training, financial support, technical assistance, marketing assistance, and/or administrative assistance to facilitate the development and operation of car and van pools. Supports capital improvements to support substitution of renewable fuels or electricity for conventional transportation fuels. Supports improvements to fleet vehicle efficiency and operations.

• Clean Energy Policy support – Develops and obtains legislative, executive, or regulatory approval for policies to facilitate the completion of renewable energy facilities. Examples might include statewide zoning laws, feed-in tariffs, favorable back-up tariffs, and renewable portfolio standards.

• Building Code Development and Support – Provides technical and administrative support for the development of more energy-efficient state and federal equipment standards. Provides marketing support for products that meet the higher energy efficiency standards. Provides training to vendors in marketing and installation of products that meet the higher energy efficiency standards.

• Policy and Market Studies; Legislative Support – Develops and obtains legislative, executive, or regulatory approval for policies to facilitate the completion of renewable energy facilities. Examples might include statewide zoning laws, feed-in tariffs, favorable back-up tariffs, renewable portfolio standards.

• Non-retrofit Targeted Training and/or Certification (Participants are traceable) – Training for facility managers, trades contractors, and engineering/design professionals. Topics for EE, renewables, transportation and other technical topics. i.e. roundtable discussions etc. Includes certification programs and other pre-requisite type training & education programs.

• Non-retrofit Generalized Workshops and Demonstrations – Marketing and outreach support to raise awareness, provide general information, encourage behavior change, etc. Not targeted at a specific site or project, but may be targeted at specific segments or types of projects. Recipients of the education are not traceable.

• Non-retrofit Technical Assistance to Building Owners – Provides technical assistance other than audits for building retrofit or equipment replacement projects: e.g. technical studies for specific improvements, building modeling, project financial analysis, support in negotiating with contractors. Open to commercial, industrial, and agricultural facility owners or specified subgroups thereof. May be combined with financial incentives.

• Other [DEFINE] _______________________________________________________________

8b. [IF 8a = “YES” FOR ANY ACTIVCITY TYPE, RECORD AMOUNT]

[IF 8a = “YES” FOR ANY ACTIVCITY TYPE ask 8c]

8c. Can we contact the appropriate program manager for this activity? [RECORD NAME]

RESPONSE MATRIX A: Questions 8a – 8c

Activity	8a. Funds applied	8b. Funding	8c. Permission
Residential retrofit programs	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Non-residential renewable energy projects	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Residential renewable energy projects	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Renewable energy manufacturing	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Alternative Fuels, Ride Share and Traffic Optimization	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Building Code Development and Support	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Policy and Market Studies; Legislative Support	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Targeted Training and/or Certification (Participants are traceable)	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Generalized Workshops and Demonstrations	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Technical Assistance to Building Owners	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Other [DEFINE]	[YES/NO]	[AMOUNT ]	[RECORD NAME]

[PLEASE READ]

For the remainder of this interview, I’d like to focus on just the non-residential retrofit programs that received SEP/ARRA.

9. Were the SEP/ARRA funds your organization received used to fund Non-Residential retrofit programs in [YEAR] distributed across multiple programs?

[IF YES ASK Q10 – 14; ELSE GO TO Q15]

10. How many separate programs? ________________

FOR EACH PROGRAM MENTIONED ASK 12-15

11. What are the names of these programs?

12. What percent of the total SEP/ARRA funds used to for Non-Residential retrofit programs was used for each of these programs?

13. Are you or another program manager in charge of any of these programs? [IF THERE ARE OTHER PROGRMAM MANGERS, ASK Q15]

14. Can we contact the program manager regarding [PROGRAM NAME]?

[IF YES, RECORD THEIR NAME AND CONTACT INFORMAITON]

Response Matrix C: FOR QUESTIONS 12-15, USE REPSONSE GRID BELOW

12. Program Name	13. Percent	14. Additional program manager	15. Permission to contact
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]

PART 3: PROGRAM DESIGN (LOGIC MODEL)

[ASK SECTIONS 3 – 6 FOR EACH OF THE PROGRAMS LISTED IN RESPONSE MATRIX A. IF NECESSARY, SCHEDULE A CALLBACK WITH THE APPROPRIATE PROGRAM MANAGER]

[PLEASE READ] In this next set of questions, I’d like to understand the rationale behind the program design.

[THESE QUESTIONS ARE DESIGNED TO FLESH OUT PROGRAM DETAILS TO DEVELOP THE PROGRAM LOGIC MODEL. THE LOGIC MODEL ARTICULATES WHAT THE PROGRAM IS TRYING TO ACHIEVE (AND DID ACHIEVE), THROUGH WHAT INTERVENTIONS, AND WITH RESPECT TO WHICH MARKET ACTORS. THIS INFORMATION WILL BE USED TO DEVELOP A MAP OF THE PROGRAM’S DOMAIN FOR ATTRIBUTION PURPOSES.]

15. Has your organization developed a formal logic model for [PROGRAM NAME]?

[IF 16 = YES, then ASK 16a]

16a. Can you please share it with us?

16. What were the ultimate objectives of [PROGRAM NAME]?

________________________________________________________________________

________________________________________________________________________

17a. In what year did the program begin?

________________________________________________________________________

17. Please describe how the program is operating now in regard to its activities, the resources it uses, and the outcomes it is achieving.

17A. Let’s start with ACTIVITIES. PROBE:

MARKETING AND OUTREACH TO PARTICIPANTS:

RECRUITMENT OF PARTICIPANTS:

RECRUITMENT OF TRADE ALLIES AND OTHER DELIVERY PARTNERS

DELIVERY OF INFORMATION AND TECHNICAL SERVICES

PROVISION OF FINANCIAL SUPPORT: INCENTIVES, LOANS, LOAN GUARANTEES

INSPECTION, QUALITY CONTROL OF INSTALLED PROJECTS

17B. INPUTS: What resources are being used to deliver the program? PROBE:

STATE ENERGY OFFICE STAFF: NUMBER, TYPE, PERCENT OF TIME

OTHER GOVERNMENT STAFF: NUMBER, TYPE, PERCENT OF TIME

STAFF OF OTHER AGENCIES OR ORGANIZATIONS: NUMBER, TYPE, PERCENT OF TIME

CONTRACTED SERVICES: PROBE NAME OF PROVIDER AND TYPE OF SERVICE, APPROXIMATE NUMBER OF CONTRACTOR STAFF OR SIZE OF CONTRACT

TECHNICAL SERVICES

FINANCIAL SERVICES

MARKETING SERVICES

ADMINISTRATIVE SERVICES

17C. OUTSIDE FUNDING: Are organizations other than the U. S. Department of Energy and the state government contributing funding to this program? IF YES, PROBE:

NAME OF THE ORGANIZATION(S)

AMOUNT OF OUTSIDE FUNDING PROVIDED IN THE PERIOD UNDER EVALUATION (2008 OR ARRA PERIOD)

PERCENTAGE OF TOTAL OVERALL FUNDING FOR THE PA UNDER EVALUATION IN 2007, 2008, ARRA PERIOD, AS RELEVANT REPRESENTED BY OUTSIDE FUNDING.

USES OF THE FUNDING PROVIDED

17D. RESULTS/OUTPUTS: Could you summarize the results your program has achieved so far? PROBE: Are organizations other than the U. S. Department of Energy and the state government contributing funding to this program? IF YES, PROBE:

NUMBER OF PARTICIPANTS ENROLLED

NUMBER OF UNITS OF SERVICE DELIVERED (AUDITS, WORKSHOP ATTENDANCE, ETC)

NUMBER OF INSTALLATION PROJECTS SUPPORTED

ESTIMATED SAVINGS FROM INSTALLATIONS SUPPORTED

17E. RESULTS VERSUS GOALS: Were you able to (Do you anticipate being able to) achieve the level of results you included in your program plans and applications to the U. S. Department of Energy?

IF NO: What circumstances prevented you (are preventing you) from achieving those goals?

18. [IF PROGRAM EXISTED PRIOR TO 2008 ASK] Did you make any important changes to program operations or the resources available to it between 2007 and 2008? IF PROGRAM EXISTED PRIOR TO ARRA AND RECEIVED ARRA FUNDING ALSO ASK] Did you make any important changes to program operations or the resources available to it between 2008 and the period funded primarily by ARRA?

19a. Why did you make those changes?

19b. Did you achieve the results you were seeking through those changes? PROBE SPECIFIC RESULTS.

19c. FOR ARRA PERIOD PROGRAMS ASK. Would your organization have been able to make these changes without the support provided by the ARRA round of funding?

YES/NO

PROBE REASONS FOR THIS ASSESSMENT.

PART 4. RESOURCE CONTRIBUTIONS FROM OTHER PROGRAMS

[ASK PART 4 IF OTHER ORGANIZATIONS SUCH AS UTILITIES OR OTHER GOVERNMENT COOPERATED IN THE DELIVERY OF THE PA THROUGH CONTRIBUTIONS OF STAFF TIME, FUNDING, OR OTHER SERVICES.]

LEAD-IN: Earlier we spoke about contributions that other organizations made to the non-residential retrofit programmatic activity. I’d like to ask a few more questions about that.

19. First, prior to 2008, had you cooperated with this (these) organization(s) in delivering energy programs to increase energy efficiency programs?

IF YES, ASK

1. Please describe how you worked together with the other organization(s) and what specific activities they undertook.

2. What kinds of resources did the other organizations provide?

1. Did your organization take the initiative to arrange for cooperative activities related to the program? Did the other organizations take the initiative? Or did the cooperation arise through some other mechanism?

21. Did representatives of the two (or more) organizations meet to jointly plan how the organizations would cooperate in the development of the program?

22. Do representatives of the organization meet on a regular basis to review program accomplishments and plan future activities?

IF YES: How often does that occur?

23. Is there regular interaction among the organizations in delivering the program services?

IF YES: Could you describe that interaction for me?

24. What do you believe are the other organizations’ main motivations for contributing to the delivery of the program?
    
    
    
    

25. If the opportunity to offer joint programming with your agency had not been available, do you believe the other organizations would offer programs to promote energy efficiency in non-residential facilities?

YES/NO

26a. Why do you say that?

26. If the other organizations had not cooperated in delivering the program as they did, would your organization have changed the type of services it provides to promote energy efficiency in non-residential facilities?
    
    YES/NO

IF YES, PROBE:

27a. How would your organization have changed the roster of services provided?

27b. In the absence of cooperation from other organizations, would the level of resources your organization dedicated to these services been less, the same, or greater?

27c. Why do you say that?

PART 5: SEP INFLUENCE AND ROLE IN THE MARKETPLACE

27. Do you believe that this program has had an impact on the long-term capabilities of non-residential facility owners in the state to carry out energy efficiency retrofit projects?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?

1. Do you believe that this program has had an impact on the long-term capabilities of equipment vendors, designers, contractors, and other firms on the “supply side” of the market to sell and install energy-efficient retrofit projects in non-residential facilities?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?
   
   
   

1. Did your organization add any positions to carry out or administer the program(s) we have been discussing? Please include positions that may be split between a number of different programmatic and administrative activities.
   
   IF YES:

   1. How many positions did your organization add?

   2. How many FTEs were added in each of the following job classifications: administrative, technical services, professional, management?

   3. How many of these FTEs will be retained after 2008 or the ARRA period, depending on which is relevant?

   4. IF c > 0: What sources of funding do you plan to use to sustain this (these) position(s)?

30. In the course of carrying out this program, has your organization developed any new capabilities or resources that will support the delivery of energy efficiency services to non-residential facilities in the future. PROBE:

    1. Contacts and relationships with market actors

    2. Tools such as building analysis software

    3. Project financing facilities such as loan funds, loan loss reserve programs, etc.

    4. Technical education materials and curricula

PART 6: DATA AVAILABILITY

[ONLY ASK THIS SECTION IF DATA ARE NOT AVIALABLE ALREADY]

[ASK SECTION 4 FOR EACH OF THE PROGRAMS LISTED IN REPSONSE MATRIX A. IF NECESSARY, SCHEDULE A CALLBACK WITH THE APPROPRIATE PROGRAM MANAGER]

NOTE: QUESTIONS 31-34 ON DATA AVAILABILITY WILL BE REVIEWED WITH PROGRAM SPONSORS PRIOR TO FINAL SELECTION OF THE PROGRAM INTO THE SAMPLE. WE WILL ALSO ASK TO RECEIVE A COPY OF THE DATABASES OR FILE FORMATS SO THAT WE CAN MAKE A JUDGMENT REGARDING THE ADEQUACY OF THE DATABASE TO SUPPORT EVALUATION ANALYSIS. THESE QUESTIONS WILL BE FOR VALIDATION OF OUR REVIEW.

31. Please describe the types of data that you tracked in [PROGRAM YEAR] on program participants. Do you have lists of:

1. Key contacts from the implementation organization (including contact information)

2. Program participants, that is, facility owners/renters, non-residential property managers who received support with the project, workshop attendees or training recipients, recipients of technical assistance, with contact information

3. Measures installed by participant, including the type and quantity of measures installed with program support, energy savings, incentive levels, other services received by participants, workshop or training description. [PROBE TO CLARIFY DATA THAT THE PM HAS ACCESS TO (EITHER IN-HOUSE OR AT A SEP FUNDED/PARTNERED ORGANIZATION) VERSUS DATA ANY EXTERNAL LEVERAGED ORGANIZATION MIGHT BE KEEPING.]

4. Participating trade allies (including contact information)

(IF DATA IS AVAILABLE BUT RESPONDENT IS NOT KNOWLEDGEABLE/DOES NOT HAVE ACCESS TO IT, RECORD NAME OF CONTACT WHO CAN ANSWER FOLLOW-UP QUESTIONS ABOUT DATA AVAILABILITY. PROBE FOR WHETHER ANYONE MAINTIANS SUCH DATA – POTENTIALLY EXTERNAL TO THE PA). IF THERE IS NO ONE CAN PROVIDE SUCH DATA THEN THANK AND TERMINATE).

32. Have there been any evaluations of this program? Are data from these evaluations available?

33. (FOR EACH TYPE OF DATA AVAILABLE) Is this data in electronic format? (If yes, discuss steps needed to secure permission to access the data via secure file transfer. If no, determine how data records are maintained and how they could be accessed.)

Are you aware of other data available that would be useful for our evaluation? If so could you describe it for me? Is it possible to access this information? Probe for contacts. If making copies is an issue, ask whether we can make copies. if still reluctant, ask whether they have a summary of retrofitted properties with minimal infoormaiton so we can select a sample and copy only those records.]

Thank you for your time and insights

14

OMB Control No. XXXXXXXX