State Energy Program Evaluation

OMB Control No. XXXXXXXX

ID-12A-R: Retrofit (Program Delivery Contractors) Residential

OMB Control No. XXXXXXXX

This interview guide is designed to provide direction to the interviewer to ensure that all relevant topics are explored to the extent possible and appropriate with the respondent. Note that our interviews are meant to be somewhat informal and open ended – not all topics will be covered in all interviews and we expect that some interviews will lead to the exploration of topics not included in this guide.

Information in [BRACKETS] will be customized to reflect the unique PA program [PROGRAM] and year of the PA offering [PROGRAM YEAR].

BACKGROUND INFORMATION (to be filled in prior to interview):

Program Administrator Name:
Year:
BPAC Area
2008 Budget:
2008 Market Title Sampled
2009-2010 ARRA budget
2009-2010 ARRA Market Title Sampled
Known programmatic activities prior to interview:
Description of target markets:
Structure of SEP/ARRA funded activities from informal discussions with SEP representative from first round discussions (from database)
Contact Name:
Contact Company:
Contact Phone:
Contact Disposition:

PART 1. INTRODUCTION

My name is [INTERVIEWER NAME] from [INTERVIEWER ORGANIZATION]. The US Department of Energy’s State Energy Program has hired us to gather information on the results of efforts to promote the development retrofit and equipment replacement projects to reduce energy consumption in single family homes and other residential facilities that received funding from the State Energy Program and the American Recovery and Reinvestment Act. This interview is being conducted as part of an evaluation of the State Energy Program being conducted by Oak Ridge National Laboratory on behalf of the U. S. Department of Energy.

The U.S. Department of Energy (DOE) would like to inform each individual that the information requested here is being solicited under the statutory authority of Title III of the Energy Policy and Conservation Act of 1975, as amended, which authorizes DOE to administer the State Energy Program (SEP). This information is being sought as part of a national evaluation of SEP, the purpose of which is to reliably quantify Program accomplishments and help inform decisions on future operations. The sole use of the information collected will be for an analysis of national-level Program impacts. Disclosure of this information is voluntary and there will be no adverse effects associated with not providing all or any part of the requested information.

The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. If you have any questions about this study, you can contact [MAIN STUDY CONTACT?].

PART 2. IDENTIFY THE APPROPRIATE RESPONDENT(S)

Our understanding is that your organization was responsible for delivering a program to promote the development of retrofit energy projects in residential facilities or as free-standing projects to sell electricity. Is that correct?

Yes: CONTINUE

No: PROBE FOR REASON FOR MISUNDERSTANDING. THANK, TERMINATE, AND CONSULT WITH STATE ENERGY OFFICE CONTACT

Our understanding is that you served or serve as the program manager for Residential Retrofit programs that received SEP/ARRA funding in [YEAR]? Is this correct:

Yes – Correct person and year for SEP/ARRA funded Residential retrofit programs	[Continue]
Yes – Correct person, wrong year for SEP/ARRA funded Residential retrofit programs	[Correct year and Continue]
No – Incorrect person for SEP/ARRA funded Residential retrofit programs	[Ask for correct person for Residential Retrofit Project Development programs]
No – There were no SEP/ARRA funded Residential retrofit programs	[Thank you and terminate. Contact main state energy office representative to verify that there were no programs that corresponded to description in the PAGE database.]

Your participation in this survey is voluntary, and your answers are confidential. The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. The length of the interview varies from person to person, but most interviews last about 50 minutes. For quality control purposes, this call will be recorded and monitored. If you have any questions about this study, you can contact [ORNL CONTACT INFORMATION].

PRIOR TO THE INTERVIEW, REVIEW THE DELIVERY ORGANIZATION’S CONTRACT AND WEB SITE TO GATHER INFORMATION TO ADDRESS THE FOLLOWING QUESTIONS. POSE THEM TO THE RESPONDENT ONLY IF NEEDED FOR VERIFICATION OR TO FILL IN MISSING INFORMATION.

1. What are the primary services that your organization delivers?

________________________________________________________________

________________________________________________________________

________________________________________________________________

2. How long has your organization been in business?

3. How long has your organization provided the specific services you delivered for the [PROGRAM]?

4. Were you involved in the management of retrofit programs that received SEP/ARRA funding in [PROGRAM YEAR]?

1 Yes

2 No [IF NOT, COLLECT CONTACT NAME AND TELEPHONE NUMBER. IF NO ONE AT ORGANIZATION IS KNOWLEDGEABLE ABOUT THE PROGRAM, STATE DETAILED REASON. CONTINUE THE INTERVIEW WITH THAT INDIVIDUAL.]

Yes	[GO TO Q3]
No	[ASK Q2]

5. What were your responsibilities in regard to retrofit programs that received SEP/ARRA funding in [PROGRAM YEAR]?

6. In what year did you first become involved with the retrofit programs that received SEP ARRA funding?

7. Are you still involved with managing this program? If not, when did your involvement end?

8. According to information provided by the State Energy Office, you received a contract valued a roughly $__________ to operate programs that promote the development of Residential Retrofit projects. Is this correct?

Listed Amount	Corrected Amount
[AMOUNT ]_______	[AMOUNT ]
	[REMAINING AMOUNT]

PART 3: PROGRAM DESIGN (LOGIC MODEL)

9. As you understand them, what were the ultimate objectives of [PROGRAM NAME]?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

10. Did your organization commit to any quantitative goals for the program, such as number of projects developed or kW of capacity installed?

    1. Yes

    2. No

    3. DK
       
       
       

11. IF 10 = YES: What were those goals?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

12. In terms of the timeline of your contract:

    1. When did you initiate development of the program?

    2. When did you launch the program to the public?

13. Please describe how the program is operating now in regard to its activities, the resources it uses, and the outcomes it is achieving.

13A. Let’s start with ACTIVITIES. PROBE:

MARKETING AND OUTREACH TO PARTICIPANTS

RECRUITMENT OF PARTICIPANTS

RECRUITMENT OF TRADE ALLIES AND OTHER DELIVERY PARTNERS

DELIVERY OF INFORMATION AND TECHNICAL SERVICES TO HOMEOWNERS

DELIVERY OF INFORMATION AND TECHNICAL SUPPORT TO VENDORS AND INSTALLATION CONTRACTORS INVOLVED IN THE PROGRAM

PROVISION OF TECHNICAL AND PROJECT DEVELOPMENT SERVICES, SUCH AS SITE ASSESSMENTS, ENGINEERING SUPPORT, FEASIBILITY STUDIES, FINANCIAL MODELING, ETC.

ADVOCACY AND TECHNICAL SUPPORT FOR LAWS AND REGULATIONS TO SUPPORT RETROFIT PROJECT DEVELOPMENT

PROVISION OF FINANCIAL SUPPORT: INCENTIVES, LOANS, LOAN GUARANTEES TO SUPPORT PROJECTS

INSPECTION, QUALITY CONTROL OF INSTALLED PROJECTS

OTHER (SPECIFY)

13B. INPUTS: What resources are being used to deliver the program? PROBE:

STATE ENERGY OFFICE STAFF: NUMBER, TYPE, PERCENT OF TIME

OTHER GOVERNMENT STAFF: NUMBER, TYPE, PERCENT OF TIME

STAFF OF OTHER AGENCIES OR ORGANIZATIONS: NUMBER, TYPE, PERCENT OF TIME

CONTRACTED SERVICES: PROBE NAME OF PROVIDER AND TYPE OF SERVICE, APPROXIMATE NUMBER OF CONTRACTOR STAFF, DOLLAR SIZE OF CONTRACT

TECHNICAL SERVICES

FINANCIAL SERVICES

MARKETING SERVICES

ADMINISTRATIVE SERVICES

13C. OUTSIDE FUNDING: Are organizations other than the State Energy Office contributing funding to this program? IF YES, PROBE:

NAME OF THE ORGANIZATION(S)

AMOUNT OF FUNDING PROVIDED IN THE PERIOD UNDER EVALUATION (2008 OR ARRA PERIOD, FOR EACH FUNDER IF AVAILABLE)

PERCENTAGE OF TOTAL FUNDING FOR THE PA UNDER EVALUATION IN 2007, 2008, ARRA PERIOD, AS RELEVANT.

USES OF THE FUNDING PROVIDED

13D. RESULTS/OUTPUTS: Could you summarize the results your program has achieved so far? PROBE:

NUMBER OF PARTICIPANTS ENROLLED

NUMBER OF UNITS OF SERVICE DELIVERED (TECHNICAL SERVICES, LOANS AND GRANTS, DOLLAR VOLUME OF LOANS AND GRANTS)

NUMBER OF INSTALLATION PROJECTS SUPPORTED

ESTIMATED SAVINGS FROM INSTALLATIONS SUPPORTED

13E. RESULTS VERSUS GOALS: Were you able to (Do you anticipate being able to) achieve the level of results you included in your program plans or quantitative goals?

[IF NO ASK] What circumstances prevented you (are preventing you) from achieving those goals?

14. [IF PROGRAM EXISTED PRIOR TO 2008 ASK] Did you make any important changes to program operations or the resources available to it between 2007 and 2008? [IF PROGRAM EXISTED PRIOR TO ARRA AND RECEIVED ARRA FUNDING ALSO ASK] Did you make any important changes to program operations or the resources available to it between 2008 and the period funded primarily by ARRA?

    1. Why did you make those changes?

    2. Did achieve the results you were seeking through those changes? PROBE SPECIFIC RESULTS.

3. [FOR ARRA PERIOD PROGRAMS ASK]. Would your organization have been able to make these changes without the support provided by the ARRA round of funding?

YES/NO

PROBE REASONS FOR THIS ASSESSMENT.

[THESE QUESTIONS ARE DESIGNED TO FLESH OUT PROGRAM DETAILS TO DEVELOP THE PROGRAM LOGIC MODEL. THE LOGIC MODEL ARTICULATES WHAT THE PROGRAM IS TRYING TO ACHIEVE (AND DID ACHIEVE), THROUGH WHAT INTERVENTIONS, AND WITH RESPECT TO WHICH MARKET ACTORS. THIS INFORMATION WILL BE USED TO DEVELOP A MAP OF THE PROGRAM’S DOMAIN FOR ATTRIBUTION PURPOSES.]

15. Has your organization or the state energy office developed a formal logic model for [PROGRAM NAME]?

    1. Can you share it with us?

PART 4: SEP INFLUENCE AND ROLE IN THE MARKETPLACE

16. Do you believe that this program has had an impact on the long-term capabilities of homeowners to develop retrofit energy projects in [STATE]?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?

17. Do you believe that this program has had an impact on the long-term capabilities of equipment vendors, designers, contractors, and other firms on the “supply side” of the market to sell and install energy-efficient retrofit projects in residential facilities?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?
   
   
   

18. Did your organization add any positions to carry out or administer the program(s) we have been discussing? Please include positions that may be split between a number of different programmatic and administrative activities.
    
    IF YES:

    1. How many positions did your organization add?

    2. How many FTEs were added in each of the following job classifications: administrative, technical services, professional, management?

    3. How many of these FTEs will be retained after 2008 or the ARRA period, depending on which is relevant?

    4. IF c > 0: What sources of funding do you plan to use to sustain this (these) position(s)?

PART 5: DATA AVAILABILITY

[ONLY ASK THIS SECTION IF WE HAVE NOT BEEN ABLE TO ACQUIRE PROGRAM DATA FROM THE STATE ENERGY OFFICE]

NOTE: QUESTIONS 36-38 ON DATA AVAILABILITY WILL BE REVIEWED WITH PROGRAM SPONSORS PRIOR TO FINAL SELECTION OF THE PROGRAM INTO THE SAMPLE. WE WILL ALSO ASK TO RECEIVE A COPY OF THE DATABASES OR FILE FORMATS SO THAT WE CAN MAKE A JUDGMENT REGARDING THE ADEQUACY OF THE DATABASE TO SUPPORT EVALUATION ANALYSIS. THESE QUESTIONS WILL BE FOR VALIDATION OF OUR REVIEW.

19. Please describe the types of data that you tracked in [PROGRAM YEAR] on program participants. Do you have lists of:

1. Program participants, that is, facility owners/renters, residential property managers who received support with the project, workshop attendees or training recipients, recipients of technical assistance, with contact information

2. Measures installed by participant, including the type and quantity of measures installed with program support, energy savings, incentive levels, other services received by participants, workshop or training description. [PROBE TO CLARIFY DATA THAT THE PM HAS ACCESS TO (EITHER IN-HOUSE OR AT A SEP FUNDED/PARTNERED ORGANIZATION) VERSUS DATA ANY EXTERNAL LEVERAGED ORGANIZATION MIGHT BE KEEPING.]

3. Participating trade allies (including contact information)

(IF DATA ARE AVAILABLE BUT RESPONDENT IS NOT KNOWLEDGEABLE OR DOES NOT HAVE ACCESS TO IT, RECORD NAME OF CONTACT WHO CAN ANSWER FOLLOW-UP QUESTIONS ABOUT DATA AVAILABILITY. PROBE FOR WHETHER ANYONE MAINTIANS SUCH DATA – POTENTIALLY EXTERNAL TO THE PA). IF THERE IS NO ONE WHO CAN PROVIDE SUCH DATA THEN THANK AND TERMINATE).

20. Have there been any evaluations of this program? Are data from these evaluations available?

21. (FOR EACH TYPE OF DATA AVAILABLE) Are these data in electronic format? (If yes, discuss steps needed to secure permission to access the data via secure file transfer. If no, determine how data records are maintained and how they could be accessed.)

Are you aware of other data available that would be useful for our evaluation? If so could you describe it for me? Is it possible to access this information? Probe for contacts. If making copies is an issue, ask whether we can make copies. if still reluctant, ask whether they have a summary of retrofitted properties with minimal INFORMATION so we can select a sample and copy only those records.]

Thank you for your time and insights

9

OMB Control No. XXXXXXXX