State Energy Program Evaluation

OMB Control No. XXXXXXXX

ID-12B-R: Renewables (Program Delivery Contractors) Residential

OMB Control No. XXXXXXXX

This interview guide is designed to provide direction to the interviewer to ensure that all relevant topics are explored to the extent possible and appropriate with the respondent. Note that our interviews are meant to be somewhat informal and open ended – not all topics will be covered in all interviews and we expect that some interviews will lead to the exploration of topics not included in this guide.

Information in [BRACKETS] will be customized to reflect the unique PA program [PROGRAM] and year of the PA offering [PROGRAM YEAR].

BACKGROUND INFORMATION (to be filled in prior to interview):

Program Administrator Name:
Year:
BPAC Area
2008 Budget:
2008 Market Title Sampled
2009-2010 ARRA budget
2009-2010 ARRA Market Title Sampled
Known programmatic activities prior to interview:
Description of target markets:
Structure of SEP/ARRA funded activities from informal discussions with SEP representative from first round discussions (from database)
Contact Name:
Contact Company:
Contact Phone:
Contact Disposition:

PART 1. INTRODUCTION

My name is [INTERVIEWER NAME] from [INTERVIEWER ORGANIZATION]. The US Department of Energy’s State Energy Program has hired us to gather information on the results of efforts to promote the development of Residential renewable energy projects in [STATE] that received funding from the State Energy Program and the American Recovery and Reinvestment Act. This interview is being conducted as part of an evaluation of the State Energy Program being conducted by Oak Ridge National Laboratory on behalf of the U. S. Department of Energy.

The U.S. Department of Energy (DOE) would like to inform each individual that the information requested here is being solicited under the statutory authority of Title III of the Energy Policy and Conservation Act of 1975, as amended, which authorizes DOE to administer the State Energy Program (SEP). This information is being sought as part of a national evaluation of SEP, the purpose of which is to reliably quantify Program accomplishments and help inform decisions on future operations. The sole use of the information collected will be for an analysis of national-level Program impacts. Disclosure of this information is voluntary and there will be no adverse effects associated with not providing all or any part of the requested information.

The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. If you have any questions about this study, you can contact [MAIN STUDY CONTACT?].

PART 2. IDENTIFY THE APPROPRIATE RESPONDENT(S)

Our understanding is that your organization was responsible for delivering a program to promote the development of renewable energy projects in Residential facilities or as free-standing projects to sell electricity. Is that correct?

Yes: CONTINUE

No: PROBE FOR REASON FOR MISUNDERSTANDING. THANK, TERMINATE, AND CONSULT WITH STATE ENERGY OFFICE CONTACT

Our understanding is that you served or serve as the program manager for Residential Renewables programs that received SEP/ARRA funding in [YEAR]? Is this correct:

Yes – Correct person and year for SEP/ARRA funded Residential renewable programs	[Continue]
Yes – Correct person, wrong year for SEP/ARRA funded Residential renewable programs	[Correct year and Continue]
No – Incorrect person for SEP/ARRA funded Residential renewable programs	[Ask for correct person for Residential Renewables Project Development programs]
No – There were no SEP/ARRA funded Residential renewable programs	[Thank you and terminate. Contact main state energy office representative to verify that there were no programs that corresponded to description in the PAGE database.]

Your participation in this survey is voluntary, and your answers are confidential. The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. The length of the interview varies from person to person, but most interviews last about 70 minutes. For quality control purposes, this call will be recorded and monitored. If you have any questions about this study, you can contact [ORNL CONTACT INFORMATION].

I would like to ask you some questions about the activities to promote Residential Renewable energy projects that received SEP/ARRA funding in [PROGRAM YEAR].

PRIOR TO THE INTERVIEW, REVIEW THE DELIVERY ORGANIZATION’S CONTRACT AND WEB SITE TO GATHER INFORMATION TO ADDRESS THE FOLLOWING QUESTIONS. POSE THEM TO THE RESPONDENT ONLY IF NEEDED FOR VERIFICATION OR TO FILL IN MISSING INFORMATION.

1. What are the primary services that your organization delivers?

________________________________________________________________

________________________________________________________________

________________________________________________________________

2. How long has your organization been in business?

3. How long has your organization provided the specific services you delivered for the [PROGRAM]?

4. Were you involved in the management of renewables programs that received SEP/ARRA funding in [PROGRAM YEAR]?

1 Yes

2 No [IF NOT, COLLECT CONTACT NAME AND TELEPHONE NUMBER. IF NO ONE AT ORGANIZATION IS KNOWLEDGEABLE ABOUT THE PROGRAM, STATE DETAILED REASON. CONTINUE THE INTERVIEW WITH THAT INDIVIDUAL.]

Yes	[GO TO Q3]
No	[ASK Q2]

5. What were your responsibilities in regard to renewables programs that received SEP/ARRA funding in [PROGRAM YEAR]?

6. In what year did you first become involved with the renewables programs that received SEP ARRA funding?

7. Are you still involved with managing this program? If not, when did your involvement end?

8. According to information provided by the State Energy Office, you received a contract valued a roughly $__________ to operate programs that promote the development of Residential Renewables projects. Is this correct?

Listed Amount	Corrected Amount
[AMOUNT ]_______	[AMOUNT ]
	[REMAINING AMOUNT]

PART 3: PROGRAM DESIGN (LOGIC MODEL)

9. As you understand them, what were the ultimate objectives of [PROGRAM NAME]?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

10. Did your organization commit to any quantitative goals for the program, such as number of projects developed or kW of capacity installed?

    1. Yes

    2. No

    3. DK
       
       
       

11. IF 10 = YES: What were those goals?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

12. In terms of the timeline of your contract:

    1. When did you initiate development of the program?

    2. When did you launch the program to the public?

13. Please describe how the program is operating now in regard to its activities, the resources it uses, and the outcomes it is achieving.

13A. Let’s start with ACTIVITIES. PROBE:

MARKETING AND OUTREACH TO PARTICIPANTS

RECRUITMENT OF PARTICIPANTS

RECRUITMENT OF TRADE ALLIES AND OTHER DELIVERY PARTNERS

DELIVERY OF INFORMATION AND TECHNICAL SERVICES

PROVISION OF TECHNICAL AND PROJECT DEVELOPMENT SERVICES, SUCH AS SITE ASSESSMENTS, ENGINEERING SUPPORT, FEASIBILITY STUDIES, FINANCIAL MODELING, ETC.

ADVOCACY AND TECHNICAL SUPPORT FOR LAWS AND REGULATIONS TO SUPPORT RENEWABLES PROJECT DEVELOPMENT, SUCH AS RENEWABLE PORTFOLIO STANDARDS, INTERCONNECTION RULES, NET METERING.

PROVISION OF FINANCIAL SUPPORT: INCENTIVES, LOANS, LOAN GUARANTEES TO SUPPORT PROJECTS

INSPECTION, QUALITY CONTROL OF INSTALLED PROJECTS

OTHER (SPECIFY)

13B. INPUTS: What resources are being used to deliver the program? PROBE:

STATE ENERGY OFFICE STAFF: NUMBER, TYPE, PERCENT OF TIME

OTHER GOVERNMENT STAFF: NUMBER, TYPE, PERCENT OF TIME

STAFF OF OTHER AGENCIES OR ORGANIZATIONS: NUMBER, TYPE, PERCENT OF TIME

CONTRACTED SERVICES: PROBE NAME OF PROVIDER AND TYPE OF SERVICE, APPROXIMATE NUMBER OF CONTRACTOR STAFF, DOLLAR SIZE OF CONTRACT

TECHNICAL SERVICES

FINANCIAL SERVICES

MARKETING SERVICES

ADMINISTRATIVE SERVICES

13C. OUTSIDE FUNDING: Are organizations other than the State Energy Office contributing funding to this program? IF YES, PROBE:

NAME OF THE ORGANIZATION(S)

AMOUNT OF FUNDING PROVIDED IN THE PERIOD UNDER EVALUATION (2008 OR ARRA PERIOD, FOR EACH FUNDER IF AVAILABLE)

PERCENTAGE OF TOTAL FUNDING FOR THE PA UNDER EVALUATION IN 2007, 2008, ARRA PERIOD, AS RELEVANT.

USES OF THE FUNDING PROVIDED

13D. RESULTS/OUTPUTS: Could you summarize the results your program has achieved so far? PROBE:

NUMBER OF PARTICIPANTS ENROLLED

NUMBER OF UNITS OF SERVICE DELIVERED (TECHNICAL SERVICES, LOANS AND GRANTS, DOLLAR VOLUME OF LOANS AND GRANTS)

NUMBER OF INSTALLATION PROJECTS SUPPORTED

KW INSTALLED

ESTIMATED SAVINGS FROM INSTALLATIONS SUPPORTED

13E. RESULTS VERSUS GOALS: Were you able to (Do you anticipate being able to) achieve the level of results you included in your program plans or quantitative goals?

IF NO: What circumstances prevented you (are preventing you) from achieving those goals?

14. [IF PROGRAM EXISTED PRIOR TO 2008 ASK] Did you make any important changes to program operations or the resources available to it between 2007 and 2008? [IF PROGRAM EXISTED PRIOR TO ARRA AND RECEIVED ARRA FUNDING ALSO ASK] Did you make any important changes to program operations or the resources available to it between 2008 and the period funded primarily by ARRA?

    1. Why did you make those changes?

    2. Did achieve the results you were seeking through those changes? PROBE SPECIFIC RESULTS.

3. [FOR ARRA PERIOD PROGRAMS ASK] Would your organization have been able to make these changes without the support provided by the ARRA round of funding?

YES/NO

PROBE REASONS FOR THIS ASSESSMENT.

[THESE QUESTIONS ARE DESIGNED TO FLESH OUT PROGRAM DETAILS TO DEVELOP THE PROGRAM LOGIC MODEL. THE LOGIC MODEL ARTICULATES WHAT THE PROGRAM IS TRYING TO ACHIEVE (AND DID ACHIEVE), THROUGH WHAT INTERVENTIONS, AND WITH RESPECT TO WHICH MARKET ACTORS. THIS INFORMATION WILL BE USED TO DEVELOP A MAP OF THE PROGRAM’S DOMAIN FOR ATTRIBUTION PURPOSES.]

15. Has your organization or the state energy office developed a formal logic model for [PROGRAM NAME]?

    1. Can you share it with us?

PART 4 – ASSESSMENT OF PROGRAM EFFECT

This next set of questions seeks your opinion on the effect of the program on targeted homeowners and vendors.

16. IF THE PROGRAM SUPPORTED ISNTALLATIONS IN INDIVIDUAL HOMES, ASK: Now, thinking about the homeowners who received support from your program for installation of renewable energy systems -- On a scale of 1 to 10, where 1 means “not at all important” and 10 means “very important”, how important were the services provided [PROGRAM NAME] in convincing and enabling those participants to undertake the renewable energy projects

ENTER 1 – 10, 99 FOR DK/REF: ________________________________

17. IF 16 = 1 – 10, ASK: Why do you say that? PROBE SPECIFIC BARRIERS OR ISSUES THE SERVICES FROM THE SEP PA HELPED TO ADDRESS, HELP FROM OTHER PROGRAMS, PREDISPOSITION OF THE FACILITY OWNERS.

_______________________________________________________________

________________________________________________________________

________________________________________________________________

18. In your opinion, what percentage of these projects would facility owners and developers have completed in the absence of the support from [PROGRAM NAME]?

ENTER PERCENT: _______________________%

19. What observations lead you to that estimate?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

20. Generally speaking, has the annual number of residential renewable projects completed in [STATE] increased, decreased, or stayed about the same since [YEAR OF PROGRAM LAUNCH]?

21. IF 20 = INCREASED OR DECREASED: By approximately what percentage would you say the number of residential renewable projects has increased/decreased?

ENTER PERCENT, 999 FOR DK/REF:________________________

22. Which factors or conditions do you believe most influenced the volume of residential renewable energy systems installed in [STATE] since [YEAR OF PROGRAM LAUNCH]? [DO NOT READ; MARK ALL FACTORS MENTIONED]

    1. Electricity rates

    2. Federal tax laws and policies

    3. Changes in state rules and regulations related to renewables projects, such as interconnection and net metering

    4. Changes in public awareness of global warming and other environmental issues related to energy use

    5. Reduction in the costs of renewable energy equipment and installation

    6. Improved performance of renewable energy equipment

    7. PROGRAM NAME

    8. Programs offered by utilities and other sponsors

    9. Other (Specify) ______________________________

23. ASK IF [PROGRAM NAME] MENTIONED, ELSE SKIP TO 25: Compared to the other factors you mentioned, would you consider [PROGRAM NAME] the one with:

    1. The greatest influence on the volume of residential renewable energy project installations

    2. Among the most amount of influence

    3. Roughly equal influence as the others

    4. Somewhat less influence than the others

    5. Much less influence than the others

    6. DK/REF

24. ASK IF 23 = a – e: Why do you say that?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

25. IF [PROGRAM NAME] NOT MENTONED IN 22, ASK: What was the main reason you did not mention [PROGRAM NAME] as a factor affecting changes in the volume of residential renewable projects installed in [STATE]?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

26. On a scale of 1 to 10 where 1 means “not at all favorable” and 10 means “very favorable”, how would you characterize the conditions for the development of residential and utility-scale renewable energy projects in [STATE]?

ENTER 1 – 10, 99 FOR DK/REF: ________________

27. IF 26 = 1 – 10, ASK: What factors would you say continue to constitute barriers to the development of renewable energy projects in [STATE]?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

28. IF 26 = 1 – 10, ASK: What rules, regulations, and market conditions would you say contribute to a favorable climate for development of renewable energy projects in [STATE]?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

29. IF ANY ANSWERS GIVEN TO 28, ASK: Are you aware of any activities that [STATE ENERGY OFFICE] has taken to promote rules, regulations, and market conditions that are favorable for the development of renewable energy projects in [STATE]? IF YES: What specific activities are you aware of?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

30. IF ANY ANWERS GIVEN TO 28, ASK: On a scale of 1 to 10, where 1 means “not at all important” and 10 means “very important”, how important do you believe these activities have been to the development of regulations and market conditions that are favorable to the development of renewable energy projects in [STATE]?

ENTER 1 – 10, 99 FOR DK/REF: __________________________

31. IF 30= 1 – 10: What factors lead you to give that rating?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

PART 4: SEP INFLUENCE AND ROLE IN THE MARKETPLACE

32. Do you believe that this program has had an impact on the long-term demand exercised by homeowners for residential renewable energy projects in [STATE]?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?

33. Do you believe that this program has had an impact on the long-term capabilities of equipment vendors, designers, contractors, and other firms on the “supply side” of the market to sell and install renewable projects in Residential facilities?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?
   
   
   

34. Did your organization add any positions to carry out or administer the program(s) we have been discussing? Please include positions that may be split between a number of different programmatic and administrative activities.
    
    IF YES:

    1. How many positions did your organization add?

    2. How many FTEs were added in each of the following job classifications: administrative, technical services, professional, management?

    3. How many of these FTEs will be retained after 2008 or the ARRA period, depending on which is relevant?

    4. IF c > 0: What sources of funding do you plan to use to sustain this (these) position(s)?

PART 5: DATA AVAILABILITY

[ONLY ASK THIS SECTION IF WE HAVE NOT BEEN ABLE TO ACQUIRE PROGRAM DATA FROM THE STATE ENERGY OFFICE]

NOTE: QUESTIONS 36-38 ON DATA AVAILABILITY WILL BE REVIEWED WITH PROGRAM SPONSORS PRIOR TO FINAL SELECTION OF THE PROGRAM INTO THE SAMPLE. WE WILL ALSO ASK TO RECEIVE A COPY OF THE DATABASES OR FILE FORMATS SO THAT WE CAN MAKE A JUDGMENT REGARDING THE ADEQUACY OF THE DATABASE TO SUPPORT EVALUATION ANALYSIS. THESE QUESTIONS WILL BE FOR VALIDATION OF OUR REVIEW.

35. Please describe the types of data that you tracked in [PROGRAM YEAR] on program participants. Do you have lists of:

1. Program participants, that is, facility owners/renters, Residential property managers who received support with the project, workshop attendees or training recipients, recipients of technical assistance, with contact information

2. Measures installed by participant, including the type and quantity of measures installed with program support, energy savings, incentive levels, other services received by participants, workshop or training description. [PROBE TO CLARIFY DATA THAT THE PM HAS ACCESS TO (EITHER IN-HOUSE OR AT A SEP FUNDED/PARTNERED ORGANIZATION) VERSUS DATA ANY EXTERNAL LEVERAGED ORGANIZATION MIGHT BE KEEPING.]

3. Participating trade allies (including contact information)

(IF DATA ARE AVAILABLE BUT RESPONDENT IS NOT KNOWLEDGEABLE OR DOES NOT HAVE ACCESS TO IT, RECORD NAME OF CONTACT WHO CAN ANSWER FOLLOW-UP QUESTIONS ABOUT DATA AVAILABILITY. PROBE FOR WHETHER ANYONE MAINTIANS SUCH DATA – POTENTIALLY EXTERNAL TO THE PA). IF THERE IS NO ONE WHO CAN PROVIDE SUCHA DATA THEN THANK AND TERMINATE).

36. Have there been any evaluations of this program? Are data from these evaluations available?

37. (FOR EACH TYPE OF DATA AVAILABLE) Are these data in electronic format? (If yes, discuss steps needed to secure permission to access the data via secure file transfer. If no, determine how data records are maintained and how they could be accessed.)

Are you aware of other data available that would be useful for our evaluation? If so could you describe it for me? Is it possible to access this information? Probe for contacts. If making copies is an issue, ask whether we can make copies. if still reluctant, ask whether they have a summary of PROJECTS with minimal INFORMATION so we can select a sample and copy only those records.]

Thank you for your time and insights

14

OMB Control No. XXXXXXXX