State Energy Program Evaluation

OMB Control No. XXXXXXXX

ID-11B: Renewable Energy Market Development (Program Managers)

OMB Control No. XXXXXXXX

This interview guide is designed to provide direction to the interviewer to ensure that all relevant topics are explored to the extent possible and appropriate with the respondent. Note that our interviews are meant to be somewhat informal and open ended – not all topics will be covered in all interviews and we expect that some interviews will lead to the exploration of topics not included in this guide.

Information in [BRACKETS] will be customized to reflect the unique PA program [PROGRAM] and year of the PA offering [PROGRAM YEAR].

BACKGROUND INFORMATION (to be filled in prior to interview):

Program Administrator Name:
Year:
BPAC Area
2008 Budget:
2008 Market Title Sampled
2009-2010 ARRA budget
2009-2010 ARRA Market Title Sampled
Known programmatic activities prior to interview:
Description of target markets:
Structure of SEP/ARRA funded activities from informal discussions with SEP representative from first round discussions (from database)
Contact Name:
Contact Company:
Contact Phone:
Contact Disposition:

PART 1. INTRODUCTION

My name is [INTERVIEWER NAME] from [INTERVIEWER ORGANIZATION]. The US Department of Energy’s State Energy Program has hired us to gather information on the results of efforts to promote the development of non-residential renewable energy projects in [STATE] that received funding from the State Energy Program and the American Recovery and Reinvestment Act. This interview is being conducted as part of an evaluation of the State Energy Program being conducted by Oak Ridge National Laboratory on behalf of the U. S. Department of Energy.

The U.S. Department of Energy (DOE) would like to inform each individual that the information requested here is being solicited under the statutory authority of Title III of the Energy Policy and Conservation Act of 1975, as amended, which authorizes DOE to administer the State Energy Program (SEP). This information is being sought as part of a national evaluation of SEP, the purpose of which is to reliably quantify Program accomplishments and help inform decisions on future operations. The sole use of the information collected will be for an analysis of national-level Program impacts. Disclosure of this information is voluntary and there will be no adverse effects associated with not providing all or any part of the requested information.

The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. If you have any questions about this study, you can contact [MAIN STUDY CONTACT?]. If you have any questions about this study, you can contact [ORNL CONTACT INFORMATION].

PART 2. IDENTIFY THE APPROPRIATE RESPONDENT(S)

Our understanding is that you are the program manager for Non-Residential Renewables programs that received SEP/ARRA funding in [YEAR]? Is this correct:

Yes – Correct person and year for SEP/ARRA funded Non-Residential renewable programs	[Continue]
Yes – Correct person, wrong year for SEP/ARRA funded Non-Residential renewable programs	[Correct year and Continue]
No – Incorrect person for SEP/ARRA funded Non-Residential renewable programs	[Ask for correct person for Non-Residential Renewables Project Development programs]
No – There were no SEP/ARRA funded Non-Residential renewable programs	[Thank you and terminate. Contact main state energy office representative to verify that there were no programs that corresponded to description in the PAGE database.]

Your participation in this survey is voluntary, and your answers are confidential. The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. The length of the interview varies from person to person, but most interviews last about 75 minutes. For quality control purposes, this call will be recorded and monitored. If you have any questions about this study, you can contact [ORNL CONTACT INFORMATION].

I would like to ask you some questions about the activities to promote Non-Residential Renewable energy projects that received SEP/ARRA funding in [PROGRAM YEAR].

1. Were you involved in the design and/or management of renewables programs that received SEP/ARRA funding in [PROGRAM YEAR]?

1 Yes

2 No [IF NOT, COLLECT CONTACT NAME AND TELEPHONE NUMBER. IF NO ONE AT ORGANIZATION IS KNOWLEDGEABLE ABOUT THE PROGRAM, STATE DETAILED REASON]

Yes	[GO TO Q3]
No	[ASK Q2]

2. Who else at your organization is most knowledgeable about the overall allocation of funding to Non-Residential Renewables programs that received SEP/ARRA funding?

Name/contact info/role: ________________________________________

Name/contact info/role: ________________________________________

Name/contact info/role: ________________________________________

3. [IF OTHERS MENTIONED IN Q2] Among you and [INSERT QUESTION 2 RESPONSE], who is most familiar with the activities conducted by your organization for [PROGRAM] in [PROGRAM YEAR]?

[IF NOT RESPONDENT, THANK AND TERMINATE AND CONTACT MOST KNOWLEGEABLE PERSON]

4. What were your responsibilities in regard to renewables programs that received SEP/ARRA funding in [PROGRAM YEAR]?

5. In what year did you first become involved with the renewables programs that received SEP ARRA funding?

6. Are you still involved with managing this program? If not, when did your involvement end?

7. How long have you been involved in administering programs funded by SEP?

8. Based on information provided by DOE, you received $__________ in SEP/ARRA funding for activities that promoted the development of Non-Residential Renewables projects. Our understanding is that these funds may have been applied to programmatic activities supporting non-residential renewables through either direct payment to targeted renewable projects or through loans, grant or incentive based renewable programs. Is this correct?

Listed Amount	Corrected Amount
[AMOUNT ]_______	[AMOUNT ]
	[REMAINING AMOUNT]

[IF LESS FUNDS WERE APPLIED TO NON-RESIDENTIAL RENEWABLES THAN INDICATED IN THE TRACKING DATA ASK 8a-8c]

8a. To which of the following other types of programmatic activities was the remaining SEP/ARRA funding applied?

• Renewables projects in residential facilities

• Renewable Energy Manufacturing – PAs develop or expand existing manufacturing capacity for renewable energy equipment or components. At least some portion of the output of the new or expanded capacity is intended for domestic installation.

• Clean Energy Policy support – Develops and obtains legislative, executive, or regulatory approval for policies to facilitate the completion of renewable energy facilities. Examples might include statewide zoning laws, feed-in tariffs, favorable back-up tariffs, renewable portfolio standards.

• Building Code Development and Support – Provides technical and administrative support for the development of more energy-efficient state and federal equipment standards. Provides marketing support for products that meet the higher energy efficiency standards. Provides training to vendors in marketing and installation of products that meet the higher energy efficiency standards.

• Policy and Market Studies; Legislative Support – Develops and obtains legislative, executive, or regulatory approval for policies to facilitate the completion of renewable energy facilities. Examples might include statewide zoning laws, feed-in tariffs, favorable back-up tariffs, renewable portfolio standards.

• Non-retrofit Targeted Training and/or Certification (Participants are traceable) – Training for facility managers, trades contractors, and engineering/design professionals. Topics for EE, renewables, transportation and other technical topics. i.e. roundtable discussions etc. Includes certification programs and other pre-requisite type training & education programs.

• Non-retrofit Generalized Workshops and Demonstrations – Marketing and outreach support to raise awareness, provide general information, encourage behavior change, etc. Not targeted at a specific site or project, but may be targeted at specific segments or types of projects. Recipients of the education are not traceable.

• Non-retrofit Technical Assistance to Building Owners – Provides technical assistance other than audits for building retrofit or equipment replacement projects: e.g. technical studies for specific improvements, building modeling, project financial analysis, support in negotiating with contractors. Open to commercial, industrial, and agricultural facility owners or specified subgroups thereof. May be combined with financial incentives.

• Other [DEFINE] _______________________________________________________________

8b. [IF 8a = “YES” FOR ANY ACTIVITY TYPE, RECORD AMOUNT]

[IF 8a = “YES” FOR ANY ACTIVITY TYPE ask 8c]

8c. Can we contact the appropriate program manager for this activity? [RECORD NAME]

9. Please describe each type of programmatic activity that received ARRA/SEP funds:

RESPONSE MATRIX A: Questions 8a – 8c and 9

Activity	8a. Funds applied	8b. Funding	8c. Permission	9. Program description
Residential renewable energy projects	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Renewable energy manufacturing	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Building Code Development and Support	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Policy and Market Studies; Legislative Support	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Targeted Training and/or Certification (Participants are traceable)	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Generalized Workshops and Demonstrations	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Technical Assistance to Building Owners	[YES/NO]	[AMOUNT ]	[RECORD NAME]
Other [DEFINE]	[YES/NO]	[AMOUNT ]	[RECORD NAME]

[PLEASE READ]

For the remainder of this interview, I’d like to focus on just the non-residential renewables programs that received SEP/ARRA.

10. Were the SEP/ARRA funds your organization received used to fund Non-Residential renewable programs in [YEAR] distributed across multiple programs?

[IF YES ASK Q11 – 14; ELSE GO TO Q16]

11. How many separate programs? ________________

FOR EACH PROGRAM MENTIONED ASK 12-15

12. What are the names of these programs?

13. What percent of the total SEP/ARRA funds used to for Non-Residential renewable programs was used for each of these programs?

14. Are you or another program manager in charge of any of these programs? [IF THERE ARE OTHER PROGRMAM MANGERS, ASK Q15]

15. Can we contact the program manager regarding [PROGRAM NAME]?

[IF YES, RECORD THEIR NAME AND CONTACT INFORMATION]

Response Matrix C: FOR QUESTIONS 12-15, USE REPSONSE GRID BELOW

12. Program Name	13. Percent	14. Additional program manager	15. Permission to contact
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]
		[YES/NO]	[RECORD CONTACT INFO]

PART 3: PROGRAM DESIGN (LOGIC MODEL)

[ASK SECTIONS 3 – 6 FOR EACH OF THE PROGRAMS LISTED IN RESPONSE MATRIX A. IF NECESSARY, SCHEDULE A CALLBACK WITH THE APPROPRIATE PROGRAM MANAGER]

[PLEASE READ] In this next set of questions, I’d like to understand the rationale behind the program design.

[THESE QUESTIONS ARE DESIGNED TO FLESH OUT PROGRAM DETAILS TO DEVELOP THE PROGRAM LOGIC MODEL. THE LOGIC MODEL ARTICULATES WHAT THE PROGRAM IS TRYING TO ACHIEVE (AND DID ACHIEVE), THROUGH WHAT INTERVENTIONS, AND WITH RESPECT TO WHICH MARKET ACTORS. THIS INFORMATION WILL BE USED TO DEVELOP A MAP OF THE PROGRAM’S DOMAIN FOR ATTRIBUTION PURPOSES.]

16. Has your organization developed a formal logic model for [PROGRAM NAME]?

[IF 16 = YES, then ASK 16a]

16a. Can you please share it with us?

17. What were the ultimate objectives of [PROGRAM NAME]?

________________________________________________________________________

________________________________________________________________________

17a. In what year did the program begin?

________________________________________________________________________

18. Please describe how the program is operating now in regard to its activities, the resources it uses, and the outcomes it is achieving.

18A. Let’s start with ACTIVITIES. PROBE:

MARKETING AND OUTREACH TO PARTICIPANTS:

RECRUITMENT OF PARTICIPANTS:

RECRUITMENT OF TRADE ALLIES AND OTHER DELIVERY PARTNERS

DELIVERY OF INFORMATION AND TECHNICAL SERVICES

PROVISION OF FINANCIAL SUPPORT: INCENTIVES, LOANS, LOAN GUARANTEES TO SUPPORT PROJECTS

INSPECTION, QUALITY CONTROL OF INSTALLED PROJECTS

OTHER (SPECIFY)

18B. INPUTS: What resources are being used to deliver the program? PROBE:

STATE ENERGY OFFICE STAFF: NUMBER, TYPE, PERCENT OF TIME

OTHER GOVERNMENT STAFF: NUMBER, TYPE, PERCENT OF TIME

STAFF OF OTHER AGENCIES OR ORGANIZATIONS: NUMBER, TYPE, PERCENT OF TIME

CONTRACTED SERVICES: PROBE NAME OF PROVIDER AND TYPE OF SERVICE, APPROXIMATE NUMBER OF CONTRACTOR STAFF, DOLLAR SIZE OF CONTRACT

TECHNICAL SERVICES

FINANCIAL SERVICES

MARKETING SERVICES

ADMINISTRATIVE SERVICES

18C. OUTSIDE FUNDING: Are organizations other than the U. S. Department of Energy and the state government contributing funding to this program? IF YES, PROBE:

NAME OF THE ORGANIZATION(S)

AMOUNT OF OUTSIDE FUNDING PROVIDED IN THE PERIOD UNDER EVALUATION (2008 OR ARRA PERIOD)

PERCENTAGE OF TOTAL OVERALL FUNDING FOR THE PA UNDER EVALUATION IN 2007, 2008, ARRA PERIOD, AS RELEVANT REPRESENTED BY OUTSIDE FUNDING.

USES OF THE FUNDING PROVIDED

18D. RESULTS/OUTPUTS: Could you summarize the results your program has achieved so far? PROBE: Are organizations other than the U. S. Department of Energy and the state government contributing funding to this program? IF YES, PROBE:

NUMBER OF PARTICIPANTS ENROLLED

NUMBER OF UNITS OF SERVICE DELIVERED (AUDITS, WORKSHOP ATTENDANCE, ETC)

NUMBER OF INSTALLATION PROJECTS SUPPORTED

ESTIMATED SAVINGS FROM INSTALLATIONS SUPPORTED

18E. RESULTS VERSUS GOALS: Were you able to (Do you anticipate being able to) achieve the level of results you included in your program plans and applications to the U. S. Department of Energy?

IF 18E=NO: What circumstances prevented you (are preventing you) from achieving those goals?

19. [IF PROGRAM EXISTED PRIOR TO 2008 ASK] Did you make any important changes to program operations or the resources available to it between 2007 and 2008? IF PROGRAM EXISTED PRIOR TO ARRA AND RECEIVED ARRA FUNDING ALSO ASK] Did you make any important changes to program operations or the resources available to it between 2008 and the period funded primarily by ARRA?

19a. Why did you make those changes?

19b. Did achieve the results you were seeking through those changes? PROBE SPECIFIC RESULTS.

19c. [FOR ARRA PERIOD PROGRAMS ASK]. Would your organization have been able to make these changes without the support provided by the ARRA round of funding?

YES/NO

PROBE REASONS FOR THIS ASSESSMENT.

PRIOR TO INTERVIEW, IDENTIFY ITEMS IN THE GRID BELOW THAT APPLY TO THE JURISDICTION USING INFORMATION AVAILABLE AT WWW.DSIREUSA.ORG/SUMMARYTABLES/RRPFE.CFM.

20. We understand that your [state legislature] and [regulators] have put in place statutes, rules, and regulations designed to facilitate the development of renewable energy facilities. FOR EACH OF THE LAWS OR REGULATIONS LISTED IN THE TABLE BELOW, ASK:

    1. Was your organization involved in advocating or providing technical support for the development or adoption of [LAW OR REGULATION]?

1. Yes

2. No

3. DK
   
   

2. Could you briefly describe the activites your organization undertook to support the development or adoption of this [law, rule, regulation]?

3. Was your organization funded by the U. S. Department of Energy’s State Energy Program or by ARRA funds received from the U. S. Department of Energy?

1. Yes

2. No

3. DK

ANSWER GRID FOR QUESTION 20

Law or Regulation	20a	20b	20c
Use of Public Benefits Funds to support renewable projects
Renewable Portfolio Standard
Net metering of renewable installations
Interconnection rules to facilitate renewables projects
Licensure of renewables installation contractors
Certification of renewable energy equipment
Access laws, such as provisions for solare easements
Building codes that support renewable energy installations
Mandates for utilities to off “green power” options

21. What other organizations played important roles in advocating or providing technical support for the development and adoption of these laws and regulations?

________________________________________________________________

________________________________________________________________

________________________________________________________________

22. FOR EACH LAW/REGULATION/RULE LISTED IN QUESTION 20 GRID ASK: In comparison to the work of other organizations, how would you compare the work your organization did to support [LAW, REGULATION, RULE] in terms of importance to the its ultimate development and adoption. Would you say it was:

    1. Much less important

    2. Somewhat less important

    3. Of about the same importance

    4. More important

    5. Much more important

    6. DK/Refused

23. FOR EACH RESPONSE a – e TO QUESTION 22, ASK:Why do you say that?

ANSWER GRID FOR QUESTIONS 22 AND 23

Law or Regulation	22	23
Use of Public Benefits Funds to support renewable projects
Renewable Portfolio Standard
Net metering of renewable installations
Interconnection rules to facilitate renewables projects
Licensure of renewables installation contractors
Certification of renewable energy equipment
Access laws, such as provisions for solare easements
Building codes that support renewable energy installations
Mandates for utilities to off “green power” options

PART 4. RESOURCE CONTRIBUTIONS FROM OTHER PROGRAMS

[ASK PART 4 IF OTHER ORGANIZATIONS SUCH AS UTILITIES OR OTHER GOVERNMENT COOPERATED IN THE DELIVERY OF THE PA THROUGH CONTRIBUTIONS OF STAFF TIME, FUNDING, OR OTHER SERVICES.]

LEAD-IN: Earlier we spoke about contributions that other organizations made to the non-residential renewable programmatic activity. I’d like to ask a few more questions about that.

24. First, prior to 2008, had you cooperated with this (these) organization(s) in delivering energy programs to increase energy efficiency programs?

IF YES, ASK

1. Please describe how you worked together with the other organization(s) and what specific activities they undertook.

2. What kinds of resources did the other organizations provide?

1. Did your organization take the initiative to arrange for cooperative activities related to the program? Did the other organizations take the initiative? Or did the cooperation arise through some other mechanism?

26. Did representatives of the two (or more) organizations meet to jointly plan how the organizations would cooperate in the development of the program?

27. Do representatives of the organization meet on a regular basis to review program accomplishments and plan future activities?

IF YES: How often does that occur?

28. Is there regular interaction among the organizations in delivering the program services?

IF YES: Could you describe that interaction for me?

29. What do you believe are the other organizations’ main motivations for contributing to the delivery of the program?
    
    
    
    

30. If the opportunity to offer joint programming with your agency had not been available, do you believe the other organizations would offer programs to promote energy efficiency in non-residential facilities?

YES/NO

1. Why do you say that?

1. If the other organizations had not cooperated in delivering the program as they did, would your organization have changed the type of services it provides to promote energy efficiency in non-residential facilities?

YES/NO

IF YES, PROBE:

1. How would your organization have changed the roster of services provided?
   
   
   

2. In the absence of cooperation from other organizations, would the level of resources your organization dedicated to these services been less, the same, or greater?

3. Why do you say that?

PART 5: SEP INFLUENCE AND ROLE IN THE MARKETPLACE

32. Do you believe that this program has had an impact on the long-term capabilities of facility owners and renewable energy developers to develop renewable energy projects in [STATE]?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?

1. Do you believe that this program has had an impact on the long-term capabilities of equipment vendors, designers, contractors, and other firms on the “supply side” of the market to sell and install energy-efficient renewable projects in non-residential facilities?

YES/NO

1. In what specific ways did your program contribute to that result?

2. Can you provide any examples of these kinds of results?
   
   
   

1. Did your organization add any positions to carry out or administer the program(s) we have been discussing? Please include positions that may be split between a number of different programmatic and administrative activities.
   
   IF YES:

   1. How many positions did your organization add?

   2. How many FTEs were added in each of the following job classifications: administrative, technical services, professional, management?

   3. How many of these FTEs will be retained after 2008 or the ARRA period, depending on which is relevant?

   4. IF c > 0: What sources of funding do you plan to use to sustain this (these) position(s)?

35. In the course of carrying out this program, has your organization developed any new capabilities or resources that will support the delivery of energy efficiency services to non-residential facilities in the future. PROBE:

    1. Contacts and relationships with market actors

    2. Tools such as site or financial analysis software

    3. Project financing facilities such as loan funds, loan loss reserve programs, etc.

    4. Technical education materials and curricula

PART 6: DATA AVAILABILITY

[ONLY ASK THIS SECTION IF DATA ARE NOT AVIALABLE ALREADY]

[ASK SECTION 4 FOR EACH OF THE PROGRAMS LISTED IN REPSONSE MATRIX A. IF NECESSARY, SCHEDULE A CALLBACK WITH THE APPROPRIATE PROGRAM MANAGER]

NOTE: QUESTIONS 36-38 ON DATA AVAILABILITY WILL BE REVIEWED WITH PROGRAM SPONSORS PRIOR TO FINAL SELECTION OF THE PROGRAM INTO THE SAMPLE. WE WILL ALSO ASK TO RECEIVE A COPY OF THE DATABASES OR FILE FORMATS SO THAT WE CAN MAKE A JUDGMENT REGARDING THE ADEQUACY OF THE DATABASE TO SUPPORT EVALUATION ANALYSIS. THESE QUESTIONS WILL BE FOR VALIDATION OF OUR REVIEW.

36. Please describe the types of data that you tracked in [PROGRAM YEAR] on program participants. Do you have lists of:

1. Key contacts from the implementation organization (including contact information)

2. Program participants, that is, facility owners/renters, non-residential property managers who received support with the project, workshop attendees or training recipients, recipients of technical assistance, with contact information

3. Measures installed by participant, including the type and quantity of measures installed with program support, energy savings, incentive levels, other services received by participants, workshop or training description. [PROBE TO CLARIFY DATA THAT THE PM HAS ACCESS TO (EITHER IN-HOUSE OR AT A SEP FUNDED/PARTNERED ORGANIZATION) VERSUS DATA ANY EXTERNAL LEVERAGED ORGANIZATION MIGHT BE KEEPING.]

4. Participating trade allies (including contact information)

(IF DATA IS AVAILABLE BUT RESPONDENT IS NOT KNOWLEDGEABLE OR DOES NOT HAVE ACCESS TO IT, RECORD NAME OF CONTACT WHO CAN ANSWER FOLLOW-UP QUESTIONS ABOUT DATA AVAILABILITY. PROBE FOR WHETHER ANYONE MAINTIANS SUCH DATA – POTENTIALLY EXTERNAL TO THE PA). IF THERE IS NO ONE CAN PROVIDE SUCH DATA THEN THANK AND TERMINATE).

37. Have there been any evaluations of this program? Are data from these evaluations available?

38. (FOR EACH TYPE OF DATA AVAILABLE) Is this data in electronic format? (If yes, discuss steps needed to secure permission to access the data via secure file transfer. If no, determine how data records are maintained and how they could be accessed.)

Are you aware of other data available that would be useful for our evaluation? If so could you describe it for me? Is it possible to access this information? Probe for contacts. If making copies is an issue, ask whether we can make copies. if still reluctant, ask whether they have a summary of renewable properties with minimal informtion so we can select a sample and copy only those records.]

Thank you for your time and insights

18

OMB Control No. XXXXXXXX