State Energy Program Evaluation

OMB Control No. 1910-XXXX

ID-14: Training and Technical Assistance (Program Delivery Contractors)

OMB Control No. 1910-XXXX

Burden Disclosure Statement

Public reporting burden for this collection of information is estimated to average 30 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Office of the Chief Information Officer, Records Management Division, IM-23, Paperwork Reduction Project (1910-XXXX), U.S. Department of Energy, 1000 Independence Ave SW, Washington, DC, 20585-1290; and to the Office of Management and Budget (OMB), OIRA, Paperwork Reduction Project (1910-XXXX), Washington, DC  20503.

This interview guide is designed to provide direction to the interviewer to ensure that all relevant topics are explored to the extent possible and appropriate with the respondent. Note that our interviews are meant to be somewhat informal and open ended – not all topics will be covered in all interviews and we expect that some interviews will lead to the exploration of topics not included in this guide.

Information in [BRACKETS] will be customized to reflect the unique PA program [PROGRAM] and year of the PA offering [PROGRAM YEAR].

BACKGROUND INFORMATION (to be filled in prior to interview):

Program Administrator Name:
Year:
BPAC Area:
2008 Budget:
2008 Market Title Sampled:
2009-2010 ARRA Budget:
2009-2010 ARRA Market Title Sampled:
Known programmatic activities prior to interview:
Description of target markets:
Structure of SEP/ARRA funded activities from informal discussions with SEP representative from first round discussions (from database):
Contact Name:
Contact Company:
Contact Phone:
Contact Disposition:

PART 1. INTRODUCTION

My name is [INTERVIEWER NAME] from [INTERVIEWER ORGANIZATION]. The US Department of Energy’s State Energy Program has hired us to talk with organizations involved with [TECHNICAL ASSISTANCE / TRAINING] programs that received funding from the State Energy Program and the American Recovery and Reinvestment Act. This interview is being conducted as part of an evaluation of the State Energy Program being conducted by Oak Ridge National Laboratory on behalf of the U. S. Department of Energy.

PART 2. IDENTIFY THE APPROPRIATE RESPONDENT(S)

Our understanding is that your organization was responsible for delivering [TECHNICAL ASSISTANCE/ TRAINING] program. Is that correct?

Yes: CONTINUE

No: PROBE FOR REASON FOR MISUNDERSTANDING. THANK, TERMINATE, AND CONSULT WITH STATE ENERGY OFFICE CONTACT

Our understanding is that you served or serve as the program manager for [TECHNICAL ASSISTANCE/ TRAINING] programs that received SEP/ARRA funding in [YEAR]? Is this correct:

Yes – Correct person and year for SEP/ARRA funded [TECHNICAL ASSISTANCE/ TRAINING] programs	[Continue]
Yes – Correct person, wrong year for SEP/ARRA funded [TECHNICAL ASSISTANCE/ TRAINING] programs	[Correct year and Continue]
No – Incorrect person for SEP/ARRA funded [TECHNICAL ASSISTANCE/ TRAINING] programs	[Ask for correct person for [TECHNICAL ASSISTANCE/ TRAINING] programs]
No – There were no SEP/ARRA funded [TECHNICAL ASSISTANCE/ TRAINING] programs	[Thank you and terminate. Contact main state energy office representative to verify that there were no programs that corresponded to description in the PAGE database.]

Your participation in this survey is voluntary, and your answers are confidential. The information from all respondents will be combined for analysis purposes and data will not be released in a way that would reveal an individual respondent. If you prefer not to answer a question, just let me know and we’ll go on to the next question. The length of the interview varies from person to person, but most interviews last about 60 minutes. For quality control purposes, this call will be recorded and monitored. If you have any questions about this study, you can contact [ORNL CONTACT INFORMATION].

I would like to ask you some questions about the [TECHNICAL ASSISTANCE/ TRAINING] programs activities to promote [RESIDENTIAL/RENEWABLE/NON-RESIDENTIAL RETROFIT] projects that received SEP/ARRA funding in [PROGRAM YEAR].

PRIOR TO THE INTERVIEW, REVIEW THE DELIVERY ORGANIZATION’S CONTRACT AND WEB SITE TO GATHER INFORMATION TO ADDRESS THE FOLLOWING QUESTIONS. POSE THEM TO THE RESPONDENT ONLY IF NEEDED FOR VERIFICATION OR TO FILL IN MISSING INFORMATION.

1. What are the primary services that your organization delivers?

________________________________________________________________

________________________________________________________________

2. Were you involved in the management of [TECHNICAL ASSISTANCE/ TRAINING] programs that received SEP/ARRA funding in [PROGRAM YEAR]?

1 Yes

2 No [IF NOT, COLLECT CONTACT NAME AND TELEPHONE NUMBER. IF NO ONE AT ORGANIZATION IS KNOWLEDGEABLE ABOUT THE PROGRAM, STATE DETAILED REASON. CONTINUE THE INTERVIEW WITH THAT INDIVIDUAL.]

3. What were your responsibilities in regard to [TECHNICAL ASSISTANCE/ TRAINING] programs that received SEP/ARRA funding in [PROGRAM YEAR]?

4. In what year did you first become involved with the [TECHNICAL ASSISTANCE/ TRAINING] programs programs that received SEP ARRA funding?

5. Are you still involved with managing this program? If not, when did your involvement end?

6. According to information provided by the State Energy Office, you received a contract valued a roughly $__________ to operate programs that promote the development of [RESIDENTIAL/RENEWABLE/NON-RESIDENTIAL RETROFIT] projects. Is this correct?

Listed Amount	Corrected Amount
[AMOUNT ]_______	[AMOUNT ]
	[REMAINING AMOUNT]

7. Did you spend the total amount budgeted? If not, why wasn’t the budget spent?

PART 3: PROGRAM DESIGN (LOGIC MODEL)

8. As you understand them, what were the ultimate objectives of [TECHNICAL ASSISTANCE/ TRAINING] program(s)?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

9. Did your organization commit to any quantitative goals for the [TECHNICAL ASSISTANCE/ TRAINING] program(s), such as number of projects developed, students taught, number of classes or projects, kWh or kW of capacity installed?

   1. Yes

   2. No

   3. DK
      
      
      

10. IF 10 = YES: What were those goals?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

11. In terms of the timeline of your contract:

    1. When did you initiate development of the [TECHNICAL ASSISTANCE/ TRAINING] program(s)?

    2. When did you launch the [TECHNICAL ASSISTANCE/ TRAINING] program(s) to the public?

12. Please describe how the [TECHNICAL ASSISTANCE/ TRAINING] program(s) operated in [PROGRAM YEAR] in regard to its activities, the resources it uses, and the outcomes it is achieving.

    1. Let’s start with ACTIVITIES. PROBE:

MARKETING AND OUTREACH TO PARTICIPANTS

RECRUITMENT OF PARTICIPANTS

RECRUITMENT OF TRADE ALLIES AND OTHER DELIVERY PARTNERS

DELIVERY OF INFORMATION AND TECHNICAL SERVICES TO [TARGET AUDIENCE, E.G. OWNERS, HOUSEHOLDS, MARKET ACTORS]

DELIVERY OF INFORMATIO AND TECHNICAL TRAINING TO VENDORS AND INSTALLERS, IF APPLICABLE

PROVISION OF TECHNICAL AND PROJECT DEVELOPMENT SERVICES, SUCH AS SITE ASSESSMENTS, ENGINEERING SUPPORT, FEASIBILITY STUDIES, FINANCIAL MODELING, ETC.

[ASK IF RENEWABLES] ADVOCACY AND TECHNICAL SUPPORT FOR LAWS AND REGULATIONS TO SUPPORT RENEWABLES PROJECT DEVELOPMENT, SUCH AS RENEWABLE PORTFOLIO STANDARDS, INTERCONNECTION RULES, NET METERING.

PROVISION OF FINANCIAL SUPPORT: INCENTIVES, LOANS, LOAN GUARANTEES TO SUPPORT PROJECTS

INSPECTION, QUALITY CONTROL OF INSTALLED PROJECTS

OTHER (SPECIFY)

2. INPUTS: What resources are being used to deliver the [TECHNICAL ASSISTANCE/ TRAINING] program(s)? PROBE:

STATE ENERGY OFFICE STAFF: NUMBER, TYPE, PERCENT OF TIME

OTHER GOVERNMENT STAFF: NUMBER, TYPE, PERCENT OF TIME

STAFF OF OTHER AGENCIES OR ORGANIZATIONS: NUMBER, TYPE, PERCENT OF TIME

CONTRACTED SERVICES: PROBE NAME OF PROVIDER AND TYPE OF SERVICE, APPROXIMATE NUMBER OF CONTRACTOR STAFF, DOLLAR SIZE OF CONTRACT

TECHNICAL SERVICES

FINANCIAL SERVICES

MARKETING SERVICES

ADMINISTRATIVE SERVICES

3. OUTSIDE FUNDING: Are organizations other than the State Energy Office contributing funding to this [TECHNICAL ASSISTANCE/ TRAINING] program(s)? IF YES, PROBE:

NAME OF THE ORGANIZATION(S)

AMOUNT OF FUNDING PROVIDED IN THE PERIOD UNDER EVALUATION (2008 OR ARRA PERIOD, FOR EACH FUNDER IF AVAILABLE)

PERCENTAGE OF TOTAL FUNDING FOR THE PA UNDER EVALUATION IN 2007, 2008, ARRA PERIOD, AS RELEVANT.

USES OF THE FUNDING PROVIDED

4. RESULTS/OUTPUTS: Could you summarize the results your [TECHNICAL ASSISTANCE/ TRAINING] program(s) has achieved so far? PROBE:

NUMBER OF PARTICIPANTS ENROLLED

DESCRIPTION OF PARTICIPANTS (E.G. BACKGROUND)

NUMBER OF UNITS OF SERVICE DELIVERED (TECHNICAL SERVICES, LOANS AND GRANTS, DOLLAR VOLUME OF LOANS AND GRANTS)

NUMBER OF TEACHERS TRAINED

NUMBER OF TEACHERS OFFERING TRAINING

NUMBER OF TECHNICAL ASSISTANCE PROVIDERS

NUMBER OF TA PROVIDERS OFFERING TA

NUMBER OF INSTALLATION PROJECTS SUPPORTED

KW INSTALLED

ESTIMATED SAVINGS FROM INSTALLATIONS SUPPORTED

ANYTHING ELSE?

5. RESULTS VERSUS GOALS: Were you able to (Do you anticipate being able to) achieve the level of results you intended? (Refer to program plans or quantitative goals)

IF NO: What circumstances prevented you (are preventing you) from achieving those goals?

13. [IF PROGRAM EXISTED PRIOR TO 2008 ASK] Did you make any important changes to [TECHNICAL ASSISTANCE/ TRAINING] program(s) operations or the resources available to it between 2007 and 2008? IF PROGRAM EXISTED PRIOR TO ARRA AND RECEIVED ARRA FUNDING ALSO ASK] Did you make any important changes to program operations or the resources available to it between 2008 and the period funded primarily by ARRA?

    1. Why did you make those changes?

    2. Did achieve the results you were seeking through those changes? PROBE SPECIFIC RESULTS.

3. FOR ARRA PERIOD PROGRAMS ASK. Would your organization have been able to make these changes without the support provided by the ARRA round of funding?

YES/NO

PROBE REASONS FOR THIS ASSESSMENT.

[THESE QUESTIONS ARE DESIGNED TO FLESH OUT PROGRAM DETAILS TO DEVELOP THE PROGRAM LOGIC MODEL. THE LOGIC MODEL ARTICULATES WHAT THE PROGRAM IS TRYING TO ACHIEVE (AND DID ACHIEVE), THROUGH WHAT INTERVENTIONS, AND WITH RESPECT TO WHICH MARKET ACTORS. THIS INFORMATION WILL BE USED TO DEVELOP A MAP OF THE PROGRAM’S DOMAIN FOR ATTRIBUTION PURPOSES.]

14. Has your organization or the state energy office developed a formal logic model for [PROGRAM NAME]?

    1. Can you share it with us?

PART 4 – ASSESSMENT OF PROGRAM EFFECT

This next set of questions seeks your opinion on the effect of the program on targeted facilities, homeowners and vendors.

15. What types of changes are being made / have occurred as a result of this intervention?

    1. ENERGY SAVINGS (e.g. kWh, kW)

    2. NUMBER OF BUILDINGS/HOUSEHOLDS WITH INSTALLATION/CHANGE IN PRACTICES

    3. SKILL SETS (e.g. operations and maintenance, new markets, new EE measures, etc).

    4. JOBS

    5. PROFESSIONAL CERTIFICATION

    6. OTHER: SPECIFY

16. IF THE PROGRAM SUPPORTED INSTALLATIONS IN INDIVIDUAL FACILITIES, ASK: Now, thinking about the [FACILITY OWNERS/HOUSEHOLDS/MARKET ACTORS] who received support from your program for installation of [RESIDENTIAL/RENEWABLE/NON-RESIDENTIAL RETROFIT] projects -- On a scale of 1 to 10, where 1 means “not at all important” and 10 means “very important”, how important were the services provided [TECHNICAL ASSISTANCE/ TRAINING] program(s) in convincing and enabling those participants to undertake the [RESIDENTIAL/RENEWABLE/NON-RESIDENTIAL RETROFIT] projects?

ENTER 1 – 10, 99 FOR DK/REF: ________________________________

17. IF 9 = 1 – 10, ASK: Why do you say that? PROBE SPECIFIC BARRIERS OR ISSUES THE SERVICES FROM THE SEP PA HELPED TO ADDRESS, HELP FROM OTHER PROGRAMS, PREDISPOSITION OF THE [FACILITY OWNERS/HOUSEHOLDS/MARKET ACTORS].

_______________________________________________________________

________________________________________________________________

________________________________________________________________

18. In your opinion, how many of these projects would [FACILITY OWNERS/ HOUSEHOLDS/MARKET ACTORS] have completed in the absence of the support from the [TECHNICAL ASSISTANCE/ TRAINING] program(s)?

ENTER HOW MANY (probe for percentage or qualitative statement): _______________________

19. What observations lead you to that estimate?

_______________________________________________________________

________________________________________________________________

________________________________________________________________

20. Generally speaking, has the annual number of [RESIDENTIAL/RENEWABLE/NON-RESIDENTIAL RETROFIT] projects completed in [STATE] increased, decreased, or stayed about the same since [YEAR OF PROGRAM LAUNCH]?

21. IF 11 = INCREASED OR DECREASED: By approximately what percentage would you say the number of non-non-residential renewable projects has increased/decreased?

ENTER PERCENT, 999 FOR DK/REF:________________________

PART 4: SEP INFLUENCE AND ROLE IN THE MARKETPLACE

22. Would your organization have provided [TECHNICAL ASSISTANCE/ TRAINING] in the absence of SEP funding?

    1. How many [courses/classes/trainings/assistance] would have been offered in absence of the program? ENTER PERCENTAGE AMOUNT: %

23. Are there any other organizations/entities who provide similar provided [TECHNICAL ASSISTANCE/ TRAINING] offerings? (If yes, ask for name of entities and probe to determine similarity of offerings]

24. In your opinion, what did the [TECHNICAL ASSISTANCE/ TRAINING] program(s) accomplish?

25. Do you believe that this [TECHNICAL ASSISTANCE/ TRAINING] program(s) has had an impact on the long-term demand exercised by [FACILITY OWNERS/ HOUSEHOLDS] [RESIDENTIAL/RENEWABLE/NON-RESIDENTIAL RETROFIT] projects in [STATE]?

YES/NO

1. In what specific ways did your [TECHNICAL ASSISTANCE/ TRAINING] program contribute to that result?

2. Can you provide any examples of these kinds of results?

1. Do you believe that this program has had an impact on the long-term capabilities of equipment vendors, designers, contractors, and other firms on the “supply side” of the market to sell and install [RESIDENTIAL RETROFIT /RENEWABLE ENERGY /NON-RESIDENTIAL RETROFIT] projects in [FACILITIES / HOMES]?

YES/NO

1. In what specific ways did your [TECHNICAL ASSISTANCE/ TRAINING] program contribute to that result?

2. Can you provide any examples of these kinds of results?
   
   
   

1. Did your organization add any positions to carry out or administer the program(s) we have been discussing? Please include positions that may be split between a number of different programmatic and administrative activities.
   
   IF YES:

   1. How many positions did your organization add?

   2. How many FTEs were added in each of the following job classifications: administrative, technical services, professional, management?

   3. How many of these FTEs were retained after 2008 or the ARRA period, depending on which is relevant?

   4. IF c > 0: What sources of funding do you plan to use to sustain this (these) position(s)?

PART 5: DATA AVAILABILITY

[ONLY ASK THIS SECTION IF WE HAVE NOT BEEN ABLE TO ACQUIRE PROGRAM DATA FROM THE STATE ENERGY OFFICE]

NOTE: QUESTIONS 23-25 ON DATA AVAILABILITY WILL BE REVIEWED WITH PROGRAM SPONSORS PRIOR TO FINAL SELECTION OF THE PROGRAM INTO THE SAMPLE. WE WILL ALSO ASK TO RECEIVE A COPY OF THE DATABASES OR FILE FORMATS SO THAT WE CAN MAKE A JUDGMENT REGARDING THE ADEQUACY OF THE DATABASE TO SUPPORT EVALUATION ANALYSIS. THESE QUESTIONS WILL BE FOR VALIDATION OF OUR REVIEW.

28. Please describe the types of data that you tracked in [PROGRAM YEAR] on program participants. Do you have lists of:

1. Program participants, that is, workshop attendees or training recipients, recipients of technical assistance, with contact information

2. Services received by participants, workshop or training description

3. Curriculum

4. Participating trade allies (including contact information)

5. Training Materials

6. Contacts at partnering organizations (including contact information)

7. Program reports (e.g. status reports, quarterly, reports, etc)

1. Measures installed by participant, including the type and quantity of measures installed with program support, energy savings, incentive levels, other services received by participants, workshop or training description. [PROBE TO CLARIFY DATA THAT THE PM HAS ACCESS TO (EITHER IN-HOUSE OR AT A SEP FUNDED/PARTNERED ORGANIZATION) VERSUS DATA ANY EXTERNAL LEVERAGED ORGANIZATION MIGHT BE KEEPING.]

(IF DATA ARE AVAILABLE BUT RESPONDENT IS NOT KNOWLEDGEABLE OR DOES NOT HAVE ACCESS TO IT, RECORD NAME OF CONTACT WHO CAN ANSWER FOLLOW-UP QUESTIONS ABOUT DATA AVAILABILITY. PROBE FOR WHETHER ANYONE MAINTIANS SUCH DATA – POTENTIALLY EXTERNAL TO THE PA). IF THERE IS NO ONE CAN PROVIDE SUCHA DATA THEN THANK AND TERMINATE).

29. Have there been any evaluations of this program? Are data from these evaluations available?

30. (FOR EACH TYPE OF DATA AVAILABLE) Are these data in electronic format? (If yes, discuss steps needed to secure permission to access the data via secure file transfer. If no, determine how data records are maintained and how they could be accessed.)

Are you aware of other data available that would be useful for our evaluation? If so could you describe it for me? Is it possible to access this information? Probe for contacts. If making copies is an issue, ask whether we can make copies. if still reluctanct, ask whether they have a summary of PROJECTS with minimal INFORMATION so we can select a sample and copy only those records.]

Thank you for your time and insights

10

OMB Control No. 1910-XXXX