Evaluation of the Veteran Business Outreach Centers

(OMB Control Number: 3245 - XXXX )

Appendix D-1 Site Visit Invite E-mail, Follow-up E-mail, and Consent Form, Director Interview Consent Form, Protocol, and Construct Justification

Site Visit Invite E-mail

Dear [VBOC Director’s name],

Greetings from the U.S. Small Business Administration (SBA)’s Office of Veterans Business Development (OVBD), and thank you for all the work that you have completed for the Veterans Business Outreach Center (VBOC) program. SBA is conducting a performance evaluation of the VBOCs as we develop plans for improving the program going forward, and for assessing its impact on the business related goals of clients, veterans, and their families served by VBOCs.

In order to offer a fair and independent study, OVBD has contracted Optimal Solutions Group, LLC (Optimal) to complete the evaluation. As part of this assessment, Optimal is conducting site visits at selected VBOCs. Your VBOC has been selected as one of the sites that Optimal would like to visit. During the site visit, Optimal would like to conduct interviews with you, your staff, and a few of your clients. Up to four staff members are invited to participate in an interview. If you consent, Optimal will follow up to identify four “typical” clients to invite for an interview. These site visits will provide Optimal with the in-depth qualitative, process-oriented information that will assist in the interpretation of quantitative data, the implementation of performance metrics, and the overall performance assessment of the VBOC program. During the site visits, Optimal aims to collect information on VBOC client demographics, client goals and challenges, services provided to clients, referral network, intake protocol, reporting protocols, client feedback mechanisms, client progress, client attitudes about VBOC services received, and VBOC best practices, lessons learned, and challenges.

Directors are in no way required to participate in the site visit, but are encouraged to do so because OVBD values your input as well as the opinions of your staff and clients. The information attributed to any respondent during interviews will not be reported in any reports provided to SBA without their explicit permission. Optimal’s reporting of the data collected from the site visits will use phrases such as, “A director/staff member/client explained that…” or “A center reported that…” to ensure anonymity.

The site visit will last no more than 2 days in early December, and Optimal will be flexible about scheduling meetings with you, your staff, and your clients during the 2 days. If you are willing to host Optimal for a site visit of your VBOC, please read, sign, and return the attached site visit consent form to the following e-mail address: [email protected]. Optimal will follow-up with you to arrange the days and times for the visit.

Thank you again for your participation and work!

Respectfully,

Barbara Carson

On behalf of the SBA Office of Veterans Business Development

Follow-up E-mail from Optimal to VBOC Director

Dear [VBOC director’s name],

Thank you for agreeing to host a site visit as part of the U.S. Small Business Administrations (SBA)’s Office of Veterans Business Development (OVBD)’s performance evaluation of the Veterans Business Outreach Program. OVBD has contracted Optimal Solutions Group, LLC (Optimal) to conduct the site visits. Please choose a 2-day window during the weeks listed below where you will be available to host the Optimal researcher who will be conducting the site visit:

December 7-11, 2015 and December 14-18, 2015

During the site visit, Optimal would like to conduct interviews with you (95 minutes), your staff (65 minutes each), and a few of your clients (40 minutes each). Interviews with staff will be arranged on-site; please provide the names of the current staff. Optimal is also requesting interviews with four VBOC clients who received services since April 2015, and is asking directors to help identify clients to be interviewed. Optimal is seeking to interview two clients who have received counseling services (one pre-venture client and one in-business client) and two clients who received training services only (one client from each of the two most highly attended training sessions). Each client should be a “typical” client in terms of needs and interactions with the VBOC. Please provide the names and contact information for the clients to be interviewed, and Optimal will arrange the interview times.

Attached below you will find copies of the director, staff, and client interview consent form, and the interview protocols. An Optimal researcher will collect the consent forms on site before the interviews begin; there is no need to send the consent forms back to Optimal ahead of time. Please let me know if you have any questions; I am available via phone (301-306-1170) or e-mail ([email protected]).

Thank you and we look forward to speaking with you soon.

Respectfully,

Allison Holmes, Ph.D.

Optimal Solutions Group, LLC (Optimal)

Site Visit Consent Form

The U.S. Small Business Administration (SBA)’s Office of Veterans Business Development (OVBD) has contracted Optimal Solutions Group, LLC (Optimal) to conduct a fair and independent performance evaluation of Veterans Business Outreach Centers (VBOCs). This evaluation comprises a landscape and location gap analysis of VBOCs, the development of performance metrics, an assessment of client attitudes about the impact of VBOC services, and the development of a framework that will allow for the implementation of impact evaluations.

As part of this analysis, Optimal is conducting site visits at selected VBOCs. These site visits will provide Optimal with the in-depth qualitative, process-oriented data that will assist in the landscape and gap analysis, the development of performance metrics, and the interpretation of quantitative data. During the site visits, Optimal aims to compile VBOC client demographics, services provided, referral network, intake protocol, reporting protocols, client feedback mechanisms, and VBOC best practices, lessons learned, and challenges.

Confidentiality and Usage

Site visits will be used in part to develop a publically available report for SBA at the completion of the project. Any quotations or information “on-the-record” will be attributed to the original source in the reports. The research team will provide the option of going “off-the-record” if requested by the interviewee, in which case the information in reports submitted to SBA will not be attributed to a person or center. Moreover, the name or region of the VBOC will not be identified, but instead be referred to as “a VBOC.” Written notes and audio recordings of interviews will be taken to document site visit findings; separate consent forms will be completed for each interview participant. Directors, staff, or clients being interviewed may decline to answer any particular interview question and they may also discontinue their interview at any time if they wish to do so. As required by the Paperwork Reduction Act, SBA (OVBD) may not conduct this site visit unless it has been approved by the Office of Management and Budget (OMB). SBA has obtained that approval under OMB control #3245-xxxx, Expiration date xx/ xx/ xxxx. The total estimated burden for directors, staff, and clients is 75.8 hours.

Storage and Access

The written notes and audio recordings will be stored on a secure server accessible only to the Optimal research team and will not be provided to SBA staff. The information will be retained for no less than 3 years after the completion of this program evaluation.

Your signature on this form grants the Optimal research team permission to conduct the site visit as described above for the above-referenced study. The research team will not use the information provided for any reason other than those stated in this consent form without your permission.

Signature Date

Printed Name

Interview Consent Form for Directors

The U.S. Small Business Administration (SBA)’s Office of Veterans Business Development (OVBD) has contracted Optimal Solutions Group, LLC (Optimal) to conduct a fair and independent performance evaluation of Veterans Business Outreach Centers (VBOCs). This evaluation comprises a landscape and location gap analysis of VBOCs, the development of performance metrics, an assessment of client attitudes about the impact of VBOC services, and the development of a framework that will allow for the implementation of impact evaluations.

As part of this analysis, the Optimal research team will be conducting individual interviews with several VBOC directors. The primary purpose of these interviews is to collect information from directors on the processes used for the implementation of the grant, as well as the successes and challenges they have experienced while running their centers. Other topics covered during the interviews include information about your role as a director, staff capacity, intake and reporting protocols, client characteristics, services provided, client tracking, and referral network.

Confidentiality and Usage

This interview will be documented through written notes and an audio recording, which will be used in part to develop an internal SBA report and a publically available report for SBA at the completion of the project. Any quotations or information “on-the-record” will be attributed to the original source in the reports. The research team will provide the option of going “off-the-record” if requested by the interviewee, in which case the information in reports submitted to SBA will not be attributed to a person or center. Instead, the information will allude to the respondent’s information through phrases such as “a VBOC director” or a “VBOC center.” As required by the Paperwork Reduction Act, SBA (OVBD) may not conduct this interview unless it has been approved by the Office of Management and Budget (OMB). SBA has obtained that approval under OMB control #3245-xxxx, Expiration date xx/ xx/ xxxx. The estimated burden is 95 minutes.

Storage and Access

The interview notes and recordings will be stored on a secure server accessible only to the Optimal research team, and will not be provided to SBA staff. The interview notes and recordings will be retained for no less than 3 years after the completion of the interviews.

Your signature on this form grants the Optimal research team permission to conduct the interview as described above for the above-referenced study. The research team will not use the information provided for any reason other than those stated in this consent form without your permission.

Signature Date

Printed Name

Director Interview Protocol

General Information about the Interview

Interview date and time: Eastern Standard Time

Interview Questions

The interview questions below are provided for your review in preparation for the [VBOC’s] upcoming director interview. The questions are asked about you and your organization’s work at the VBOC. Please note that some of these questions may be skipped during the interview because they are not applicable to your program or because additional follow-up questions not listed here may be asked based on the information provided during the interview.

Notation:

• [F] indicates a follow-up question.

VBOC Resources

1. What drew you to working at a VBOC?

2. Please describe how you selected any consultants (e.g., experts to provide one-time training), subcontractors, or strategic alliances?

3. Please describe the standardized training for the staff.

[F] How do you ensure that staff delivers consistent, quality services?

What you do

Overall program approach

1. To the best of your knowledge, are there similar business assistance programs in your service region?

[F-Yes] To the best of your knowledge, please give examples of these other programs (e.g., organization names, location).

[F-Yes] To the best of your knowledge, do you consider these similar business assistance programs to be alternatives or complements to your VBOC?

2. What makes your VBOC unique compared to these other business assistance programs?

3. How do you address and include military culture in the program?

Program outreach

1. In the web survey, you outlined your outreach and marketing strategy [INTERVIEWER REFERENCE RESPONSES]. How effective has this strategy been?

[F] Have you considered increasing or expanding efforts with your most common sources that generate new clients? Why or why not?

[F] Have any of the results of your strategies surprised you? How so? Can you give me a specific example?

[F] Is there anything that SBA could do to assist your outreach efforts?

Program services

1. In the web survey, you outlined the services and delivery formats your VBOC offers [INTERVIEWER REFERENCE RESPONSES]. Is there anything about the service utilization that has surprised you (e.g., high utilization of topics/modes)?

[F] Have you considered adjusting your services based on the utilization (e.g., extra courses of highly utilized training topic)? Why or why not?

[F] Do you find that you have different utilization of services based on location? Can you give me a specific example?

Client intake

1. Do you notice any patterns in the flow of new clients? (For instance, consider times of year or activities at the local military installation when there is a surge of new clients.)

2. Please describe any formal or informal screening processes used by your VBOC for providing client services.

3. Please describe client in-take procedures.

[F] Do you use the counseling questions (in WebCATS/Neoserra) for each client?

[F] Do you use the training questions (in WebCATS/Neoserra) for each client?

[F] Do you record client-level information (e.g., name, address, e-mail, business stage, etc.) for clients attending training sessions?

[F] Do you use any other forms to record data about clients?

[F] How would you rate your staff’s compliance in completing these forms?

[F] Briefly describe any challenges that staff face in completing these forms.

[F] How could your intake procedure be improved?

4. Now I’d like to discuss the general process used by your VBOC for assessing a client’s needs and providing appropriate service. Specifically:

[F] What questions are asked of the client? Are the questions standard across staff and client?

[F] How are decisions made about assistance to provide to the client? Is a standard decision tree used by staff for all clients?

[F] If there is a standard process for this assistance process, is it consistently implemented by staff?

[F] If a client asks for a particular service or kind of assistance but staff feel that it is not necessary or sufficient to meet the client’s need, will staff recommend a different service?

[F] How often do you re-assess a client’s needs or goals?

Referrals and community connections

1. In the web survey, you described the types of collaborators your VBOC works with [INTERVIEWER REFERENCES RESPONSES]. What are some lessons you have learned establishing your collaborator list and referring clients out? Consider the experiences of your staff and clients.

[F] How could the referral process be improved?

[F] Is there anything that SBA could do to facilitate the process?

Clients

1. Are there certain business industries that are frequently represented among your clients?

2. Approximately what percentage of clients receives multiple types of services (e.g., counseling and training)?

[F] Is service enhanced when training clients come for an individual counseling session or counseling clients come to a training session?

[F] Do you find that certain services or combinations of services have better effects on clients’ outcomes?

3. Approximately what percentage of clients receives multiple assistance sessions (e.g., attends multiple trainings, receives multiple counseling sessions, etc.)?

[F] Are there any trends in types of clients who receive multiple sessions (e.g., business status, general business knowledge, industry)?

Closing

Is there anything else we should know about your VBOC?

Thank you for your participation!

Director Interview Construct Justification

Construct	Justification
VBOC Resources	These questions assess staff expertise and capacity, and catalog the skills of other contractors or strategic alliances that the VBOC uses. OVBD is collecting this information to assess the staffing needs and resource allocation of each center.
Overall Program Approach	These questions ask directors if they have found that their program is distinct from other business assistance programs in their region, and whether one of those core distinctions is having knowledge of military culture and using this knowledge throughout its operations. OVBD would like to continue to distinguish VBOCs from other business assistance programs, and would want to confirm with directors that the models used by VBOCs and clients served do not overlap with others.
Program Outreach	These questions assess the modes and avenues of VBOC program outreach, including geographic focus, industries, and veteran sub-groups. By asking these questions, OVBD can determine if VBOCs are serving a representative sample of the veteran population and if they are reaching clients throughout their designated region.
Program Services	This section asks directors about common services, modes of delivery, and special initiatives developed by the VBOC. Depending on staff expertise and client need, services may vary in availability and frequency across VBOCs. OVBD would like to know what specialized initiatives have been developed by each VBOC to match unique client needs and the method used for developing these initiatives.
Client Intake	These questions pertain to how new clients are entered into the client database and how their initial needs are assessed. There is no standard client intake process across VBOCs, so each VBOC may have its own standard operating procedure that each staff member uses when assessing clients' needs. OVBD wants to know each center’s strategy in order to offer suggestions for improvement and share best practices across VBOCs.
Referrals and Community Connections	OVBD would like to have a picture of the resources and ongoing partnerships that VBOCs have developed in order to assess whether or not there are limitations in resources either in services offered or with referral networks. In order for VBOCs to succeed, they have to have relationships with business assistance programs and SBA resource partners across their geographic span.
Client Characteristics	OVBD would like to have a general idea of client trends, preferences, and needs in order to determine what resources are needed and where they should be concentrated.
Client Tracking and Feedback	VBOCs are required to submit their WebCATS/Neoserra data and reports to OVBD on a quarterly basis. Other than that requirement, each VBOC can independently determine how they collect client feedback and track client outcomes. OVBD would like to know what tracking and feedback methods are used by VBOCs in order to make recommendations for the kinds of information collected, and on how to collect accurate and timely information.
Moving Forward	This portion allows directors to provide OVBD with suggestions for how the VBOC program can be improved moving forward.