Site Visit Interview Logistics & Protocol

SITE VISIT AND PHONE INTERVIEW PROTOCOL AND LOGISTICS

Logistics: The Big Picture

Purpose of the Site Visits and Phone Interviews

The site visits and phone interviews will allow us to learn how the DEI Initiative unfolds in real settings, as well as whether and how DEI sites vary from comparison sites. The ultimate goal of the DEI is to improve the employment outcomes of people with disabilities by using seven targeted strategies to facilitate system-wide change. Site visits and phone interviews are our means for gauging the extent to which systems change is occurring. Your Role

You will be working under the guidance of the State Evaluation Liaison to gather information relevant to the DEI project goals. Each team member has unique experiences with and knowledge of the DEI, and this makes you an invaluable member of your site visit or phone interview team. You must remain objective, nonjudgmental, and neutral throughout the project. Furthermore, you must try to view each site (state/LWIA/ American Job Center) as a separate entity.

Site Visit Activities & Specs

 7 states (R2) will receive site visits.

 Visits are approximately 1 week per state.

 The approved question protocol will guide the process.

 Activities can include:

o Interviews (Group and Individual);

^{o Observations; and}

^{o Meetings.}

Site Visit Teamwork

Site visits (SV) will be conducted by teams of 3–6 people. Each SV team will include, at minimum, the State Evaluation Liaison, a BPA Lead, and an additional research staff member who will act as participant note-taker. This ensures state-specific expertise, workforce and disability expertise, and the ability to focus on the SV while thorough notes are recorded. SV personnel are as follows:

DEI State Evaluation Liaison	BPA Leads	Research Staff/Participant Note-Takers	Other Senior Staff Available
Doug Klayman (CA, WI)	Kay Magill	Kristin Bard	Anne Chamberlain (AK, AR, ME)
Bob Bleimann (WA, HI)	Linda Toms Barker	Kelley Akiya	Kim Barlow (IL, KS, NJ)
Becca Jackson (SD, OH, TN)	Sherry Almandsmith	Paola Rochabrun Oliveira	Marilia Mochel (DE, NY, VA)
		Hannah Zangwill	Laura Skaff
		Ofelia Cabrera
		Halima Ahmadi
		Danielle Herbert

Each SV will begin with all researchers meeting together on the first day to interview key state-level personnel who work with DEI. Subsequent days will be spent visiting (on average) 1 LWIA/2 American Job Centers per day. Typically, on each day, researchers will interview LWIB members, American Job Center managers and staff, and additional key partners or employers. In states with several LWIAs involved in DEI (CA, TN, OH), research teams will split into pairs to conduct observations and interviews at more LWIAs.

Phone Interview Activities & Specs

 9 states (R1) will participate in phone interviews.

 Phone interviews will follow up with some participants from last year’s Getting to Know You Visits and will also connect researchers with stakeholders from DEI and comparison sites that we were not able to visit last year.

 We hope to complete each state’s interviews within a 2-week window per state. While we will have a “window” for each state’s phone interviews, this will be somewhat flexible and will be determined independently by interview participants.

Phone Interview Teamwork

 Phone interviews (PI) will be conducted by a team of 2–3 people. In general, PIs at the state or

LWIB level will include the State Evaluation Liaison, a BPA Lead, and a participant note-taker.

Pre-Site Visit or Phone Interview

Tracking Our Availability

 We will use Google Calendar to display everyone’s availability during the SV window. Window

TBD, depending on OMB clearance. It is critical that you keep this schedule up-to-date.

Communicating with State Personnel

 The State Evaluation Liaison will begin discussing the SV or PI with the DEI State Lead in early February. This will involve an initial explanatory e-mail (Purpose, What to Expect) and at least 1 conference call Q & A. Multiple Q & A calls may be necessary if the State Lead determines that this would benefit (i.e., set at ease) local site personnel.

 As soon as possible, states will be given the dates of our SV/PI “window” (8–10 weeks) during which we will complete all SVs and PIs. States will work with BPA to schedule interviews. The sooner they settle on a date, the more options they will have.

Reviewing 2011 Notes

 Notes from 2011 Getting to Know You Visits will NOT be included in the state information binder, as they are confidential. For this reason, please thoroughly review both the summaries and the notes.

 During this review, record any specific follow-up questions that you have.

 If you need to access 2011 notes during the SV, you may do so using the limited-access dropbox files. Do not print out notes while travelling, as they are confidential.

Binders (See Sample)

 Each state has a “binder” that should be considered a living document. That is, it will be augmented on a continual basis as we learn more about the state. For SVs, each researcher participating on the visit will have a binder. For PIs, researchers at a single location (i.e., Altarum, BPA, SD) can share a binder.

 Your state binder will be ready at least 2 weeks prior to your SV/PI.

2

 Binders content includes:

o Face sheet outlining basic grant information including grantee, grant type (adult/youth), which of the 7 strategies are being employed, grant amount;

o Names, addresses, and maps for all DEI sites;

^{o Confidentiality Protocol;}

^{o State annual WIA report;}

^{o LWIA or American Job Center newsletters, when available;}

^{o Glossary of federal and state-specific terms and acronyms;}

^{o Site Visit/Phone Interview Questions and Probes; and}

^{o Sample Note Format.}

 Well before a SV or PI, research teams should review all binder content: State-level, LWIA-level,

and American Job Center-level. Tab, highlight, or make notes to remind yourself of questions specific to what you have reviewed.

 For those who are technologically inclined, you may wish to use OneNote for your binder. This would allow you to access all binder content from a tablet or laptop. If you wish to pilot this option:

o Hard-copy binders will still need to be available for any team members who do not wish to use OneNote.

o Any adjustments/additions made to the “binder” in OneNote will need to be made to hard copy versions, as well.

o A demonstration of OneNote will be arranged in the coming week for anyone interested.

Pre-Visit/Pre-Interview Meetings

 Prior to each SV or series of PIs, the research team for that state will meet to discuss the agenda, any special considerations, and impressions from reading the material in the binder. Research team members must read the binder contents and 2011 notes (where applicable) prior to this meeting.

 The lead interviewer(s) should talk to the note taker(s) about their interviewing style, and any expectations or preferences that they have (for example, is jumping in with questions ok?).

 Before each state’s visit or interview, the SV/PI team will have a conference call meeting with that state’s ETA regional representatives. This meeting will serve as an opportunity to inform ETA personnel of the SV or PI agenda, as well as to learn their perspective on state progress made in working with customers with disability.

Privacy

 Prior to SVs or PIs, it is critical that all team members are aware of the privacy protocols for this project:

o Between respondents: Project Leads, Disability Resource Coordinators, site contact people, etc., do not participate in interviews with other respondents. Site visitors do not repeat something said in one interview to another respondent in another interview.

o Outside site visit: We do not discuss either the information obtained or our opinions of it with people outside of the research team.

o In the report: In order to honor our promise of confidentiality to each interviewed respondent, we do not attribute observations and comments to specific individuals nor reference their names, titles, or organizational affiliations in the written report.

3

During the Site Visit or Phone Interview

The Basics

 Arrive 5 minutes early. Do NOT keep respondents waiting.

 Dress appropriately (business attire) on SVs.

Stay Neutral

Being perceived as biased in any way can jeopardize your ability to gather important information. It might be difficult to stop yourself from making comparisons or judgments, but you must try to be aware when you are doing so and not allow your judgment to show.

 Resist the temptation to describe your own program or experiences when on a SV/PI. This can artificially “shape” the interview.

 Use neutral responses. Do not express criticism, surprise, approval, or disapproval. For instance,

say “I see,” rather than “That’s great!” or “That sounds very difficult,” instead of “How awful!”

 Maintain neutral facial expressions and body language (SV), even if the respondent says something you find disturbing, shocking, or exciting.

Introductions

At the beginning of each SV/PI, the team and the DEI Evaluation should be adequately introduced. In most cases, the State Evaluation Liaison will provide introductions. In case you are asked to lead an interview, you should be prepared to do so.

 Each team member should introduce him/herself by name, title, and affiliation.

 Explain the purpose of DEI Evaluation as a whole, including its general purpose and how the

^{SV/PI fits into the overall study.}

 Explain the purpose of this SV/PI. Tell respondents that we are trying to understand how people with different roles in American Job Centers, LWIBs, and state offices work with customers with disabilities.

 Tell the respondent that notes will be taken during the SV/PI.

 Describe how information will be used. Tell the respondent that we will take the information obtained from all documents and interviews, write a report on our findings, and distribute this report to DOL. Be sure to tell respondents that they will receive a summary of the interview for their review.

 Clearly explain confidentiality policies: No discussion of the SV/PI with anyone outside the research team, maintaining confidentiality among respondents, and not attributing comments to any individuals, either by title, position, or name.

SAMPLE INTRODUCTION:

Hi, my name is . I am from [SD/BPA/Altarum] and a member of the DEI Evaluation Team. Other members of the team are [allow each team member to introduce themselves]. We are meeting with you as part of the DEI Evaluation. The interview should last about minutes/hour(s). We are also meeting with other staff, etc.

The overall goal of these interviews is to gather information that will help us understand your experience working with customers with disabilities and [for DEI sites] implementing DEI strategies. The result of our visit will be a summary report on our findings.

Responses to this data collection are voluntary and will be used only for statistical purposes. Reports prepared for this study will summarize findings across the sample and will not associate responses with a specific firm/establishment/state/district or individual. We will not provide information that identifies you or your firm/establishment/state/district to anyone outside the study team, except as required by law.

The Paperwork Reduction Act of 1995 provides that persons cannot be required to respond to a collection of information unless such collection displays a valid Office of Management and Budget Control Number. The Control Number for the DEI Evaluation is 1230-XXXX, and it expires XX/XX/XXXX. Your response is estimated to take about 5 minutes. Send comments regarding the burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to the U.S. Department of Labor, Office of Disability Employment Policy, 200 Constitution Ave., Room S-1303, Washington DC, 20210.

Before we begin, do you have any questions about the purpose of the study or our privacy policy?

Interview Structure

 Tell the respondent the range of topics to be covered, giving a general overview of the structure of the interview: “First, let’s talk about your role, and then I want to get some history about the program. After that, we can cover program operations, policies, successes, and challenges.”

 Apologize in advance for items that may seem redundant –they do help us understand.

 Be prepared with some strategies for refocusing the conversation if it goes off track. Develop a technique for politely interrupting someone if they become long-winded or stray off the topic:

^{o “I want to make sure I ask you about … before we finish up.”}

^{o “Can I ask you to hold on to that thought?”}

^{o “Before we go there, I’d like to back up/I wonder if you could explain/tell me more….”}

Using the DEI Site Visit/Phone Interview Questions

 The DEI SV/PI questions do not need to be followed verbatim, but all questions/topics must be addressed. Check the list of questions frequently throughout the SV/PI process to ensure that you are covering all topics. You may wish to mark any questions/topics that are missed as the interview is being conducted.

 At the end of each interview, there will be an opportunity to ask follow-up questions. This is an appropriate time to ask any missed questions.

Questions On- and Off-the-List

 Ask questions in a friendly and nonthreatening manner.

 Try to ask “how” and “what” questions—these make people feel less defensive than “why” questions. For example, instead of “Why did you do that?” ask “How did you handle that?” or “How does that work?” or “What do you do when that happens?”

 Acknowledge the challenges described by respondents: “That sounds quite complicated/difficult. How did you handle that?”

 Try to get the respondent to distinguish between ideas, theories, or plans, and the reality of how the program is actually operating. For example, “Where does that plan stand right now?” or “Is that process fully implemented, or are you still working on that?”

 Use neutral probes to dig deeper and avoid generalities:

o “Could you give me an example?”

^{o “Could you be more specific about that?”}

^{o “What else do you think about that?”}

^{o “Tell me more.”}

^{o “Now how does that really work?”}

^{o “Could you give me an example of that?”}

^{o “You’ve given me a general sense of..., but I’d like to hear a bit more detail about…to be}

sure I understand.”

Active Listening

 Show interest in what the respondent is saying. Make eye contact. Nod. Encourage the respondent (“okay” or “tell me more” or “yes”).

 Test your understanding of what a person is saying by summarizing and reflecting back your version of what he/she has just said. Ask, “Did I understand that correctly?”

Completing the Interview

End the interview on a positive note by planning ahead for a general, big picture question that allows the respondent to tell you something exciting, positive, or of special interest to him/her.

 Ask a big picture question, such as, “On a final note, what would you say was the most successful experience….important contribution…biggest concern related to your experience with the DEI program?”

 Obtain permission. After a SV/PI, there are often items that need clarification. Ask permission to phone the respondent if needed in the future.

 Exchange business cards during a SV. This ensures the correct spelling of the respondent’s

^{name, accurate title, and current contact information.}

 Express your gratitude for the time s/he has taken to meet with you and answer your questions.

Note-Taking

 Participant note-takers have the lead responsibility for recording notes from SVs and PIs;

however, all team members are encouraged to make notes.

 You do not need to write down everything the respondent says, but try to write down important pieces of information.

 Be careful not to only take notes when what is being said is disturbing or shocking. This can cause the respondent to feel more cautious about openly answering questions.

 Please try to take notes as legibly as possible.

 As soon as possible after the interview, review your notes, finish incomplete sentences, clarify confusing thoughts, and complete anything you failed to write down.

Post-Site Visit or Phone Interview

Notes and Summaries

 Whenever possible, the team should debrief immediately following a visit or interview. This often happens as the team drives to the next locale. During this debrief, someone needs to take notes. What were the general impressions? Do these vary across the group?

 As soon as possible, the participant note-taker should begin typing up his/her notes. These should be detailed and include your own thoughts (use italics and your initials to show when something is your opinion).

 Completed (typed) notes are due to be shared with the SV/PI team within 1 week of completing the SV/PI.

 All team members must read through the notes and provide the note-taker with feedback in

Track Changes within 1 week of receiving them.

 Final notes, incorporating everyone’s feedback, must be completed by the 2.5 week mark.

 Once the first version of the notes has been circulated, the State Evaluation Liaison will begin working on a summary of the visit. This is due to ODEP within 30 days of the SV/PI. The format for the summary is in the State Binder.

Task	Due Date
Distribute typed-up notes to SV/PI team	1 week after SV/PIs completed
Provide feedback to note-taker	2 weeks after SV/PIs completed
Final version of notes completed	2.5 weeks after SV/PIs completed
Distribute draft state summary to team	3 weeks after SV/PIs completed
Provide feedback on the summary	ASAP

6

Site Visit and Phone Interview Protocol and Logistics

Summary to ODEP 4 weeks after SV/PIs completed

Team Debriefing

Within a week of each state’s SV or PI, the State Evaluation Liaison will organize a one hour conference call to debrief. This time will be used to discuss each team member’s overall impression of the site and to discuss any particular aspects of the SV/PI. You can reference your notes during this meeting.

Sharing Information with the States (Verification)

Within 4 weeks of a SV or state PI, we will share the state summary with the DEI State Lead. This is an opportunity for him/her to share additional ideas, correct misconceptions, answer follow-up questions, etc. The summary shared with the state will be the same summary shared with ODEP. Recipients at ODEP will be informed that they are receiving an unverified report.

In addition to sharing summaries with the State Leads, notes from each interview (not including comments or opinions expressed by the research team) will be shared with the interviewee for the same verification purposes.

Some interviewees may ask you for feedback on the visit.

 Remind respondents that they will receive a copy of our notes for comment.

 Verbal feedback should be limited to an expression of our appreciation for the program’s assistance with the DEI Evaluation and the participation of the interviewed respondents. We are not there to judge or provide feedback on the level of compliance or specific findings.

 Answer questions such as “Did we pass the test?” or “How did I do?” with a neutral response, such as “The interviews went very well,” or “I greatly appreciate your cooperation in discussing these topics.” If pushed, explain that it is too early to present findings since the team has not yet had a chance to process all the information gathered during your SV/PI. You may redirect people to the State Evaluation Liaison, who can explain our feedback policy.

Annual Synthesis Report

At the conclusion of the grant year, the Evaluation Team must submit an annual synthesis report to ODEP. This report summarizes everything that has been learned to date on the DEI Evaluation from both R1 and R2 sites. Writing this report will be a team effort. A timeline allowing for a few internal drafts will be distributed at the beginning of the summer.

In addition to the information from the notes and summaries for each state, the synthesis report will include a “model” for each state. This will be a baseline picture of how the current American Job Center system is operating in that state. Subsequent models will be created in the following grant years in order to map systems change.

7