STEM Maker Ed Support Part B Revised 2017-10-13

Section B. Description of Statistical Methodology

B. 1. Respondent Universe

The Maker/STEM Education Support for 21^st Century Community Learning Centers (CCLC) program model consists of six museum/science center partners, each partnering with approximately five 21^st CCLC sites for an approximate total of 30 21^st CCLC sites. Each 21^st CCLC site expects to serve approximately 15 upper elementary and/or middle school youth participants with the Making activities for an approximate total of 450 youth participants.

In order to understand variety in implementation and the experiences of museum/science center partners, 21^st CCLC facilitators, and the 21^st CCLC youth participating in program, formative evaluation data will be collected from participants at all levels of the program model via the following methodologies: Museum/science center partners (interviews); 21^st CCLC facilitators (surveys and interviews); and 21^st CCLC youth participants (interviews; table 1).

Table 1: Data collection summary

Participant group	Data collection method	Sample size
Museum staff	Interviews	6 (one per museum/science center partner, all partners)
21st CCLC facilitators	Online survey	30 (one per site, all sites)
	Interviews	12 (one per site for the 12 selected sites)
21st CCLC youth	Interviews	48 (four per site for the 12 selected sites)

The results of this evaluation are intended to benefit the museum field as well as the public. As a formative evaluation, NYSCI and IMLS are interested in documenting the implementation of the program model and gaining a general understanding of the experiences of all museum/science center partners, all 21^st CCLC facilitators involved in the program, and a sample of the youth participants. To gather this information, the following two data collection methods will be employed:

Museum/science center partner interviews

At the conclusion of the 21^st CCLC site implementations, the evaluation team will conduct interviews with the lead staff member from each of the six museum/science centers (n=6). Each interview will last no longer than 60 minutes. Using a semi-structured protocol, interviews will gather feedback on the museum/science center partner training offered by NYSCI; the trainings the museums/science center provided for their 21^st CCLC facilitators; the museum/science center staff role in implementation at 21^st CCLC sites; the opportunities and challenges faced during each component of the program model; and suggestions for improvement.

21^st CCLC facilitator survey

Twenty-first CCLC facilitators responsible for co-implementing the Making activities at all 30 participating sites (n=30) will be invited to complete an online survey through Qualtrics © survey software or via a paper survey if requested. The survey will take approximately 15 minutes to complete. We believe since the first ten of the fifteen close-ended questions ask for basic background information or details on the program implementation, based on our experience with this group and prior surveys, the survey can be completed in this amount of time.

The survey will consist of 15 close-ended questions and six open-ended questions. The evaluation team will use the survey to gather broad feedback regarding facilitators’ training experiences and implementation experiences and about facilitator suggestions for improving the program model in the future.

As the program model is designed to allow for variation in implementation of the Making program, NYSCI and IMLS also are interested in gaining additional in-depth understanding of the variations that occur within the program model (e.g., variations in 21^st CCLC facilitator trainings provided by the museum/science center partners, the variations in 21^st CCLC site implementations, and the resulting experiences of youth based on the variations in training and site implementation; see Part A for the initial description of the program model). The short implementation timeframe (summer 2017) and the total number of 21^st CCLC sites (approximately 30), however, make it prohibitive to collect in-depth information on all 30 implementing sites.

Therefore, the evaluation team has selected a purposive sampling procedure, called maximum variation sampling (Schwandt, 2007), to gather in-depth information on the program model across the various 21^st CCLC implementations. For the purposes of this formative evaluation, maximum variation sampling will allow the evaluation team to select “cases” or sites based on the diversity of a range of specific characteristics in order to ensure the data collected explores the maximum variability in the implementation of the program model. A maximum variation sampling procedure will allow the evaluation team to gather additional in-depth feedback from a range of 21^st CCLC facilitators and youth participating in the program. A purposive sample of 12 of the approximately 30 participating 21^st CCLC sites (i.e., approximately two sites per museum/science center partner) will be selected for in-depth investigation using the following five characteristics:

• Assigned museum/science center partner

• Prior experience implementing Making programs at the 21^st CCLC site

• Format of the training provided by the museum/science center partner to the 21^st CCLC facilitators (e.g., one full day training, trainings just prior to activity implementation)

• Format/dosage of the 21^st CCLC site implementation (e.g., one activity per session, multiple activities per session)

• Age of youth participating at the 21^st CCLC site

Once the sample of sites is determined, the following formative data will be collected only from the 12 identified sites:

21^st CCLC facilitator interviews

One 21^st CCLC facilitator from each of the 12 identified sites (n=12) will be contacted for an in-depth interview. Interviews will last no longer than 45 minutes each. Using a twenty-three question semi-structured protocol (see attachment for facilitator interview protocol), interviews will gather in-depth information regarding the training 21^st CCLC facilitators received from the museum/science center partner, the context of implementation of the activities at their site, their feedback on being a part of the program, and their challenges and suggestions for improvement.

21^st CCLC youth participant interviews

Approximately four youth participants per the same 12 identified sites (n=48) will be invited to participate in an in-person interview. Written consent will first be requested and obtained from their parents/guardians before the youth interviews can be commenced. In consultation with the museum/science center partners and 21^st CCLC facilitators, youth will then be selected for interviews based on the following criteria: 1) returned, signed parent/guardian consent form; 2) distribution of age; 3) distribution of gender; and 4) range of participation dosage in the program. Interviews will be administered by a member of the evaluation team during the final scheduled 21^st CCLC program activity. The interview will take approximately 10 minutes depending on the age of participant (expected range from upper elementary to middle school grades). Using a nine-question semi-structured protocol (see attachment for youth interview protocol), the evaluation team will use the interviews to gather information on youth experiences and interest in the program as well as youth participants’ suggestions for improvement.

B.2. Procedures for Data Collection

The following data collection procedures will be implemented:

Museum/science center partner interviews

Once all 21^st CCLC sites have completed implementations, a lead staff member at each of the museum/science center partners will be invited via email to set up a time to participate in a

60 minute interview. Interviews will be scheduled at a time that is convenient for the staff member during working hours. Interviews will be conducted in-person or via telephone depending on the availability of the staff member. The staff member will be read a consent to participate statement at the start of the session and be asked to record his/her verbal consent prior to beginning the interview. Upon verbal consent, interview data will be audio recorded and uploaded and securely stored once each interview is complete. Once the interview is transcribed by a professional transcription service, the audio data will be destroyed.

21^st CCLC facilitator survey

At the conclusion of their 21^st CCLC site implementations, the 21^st CCLC facilitators responsible for co-implementing the Making activities in each of the approximately 30 participating sites (n=30 facilitators) will be invited to complete an online survey through Qualtrics © survey software. The survey will take approximately 15 minutes to complete. Museum/science center partners will inform the facilitators that they will be contacted by the evaluation team for their feedback via online survey. Facilitators then will be contacted through an initial email invitation that includes an individualized link to the survey. Facilitators will also be instructed that they can request a paper version of the survey by replying to the email. A paper-based survey and self-addressed stamped envelope then will be sent to the requesting facilitator. Paper surveys will be entered into the system by project staff. Whether completing an online or paper survey, facilitators will be reminded a maximum of two times via email to complete the survey. The facilitator will complete a consent to participate within the survey software or on the paper version prior to beginning the survey. If he or she does not provide consent, he or she will not be asked to complete the survey.

21^st CCLC facilitator interviews

At the conclusion of implementation at their site, 12 21^st CCLC facilitators will be contacted via email to schedule a 45 minute follow-up interview. Interviews will last no longer than 45 minutes each. Interviews will be scheduled at a time that is convenient for the facilitator during working hours. Interviews will be conducted in-person or via telephone depending on the availability of the facilitator. The facilitator will be read a consent to participate statement at the start of the session and be asked to record his/her verbal consent prior to beginning the interview. Upon receiving verbal consent, interview data will be audio recorded and uploaded and securely stored once each interview is complete. Once the interview is transcribed by a professional transcription service, the audio data will be destroyed.

21 CCLC youth participant interviews

After implementation of the final Making activities, a select number of youth (approximately 4 per each of the twelve sites with more details about their selection described above in B.1) will participate in a ten minute, in-person interview administered by a member of the evaluation team. Prior to visiting any 21^st CCLC site, the evaluation team will work with NYSCI, the museum/science center partners and the 21^st CCLC staff to secure permission for youth to participate in interviews regarding their experiences in the program. Youth of parents/guardians who have not provided consent for their child to participate in an interview will not be offered the opportunity to participate in an interview session. Interview participants will be selected based on consents received, age, and gender. Once consents have been collected and potential interviewees identified by museum/science center partner staff, 21^st CCLC staff and the evaluation team, interview data will be audio recorded. Each recording will be uploaded and securely stored once each interview is complete. Once the interview is transcribed by a professional transcription service, the audio data will be destroyed.

B.3. Response Rates and Non-Responses

For the 21^st CCLC facilitator surveys, the evaluation team estimates a response rate of at least 85% of the total participating facilitator population. For 21^st CCLC facilitator interviews and interviews with staff at the six museum/science partners, the evaluation team estimates a 100% response rate. For the youth interviews, the evaluation team expects to collect enough parent/guardian consents in order to meet the goal of four interviews per site selection. Participation in all surveys and interviews will be strictly voluntary.

Based on the proposed plan and the evaluation team’s past experience with data collection related to Making programs and 21^st CCLC sites, we do not expect non-response to be an issue for this formative evaluation study. To encourage participation, NYSCI and EDC will provide an overview of the evaluation plan to all museum/science center partners during the museum/science center partner training and will work closely with the museum/science center staff and the staff at 21^st CCLC sites to promote participation of the youth interviews and facilitator surveys. Additional incentives for survey completion or interview participation will not be provided.

B.4. Tests of Procedures and Methods

Survey instruments and interview protocols for this project have been adapted based on instruments developed for a previous Making-related program evaluation (Goodyear & Greller, 2015). Additionally, the evaluation team is working closely with a team of project advisors from EDC, NYSCI, and IMLS who are familiar with quantitative and qualitative evaluation methodologies, and with procedures for collecting data from upper elementary and middle school youth and within museum/science center and 21^st CCLC settings.

B.5. Contact Information for Statistical or Design Consultants

Senior Advisor: Leslie Goodyear, Principal Research Scientist, EDC

Project Lead: Erin Stafford, Research Associate II, EDC

Expert Consultant: Sarah Ryan, Senior Research Associate, EDC

Federal Contact: Christopher J. Reich, Chief Administrator, Office of Museum Services, Institute of Museum and Library Services

Federal Contact: Matthew Birnbaum, Supervising Social Scientist, Office of Impact Assessment and Learning, Institute of Museum and Library Services

3