Vol. I IES Modernization Interviews and Surveys Phase 2

Volume I

Supporting Statement

IES Modernization Interviews and Surveys – Phase 2

OMB# 1880-0542

June 2020

Contents

1 Submittal-Related Information 1

2 Background and Study Rationale 1

3 Recruitment and Data Collection 1

4 Consultations outside The Agency 3

5 Justification for Sensitive Questions 3

6 Paying Respondents 4

7 Assurance of Confidentiality 4

8 Estimate of Hourly Burden 4

9 Cost to Federal Government 5

10 Project Schedule 5

Attachment 1 – IES Modernization Interviews and Surveys – Phase 2 Communication Materials

Attachment 2 – IES Modernization Interviews and Surveys – Phase 2 Interview Guides and Surveys

1. Submittal-Related Information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB# 1880-0542), which provides for the Department to conduct surveys and other studies regarding customer satisfaction.

The goal of this research is to use interviews, focus groups and surveys to gain insights into current audiences of the Institute of Education Sciences (IES) and how IES might better serve them. The results will be used to refine the understanding of key IES user groups and determine what information is most useful to them so we can better achieve the mission of IES.

2. Background and Study Rationale

The Institute of Education Sciences (IES) is the nation's premier source for research, evaluation and statistics for educators, policymakers, and stakeholders seeking to improve outcomes for all students. IES supports a diverse but critical set of services for the nation's educators and researchers, including acting as an independent non-partisan curator of the survey and performance data of the Department of Education as well as directly funding rigorous testing of new approaches to improving education outcomes. IES’s mission is to provide national leadership to expand the fundamental knowledge and understanding of (A) the condition and progress of education in the United States; (B) educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and (C) the effectiveness of federal and other education programs.

IES users are as diverse as the services offered by the Institute, which presents a particular challenge when attempting to deliver world-class experiences through IES digital channels. IES is comprised of four centers, many of which operate and maintain different data tools, content retrieval systems, intranets, and other digital properties that are often siloed according to center and—in many cases—program. Currently, IES’s website is built according to its organizational chart and doesn’t reflect customer/user experience research and mission-driven, agile development practices. Previous attempts to redesign the digital experience have been based solely on internal input, without the benefit of external user research and insight. Internal resources are scarce and in high demand, making it even more challenging than usual to distribute innovation across dozens of vendor and contractor relationships.

The IES Customer Experience effort addresses these challenges through a user-first understanding of IES. A previous phase of this research was approved by OMB in March 2020 (OMB# 1880-0542), and the associated interviews and focus groups were completed in March and April of 2020. This work can be considered the second phase of that research and uses findings from those interviews to further define our user groups and their needs, which informed the respondent types as well as survey and interview questions for this study. The goal of this research effort is to employ interviews and surveys to gain insights into current audiences and how IES might better serve them. IES and contracting Grant Thornton/McKinsey research team have provided a pool of potential research participants. This project will conduct individual interviews and focus groups with those targeted participants. The project will also distribute a survey to the targeted group and the targeted group’s networks. The results of the research will be used to understand the needs, pain points, preferences and opportunity areas within key IES user groups so we can better achieve the mission of IES. We will also use these findings to inform future user research; the questions we ask now may inform future surveys and interviews, which will be used to evaluate potential solutions for IES to serve these key user groups.

3. Recruitment and Data Collection

Leadership at IES and the research team have together compiled a list of key user groups – local education leaders, state education leaders, institute of higher education leaders, researchers, associations, foundations, thinktanks, media and educators. The list was created by IES leadership, their team, and the research team based on previous research. We will use this list to distribute the survey to additional users in these segments. We will also use two 3^rd party recruitment services – Schlesinger for surveys and interviews and Coleman for interviews, to recruit any additional participants we cannot attain through our list. The maximum number of potential respondents we will contact is 2479.

The projected groups of participants are defined by their relationships to IES, and are as follows:

• Policy makers, consumers of IES data and targets in the IES mission statement (e.g., State Educational Agencies (SEAs), Local Education Agencies (LEAs), Institute of Higher Education Leaders (IHEs))

• Researchers, consumers of IES data, grant recipients, and targets in the IES mission statement (e.g., education PhDs, IES grant applicants)

• Associations, Foundations, Thinktanks – translators of IES data to educators, SEAs, LEAs, IHEs (e.g., Teacher, superintendent, and other collective representation at the national level; funders and evaluators of education research)

• Media – translators of IES data to the general public (e.g., EdWeek leadership and staff, via introductions from IES leadership)

• Educators, targets in the IES mission statement (e.g., classroom teachers and higher education faculty)

The research team will determine which users should be placed in focus groups versus individual interviews based on quotas and respondent availability. There will be 5 available interview times for each user group – policy makers (local education leaders, state education leaders, institute of higher education leaders), researchers, channels (associations, foundations, thinktanks), media and educators. The remainder will be scheduled in focus groups. Any respondents who are recruited by Coleman (who are focusing on recruiting respondents from groups who are usually more difficult to recruit) will automatically be scheduled for 1:1 interviews so the researchers can get the very most from those interviews.

Recruitment

Under a contract to IES, The Grant Thornton Team (Grant Thornton and subcontractor, McKinsey & Company) developed the interview and focus group guides and surveys and will be responsible for the overall management of the interviews, focus groups and survey administration.

All participants will be recruited based on the pre-identified user segments - local education leaders, state education leaders, institute of higher education leaders, researchers, associations, foundations, thinktanks, media and educators – by speaking with those user groups, through social media posts, through recommendations by IES and previous interviewees, and through third party services for both surveys and interviews (e.g., teachers of Fairfax County Facebook Group, Association Twitter).

The research team will receive warm introductions from the IES leadership or, for pre-existing relationships from a previous set of interviews, conduct direct outreach via email (see Attachment 1) based on a list created by IES leadership and previous interviewees. As we conduct interviews and focus groups, the interviewees may also assist in identifying additional users within the segments via email (see Attachment 1). After initial email introductions, the research team will follow up to communicate details of the study and interview session. (Please see Attachment 1 for full communication materials).

Two 3^rd party research agencies will be used to recruit participants. Both agencies have pre-existing relationships with interviewees, who have opted in to receiving communication from the research agencies to participate in potential interviews and surveys. For interviews, we will use Coleman Research Group, an expert network service that McKinsey has used in previous research efforts. Coleman will use the email outreach (see Attachment 1) to contact their network of participants. Schlesinger will be used to recruit survey participants. They will be contacted via email (see Attachment 1). (Please see Attachment 1 for full communication materials).

Social Media will be used to recruit educators based on a list of Facebook, Twitter, and other social media groups recommended by IES leadership and previous interviewees. These will be posted by either IES or an interviewee. This will be decided based on who has a pre-existing relationship with the social media group (e.g., if an Association has a Facebook page, they will post the request on social media) (Please see Attachment 1 for social media sample posts)

The recruitment and data collection procedure will ensure that:

• all participating staff are made aware of all steps and procedures, the overall objectives, protocols and restrictions, and a review of the recruitment and data collection materials;

• during outreach and data collection, participants will be informed of the objectives and purpose of the data collection effort;

• participants are made aware that this is a voluntary effort and provide express consent for any audio or video recording, or photography (see Attachment 1 for interviews and focus groups & Attachment 2 for surveys); and

• relevant participants receive a thank you for their participation and an email reminder to confirm their participation and to field any questions they may have regarding their participation (see Attachments 1 & 2).

Data Collection

The surveys will be administered via the Qualtrics platform and should take no longer than 30 minutes to complete.

The interviews and focus groups will be conducted via phone or video conference. Video conference is preferred, but we will accept phone if video is not available. Participants will participate in sessions with an interviewer and notetaker. For focus groups, there will be a maximum of 2 facilitators and a notetaker. Interviews will last 30-60 minutes, while focus groups will last 60 minutes. Interviews will also include an integration with a prioritization tool, Optimal Sort, which allows us to collect user input on how a website could be organized and usertesting.com to collect feedback on prototypes (see Attachment 2).

The full interview and focus group guides and surveys can be found in Attachment 2. In general, participants will be asked questions that fall into these categories:

1. What are the qualities, characteristics, examples of preferred tools and channels for research?

2. What IES products do they use (includes specific reasons for use or lack of use)?

3. How satisfied are they with IES (across multiple measures)?

4. What are their pain points and why?

5. What potential future IES focus areas resonate with them most and why?

4. Consultations outside the agency

These interviews and focus groups will be administered under a contract to IES, The Grant Thornton Team (Grant Thornton and subcontractor, McKinsey & Company). Qualtrics will be used to host the survey and collect survey data. Coleman Search Group and Schlesinger group will be used to recruit additional interviewees and survey participants, respectively.

5. Justification for Sensitive Questions

Throughout the user survey development processes, effort has been made to avoid asking for information that would be sensitive or offensive. Further, respondents are reminded that participation is voluntary and that any responses can be refused at any time in the process.

6. Paying Respondents

The McKinsey-Grant Thornton team is not offering incentives for this study. The 3rd party recruiting services may choose to pay their participants as part of their contract with their panels. Schlesinger Group will pay a maximum of $20 and Coleman Research Group will pay a maximum of $1000. Coleman will only be used as needed for hard to reach participants for an in-depth interview, such as certain state superintendents. This will incur no additional cost to the federal government.

7. Assurance of Confidentiality

The data collected will be used for research analysis internal to IES and its contractors and will not be distributed publicly. All participants will be made aware of the research intent and we will ask explicit permission to record, photograph, or video. Respondents will be notified that all the information they provide may be used for research purposes only and may not be disclosed. The assurances they are given and the consent forms they are asked to sign are in Attachment 1.

8. Estimate of Hourly burden

The total hours below include actual interview, focus group and survey times. Recruitment will take, on average, 5 minutes.

Interviews and Focus Groups

Due to the nature of the interview and focus group introductions, through direct connections, we anticipate high levels of participation. For those recruited through Coleman Research Group, we also anticipate high levels of participation because of the activity levels of their network.

Surveys

With surveys sent through warm introductions, we estimate an 80% response rate due to the number of direct IES contacts. Social media will be used primarily for educator survey outreach and we estimate a 20% response rate due to the targeted outreach. For surveys sent through Schlesinger Group, we estimate a 20% response rate due to the activity levels of their network. These proportions guide how we estimate the number of responses from the total pool of potential respondents in Table 1 below.

Table 1. Estimate of Hourly Burden

	No. of respondents	No. of responses	Participation Time	Burden Hours
Total Recruitment	2479	891	5 min	207
Surveys (subtotal)	790	790	-	395
Policy Makers	200	200	30 min	100
Researchers	200	200	30 min	100
Association leaders	40	40	30 min	20
Educators	350	350	30 min	175
Interviews and Focus Groups (subtotal)	116	116	-	116
Policy Makers	44	44	1 hour	44
Researchers	22	22	1 hour	22
Associations, Foundations, Thinktanks	20	20	1 hour	20
Media	10	10	1 hour	10
Educators	20	20	1 hour	20
Totals	2479	1797		718

Italicized rows provide more detail and should not be considered as duplicative of rows providing subtotals. All respondents are part of the total number recruited through email or social media, as indicated in the recruitment numbers.

9. Cost to federal government

For this study there is a $0 incremental cost because it is a part of a larger IES Customer Experience effort and will be incurred by the McKinsey-Grant Thornton Team.

10. Project Schedule

The below timeline is based on an estimate of 1-2 weeks to recruit participants followed by 2 – 6 weeks of interviews, focus groups and surveys

Table 2: Estimated Schedule

Activity	Dates
Recruitment and administration of interviews, focus groups and surveys	May 2020 – June 2020
Study summary and synthesis	June 2020 – July 2020