Vol. I - IES Modernization P2

Volume I

Supporting Statement

IES Modernization Interviews and Surveys – Phase 2

OMB# 1880-0542

April 2020

Contents

1 Submittal-Related Information 1

2 Background and Study Rationale 1

3 Recruitment and Data Collection 1

4 Consultations outside The Agency 3

5 Justification for Sensitive Questions 3

6 Paying Respondents 3

7 Assurance of Confidentiality 3

8 Estimate of Hourly Burden 4

9 Cost to Federal Government 5

10 Project Schedule 5

Attachment 1 – IES Modernization Interviews and Surveys Communication Materials

Attachment 2 – IES Modernization Interview Guides and Surveys

1. Submittal-Related Information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB# 1880-0542), which provides for the Department to conduct surveys and other studies regarding customer satisfaction.

The goal of this research is to use interviews and surveys to gain insights into current audiences of the Institute of Education Sciences (IES) and how IES might better serve them. The results will be used to refine the understanding of key IES user groups and determine what information is most useful to them so we can better achieve the mission of IES.

2. Background and Study Rationale

The Institute of Education Sciences (IES) is the nation's premier source for research, evaluation and statistics for educators, policymakers, and stakeholders seeking to improve outcomes for all students. IES supports a diverse but critical set of services for the nation's educators and researchers, including acting as an independent non-partisan curator of the survey and performance data of the Department of Education as well as directly funding rigorous testing of new approaches to improving education outcomes. IES’s mission is to provide national leadership to expand the fundamental knowledge and understanding of (A) the condition and progress of education in the United States; (B) educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and (C) the effectiveness of federal and other education programs.

IES users are as diverse as the services offered by the Institute, which presents a particular challenge when attempting to deliver world-class experiences through IES digital channels. IES is comprised of four centers, many of which operate and maintain different data tools, content retrieval systems, intranets, and other digital properties that are often siloed according to center and—in many cases—program. Currently, IES’s website is built according to its organizational chart and doesn’t reflect customer/user experience research and mission-driven, agile development practices. Previous attempts to redesign the digital experience have been based solely on internal input, without the benefit of external user research and insight. Internal resources are scarce and in high demand, making it even more challenging than usual to distribute innovation across dozens of vendor and contractor relationships.

The IES Customer Experience effort addresses these challenges through a user-first understanding of IES. A previous phase of this research was approved by OMB in March 2020 (OMB# 1880-0542), and the associated interviews and focus groups were completed in March and April of 2020. This work can be considered the second phase of that research and uses findings from those interviews to further define our user groups and their needs, which informed the respondent types as well as survey and interview questions for this study. The goal of this research effort is to employ interviews and surveys to gain insights into current audiences and how IES might better serve them. IES and contracting Grant Thornton/McKinsey research team have provided a pool of potential research participants. This project will conduct individual interviews with those targeted participants. The project will also distribute a survey to the targeted group and the targeted group’s networks. The results of the research will be used to understand the needs, pain points, preferences and opportunity areas within key IES user groups so we can better achieve the mission of IES. We will also use these findings to inform future user research; the questions we ask now may inform future surveys and interviews, which will be used to evaluate potential solutions for IES to serve these key user groups.

3. Recruitment and Data Collection

Leadership at IES and the research team have together compiled a list of key user groups – local education leaders, state education leaders, institute of higher education leaders, researchers, associations, foundations, thinktanks, media and educators. The list was created by IES leadership, their team, and the research team based on previous research. We will use this list to distribute the survey to additional users in these segments. The maximum number of potential respondents we will contact is 2479.

The projected groups of participants are defined by their relationships to IES, and are as follows:

• Policy makers, consumers of IES data and targets in the IES mission statement (e.g., State Educational Agencies (SEAs), Local Education Agencies (LEAs), Institute of Higher Education Leaders (IHEs))

• Researchers, consumers of IES data, grant recipients, and targets in the IES mission statement (e.g., education PhDs, IES grant applicants)

• Associations, Foundations, Thinktanks – translators of IES data to educators, SEAs, LEAs, IHEs (e.g., Teacher, superintendent, and other collective representation at the national level; funders and evaluators of education research)

• Media – translators of IES data to the general public (e.g., EdWeek leadership and staff, via introductions from IES leadership)

• Educators, targets in the IES mission statement (e.g., classroom teachers and higher education faculty)

Recruitment

Under a contract to IES, The Grant Thornton Team (Grant Thornton and subcontractor, McKinsey & Company) developed the interview guides and surveys and will be responsible for the overall management of the interviews and survey administration.

All participants will be recruited based on the pre-identified user segments - local education leaders, state education leaders, institute of higher education leaders, researchers, associations, foundations, thinktanks, media and educators – by speaking with those user groups, through social media posts, through recommendations by IES and previous interviewees, for both surveys and interviews (e.g., teachers of Fairfax County Facebook Group, Association Twitter).

The research team will receive warm introductions from the IES leadership or, for pre-existing relationships from a previous set of interviews, conduct direct outreach via email (see Attachment 1) based on a list created by IES leadership and previous interviewees. As we conduct interviews, the interviewees may also assist in identifying additional users within the segments via email (see Attachment 1). After initial email introductions, the research team will follow up to communicate details of the study and interview session. (Please see Attachment 1 for full communication materials).

Social Media will be used to recruit educators based on a list of Facebook, Twitter, and other social media groups recommended by IES leadership and previous interviewees. These will be posted by either IES or an interviewee. This will be decided based on who has a pre-existing relationship with the social media group (e.g., if an Association has a Facebook page, they will post the request on social media) (Please see Attachment 1 for social media sample posts)

The recruitment and data collection procedure will ensure that:

• all participating staff are made aware of all steps and procedures, the overall objectives, protocols and restrictions, and a review of the recruitment and data collection materials;

• during outreach and data collection, participants will be informed of the objectives and purpose of the data collection effort;

• participants are made aware that this is a voluntary effort and provide express consent for any audio or video recording, or photography (see Attachment 1 for interviews & 2 for surveys); and

• relevant participants receive a thank you for their participation and an email reminder to confirm their participation and to field any questions they may have regarding their participation (see Attachments 1 & 2).

Data Collection

The surveys will be administered via the Qualtrics platform and should take no longer than 30 minutes to complete.

The interviews will be conducted via phone or video conference. Video conference is preferred, but we will accept phone if video is not available. Participants will participate in interviews with an interviewer. Sessions will last 30 - 60 minutes. Interviews will also include an integration with a prioritization tool, Optimal Sort, which allows us to collect user input on how a website could be organized (see Attachment 2).

The full interview guides and surveys can be found in Attachment 2. In general, participants will be asked questions that fall into these categories:

1. What are the qualities, characteristics, examples of preferred tools and channels for research?

2. What IES products do they use (includes specific reasons for use or lack of use)?

3. How satisfied are they with IES (across multiple measures)?

4. What are their pain points and why?

5. What potential future IES focus areas resonate with them most and why?

4. Consultations outside the agency

These interviews will be administered under a contract to IES, The Grant Thornton Team (Grant Thornton and subcontractor, McKinsey & Company). Qualtrics will be used to host the survey and collect survey data.

5. Justification for Sensitive Questions

Throughout the user survey development processes, effort has been made to avoid asking for information that would be sensitive or offensive. Further, respondents are reminded that participation is voluntary and that any responses can be refused at any time in the process.

6. Paying Respondents

No incentivizes are being offered for this study.

7. Assurance of Confidentiality

The data collected will be used for research analysis internal to IES and its contractors and will not be distributed publicly. All participants will be made aware of the research intent and we will ask explicit permission to record, photograph, or video. Respondents will be notified that all the information they provide may be used for research purposes only and may not be disclosed. The assurances they are given and the consent forms they are asked to sign are in Attachment 1.

8. Estimate of Hourly burden

The total hours below include actual interview and survey times. Recruitment will take, on average, 5 minutes.

Interviews

Due to the nature of the interview introductions, through direct connections, we anticipate high levels of participation.

Surveys

With surveys sent through warm introductions, we estimate an 80% response rate due to the number of direct IES contacts. Social media will be used primarily for educator survey outreach and we estimate a 20% response rate due to the targeted outreach. These proportions guide how we estimate the number of responses from the total pool of potential respondents in Table 1 below.

Table 1. Estimate of Hourly Burden

	No. of respondents	No. of responses	Participation Time	Burden Hours
Total Recruitment	2479	891	5 min	207
Surveys (subtotal)	825	825	-	413
Policy Makers	205	205	30 min	103
Researchers	210	210	30 min	105
Association leaders	40	40	30 min	20
Educators	370	370	30 min	185
Interviews (subtotal)	66	66	-	66
Policy Makers	24	24	1 hour	24
Researchers	12	12	1 hour	12
Associations, Foundations, Thinktanks	10	10	1 hour	10
Media	10	10	1 hour	10
Educators	10	10	1 hour	10
Totals	2479	1782		686

Italicized rows provide more detail and should not be considered as duplicative of rows providing subtotals. All respondents are part of the total number recruited through email or social media, as indicated in the recruitment numbers.

9. Cost to federal government

For this study there is a $0 incremental cost because it is a part of a larger IES Customer Experience effort and will be incurred by the Grant Thornton/McKinsey Team. The cost is also $0 because many of the interviewees are from known introductions vs cold outreach.

10. Project Schedule

The below timeline is based on an estimate of 1-2 weeks to recruit participants followed by 2 – 6 weeks of interviews and surveys

Table 2: Estimated Schedule

Activity	Dates
Recruitment and administration of interviews and surveys	May 2020 – June 2020
Study summary and synthesis	June 2020 – July 2020