Change Memo

DATE: June 15, 2020

TO: Robert Sivinski, OMB

THROUGH: Carrie Clarady, Avar Consulting, in contract to NCES

SUBJECT: IES Modernization Phase 2 Change Memo (OMB #1880-0542)

The Institute of Education Sciences (IES) is the nation's premier source for research, evaluation and statistics for educators, policymakers, and stakeholders seeking to improve outcomes for all students. IES supports a diverse but critical set of services for the nation's educators and researchers, including acting as an independent non-partisan curator of the survey and performance data of the Department of Education as well as directly funding rigorous testing of new approaches to improving education outcomes. IES’s mission is to provide national leadership to expand the fundamental knowledge and understanding of (A) the condition and progress of education in the United States; (B) educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and (C) the effectiveness of federal and other education programs.

IES users are as diverse as the services offered by the Institute, which presents a particular challenge when attempting to deliver world-class experiences through IES digital channels. IES is comprised of four centers, many of which operate and maintain different data tools, content retrieval systems, intranets, and other digital properties that are often siloed according to center and—in many cases—program. Currently, IES’s website is built according to its organizational chart and doesn’t reflect customer/user experience research and mission-driven, agile development practices. Previous attempts to redesign the digital experience have been based solely on internal input, without the benefit of external user research and insight. Internal resources are scarce and in high demand, making it even more challenging than usual to distribute innovation across dozens of vendor and contractor relationships.

The IES Customer Experience effort addresses these challenges through a user-first understanding of IES. A previous phase of this research was approved by OMB in March 2020 (OMB# 1880-0542), and the associated interviews and focus groups were completed in March and April of 2020. In May 2020, a second phase of that research was approved; the goal is to use findings from the initial interviews to further define user groups and their needs. The project will conduct individual interviews and focus groups with targeted participants and will also distribute a survey to the targeted group and the targeted group’s networks. The results of the research will be used to understand the needs, pain points, preferences, and opportunity areas within key IES user groups to better achieve the mission of IES.

This proposal seeks to amend the May 2020 submission by supplementing recruiting procedures to include use of Coleman Research Group and Schlesinger Group. It decreases the number of quantitative research surveys we plan to administer and increases the number of interviews we hope to run, including adding some focus groups to allow us more individual contacts. It slightly increases the number of responses (+15) and the number of burden hours (+32). It does not increase the cost to the federal government.

Leaders within the organizations who were asked to give the research team access to participants (e.g., heads of associations, school leaders) now have more limited access to survey participants due to COVID-19 and less access to potential participants in their workspaces. Adding 3^rd party recruitment services will allow us to administer the survey to larger number a group of participants to meet our survey and interview completion goals.

Coleman Research Group is a recruiting service for qualitative interviews and Schlesinger Group is a recruiting service whose work focuses on quantitative surveys. Both have previously been utilized by McKinsey and Company, the contractor administering this survey on behalf of IES. These services have contracts with participant panels.

To account for the decreased respondents, the research team will conduct additional qualitative research. To account for that increased time, focus groups will be added and usertesting.com to facilitate more efficient user testing (see Attachment 2).

The approved versions of Volume 1, Attachment I and II have been revised to reflect the changes to recruiting procedures. Changes are listed below. Text added since the last approved version is marked red font color. Text deleted since the last approved version is marked in crossed-out red font color. All unchanged text is shown in black font.

The following updates were made to Vol I:

1. Submittal related information

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB# 1880-0542), which provides for the Department to conduct surveys and other studies regarding customer satisfaction.

The goal of this research is to use interviews, focus groups and surveys to gain insights into current audiences of the Institute of Education Sciences (IES) and how IES might better serve them. The results will be used to refine the understanding of key IES user groups and determine what information is most useful to them so we can better achieve the mission of IES.

2. Background and Rationale

Paragraph 3

The IES Customer Experience effort addresses these challenges through a user-first understanding of IES. A previous phase of this research was approved by OMB in March 2020 (OMB# 1880-0542), and the associated interviews and focus groups were completed in March and April of 2020. This work can be considered the second phase of that research and uses findings from those interviews to further define our user groups and their needs, which informed the respondent types as well as survey and interview questions for this study. The goal of this research effort is to employ interviews and surveys to gain insights into current audiences and how IES might better serve them. IES and contracting Grant Thornton/McKinsey research team have provided a pool of potential research participants. This project will conduct individual interviews and focus groups with those targeted participants. The project will also distribute a survey to the targeted group and the targeted group’s networks. The results of the research will be used to understand the needs, pain points, preferences and opportunity areas within key IES user groups so we can better achieve the mission of IES. We will also use these findings to inform future user research; the questions we ask now may inform future surveys and interviews, which will be used to evaluate potential solutions for IES to serve these key user groups.

3. Recruitment and Data Collection (pp. 1-2)

Leadership at IES and the research team have together compiled a list of key user groups – local education leaders, state education leaders, institute of higher education leaders, researchers, associations, foundations, thinktanks, media and educators. The list was created by IES leadership, their team, and the research team based on previous research. We will use this list to distribute the survey to additional users in these segments. We will also use two 3^rd party recruitment services – Schlesinger for surveys and interviews and Coleman for interviews, to recruit any additional participants we cannot attain through our list. The maximum number of potential respondents we will contact is 2479.

The research team will determine which users should be placed in focus groups versus individual interviews based on quotas and respondent availability. There will be 5 available interview times for each user group – policy makers (local education leaders, state education leaders, institute of higher education leaders), researchers, channels (associations, foundations, thinktanks), media and educators. The remainder will be scheduled in focus groups. Any respondents who are recruited by Coleman (who are focusing on recruiting respondents from groups who are usually more difficult to recruit) will automatically be scheduled for 1:1 interviews so the researchers can get the very most from those interviews.

Recruitment (pp. 2-3)

Under a contract to IES, The Grant Thornton Team (Grant Thornton and subcontractor, McKinsey & Company) developed the interview and focus group guides and surveys and will be responsible for the overall management of the interviews, focus groups and survey administration.

All participants will be recruited based on the pre-identified user segments - local education leaders, state education leaders, institute of higher education leaders, researchers, associations, foundations, thinktanks, media and educators – by speaking with those user groups, through social media posts, through recommendations by IES and previous interviewees, and through third party services for both surveys and interviews (e.g., teachers of Fairfax County Facebook Group, Association Twitter).

The research team will receive warm introductions from the IES leadership or, for pre-existing relationships from a previous set of interviews, conduct direct outreach via email (see Attachment 1) based on a list created by IES leadership and previous interviewees. As we conduct interviews and focus groups, the interviewees may also assist in identifying additional users within the segments via email (see Attachment 1). After initial email introductions, the research team will follow up to communicate details of the study and interview session. (Please see Attachment 1 for full communication materials).

Two 3^rd party research agencies will be used to recruit participants. Both agencies have pre-existing relationships with interviewees, who have opted in to receiving communication from the research agencies to participate in potential interviews and surveys. For interviews, we will use Coleman Research Group, an expert network service that McKinsey has used in previous research efforts. Coleman will use the email outreach (see Attachment 1) to contact their network of participants. Schlesinger will be used to recruit survey participants. They will be contacted via email (see Attachment 1). (Please see Attachment 1 for full communication materials).

Social Media will be used to recruit educators based on a list of Facebook, Twitter, and other social media groups recommended by IES leadership and previous interviewees. These will be posted by either IES or an interviewee. This will be decided based on who has a pre-existing relationship with the social media group (e.g., if an Association has a Facebook page, they will post the request on social media) (Please see Attachment 1 for social media sample posts)

The recruitment and data collection procedure will ensure that:

• all participating staff are made aware of all steps and procedures, the overall objectives, protocols and restrictions, and a review of the recruitment and data collection materials;

• during outreach and data collection, participants will be informed of the objectives and purpose of the data collection effort;

• participants are made aware that this is a voluntary effort and provide express consent for any audio or video recording, or photography (see Attachment 1 for interviews and focus groups & Attachment 2 for surveys); and

• relevant participants receive a thank you for their participation and an email reminder to confirm their participation and to field any questions they may have regarding their participation (see Attachments 1 & 2).

Data Collection (p.3)

The interviews and focus groups will be conducted via phone or video conference. Video conference is preferred, but we will accept phone if video is not available. Participants will participate in sessions interviews with an interviewer and notetaker. For focus groups, there will be a maximum of 2 facilitators and a notetaker. Interviews will last 30-60 minutes, while focus groups will last 60 minutes. Sessions will last t30 to 60 minutes. Interviews will also include an integration with a prioritization tool, Optimal Sort, which allows us to collect user input on how a website could be organized and usertesting.com to collect feedback on prototypes (see Attachment 2).

The full interview and focus group guides and surveys can be found in Attachment 2. In general, participants will be asked questions that fall into these categories:

4. Consultations outside of the agency (p. 3)

These interviews and focus groups will be administered under a contract to IES, The Grant Thornton Team (Grant Thornton and subcontractor, McKinsey & Company). Qualtrics will be used to host the survey and collect survey data. Coleman Search Group and Schlesinger Group will be used to recruit additional interviewees and survey participants, respectively.

6. Paying respondents (p. 4)

No incentivizes are being offered for this study.

The McKinsey-Grant Thornton team is not offering incentives for this study. The 3^rd party recruiting services may choose to pay their participants as part of their contract with their panels. Schlesinger Group will pay a maximum of $20 and Coleman Research Group will pay a maximum of $1000. Coleman will only be used as needed for hard to reach participants for an in-depth interview, such as certain state superintendents. This will incur no additional cost to the federal government.

8. Estimate of hourly burden (p. 4)

The total hours below include actual interview, focus group and survey times. Recruitment will take, on average, 5 minutes.

Interviews and Focus Groups

Due to the nature of the interview and focus group introductions, through direct connections, we anticipate high levels of participation. For those recruited through Coleman Research Group, we also anticipate high levels of participation because of the activity levels of their network.

Surveys

With surveys sent through warm introductions, we estimate an 80% response rate due to the number of direct IES contacts. Social media will be used primarily for educator survey outreach and we estimate a 20% response rate due to the targeted outreach. For surveys sent through Schlesinger Group, we estimate a 20% response rate due to the activity levels of their network. These proportions guide how we estimate the number of responses from the total pool of potential respondents in Table 1 below.

Table 1. Estimate of Hourly Burden

	No. of respondents	No. of responses	Participation Time	Burden Hours
Total Recruitment	2479	891	5 min	207
Surveys (subtotal)	825 790	825 790	-	413 395
Policy Makers	205 200	205 200	30 min	103 100
Researchers	205 200	205 200	30 min	105 100
Association leaders	40	40	30 min	20
Educators	370 350	370 350	30 min	185 175
Interviews and focus groups (subtotal)	66 116	66 116	-	66 116
Policy Makers	24 44	24 44	1 hour	24 44
Researchers	12 22	12 22	1 hour	12 22
Associations, Foundations, Thinktanks	10 20	10 20	1 hour	10 20
Media	10 10	10 10	1 hour	10
Educators	10 20	10 20	1 hour	10 20
Totals	2479	1782 1797		686 718

9. Cost to federal government (p. 5)

For this study there is a $0 incremental cost because it is a part of a larger IES Customer Experience effort and will be incurred by the McKinsey-Grant Thornton/McKinsey Team. The cost is also $0 because many of the interviewees are from known introductions vs cold outreach.

10. Project Schedule (p. 5)

The below timeline is based on an estimate of 1-2 weeks to recruit participants followed by 2 – 6 weeks of interviews, focus groups and surveys

Table 2: Estimated Schedule

Activity	Dates
Recruitment and administration of interviews, focus groups and surveys	May 2020 – June 2020
Study summary and synthesis	June 2020 – July 2020

The following updates were made to Attachment I:

Introductory email, Page 3

Response from research team – 1:1 interviews

Hi <<respondent>>,

We are hosting a 1 hour conversation as a part of an initiative to modernize the Customer Experience at IES. <<REFERRAL NAME>> mentioned that you would be a good person to help us validate some of what we’ve already learned and also test some new concepts and ideas in an <interview or focus group>.

Do you have time for a meeting at any of the below times? If so, please let us know in your response what time works well.

<<DATES AND TIMES>>

Once I hear back, I will send an invitation. Please let us know if we can answer any additional questions and thank you in advance for your help in this effort!

Thank you,

<<INTERVIEW COORDINATOR>>

McKinsey, as a part of the Team Grant Thornton, on behalf of IES

Consent form, Page 6

Title of Research Study: Institute for Education Sciences (IES) Modernization Interview

Thank you for your interest in participating in the research study described below. This permission form explains the research study. Please read it carefully, and feel free to ask questions about anything you do not understand. If you do not have questions now, you may ask them when they occur to you later.

1. What is this study about?

On behalf of the Institute for Education Sciences (IES), part of the U.S. Department of Education, Team Grant Thornton (Grant Thornton and subcontractor McKinsey & Company), a research company based in Washington, D.C., is conducting individual interviews and focus groups with IES current and potential future users. We are looking for people who might be interested in IES initiatives, programs, and data products to participate in interviews to discuss their needs, pain points, and explore potential opportunities to better serve this group.

2. Where will the interview or focus group take place?

The interviews and focus groups will take place online using video messaging software.

3. What will happen during the interview or focus group?

During the one-on-one interview, you will take part in an hour-long discussion with one of our researchers and a notetaker on topics related to your work and how IES can help you meet your needs. For focus groups, there will be no greater than 10 interviewees, 2 facilitators and one note taker.

4. Will information be kept confidential?

You will be assigned a unique identifier (ID), and your name will not be linked to your answers. All of the information provided may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Audio and video recordings of the interview will be recorded only for analysis, but no other personally identifying information will be recorded or retained.

5. How long will the research interview or focus group last and when will it take place?

The interview will last no more than one hour. The discussion will take place online.

6. Will the results go to my job?

Your answers will not be disclosed to your employer and will not be linked to any personally identifiable information.

7. Is there an incentive for participating?

There is no incentive for participation in interviews or focus groups unless the recruiting firm provided written confirmation of an incentive.

8. What are the possible risks of being in the study?

There are no known risks to participating in this voluntary study.

9. Who can I contact with questions or for further information?

If you have any questions about the study, you can call <NAME>, the <TITLE> at <CONTACT PHONE>.

10. Can participation end early?

Participation in this interview or focus group is completely voluntary. You have the right to refuse to answer particular questions. You may elect to withdraw from this discussion at any time and can leave the interview early if you wish.

Initial Recruitment Emails, Page 9 (new material)

Initial Recruitment Emails

Coleman outreach

For interviews/focus groups only 

 

Hi ______, 

  

I hope you’re doing well! I’m reaching out to gauge your interest in a discussion over the phone, regarding your experience education research – pains, needs and potential opportunities. I am working with a consulting firm who is hoping to better understand the space and based on your experience I thought you'd be a fantastic resource to connect them with! 

  

This would be an approximately 60-minute conversation, would not involve anything confidential or proprietary, and you would be compensated at an hourly rate. 

  

Is this something you would be open to? 

  

If so, let me know and I’d be happy to send over the information for next steps. I can be reached at your convenience on my direct line, xxx-xxx-xxxx, or via email at _________. 

  

Thanks and look forward to hearing back!  

<next step to send them the interview invitation>  

Schlesinger outreach

For surveys only – sample

You are being invited to participate in a survey on the education experience.

Time allocated: ~30 minutes

Screener: Yes

Would you like to proceed: Yes/No

<next step is to have them complete the consent form and survey>

The following additions were made to Attachment II:

Focus Groups

Focus groups will follow the same protocol as the 1:1 interviews with the following changes to the script:

1. We want you to do the talking. We would like everyone to participate. I may call on you if I haven’t heard from you in a while.

2. Please be tolerant of others’ thoughts and opinions and let’s have just one speaker at a time.

3. The discussion will last for a little over one hour. Please silence your mobile phones.  Please give everyone the chance to express his/her opinion during the conversation. You can address each other if you like. I am here to facilitate the discussion.

User Testing

As a part of the interviews, we will include a user test.

Participants will be shown a new webpage design or concept (e.g., new homepage layout). This will help the research team understand how usable the new design is for a given user group.

Script:

Now we would like to you participate in an activity. Our goal is to understand how you think about and categorize information. Just like with the rest of the interview, there are no right or wrong answers.

First, tell me what you see on the page.

Now, navigate around the site and talk through what you expect to occur.

If there are things you are expecting that do not occur, please tell us.

Questions throughout:

Why did you navigate there?

What were you expecting to occur?