IES Modernization Interviews and Surveys – Phase 2
OMB# 1880-0542
Attachment 2
Interview Guides and Surveys
April 2020
Survey Open (all surveys)
Welcome! This study is being conducted on behalf of the Institute for Education Sciences (IES), part of the U.S. Department of Education. This survey should take no longer than 30 minutes to complete and will be used to help the research team to learn about 1) experience using education research and 2) opportunities to improve education data and research. Thank you for your participation.
Your participation in this research is voluntary. You may stop the survey at any time. Your answers will be kept confidential, and your personal information will not be shared with anyone outside the research team. All of the information provided may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).
Please select the group you belong to. If you belong to multiple, choose the one that is most applicable.
Educator
State, Local, or Higher Education Leader
Researcher
Survey Close (all surveys)
Thank you for your participation! We appreciate your help and support in conducting this research.
Are you an education researcher?
Yes
No>>> terminate
Role [MULTIPLE SELECT]
PhD student
Post-Doctorate
Other graduate student
Faculty (<50%)
Institutional researcher (>20%)
Other researcher
Other --- terminate
Area of research [MULTIPLE SELECT]
K12 education (45-55%)
Higher education (45-55%)
Other (specify) (<10%)
Employer [MULTIPLE SELECT]
Private Institute of higher education (50%)
Public Institute of higher education (50%)
Consultancy -- terminate if only responses 3 – 5 are selected
Association --- terminate if only responses 3 – 5 are selected
Private Institute --- terminate if only responses 3 – 5 are selected
Foundation --- terminate if only responses 3 – 5 are selected
Other --- terminate
I consider my research institution to be _____ . [SINGLE SELECT]
Very well resourced (20%)
Well resourced (20%)
Moderately resourced (20%)
Inadequately resourced (20%)
Very inadequately resourced (20%)
Does your current research include direct partnerships with the following [MULTIPLE SELECT]
State education agencies
District education agencies
Local schools
Higher Education Institute (if applicable excluding your own institution)
Parents
Students
Other (please specify)
None of the above (20%)
Think of the resources and tools you use throughout your research, from selecting topics to disseminating. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Include specific names and webpage links where applicable. [DATA ENTRY]
Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration with other educators
Incentivizes important topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Other (specify)
Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
I am well connected to the subjects of my research (e.g., schools, state and local leaders)
I receive ample feedback on outcomes from those who have used my research
I am incentivized to conduct applied research
I am incentivized to conduct research that will make a difference in classrooms
There are sufficient grants for efficacy work (versus innovation work)
What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; I am not familiar with IES >>> skip to question 5]
The below statements may or may not describe IES. Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration
Incentivizes important topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Which of the following resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]
IES abstracts database
DataLab
EDGE
ERIC
Grants applications
IPEDS
NAEP
Publications and Product Search
Small Business Innovation Research
Survey tools
Technical Working Group Meeting
[DATA ENTRY] What other research products do you use?
[carry over responses from question 5 with a checked box, up to 5 with highest usage]
How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]
[carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]
For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration
Incentivizes important topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]
Introduction
Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.
Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”
“There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”
Any questions on context before we begin?
Background
Tell me a little about what you do. What type of organization do you work for and what is your role?
What types of decisions do you make that require research or knowing about evidence based practices?
How often? What percent of your time? Who else is involved?
What are some of your favorite experiences with products you’ve used to do accomplish your research goals?
[Probe] specific names, use case, feature, organization
Probing on insights
From our interviews, we’ve heard there are 3 top needs for people in your role when it comes to grants. Are these true for you? Say more. Why or why not?
Grant cycle, from application to receiving the grant, is too long.
[probe] how long is too long? Do any organizations offer fast/easy grant processes?
Dissemination of information, I expect my grantor to promote my work
Program Officers, I want them to be a thought partner who shapes the work and point me to related studies and potential partnerships
Favorite resources
Think of the research resources you use to help your district.
Describe the one you like most.
Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names
Testing ideas
We heard from others like you that these types of solutions might be useful.
Research drawer
Increased speed in grant process
Applied research grants
Which of those sounds helpful? Unhelpful? Why?
Wrap up
Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?
“Thanks again for allowing us to come spend this time with you today!”
What type of school do you work in?
Public (100%)
Private --- terminate
What is your school locale?
City (>25%)
Suburb (>15%)
Town (>15%)
Rural (>25%)
Does your school have title one status?
Yes (>70%)
No
What subjects do you teach?
Core subject
Special education
Other (specify)
What grade(s) do you teach?
Prek/elementary (>25%)
Middle (>25%)
Secondary (>25%)
What is your role?
PreK-12 Administrator – (20%)
PreK-12 Instructor (80%)
Select your affiliations. (60% of our people in+active in a union or asscn, and the other 40% evenly spread among the others.)
In a union and active
In a union and not active
Not in a union
In one or more professional association and active
In one or more professional association and not active
Not in a professional association
Complete the sentence. My school is _____.
Very well resourced (20%)
Well resourced (20%)
Moderately resourced (20%)
Inadequately resourced (20%)
Very inadequately resourced (20%)
Think of the research resources you use to help your classroom. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names and webpage links where applicable. [DATA ENTRY]
Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration with other educators
Incentivizes important topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Other (specify)
For which types of decisions do you use evidence-based research and how often? [MULTIPLE SELECT] [RANK never, 1 time per year, 2 – 4 times per year, 5 – 10 times per year, great than 10 times per year]
lesson plans
curriculum design
pedagogical practices
assignments
new teaching practices and tips
other (specify)
I do not apply evidence-based research
Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Education research is usable in the classroom
I have the tools and support I need to make an effective curriculum
I often read research
I know how to interpret research
There is strong school support when a new state if local standard is put into place
I understand why new standards are put in place
What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; N/A I am not familiar IES >>> skip to question 7]
The below statements may or may not describe IES. Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration among the field
Incentivizes important research topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Which of the following IES resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]
CCD
College Navigator
IES Publications and Products Search
ERIC
IPEDS
KidsZone/Educators Corner
NAEP
WWC – Study Reviews
[data entry] What other research products do you use?
[carry over responses from question 5 with a checked box, up to 5 with highest usage]
How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]
[carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]
For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration among the field
Incentivizes important research topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]
Introduction
Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.
Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”
“There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”
Any questions on context before we begin?
Background
Tell me a little about what you do. What type of organization do you work for and what is your role?
What types of decisions do you make that require research or knowing about evidence based practices?
How often? What percent of your time? Who else is involved?
What are some of your favorite experiences with products you’ve used to do accomplish your research goals?
[Probe] specific names, use case, feature, organization
Probing on insights
From our interviews, we’ve heard there are 3 top needs for people in your role when it comes to grants. Are these true for you? Say more. Why or why not?
Lack of clarity on how to execute research outputs
Research is difficult to understand
Intent is lost in translation – unclear why decisions are made
Favorite resources
Think of the research resources you use to help your classes
Describe the one you like most.
Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names
Testing ideas
We heard from others like you that these types of solutions might be useful.
Associations giving more information and taking in feedback
Pinterest and other social media level insights
Training on how to use data
Which of those sounds helpful? Unhelpful? Why?
Wrap up
Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?
“Thanks again for allowing us to come spend this time with you today!”
In what type of school do you work?
Institute of higher education (33%)
State education agency (33%)
Local education agency (33%)
What is your role?
Superintendent (15 - 20%)
School Board (10 – 15%)
Head of Academics (15 - 20%)
Researcher (15 - 20%)
Curriculum/Teaching and Learning team member (15 - 20%)
What is your locale?
City (>25%)
Suburb (>15%)
Town (>15%)
Rural (>25%)
Do you have title one status?
Yes (>70%)
No
What are your school or districts education research capabilities?
does not have (0-40%)
minimal (30 – 60%)
Adequate (30 – 40%)
Robust (20– 30%)
What is your affiliation? (0% of our people in+active in a union or asscn, and the other 40% evenly spread among the others.)
In a union and active
In a union and not active
Not in a union
In one or more professional association and active
In one or more professional association and not active
Not in a professional association
Think of the research resources you use to help your schools. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names and webpage links where applicable. [DATA ENTRY]
Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration with other educators
Incentivizes important topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Other (specify)
Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
[screener question 2: response = institute of higher education]
Instructors easily adopt new practices
Instructors are focused on being effective teachers
Instructors are incentivized to focus on the student
I am satisfied with the information I have on my students previous education
Data is easily applied to my situation
I have the research I need to support decision-making
Research is timely
Research is reliable
Research is easily applied
Research is on the topics I need most
Research reports draw helpful conclusions
I can easily learn why an intervention did or did not work
I give feedback to researchers on their work in my schools
[screener question 2: response = state education agency, local education agency]
My schools are similar to one another
I am satisfied with the information I have after my students leave my schools
I can easily customize research to my contexts
Research studies are well designed
Data is easily applied to my situation
I have the data I need to support decision-making
I have the research I need to support decision-making
Research is timely
Research is reliable
Research is easily applied
Research is on the topics I need most
Research reports draw helpful conclusions
I can easily learn why an intervention did or did not work
I give feedback to researchers on their work in my schools
I depend on relationships with other districts to find interventions for my schools
What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; N/A I am not familiar IES >>> skip to question 7]
The below statements may or may not describe IES. Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration among the field
Incentivizes important research topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Which of the following IES resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]
IES abstracts database
Annual Report: Condition of Education
CCD
College Navigator
Education Finance Statistics Center
ERIC
IPEDS
KidsZone/Educators Corner
NAEP
RELs
Technical Working Group Meeting
WWC – Practice Guides
WWC – Study Reviews
[data entry] what other research products do you use?
[carry over responses from question 5 with a checked box, up to 5 with highest usage]
How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]
[carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]
For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration among the field
Incentivizes important research topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]
Introduction
Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.
Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”
“There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”
Any questions on context before we begin?
Background
Tell me a little about what you do. What type of organization do you work for and what is your role?
What types of decisions do you make that require research or knowing about evidence based practices?
How often? What percent of your time? Who else is involved?
What are some of your favorite experiences with products you’ve used to do accomplish your research goals?
[Probe] specific names, use case, feature, organization
Probing on insights
From our interviews, we’ve heard there are 3 top needs for people in your role. Are these true for you? Say more. Why or why not?
Research and data is not easily accessible
Research does not have all of the data I need
Hard to get the right information at the right time
IHE only:
Hard to get faculty to adopt interventions
Lack of predictive data and long-term outcomes
Research does not have the narratives
Favorite resources
Think of the research resources you use to help your classes
Describe the one you like most.
Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names
Testing ideas
We heard from others like you that these types of solutions might be useful.
Associations giving more information and taking in feedback
State level research departments with permanent staffing for continuity
Repository for research that includes comments from those who have implanted it
Which of those sounds helpful? Unhelpful? Why?
Wrap up
Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?
“Thanks again for allowing us to come spend this time with you today!”
Introduction
Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.
Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”
“There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”
Any questions on context before we begin?
Background
Tell me a little about what you do. What type of organization do you work for and what is your role?
What are some of your favorite experiences with products you’ve used to do accomplish your research goals?
[Probe] specific names, use case, feature, organization
Probing on insights
From our interviews, we’ve heard there are 3 top needs for people in your role. Are these true for you? Say more. Why or why not?
There’s a lot of research that covers the same ground in the same way, I have to parse through it to find what’s new and interesting, if anything
I don’t think people in a general audience would be interested in 40 – 60% of what I’m writing, I have to tailor my stories to my core audience, usually a trade or professionals like researchers
It’s so hard to use IES tools, like NAEP data explorer. I stumble across new variables in it and have to write down the magic combination of parameters to reach it again. If it wasn’t such good data, I wouldn’t bother dealing with the tool to get it
Favorite resources
Think of the research resources you use to help your job
Describe the one you like most.
Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names
Testing ideas
We heard from others like you that these types of solutions might be useful.
Research nuggets/fast facts
Summary reports
Cleaner variables and data
Which of those sounds helpful? Unhelpful? Why?
Wrap up
Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?
“Thanks again for allowing us to come spend this time with you today!”
What is the primary make up of your membership?
Higher education staff (50%)
PreK-12 staff (50%)
Other (20%)
What group does your group serve?
Superintendent (15 - 20%)
School Board (10 – 15%)
Head of Academics (15 - 20%)
Researcher (15 - 20%)
Curriculum/Teaching and Learning team member (15 - 20%)
Other (please specify)
What is your associations research capabilities?
does not have (0-10%)
minimal (0 - 10%)
Adequate
Robust
Think of the research resources you use to help your members. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names and webpage links where applicable. [DATA ENTRY]
Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration with other educators
Incentivizes important topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Other (specify)
Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Research is relevant to today’s needs
Research is actionable
Research solutions are tailored to specific groups
Research is pragmatic
I make lasting relationships with school leaders
I make trusted relationships with school leaders
What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; N/A I am not familiar IES >>> skip to question 7]
The below statements may or may not describe IES. Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration among the field
Incentivizes important research topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Which of the following IES resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]
IES abstracts database
Annual Report: Condition of Education
CCD
DataLab
EDGE
Education Finance Statistics Center
ERIC
Grants applications
IPEDS
KidsZone/Educators Corner
NAEP
Publications and Product Search
RELs
WWC – Practice Guides
WWC – Study Reviews
Federal Program Evaluations
[data entry] what other research products do you use?
[carry over responses from question 5 with a checked box, up to 5 with highest usage]
How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]
[carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]
For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]
Is reliable
Encourages collaboration among the field
Incentivizes important research topics
Provides timely information and services
Is easy to use
Provides relevant information
Addresses problems or concerns to my satisfaction
Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]
Introduction
Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.
Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”
“There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”
Any questions on context before we begin?
Background
Tell me a little about what you do. What type of organization do you work for and what is your role?
What are some of your favorite experiences with products you’ve used to do accomplish your research goals?
[Probe] specific names, use case, feature, organization
Probing on insights
From our interviews, we’ve heard there are 3 top needs for people in your role when it comes to grants. Are these true for you? Say more. Why or why not?
Research does not the data I need in it to put into practice
Hard to sustain relationships with state and local leaders
Finding the right type of research is a challenge
Favorite resources
Think of the research resources you use to help your job
Describe the one you like most.
Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names
Testing ideas
We heard from others like you that these types of solutions might be useful.
User driven feedback systems
Specialized knowledge and forums (grade and subject specific)
Which of those sounds helpful? Unhelpful? Why?
Wrap up
Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?
“Thanks again for allowing us to come spend this time with you today!”
As a part of each of the interviews, we will include a cart sorting question.
Participants will be shown a list of various parts of the IES website to group (e.g., RELs, research grants, NAEP). This will help the research team see how the interviewee thinks about information in order to inform the website navigation.
Script:
Now we would like to you participate in an activity. Our goal is to understand how you think about and categorize information. Just like with the rest of the interview, there are no right or wrong answers.
First, group the cards in whatever way makes sense to you.
Now, name the sections. Whatever makes sense to you as a name for something that contains the contents of the group you just created.
If you see something no longer makes logical sense, you can change your groupings.
Questions throughout:
Why did you use group in that way?
Where else were you thinking about placing information?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Molin, Ed C |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |