IES Modernication Interviews and Surveys - Phase 2

Generic Clearance for the Collection of Qualitative Feedback on Agency Service Delivery

Attachment 2 IES Modernization P2

IES Modernication Interviews and Surveys - Phase 2

OMB: 1880-0542

Document [docx]
Download: docx | pdf







IES Modernization Interviews and Surveys – Phase 2

OMB# 1880-0542


Attachment 2

Interview Guides and Surveys











June 2020














Interview Guides and Surveys



Survey Open (all surveys)

Welcome! This study is being conducted on behalf of the Institute for Education Sciences (IES), part of the U.S. Department of Education. This survey should take no longer than 30 minutes to complete and will be used to help the research team to learn about 1) experience using education research and 2) opportunities to improve education data and research. Thank you for your participation.


Your participation in this research is voluntary. You may stop the survey at any time. Your answers will be kept confidential, and your personal information will not be shared with anyone outside the research team. All of the information provided may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).


Please select the group you belong to. If you belong to multiple, choose the one that is most applicable.

  • Educator

  • State, Local, or Higher Education Leader

  • Researcher


Survey Close (all surveys)

Thank you for your participation! We appreciate your help and support in conducting this research.



Researchers



Survey Screener


Are you an education researcher?

  • Yes

  • No>>> terminate


Role [MULTIPLE SELECT]

  • PhD student

  • Post-Doctorate

  • Other graduate student

  • Faculty (<50%)

  • Institutional researcher (>20%)

  • Other researcher

  • Other --- terminate


Area of research [MULTIPLE SELECT]

  • K12 education (45-55%)

  • Higher education (45-55%)

  • Other (specify) (<10%)


Employer [MULTIPLE SELECT]

  • Private Institute of higher education (50%)

  • Public Institute of higher education (50%)

  • Consultancy -- terminate if only responses 3 – 5 are selected

  • Association --- terminate if only responses 3 – 5 are selected

  • Private Institute --- terminate if only responses 3 – 5 are selected

  • Foundation  --- terminate if only responses 3 – 5 are selected

  • Other --- terminate


I consider my research institution to be _____ . [SINGLE SELECT]

  • Very well resourced (20%)

  • Well resourced (20%)

  • Moderately resourced (20%)

  • Inadequately resourced (20%)

  • Very inadequately resourced (20%)



Does your current research include direct partnerships with the following [MULTIPLE SELECT]

  • State education agencies

  • District education agencies

  • Local schools

  • Higher Education Institute (if applicable excluding your own institution)

  • Parents

  • Students

  • Other (please specify)

  • None of the above (20%)




Survey

  1. Think of the resources and tools you use throughout your research, from selecting topics to disseminating. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Include specific names and webpage links where applicable. [DATA ENTRY]



  1. Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration with other educators

    3. Incentivizes important topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction

    8. Other (specify)



  1. Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. I am well connected to the subjects of my research (e.g., schools, state and local leaders)

    2. I receive ample feedback on outcomes from those who have used my research

    3. I am incentivized to conduct applied research

    4. I am incentivized to conduct research that will make a difference in classrooms

    5. There are sufficient grants for efficacy work (versus innovation work)



  1. What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; I am not familiar with IES >>> skip to question 5]



  1. The below statements may or may not describe IES.  Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration

    3. Incentivizes important topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction







  1. Which of the following resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]

    1. IES abstracts database

    2. DataLab

    3. EDGE

    4. ERIC

    5. Grants applications

    6. IPEDS

    7. NAEP

    8. Publications and Product Search

    9. Small Business Innovation Research

    10. Survey tools

    11. Technical Working Group Meeting

    12. [DATA ENTRY] What other research products do you use?



  1. [carry over responses from question 5 with a checked box, up to 5 with highest usage]



How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]



  1. [carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]



For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration

    3. Incentivizes important topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction



  1. Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]


Discussion Guide



  1. Introduction

    1. Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.

    2. Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”

    3. There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”

    4. Any questions on context before we begin?



  1. Background

    1. Tell me a little about what you do. What type of organization do you work for and what is your role?

    2. What types of decisions do you make that require research or knowing about evidence based practices?

      1. How often? What percent of your time? Who else is involved?

    3. What are some of your favorite experiences with products you’ve used to do accomplish your research goals?

      1. [Probe] specific names, use case, feature, organization

  2. Probing on insights

    1. From our interviews, we’ve heard there are 3 top needs for people in your role when it comes to grants. Are these true for you? Say more. Why or why not?

      1. Grant cycle, from application to receiving the grant, is too long.

        1. [probe] how long is too long? Do any organizations offer fast/easy grant processes?

      2. Dissemination of information, I expect my grantor to promote my work

      3. Program Officers, I want them to be a thought partner who shapes the work and point me to related studies and potential partnerships

  3. Favorite resources

    1. Think of the research resources you use to help your district.

      1. Describe the one you like most.

      2. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names

  4. Testing ideas

    1. We heard from others like you that these types of solutions might be useful.

      1. Research drawer

      2. Increased speed in grant process

      3. Applied research grants

    2. Which of those sounds helpful? Unhelpful? Why?

  5. Wrap up

    1. Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?

    2. Thanks again for allowing us to come spend this time with you today!”







Educators



Survey Screener


What type of school do you work in?

  • Public (100%)

  • Private --- terminate


What is your school locale?

  • City (>25%)

  • Suburb (>15%)

  • Town (>15%)

  • Rural (>25%)


Does your school have title one status?

  • Yes (>70%)

  • No


What subjects do you teach?

  • Core subject

  • Special education

  • Other (specify)


What grade(s) do you teach?

  • Prek/elementary (>25%)

  • Middle (>25%)

  • Secondary (>25%)


What is your role?

  • PreK-12 Administrator – (20%)

  • PreK-12 Instructor (80%)


Select your affiliations. (60% of our people in+active in a union or asscn, and the other 40% evenly spread among the others.)

  • In a union and active

  • In a union and not active

  • Not in a union

  • In one or more professional association and active

  • In one or more professional association and not active

  • Not in a professional association


Complete the sentence. My school is _____.

  • Very well resourced (20%)

  • Well resourced (20%)

  • Moderately resourced (20%)

  • Inadequately resourced (20%)

  • Very inadequately resourced (20%)




Survey

  1. Think of the research resources you use to help your classroom. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names and webpage links where applicable. [DATA ENTRY]

  2. Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration with other educators

    3. Incentivizes important topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction

    8. Other (specify)



  1. For which types of decisions do you use evidence-based research and how often? [MULTIPLE SELECT] [RANK never, 1 time per year, 2 – 4 times per year, 5 – 10 times per year, great than 10 times per year]

    1. lesson plans

    2. curriculum design

    3. pedagogical practices

    4. assignments

    5. new teaching practices and tips

    6. other (specify)

    7. I do not apply evidence-based research



  1. Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. Education research is usable in the classroom

    2. I have the tools and support I need to make an effective curriculum

    3. I often read research

    4. I know how to interpret research

    5. There is strong school support when a new state if local standard is put into place

    6. I understand why new standards are put in place



  1. What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; N/A I am not familiar IES >>> skip to question 7]





  1. The below statements may or may not describe IES.  Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration among the field

    3. Incentivizes important research topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction







  1. Which of the following IES resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]

    1. CCD

    2. College Navigator

    3. IES Publications and Products Search

    4. ERIC

    5. IPEDS

    6. KidsZone/Educators Corner

    7. NAEP

    8. WWC – Study Reviews

    9. [data entry] What other research products do you use?



  1. [carry over responses from question 5 with a checked box, up to 5 with highest usage]



    1. How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]



  1. [carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]



    1. For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    2. Is reliable

    3. Encourages collaboration among the field

    4. Incentivizes important research topics

    5. Provides timely information and services

    6. Is easy to use

    7. Provides relevant information

    8. Addresses problems or concerns to my satisfaction



  1. Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]


Discussion Guide



  1. Introduction

    1. Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.

    2. Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”

    3. There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”

    4. Any questions on context before we begin?



  1. Background

    1. Tell me a little about what you do. What type of organization do you work for and what is your role?

    2. What types of decisions do you make that require research or knowing about evidence based practices?

      1. How often? What percent of your time? Who else is involved?

    3. What are some of your favorite experiences with products you’ve used to do accomplish your research goals?

      1. [Probe] specific names, use case, feature, organization

  2. Probing on insights

    1. From our interviews, we’ve heard there are 3 top needs for people in your role when it comes to grants. Are these true for you? Say more. Why or why not?

      1. Lack of clarity on how to execute research outputs

      2. Research is difficult to understand

      3. Intent is lost in translation – unclear why decisions are made

  3. Favorite resources

    1. Think of the research resources you use to help your classes

      1. Describe the one you like most.

      2. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names

  4. Testing ideas

    1. We heard from others like you that these types of solutions might be useful.

      1. Associations giving more information and taking in feedback

      2. Pinterest and other social media level insights

      3. Training on how to use data

    2. Which of those sounds helpful? Unhelpful? Why?

  5. Wrap up

    1. Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?

    2. Thanks again for allowing us to come spend this time with you today!”





Policy Makers


Survey Screener


In what type of school do you work?

  • Institute of higher education (33%)

  • State education agency (33%)

  • Local education agency (33%)


What is your role?

  • Superintendent (15 - 20%)

  • School Board (10 – 15%)

  • Head of Academics (15 - 20%)

  • Researcher (15 - 20%)

  • Curriculum/Teaching and Learning team member (15 - 20%)


What is your locale?

  • City (>25%)

  • Suburb (>15%)

  • Town (>15%)

  • Rural (>25%)


Do you have title one status?

  • Yes (>70%)

  • No


What are your school or districts education research capabilities?

  • does not have (0-40%)

  • minimal (30 – 60%)

  • Adequate (30 – 40%)

  • Robust (20– 30%)


What is your affiliation? (0% of our people in+active in a union or asscn, and the other 40% evenly spread among the others.)

  • In a union and active

  • In a union and not active

  • Not in a union

  • In one or more professional association and active

  • In one or more professional association and not active

  • Not in a professional association





Survey

  1. Think of the research resources you use to help your schools. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names and webpage links where applicable. [DATA ENTRY]



  1. Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration with other educators

    3. Incentivizes important topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction

    8. Other (specify)



  1. Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

[screener question 2: response = institute of higher education]



  • Instructors easily adopt new practices

  • Instructors are focused on being effective teachers

  • Instructors are incentivized to focus on the student

  • I am satisfied with the information I have on my students previous education

  • Data is easily applied to my situation

  • I have the research I need to support decision-making

  • Research is timely

  • Research is reliable

  • Research is easily applied

  • Research is on the topics I need most

  • Research reports draw helpful conclusions

  • I can easily learn why an intervention did or did not work

  • I give feedback to researchers on their work in my schools





[screener question 2: response = state education agency, local education agency]

  • My schools are similar to one another

  • I am satisfied with the information I have after my students leave my schools

  • I can easily customize research to my contexts

  • Research studies are well designed

  • Data is easily applied to my situation

  • I have the data I need to support decision-making

  • I have the research I need to support decision-making

  • Research is timely

  • Research is reliable

  • Research is easily applied

  • Research is on the topics I need most

  • Research reports draw helpful conclusions

  • I can easily learn why an intervention did or did not work

  • I give feedback to researchers on their work in my schools

  • I depend on relationships with other districts to find interventions for my schools



  1. What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; N/A I am not familiar IES >>> skip to question 7]





  1. The below statements may or may not describe IES.  Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration among the field

    3. Incentivizes important research topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction







  1. Which of the following IES resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]

    1. IES abstracts database

    2. Annual Report: Condition of Education

    3. CCD

    4. College Navigator

    5. Education Finance Statistics Center

    6. ERIC

    7. IPEDS

    8. KidsZone/Educators Corner

    9. NAEP

    10. RELs

    11. Technical Working Group Meeting

    12. WWC – Practice Guides

    13. WWC – Study Reviews

    14. [data entry] what other research products do you use?



  1. [carry over responses from question 5 with a checked box, up to 5 with highest usage]



    1. How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]



  1. [carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]



    1. For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    2. Is reliable

    3. Encourages collaboration among the field

    4. Incentivizes important research topics

    5. Provides timely information and services

    6. Is easy to use

    7. Provides relevant information

    8. Addresses problems or concerns to my satisfaction



  1. Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]




Discussion Guide



  1. Introduction

    1. Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.

    2. Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”

    3. There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”

    4. Any questions on context before we begin?



  1. Background

    1. Tell me a little about what you do. What type of organization do you work for and what is your role?

    2. What types of decisions do you make that require research or knowing about evidence based practices?

      1. How often? What percent of your time? Who else is involved?

    3. What are some of your favorite experiences with products you’ve used to do accomplish your research goals?

      1. [Probe] specific names, use case, feature, organization

  2. Probing on insights

    1. From our interviews, we’ve heard there are 3 top needs for people in your role. Are these true for you? Say more. Why or why not?

      1. Research and data is not easily accessible

      2. Research does not have all of the data I need

      3. Hard to get the right information at the right time

    2. IHE only:

      1. Hard to get faculty to adopt interventions

      2. Lack of predictive data and long-term outcomes

      3. Research does not have the narratives

  3. Favorite resources

    1. Think of the research resources you use to help your classes

      1. Describe the one you like most.

      2. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names

  4. Testing ideas

    1. We heard from others like you that these types of solutions might be useful.

      1. Associations giving more information and taking in feedback

      2. State level research departments with permanent staffing for continuity

      3. Repository for research that includes comments from those who have implanted it

    2. Which of those sounds helpful? Unhelpful? Why?

  5. Wrap up

    1. Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?

    2. Thanks again for allowing us to come spend this time with you today!”





Media

Interviews


Discussion Guide



  1. Introduction

    1. Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.

    2. Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”

    3. There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”

    4. Any questions on context before we begin?



  1. Background

    1. Tell me a little about what you do. What type of organization do you work for and what is your role?

    2. What are some of your favorite experiences with products you’ve used to do accomplish your research goals?

      1. [Probe] specific names, use case, feature, organization

  2. Probing on insights

    1. From our interviews, we’ve heard there are 3 top needs for people in your role. Are these true for you? Say more. Why or why not?

      1. There’s a lot of research that covers the same ground in the same way, I have to parse through it to find what’s new and interesting, if anything​

      2. I don’t think people in a general audience would be interested in 40 – 60% of what I’m writing, I have to tailor my stories to my core audience, usually a trade or professionals like researchers​

      3. It’s so hard to use IES tools, like NAEP data explorer. I stumble across new variables in it and have to write down the magic combination of parameters to reach it again. If it wasn’t such good data, I wouldn’t bother dealing with the tool to get it

  3. Favorite resources

    1. Think of the research resources you use to help your job

      1. Describe the one you like most.

      2. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names

  4. Testing ideas

    1. We heard from others like you that these types of solutions might be useful.

      1. Research nuggets/fast facts

      2. Summary reports

      3. Cleaner variables and data

    2. Which of those sounds helpful? Unhelpful? Why?

  5. Wrap up

    1. Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?

    2. Thanks again for allowing us to come spend this time with you today!”




Associations, Foundations, Thinktanks

Survey Screener


What is the primary make up of your membership?

  • Higher education staff (50%)

  • PreK-12 staff (50%)

  • Other (20%)


What group does your group serve?

  • Superintendent (15 - 20%)

  • School Board (10 – 15%)

  • Head of Academics (15 - 20%)

  • Researcher (15 - 20%)

  • Curriculum/Teaching and Learning team member (15 - 20%)

  • Other (please specify)


What is your associations research capabilities?

  • does not have (0-10%)

  • minimal (0 - 10%)

  • Adequate

  • Robust




Survey

  1. Think of the research resources you use to help your members. Describe the one you like most. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names and webpage links where applicable. [DATA ENTRY]



  1. Rank the benefits of the resource you discussed in question 1. [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration with other educators

    3. Incentivizes important topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction

    8. Other (specify)



  1. Rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. Research is relevant to today’s needs

    2. Research is actionable

    3. Research solutions are tailored to specific groups

    4. Research is pragmatic

    5. I make lasting relationships with school leaders

    6. I make trusted relationships with school leaders

  1. What is your overall opinion of the Institute of Educational Sciences (IES)? [RANK 1- very satisfied; 7 – very dissatisfied; N/A I am not familiar IES >>> skip to question 7]





  1. The below statements may or may not describe IES.  Please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    1. Is reliable

    2. Encourages collaboration among the field

    3. Incentivizes important research topics

    4. Provides timely information and services

    5. Is easy to use

    6. Provides relevant information

    7. Addresses problems or concerns to my satisfaction







  1. Which of the following IES resources have you used, if any? [CHECK BOX: I have used this product in the last 6 months] [Usage: never; one to three times; three to ten times; ten to twenty times; greater than twenty times]

    1. IES abstracts database

    2. Annual Report: Condition of Education

    3. CCD

    4. DataLab

    5. EDGE

    6. Education Finance Statistics Center

    7. ERIC

    8. Grants applications

    9. IPEDS

    10. KidsZone/Educators Corner

    11. NAEP

    12. Publications and Product Search

    13. RELs

    14. WWC – Practice Guides

    15. WWC – Study Reviews

    16. Federal Program Evaluations

    17. [data entry] what other research products do you use?





  1. [carry over responses from question 5 with a checked box, up to 5 with highest usage]



    1. How satisfied are you with the below products? [RANK 1- very satisfied; 7 – very dissatisfied]



  1. [carry over responses from question 6 with a checked box, up to 4 from highest and lowest satisfaction, skip if no checked boxes]



    1. For each of the below products, please rate your agreement with each statement [RANK 1- strongly disagree; 7 – strongly agree]

    2. Is reliable

    3. Encourages collaboration among the field

    4. Incentivizes important research topics

    5. Provides timely information and services

    6. Is easy to use

    7. Provides relevant information

    8. Addresses problems or concerns to my satisfaction



  1. Do you have any suggestions on how your overall experience could have been better? [DATA ENTRY]



Discussion Guide



  1. Introduction

    1. Participant and team intro, “Hi, my name is __, a designer and researcher who wants to learn more about how you use education research – your pains, needs and areas for improvement.

    2. Overview of project: “We are serving IES to help them improve customer experience and better accomplish their mission. We are interested in probing deeper on things we’ve learned from talk with other profiles similar to yours and also testing some ideas with you.”

    3. There are no right or wrong answers here, and even negative feedback will help us make better product and service experiences. We know that some aspects of your job are private, so if at any time you don’t want to answer a question please feel free to say so: we won’t be offended. Our interest is to learn from you. Your input is very important to us — we couldn’t do this without you and can’t thank you enough for taking the time to help us.”

    4. Any questions on context before we begin?



  1. Background

    1. Tell me a little about what you do. What type of organization do you work for and what is your role?

    2. What are some of your favorite experiences with products you’ve used to do accomplish your research goals?

      1. [Probe] specific names, use case, feature, organization

  2. Probing on insights

    1. From our interviews, we’ve heard there are 3 top needs for people in your role when it comes to grants. Are these true for you? Say more. Why or why not?

      1. Research does not the data I need in it to put into practice

      2. Hard to sustain relationships with state and local leaders

      3. Finding the right type of research is a challenge

  3. Favorite resources

    1. Think of the research resources you use to help your job

      1. Describe the one you like most.

      2. Be specific- What are its characteristics? Does it involve other people? Where did you discover it? Include specific names

  4. Testing ideas

    1. We heard from others like you that these types of solutions might be useful.

      1. User driven feedback systems

      2. Specialized knowledge and forums (grade and subject specific)

    2. Which of those sounds helpful? Unhelpful? Why?

  5. Wrap up

    1. Are there any things you thought we’d talk about today that we didn’t cover? Anything else to add?

    2. Thanks again for allowing us to come spend this time with you today!”





Focus Groups



Focus groups will follow the same protocol as the 1:1 interviews with the following changes to the script:

    1. We want you to do the talking. We would like everyone to participate. I may call on you if I haven’t heard from you in a while.

    2. Please be tolerant of others’ thoughts and opinions and let’s have just one speaker at a time.

    3. The discussion will last for a little over one hour. Please silence your mobile phones.  Please give everyone the chance to express his/her opinion during the conversation. You can address each other if you like. I am here to facilitate the discussion.



User testing



As a part of the interviews, we will include a user test.

Participants will be shown a new webpage design or concept (e.g., new homepage layout). This will help the research team understand how usable the new design is for a given user group.

Script:

Now we would like to you participate in an activity. Our goal is to understand how you think about and categorize information. Just like with the rest of the interview, there are no right or wrong answers.

First, tell me what you see on the page.

Now, navigate around the site and talk through what you expect to occur.

If there are things you are expecting that do not occur, please tell us.

Questions throughout:

Why did you navigate there?

What were you expecting to occur?



Card Sort



As a part of each of the interviews, we will include a cart sorting question.

Participants will be shown a list of various parts of the IES website to group (e.g., RELs, research grants, NAEP). This will help the research team see how the interviewee thinks about information in order to inform the website navigation.

Script:

Now we would like to you participate in an activity. Our goal is to understand how you think about and categorize information. Just like with the rest of the interview, there are no right or wrong answers.

First, group the cards in whatever way makes sense to you.

Now, name the sections. Whatever makes sense to you as a name for something that contains the contents of the group you just created.

If you see something no longer makes logical sense, you can change your groupings.

Questions throughout:

Why did you use group in that way?

Where else were you thinking about placing information?





File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorMolin, Ed C
File Modified0000-00-00
File Created2021-01-13

© 2024 OMB.report | Privacy Policy