Appendix C_IADA Assessment Vendor Instruments 06-12-24

Evaluation of the Innovative Assessment Demonstration Authority Pilot Program Survey Data Collections

Appendix C_IADA Assessment Vendor Instruments 06-12-24

OMB:

Document [docx]

Download: docx | pdf

6-10-24 DRAFT

Appendix C :
IADA System Assessment Vendor
Data Collection Instruments

OMB#: 1850-XXXX

Expiration Date: XX/XX/XXXX

Evaluation of the Innovative Assessment Demonstration Authority (IADA) Pilot Program:

IADA Assessment Vendor Representative Survey

Summer 2024

Notice of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Participating state assessment systems will be identified, but not individual survey participants. All of the information you provide may only be used for research purposes and may not be disclosed or used in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Paperwork Reduction Act of 1995

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-XXXX. The approximate time required to complete the survey is 20 minutes, including the time to review instructions, search existing data resources, gather the needed data, and complete and review the information collection. Your response to this collection is voluntary. If you have any comments concerning the accuracy of the time estimates or suggestions for improving this survey, please write to: U.S. Department of Education, Washington, DC 20202-4651.

Introduction to the Survey

You are invited to complete this 20-minute survey about the extent to which your organization experienced challenges developing and/or implementing the [NAME OF IADA SYSTEM]. This survey is part of the Congressionally mandated evaluation of the Innovative Assessment Demonstration Authority (IADA) program. [FOR THE IADA ASSESSMENT VENDORS FOR NH PACE, GMAP THROUGH-YEAR ASSESSMENT, OR NAVVY DISPLAY: Although [NAME OF IADA SYSTEM] is no longer part of the IADA program, we encourage you to participate in the survey because your experiences will provide valuable lessons.] Your organization’s responses will provide critical information to the U.S. Department of Education on the implementation of the IADA program and will provide valuable information to other states and vendors that may want to develop a new assessment system in the future.

The survey asks you to share how much your organization experienced specific challenges associated with six key stages of assessment development and implementation. These activities include developing the initial plan for the IADA assessment, preparing for the first operational assessment, administering an operational assessment, conducting post-administration activities, conducting continuous improvement activities, and scaling up the system.
This survey is intended to capture the perspectives of the vendor with the primary responsibility for each major assessment development or implementation activity for the [NAME OF IADA SYSTEM]. As needed, please reach out to other members of your organization to contribute to survey responses; however, submit only one completed survey for the system.
Although voluntary, we strongly encourage you to participate in this survey so that your organization’s experiences with [NAME OF IADA SYSTEM] will be represented in the IADA Best Practices report required by Congress. [FOR THE LA, NC, AND MA SURVEYS DISPLAY: Participating (or not participating) in the survey will not affect the state’s IADA demonstration authority.]
After you complete this survey, the study team will request a follow-up interview to learn more about the major implementation activity identified in the survey as the most challenging. The team will ask about the specific challenges for this activity and proposed or implemented solutions.
Survey findings will be summarized in the Best Practices report. The report may present findings separately by IADA systems, but will not name individual survey participants. Survey responses will be available to authorized researchers through a restricted-use data file maintained by the U.S. Department of Education’s Institute of Education Sciences (IES). The data file will not include individual survey participant’s names.
IES has contracted with Westat, an independent research organization, to lead the evaluation and administer this survey.

Click the Next Button to
Continue to the Survey

Section 1. Major Assessment Development and Implementation Activities for the Vendor

The IADA Assessment Director indicated that your organization had primary responsibility for the following major assessment development and/or implementation activities that [FOR LA, MA, and NC DISPLAY: the system engaged in to at least some extent so far/FOR GA NAVVY, GMAP, AND NH DISPLAY: the system engaged in to at least some extent].

[DISPLAY ONLY THOSE ACTIVTIES FOR WHICH THE VENDOR WAS IDENTIFIED AS HAVING PRIMARY RESPONSIBILITY BY THE IADA DIRECTOR.]

Developing the initial plan for the IADA assessment. Example sub-activities include establishing test and item requirements, creating an implementation timeline, providing information to parents and students, and providing training to staff on the system’s purpose and major features.
Preparing for the first operational assessment. An operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards. Example sub-activities include developing initial items, piloting items, ensuring appropriate testing accommodations, developing assessment administration procedures, and conducting professional development for educators.
Administering an operational assessment. Example sub-activities include administering full assessments and calculating annual determinations of student proficiency.
Conducting post-administration activities. Example sub-activities include ensuring the technical quality of scores, producing student score reports, generating data reports for educators (such as classroom- or school-level results to inform instruction), and identifying schools for additional support in the accountability system.
Conducting continuous improvement activities. Example sub-activities include getting feedback from participating teachers, school leaders, or local education agency staff; and conducting annual evaluations.
Scaling up the system. Example sub-activities include increasing the number of districts or schools participating in the IADA assessment system over time.

The next portion of the survey asks you to rate the extent to which your organization experienced challenges associated with each major assessment development and implementation activity for which your organization had primary responsibility. When thinking about rating challenges, please think of your organization’s perspective, not your perspective of what the state perceived as a challenge. Please keep the following response definitions in mind as you rate the challenges:

Not applicable: the potential challenge was not a possibility for your organization. For example, select ‘not applicable’ if the assessment system had not yet reached the stage for the challenge to be addressed or if the potential challenge does not apply given the type of assessments included in the IADA system.
Not involved: your organization was not involved in the activity related to the potential challenge.
Not a Challenge: the potential challenge was not a concern; for example, the potential challenge did not cause changes to the design or implementation timeline of the system.
Minor Challenge: the potential challenge was somewhat problematic; for example, the potential challenge was difficult but did not cause significant setbacks or delays in implementation of the system, and may have led to additional planning time or the reallocation of resources.
Major Challenge: the potential challenge was considerably problematic; for example, the potential challenge may have caused significant setbacks or delays in implementation of the system, led to modifications to the original IADA system design, or required additional expertise as well as additional planning time or resources.

These definitions also are available to respondents by hovering over the words that appear in blue text in the web survey. As a reminder, please reach out to other members of your organization to contribute to survey responses as needed; however, submit only one completed survey for the organization.

Section 2. Challenges Related to Developing the Initial Plan for the IADA Assessment System

[DISPLAY SECTION ONLY IF IADA DIRECTOR IDENTIFIED VENDOR AS THE PRIMARY VENDOR FOR THIS ACTIVITY; OTHERWISE, SKIP TO SECTION 3]

2-1. To what extent was each of the following a challenge for your organization while developing the initial plan for the IADA assessment system? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS DISPLAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS DISPLAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If your organization experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	NOT INVOLVED
Engaging with families	1	2	3	NA	NI
Engaging with educators	1	2	3	NA	NI
Engaging with other stakeholders such as political leaders or community members	1	2	3	NA	NI
Recruiting districts or schools for the pilot or the initial administration	1	2	3	NA	NI
Developing tools to support the assessment such as an aligned assessment blueprint or standardized or calibrated tools for accurate and reliable scoring (such as rubrics)	1	2	3	NA	NI
Determining a method for evaluating the comparability of IADA results to results from the existing statewide assessment	1	2	3	NA	NI
Developing IADA assessment(s) that align to existing content standards	1	2	3	NA	NI
Scheduling sufficient planning time to support the first administration of operational IADA assessments	1	2	3	NA	NI
Allocating funds (within budget) to support the IADA assessment system	1	2	3	NA	NI
Securing additional funds to support the IADA assessment system such as grants	1	2	3	NA	NI
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):

[GO TO 2-2:

IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 3; OR
IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 3]

2-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 3, DISPLAY ONLY THOSE ROWS FROM 2-1A THROUGH 2-1M WITH A VALUE OF 3.

IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 2-1A THROUGH 2-1M WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Engaging with families	1
Engaging with educators	1
Engaging with other stakeholders such as political leaders or community members	1
Recruiting districts or schools for the pilot or the initial administration	1
Developing tools to support the assessment such as an aligned assessment blueprint or standardized or calibrated tools for accurate and reliable scoring (such as rubrics)	1
Determining a method for evaluating the comparability of IADA results to results from the existing statewide assessment		1
Developing IADA assessment(s) that align to existing content standards		1
Scheduling sufficient planning time to support the first administration of operational IADA assessments		1
Allocating funds (within budget) to support the IADA assessment system	1
Securing additional funds to support the IADA assessment system such as grants	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 3. Challenges Related to Preparing for the First Operational Assessment

[DISPLAY SECTION ONLY IF IADA DIRECTOR IDENTIFIED VENDOR AS THE PRIMARY VENDOR FOR THIS ACTIVITY; OTHERWISE, SKIP TO SECTION 4]

3-1. To what extent was each of the following a challenge for your organization while preparing for the first operational assessment? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMSDISPLAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMSDISPLAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If your organization experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

As a reminder, an operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	NOT INVOLVED
Developing or supplementing the item bank or pilot testing the items that determine proficiency or mastery of the state-approved competencies or academic standards	1	2	3	NA	NI
Preparing administration procedures or protocols for general administration.	1	2	3	NA	NI
Developing or deploying professional development, supports, or other resources for teachers or building administrators to score the IADA assessment	1	2	3	NA	NI
Developing or deploying supports or other resources to help parents and students interpret score reports	1	2	3	NA	NI
Developing or deploying supports or other resources to help other stakeholder groups such as political leaders or community members interpret score reports	1	2	3	NA	NI
Meeting specific technical challenges required of the IADA assessment (for example, challenges related to fairness, validity, bias, or sensitivity).	1	2	3	NA	NI
(Specify the technical challenge(s)):
Obtaining sufficient funding to support activities to prepare for the first operational assessment such as professional development	1	2	3	NA	NI
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):

[GO TO 3-2:

IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 3; OR
IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 4]

3-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 3, DISPLAY ONLY THOSE ROWS FROM 3-1A THROUGH 3-1J WITH A VALUE OF 3.

IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 3-1A THROUGH 3-1J WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Developing or supplementing the item bank or pilot testing the items that determine proficiency or mastery of the state-approved competencies or academic standards	1
Preparing administration procedures or protocols for general administration.	1
Developing or deploying professional development, supports, or other resources for teachers or building administrators to score the IADA assessment	1
Developing or deploying supports or other resources to help parents and students interpret score reports	1
Developing or deploying supports or other resources to help other stakeholder groups such as political leaders or community members interpret score reports	1
Meeting specific technical challenges required of the IADA assessment (for example, challenges related to fairness, validity, bias, or sensitivity)..	1
(Specify the technical challenge(s)):
Obtaining sufficient funding to support activities to prepare for the first operational assessment such as professional development	1

Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 4. Challenges Related to Administering an Operational Assessment

[DISPLAY SECTION ONLY IF IADA DIRECTOR IDENTIFIED VENDOR AS THE PRIMARY VENDOR FOR THIS ACTIVITY; OTHERWISE, SKIP TO SECTION 5]

4-1. To what extent was each of the following a challenge for your organization while administering an operational assessment? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS DISPLAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS DISPLAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If your organization experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	NOT INVOLVED
Scheduling or administering the assessment	1	2	3	NA	NI
Ensuring that LEAs inform parents in participating schools about the IADA assessment	1	2	3	NA	NI
Troubleshooting during the administration such as maintaining a help desk, providing hardware or connectivity support, or recovering lost data	1	2	3	NA	NI
Maintaining the security of individual results for each student	1	2	3	NA	NI
Obtaining sufficient funding for the administration of an operational assessment	1	2	3	NA	NI
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):

[GO TO 4-2:

IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 3; OR
IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 5]

4-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 3, DISPLAY ONLY THOSE ROWS FROM 4-1A THROUGH 4-1H WITH A VALUE OF 3.

IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 4-1A THROUGH 4-1H WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Scheduling or administering the assessment	1
Ensuring that LEAs inform parents in participating schools about the IADA assessment	1
Troubleshooting during the administration such as maintaining a help desk, providing hardware or connectivity support, or recovering lost data	1
Maintaining the security of individual results for each student	1
Obtaining sufficient funding for the administration of an operational assessment	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 5. Challenges Related to Conducting Post-Administration Activities

[DISPLAY SECTION ONLY IF IADA DIRECTOR IDENTIFIED VENDOR AS THE PRIMARY VENDOR FOR THIS ACTIVITY; OTHERWISE, SKIP TO SECTION 6]

5-1. To what extent was each of the following a challenge for your organization while conducting post-administration activities? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS DISPLAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS DISPLAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If your organization experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	NOT INVOLVED
Collecting or gathering data for performance-based assessments	1	2	3	NA	NI
Ensuring the technical quality of the scores	1	2	3	NA	NI
Producing individual student reports	1	2	3	NA	NI
Applying student results for accountability	1	2	3	NA	NI
Using student results to improve instruction or guide decisions	1	2	3	NA	NI
Determining comparability of IADA results to results from the existing statewide assessment	1	2	3	NA	NI
Determining alignment of IADA assessment to state content standards	1	2	3	NA	NI
Obtaining sufficient funding for post-administration activities	1	2	3	NA	NI
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):

[GO TO 5-2:

IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 3; OR
IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 6]

5-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 3, DISPLAY ONLY THOSE ROWS FROM 5-1A THROUGH 5-1K WITH A VALUE OF 3.

IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 5-1A THROUGH 5-1K WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Collecting or gathering data for performance-based assessments	1
Ensuring the technical quality of the scores	1
Producing individual student reports	1
Applying student results for accountability	1
Using student results to improve instruction or guide decisions	1
Determining comparability of IADA results to results from the existing statewide assessment	1
Determining alignment of IADA assessment to state content standards	1
Obtaining sufficient funding for post-administration activities	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 6. Challenges Related to Conducting Continuous Improvement Activities

[DISPLAY SECTION ONLY IF IADA DIRECTOR IDENTIFIED VENDOR AS THE PRIMARY VENDOR FOR THIS ACTIVITY; OTHERWISE, SKIP TO SECTION 7]

6-1. To what extent was each of the following a challenge for your organization while conducting continuous improvement activities? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS DISPLAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS DISPLAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If your organization experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	NOT INVOLVED
Promoting appropriate data use by educators	1	2	3	NA	NI
Obtaining feedback from families on their satisfaction with the IADA assessment system	1	2	3	NA	NI
Obtaining feedback from educators on their satisfaction with the IADA assessment system	1	2	3	NA	NI
Obtaining feedback from other stakeholders (such as political leaders, community members) on their satisfaction with the IADA assessment system	1	2	3	NA	NI
Conducting an annual evaluation	1	2	3	NA	NI
Obtaining sufficient funding to support continuous improvement activities	1	2	3	NA	NI
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):

[GO TO 6-2:

IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 3; OR
IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 7]

6-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 3, DISPLAY ONLY THOSE ROWS FROM 6-1A THROUGH 6-1I WITH A VALUE OF 3.

IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 6-1A THROUGH 6-1I WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Promoting appropriate data use by educators	1
Obtaining feedback from families on their satisfaction with the IADA assessment system	1
Obtaining feedback from educators on their satisfaction with the IADA assessment system	1
Obtaining feedback from other stakeholders (such as political leaders, community members) on their satisfaction with the IADA assessment system	1
Conducting an annual evaluation	1
Obtaining sufficient funding to support continuous improvement activities
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 7. Challenges Related to Scaling Up the IADA Assessment System

[DISPLAY SECTION ONLY IF IADA DIRECTOR IDENTIFIED VENDOR AS THE PRIMARY VENDOR FOR THIS ACTIVITY; OTHERWISE, SKIP TO SECTION 8]

7-1. To what extent was each of the following a challenge for your organization regarding scaling up the IADA assessment system? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS DISPLAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS DISPLAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If your organization experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

(Select 'not applicable' only if the potential challenge was not a possibility for the assessment system. For example, select ‘not applicable’ if the system had not yet reached the stage for the challenge to be addressed or if the potential challenge does not apply given the type of assessments included in the system.

Select ‘not involved’ only if your organization was not involved in the activity related to the potential challenge.)

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	NOT INVOLVED
Recruiting districts or schools to participate in the IADA assessment system	1	2	3	NA	NI
Establishing IADA assessment quality or other triggers to signal readiness for statewide implementation	1	2	3	NA	NI
Scaling up the system statewide within the allotted time	1	2	3	NA	NI
Obtaining sufficient funding for scale-up activities	1	2	3	NA	NI
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):
Other challenge	1	2	3	NA	NI
(Specify):

[GO TO 7-2:

IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 3; OR
IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 8]

7-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 3, DISPLAY ONLY THOSE ROWS FROM 7-1A THROUGH 7-1G WITH A VALUE OF 3.

IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 7-1A THROUGH 7-1G WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Recruiting districts or schools to participate in the IADA assessment system	1
Establishing IADA assessment quality or other triggers to signal readiness for statewide implementation	1
Scaling up the system statewide within the allotted time	1
Obtaining sufficient funding for scale-up activities	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 8. Most Difficult Challenge and Most Difficult Activity

8-1. In the previous sections, you identified the most difficult challenge for each major assessment development and/or implementation activity. Which of these challenges was the top challenge for your organization as a whole?

SYSTEM’S MOST DIFFICULT CHALLENGE FOR EACH ACTIVITY	SELECT THE TOP CHALLENGE FOR THE SYSTEM (SELECT ONLY ONE)
[IF ONLY ONE OF 2-1A THROUGH 2-1M = 3, THEN DISPLAY THAT ROW FROM 2-1; ELSE IF ONLY ONE OF 2-1A THROUGH 2-1M = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 2-1; ELSE DISPLAY ROW SELECTED FOR ITEM 2-2]	1
[IF ONLY ONE OF 3-1A THROUGH 3-1J = 3, THEN DISPLAY THAT ROW FROM 3-1; ELSE IF ONLY ONE OF 3-1A THROUGH 3-1J = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 3-1; ELSE DISPLAY ROW SELECTED FOR ITEM 3-2]	1
[IF ONLY ONE OF 4-1A THROUGH 4-1H = 3, THEN DISPLAY THAT ROW FROM 4-1; ELSE IF ONLY ONE OF 4-1A THROUGH 4-1H = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 4-1; ELSE DISPLAY ROW SELECTED FOR ITEM 4-2]	1
[IF ONLY ONE OF 5-1A THROUGH 5-1K = 3, THEN DISPLAY THAT ROW FROM 5-1; ELSE IF ONLY ONE OF 5-1A THROUGH 5-1K = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 5-1; ELSE DISPLAY ROW SELECTED FOR ITEM 5-2]	1
IF ONLY ONE OF 6-1A THROUGH 6-1I = 3, THEN DISPLAY THAT ROW FROM 6-1; ELSE IF ONLY ONE OF 6-1A THROUGH 6-1I = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 6-1; ELSE DISPLAY ROW SELECTED FOR ITEM 6-2]	1
[IF ONLY ONE OF 7-1A THROUGH 7-1G = 3, THEN DISPLAY THAT ROW FROM 7-1; ELSE IF ONLY ONE OF 7-1A THROUGH 7-1G = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 7-1; ELSE DISPLAY ROW SELECTED FOR ITEM 7-2]	1

8-2. Thinking about your organization’s experience developing and/or implementing [NAME OF IADA SYSTEM] at any point up to or through [FOR CURRENT SYSTEMS DISPLAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS DISPLAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA], which activity was the most challenging?

[DISPLAY ONLY THOSE ACTIVITIES THAT THE IADA DIRECTOR IDENTIFIED THE VENDOR AS HAVING PRIMARY RESPONSIBILITY.]

ACTIVITIES	SELECT THE MOST CHALLENGING ACTIVITY (SELECT ONLY ONE)
Developing the initial plan for the IADA assessment	1
Preparing for the first operational assessment	1
Administering an operational assessment	1
Conducting post-administration activities	1
Conducting continuous improvement activities	1
Scaling up the system	1

Section 9. Concluding Section

9-1. Thank you for responding to the survey questions. Did other staff from your organization contribute to the survey responses?

1 □ Yes CONTINUE TO 9-2

0 □ No SKIP TO END OF SURVEY

9-2. Please describe how these other individuals contributed to the IADA assessment system and list their titles.

Shape1

This is the end of the survey. Thank you for your participation!

OMB#: 1850-XXXX

Expiration Date: XX/XX/XXXX

Evaluation of the Innovative Assessment Demonstration Authority (IADA) Pilot Program:

IADA Assessment Vendor Representative Interview Protocol

Summer 2024

Notice of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (IES) (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Participating state assessment system will be identified, but not individual interview participants. All of the information you provide may only be used for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Paperwork Reduction Act of 1995

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-XXXX. The approximate time required to participate in this interview is 60 minutes. Your participation is voluntary. If you have any comments concerning the accuracy of the time estimates or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202-465

Section 1. Introduction and Consent (5 minutes)

Thank you for taking the time to speak with me. I am [NAME] of Westat, a research organization in Rockville, MD. I am joined by my colleague [NAME] taking notes. The U.S. Department of Education’s Institute of Education Sciences (IES) has contracted with Westat to evaluate the Innovative Assessment Demonstration Authority (IADA) pilot program, which, as you know includes the [NAME OF IADA SYSTEM] in [STATE]. (For interviews with assessment vendor representatives for NH PACE, GMAP through-year assessment, or Navvy add, Although [NAME OF IADA SYSTEM] is no longer part of the IADA program, we encourage you to participate in the interview because we believe your experiences will provide valuable lessons for others.)

We greatly appreciate your time and willingness to talk about your organization’s experiences with the IADA pilot program. The interview focuses on the most challenging activities your organization encountered while implementing [NAME OF IADA SYSTEM] and your organization’s strategies for addressing these challenges. In addition, we will ask you to share what practices worked well, those that did not work well or as planned, and practices that you thought might work well but were unable to try and why.

This interview will last about 60 minutes. Participation in the interview is voluntary. [For interviews with assessment representatives from LA, NC, and MA add, Participating (or not participating) in the interview will not affect the state’s IADA demonstration authority.] However, your organization’s participation is important because your experiences will help inform policymakers and other states and vendors interesting in developing innovative student assessment systems. Westat will summarize the findings in a Congressionally mandated report that may include quotes from assessment vendor representatives without naming individual respondents. The report also may present detailed findings by assessment system. Your coded responses will be available to authorized researchers through a restricted-use file maintained by IES, but you will not be identified by name in any of these files.

(If the IADA assessment vendor representative is the only interview participant, note that here and skip to question 1.1)

If anyone is joining the IADA assessment vendor representative on the call, say: Let’s start with short introductions. Again, I am [NAME] with Westat. Please introduce yourselves and briefly describe your role related to the [NAME OF IADA SYSTEM].

Enter each participant’s name and role

Name	Role

1.1 Do you agree to participate in the interview?

Yes (Continue)
No (Thank participant and end call)

(If there are multiple participants, obtain agreement to participate from each participant. Thank and excuse any participant not willing to participate.)

Thank you.

1.2. Do you have any questions for me before we get started? (Note and respond to any questions.)

With your permission, I would like to record the interview. The recording will be transcribed, used to verify our notes, and destroyed after the study. Westat will not share the recording or transcription with anyone outside of the research team. You may ask me to stop recording at any point during the interview.

1.3. Do I have your permission to record the interview?

Yes (Start recording)
No (Do not start recording and only take notes.)

(If there are multiple participants, ask each participant if they give permission to record the interview. If any refuse, do not start recording and only take notes.)

Thank you.

(If there are multiple participants, say: As I go through the interview, I will invite everyone’s input. However, if there are differing opinions, I’ll ask the group to work towards a consensus, or final answer, once everyone has had the opportunity to share their perspective.)

Section 2. Major Assessment Implementation Activity Identified as the Most Challenging (20 minutes)

In this first portion of our interview, I would like to discuss the major development and/or implementation activity for [NAME OF IADA SYSTEM] you believe was the most challenging for your organization and the strategies your organization used to address the challenges. As you may recall, prior to this interview, we surveyed your organization about the major IADA assessment development and/or implementation activities for which your organization had primary responsibility. Also, in rating challenges, we asked that you used your organization’s perspective, not your perspective of what the state perceived as a challenge.

The major activities covered in the survey were:

[Pre-populate list of activities based on activities identified for the vendor by the IADA Assessment Director.]

Developing the initial plan for the IADA assessment,
Preparing for the first operational assessment,
Administering an operational assessment,
Conducting post-administration activities such as ensuring technical quality of scores and using reports,
Conducting continuous improvement activities, and
Scaling up the system.

The survey response indicates that [ACTIVITY SELECTED AS THE MOST CHALLENGING (RESPONSE TO SURVEY ITEM 8-2)] was the most challenging major activity. Please keep this in mind as we move to the next set of questions.

(If the most challenging activity was:

Developing the initial plan for the IADA assessment, then go to Section 2.1
Preparing for the first operational IADA assessment, then go to Section 2.2
Administering an operational assessment, then go to Section 2.3
Conducting post-administration activities, then go to Section 2.4
Conducting continuous improvement activities, then go to Section 2.5
Scaling up the system, then go to Section 2.6)

(Throughout the interview, if the interview includes multiple participants and there is disagreement on a response to any item, give each participant the opportunity to voice their opinions. Take note of agreement and disagreement to facilitate the group to focus on area(s) of disagreement, then say: It appears the group is working to decide whether [POTENTIAL ANSWER A OR POTENTIAL ANSWER B] is a consensus answer to [RESTATE THE QUESTION]. Which of these would be most useful to discuss in a best practice report that can inform other states interested in innovative assessments?)

Section 2.1. Major Assessment Implementation Activity #1: Developing the initial plan for the IADA Assessment

(Prepopulate questions 2.1.1 and 2.1.2 with the challenge identified as a major challenge from survey item 2-1. If no challenges were identified as major by the respondent in survey item 2-1, prepopulate questions 2.1.1 – 2.1.2 below with the minor challenge identified as the most difficult challenge in survey item 2-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.1.1 through 2.1.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #1: Developing the initial plan for the IADA assessment.

[If major challenges are displayed, say: Recall that we defined a major challenge as one that was considerably problematic; for example, the potential challenge may have caused significant setbacks or delays in implementation of the system, led to modifications to the original IADA system design, or required additional expertise as well as additional planning time or resources.

If the most difficult challenge is displayed, say: Recall that we defined a minor challenge as one that was somewhat problematic; for example, the potential challenge was difficult but did not cause significant setbacks or delays in implementation of the system, and may have led to additional planning time or the reallocation of resources.]

2.1.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER INITIAL PLANNING ON SURVEY ITEM 2-1/2-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: Was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of staff?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.1.2. Did your organization work with the state to identify and implement a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER DEVELOPING THE INITIAL PLAN ON SURVEY ITEM 2-1/2-2]?

(If the respondent says ‘no’, ask the probing questions below to verify that they did not identify a strategy to address the challenge, then skip to the next major challenge that was identified under this activity area if applicable.

If the respondent says ‘yes’, say: Please describe the strategy. If your organization used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your organization try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Work with the state to change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.1.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the organization used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.2. Major Assessment Implementation Activity #2: Preparing for the First Operational Assessment

(Prepopulate questions 2.2.1 and 2.2.2 with the challenge identified as a major challenge from survey item 3-1. If no challenges were identified as major by the respondent in survey item 3-1, prepopulate questions 2.2.1 – 2.2.2 below with the minor challenge identified as the most difficult challenge in survey item 3-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.2.1 through 2.2.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #2: Preparing for the first operational assessment. As a reminder, an operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards.

If the most difficult minor challenge is displayed, say: Recall that we defined a minor challenge as one that was somewhat problematic; for example, the potential challenge was difficult but did not cause significant setbacks or delays in implementation of the system, and may have led to additional planning time or the reallocation of resources.]

2.2.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER PREPARING FOR THE FIRST OPERATIONAL ASSESSMENT ON SURVEY ITEM 3-1/3-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of available time for current staff to prepare for administration)?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.2.2. Did your organization work with the state to identify and implement a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER PREPARING FOR THE FIRST OPERATIONAL ASSESSMENT ON SURVEY ITEM 3-1/3-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your organization used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your organization try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Change the timeline for the first operational assessment?
Leverage existing tools or resources?

2.2.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the organization used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.3. Major assessment implementation activity #3: Administering an operational assessment

(Prepopulate questions 2.3.1 and 2.3.2 with the challenge identified as a major challenge from survey item 4-1. If no challenges were identified as major by the respondent in survey item 4-1, prepopulate questions 2.3.1 – 2.3.2 below with the minor challenge identified as the most difficult challenge in survey item 4-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.3.1 through 2.3.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #3: Administering an operational assessment. As a reminder, an operational assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject, and that IADA results can be used to determine whether students are proficient against the state’s academic standards.

2.3.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER ADMINISTERING OPERATIONAL ASSESSMENT ON SURVEY ITEM 4-1/4-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed, For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of staff capacity?
Lack of public support, for example districts’ or schools’ unwillingness to schedule the assessment?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.3.2. Did your organization work with the state to identify and implement a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER ADMINISTERING OPERATIONAL ASSESSMENT ON SURVEY ITEM 4-1/4-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your organization used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your organization try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Leverage existing tools or resources?

2.3.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the organization used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most minor challenge, go to Section 2.7)

Section 2.4. Major Assessment Implementation Activity #4: Conducting post-administration Activities

(Prepopulate questions 2.4.1 and 2.4.2 with the challenge identified as a major challenge from survey item 5-1. If no challenges were identified as major by the respondent in survey item 5-1, prepopulate questions 2.4.1 – 2.4.2 below with the minor challenge identified as the most difficult challenge in survey item 5-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.4.1 through 2.4.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #4: Conducting post-administration activities.

2.4.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING POST-ADMINISTRATION ACTIVITIES ON SURVEY ITEM 5-1/5-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of knowledge about how to collect performance-based assessment data?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources?

2.4.2. Did your organization work with the state to identify and implement a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING POST-ADMINISTRATION ACTIVITIES ON SURVEY ITEM 5-1/5-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your organization used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your organization try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.4.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the organization used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.5. Major Assessment Implementation Activity #5: Conducting Continuous Improvement Activities

(Prepopulate questions 2.5.1 and 2.5.2 with the challenge identified as a major challenge from survey item 6-1. If no challenges were identified as major by the respondent in survey item 6-1, prepopulate questions 2.5.1 – 2.5.2 below with the minor challenge identified as the most difficult challenge in survey item 6-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.5.1 through 2.5.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #5: Conducting continuous improvement activities.

2.5.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING CONTINUOUS IMPROVEMENT ACTIVITIES ON SURVEY ITEM 6-1/6-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of data literacy skills such as accurately interpreting the scores or an understanding of how to use the scores to plan for or target instruction?
Lack of public support, for example unwillingness to participate in feedback opportunities or the annual evaluation?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources to conduct an evaluation?

2.5.2. Did your organization work with the state to identify and implement a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING CONTINUOUS IMPROVEMENT ACTIVITIES ON SURVEY ITEM 6-1/6-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your organization used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your organization try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Leverage existing tools or resources?

2.5.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the organization used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.6. Major Assessment Implementation Activity #6: Scaling Up the System

(Prepopulate questions 2.6.1 and 2.6.2 with the challenge identified as a major challenge from survey item 7-1. If no challenges were identified as major by the respondent in survey item 7-1, prepopulate questions 2.6.1 – 2.6.2 below with the minor challenge identified as the most difficult challenge in survey item 7-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.6.1 through 2.6.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #6: Scaling up the system.

2.6.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER SCALING ON SURVEY ITEM 7-1/7-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of staff capacity?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.6.2. Did your organization work with the state to identify and implement a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER SCALING ON SURVEY ITEM 7-1/7-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your organization used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your organization try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage- existing tools or resources?

2.6.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the organization used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.7 Additional Activity Seen as Most Challenging

(Ask the questions in this section only if the study team’s scoring of the challenges responses in the surveys suggested that another major activity was very challenging for the system; otherwise go to Section 3.)

2.7.1 Your survey responses suggested that [MOST CHALLENGING ACTIVITY BASED ON SCORES] also was an activity with a lot of challenges for your organization. That is, the top challenge you reported for system, [CHALLENGE LISTED IN SURVEY ITEM 8-1], did not fall under the [ACTIVITY SELECTED AS THE MOST CHALLENGING (RESPONSE TO SURVEY ITEM 8-2)]. What was it about [ACTIVITY SELECTED AS THE MOST CHALLENGING BASED ON RESPONSE TO SURVEY ITEM 8-2] that made it more challenging than [MOST CHALLENGING ACTIVITY BASED ON SCORES]?

Section 3.0 What Worked Well, What Did Not Work As Intended, and Why? (15 minutes)

In the last section, I asked you to tell me about specific challenges related to a particular activity, but now I would like you to think more broadly about the major assessment implementation activities for the [NAME OF IADA SYSTEM] that your organization [FOR CURRENT SYSTEMS SAY: WAS THE PRIMARY VENDOR AT ANY POINT UP TO OR THROUGH THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: WAS THE PRIMARY VENDOR AT ANY POINT UP TO OR THROUGH THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. Considering all relevant activities, please identify what worked well in terms of supporting the system’s objectives and why. As a reminder, the major activities are:

[Pre-populate list of activities based on activities identified for the vendor by the IADA Assessment Director.]

Developing the initial plan for the IADA assessment,
Preparation for the first operational IADA assessment,
Administering an operational assessment,
Conducting post-administration activities such as ensuring technical quality of scores and using reports,
Conducting continuous improvement activities, and
Scaling up the IADA assessment statewide.

3.1. Thinking across the activities your organization engaged in, what practices worked well? Discuss up to three practices that worked well.

Probe if needed, For example, were there particular practices your organization implemented that contributed to the success of system such as, the practice of recruiting participating districts and schools?

(As respondent is speaking, match the successful practice with the list of major activities using reference list, and then ask the respondent to confirm the match by saying: It sounds like this practice was related to the [MAJOR ACTIVITY X]. Is that correct? Is there an additional activity you would relate this practice to?)

3.2 How did the [THE RESPONDENT’S PRACTICE ANSWER] contribute to the success of the system?

(Repeat the question for each of up to three practices identified in the previous question and record answers. Then ask: Which one practice was the most important contributor to progress overall for [NAME OF IADA SYSTEM]?)

3.3 Please think about your organization’s support of the IADA system broadly. Were there aspects of the system’s theory of action or the system’s original design that did not work as intended? For example, were there aspects related to the planned grade(s) or subjects tested or planned method for calculating the annual determination of student proficiency that did not work as intended?

Add if needed: This question is related to the system’s overall design rather than about the specific challenges related to assessment activities discussed above.

3.4 Why did [THE RESPONDENT’S ASPECT ANSWER] not work as intended?

(Repeat this question for each aspect mentioned and record answers. Then ask: Which one aspect stands out as the most important one that you’d want other states and vendors to know about and learn from?)

Section 4.0 Practices That the Organization Wanted to Try, But Was Unable To (15 minutes)

4.1. I would like you to reflect on the objectives of the [NAME OF IADA SYSTEM]. Were there any practices that your organization wanted to implement to support these objectives, but was unable to? What were these practices that your organization wanted to implement, but did not, and why? (If none, go to Section 5.)

Probe if needed, For example, were there barriers that prevented your organization from implementing a practice that you felt would have supported the objectives of the [NAME OF IADA SYSTEM]?

(As respondent is speaking, match the practice they wanted to try with the list of major activities using reference list, and then ask the respondent to confirm the match by saying, ‘It sounds like this practice was related to the [MAJOR ACTIVITY X]. Is that correct? Is there an additional activity you would relate this practice to?’)

4.2. What prevented your organization from trying [THE RESPONDENT’S PRACTICE ANSWER]?

(Repeat this question for each practice mentioned and record answers. Then ask: Which one practice faced the largest barrier to trying it?)

Probe if needed, were there barriers related to:

The political climate?
Pressures from the public?
State legislation or policy?
Funding or other resources?
Technical issues?
The IADA program requirements?
The COVID pandemic?

Section 5.0 Wrap Up and Thank You (5 minutes)

5.1. Is there anything we did not discuss regarding the implementation of the [NAME OF IADA SYSTEM] that you would like to share with me?

Thank you for your time.

C-1

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2024-09-12