Evaluation of the Innovative Demonstration Authority Pilot Program

Evaluation of the Innovative Assessment Demonstration Authority Pilot Program Survey Data Collections

Appendix B_IADA TAC Member Instruments 06-12-24

Evaluation of the Innovative Demonstration Authority Pilot Program

OMB:

Document [docx]

Download: docx | pdf

06-11-24 DRAFT

Appendix B :
IADA System Technical Advisory Committee Member Data Collection Instruments

OMB#: 1850-XXXX

Expiration Date: XX/XX/XXXX

Evaluation of the Innovative Assessment Demonstration Authority (IADA) Pilot Program:

IADA Technical Advisory Committee Member Survey

Summer 2024

Notice of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Participating state assessment systems will be identified, but not individual survey participants. All of the information you provide may only be used for research purposes and may not be disclosed or used in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Paperwork Reduction Act of 1995

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-XXXX. The approximate time required to complete the survey is 20 minutes, including the time to review instructions, search existing data resources, gather the needed data, and complete and review the information collection. Your response to this collection is voluntary. If you have any comments concerning the accuracy of the time estimates or suggestions for improving this survey, please write to: U.S. Department of Education, Washington, DC 20202-4651.

Introduction to the Survey

You are invited to complete this 20-minute survey about the extent to which [FOR LA, MA, NC, NH DISPLAY: NAME OF STATE; FOR GMAP DISPLAY: the GMAP Assessment Partnership; FOR NAVVY DISPLAY: the Putnam Consortium] experienced challenges developing and implementing the [NAME OF IADA SYSTEM]. This survey is part of the Congressionally mandated evaluation of the Innovative Assessment Demonstration Authority (IADA) program. [FOR THE TAC MEMBERS FOR NH PACE, GMAP THROUGH-YEAR ASSESSMENT, OR NAVVY DISPLAY: Although [NAME OF IADA SYSTEM] is no longer part of the IADA program, we encourage you to participate in the survey because your input will provide valuable lessons.] Your responses will provide critical information to the U.S. Department of Education on the implementation of the IADA program and will provide valuable information to other states that may want to develop a new assessment system in the future.

The survey asks you to share how much the IADA system experienced specific challenges associated with six key stages of assessment development and implementation. These activities include developing the initial plan for the IADA assessment, preparing for the first operational assessment, administering an operational assessment, conducting post-administration activities, conducting continuous improvement activities, and scaling up the system.
Although voluntary, we strongly encourage you to participate in this survey so that the experiences of [NAME OF IADA SYSTEM] will be represented in the IADA Best Practices report required by Congress. [FOR THE LA, NC, AND MA SURVEYS DISPLAY: Participating (or not participating) in the survey will not affect the state’s IADA demonstration authority.]
After you complete this survey, the study team will request a follow-up interview with you and other members of the system’s technical advisory committee (TAC) to learn more about the major implementation activity identified in the survey as the most challenging. The team will ask about the specific challenges for this activity and proposed or implemented solutions.
Survey findings will be summarized in the Best Practices report. The report may present findings separately by IADA systems, but will not name individual survey participants. Survey responses will be available to authorized researchers through a restricted-use data file maintained by the U.S. Department of Education’s Institute of Education Sciences (IES). The data file will not include individual survey participant’s names.
IES has contracted with Westat, an independent research organization, to lead the evaluation and administer this survey.

Click the Next Button to
Continue to the Survey

Section 1. Major Assessment Development and Implementation Activities for the [STATE/SYSTEM]

This section asks you to identify which of six activities (stages) of assessment development and implementation [NAME OF IADA SYSTEM] was engaged in at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. The six activities to consider are listed below with example sub-activities. Please note that your system design may have had different and/or additional sub-activities.

Developing the initial plan for the IADA assessment. Example sub-activities include establishing test and item requirements, creating an implementation timeline, providing information to parents and students, and providing training to staff on the system’s purpose and major features.
Preparing for the first operational assessment. An operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards. Example sub-activities include developing initial items, piloting items, ensuring appropriate testing accommodations, developing assessment administration procedures, and conducting professional development for educators.
Administering an operational assessment. Example sub-activities include administering full assessments and calculating annual determinations of student proficiency.
Conducting post-administration activities. Example sub-activities include ensuring the technical quality of scores, producing student score reports, generating data reports for educators (such as classroom- or school-level results to inform instruction), and identifying schools for additional support in the accountability system.
Conducting continuous improvement activities. Example sub-activities include getting feedback from participating teachers, school leaders, or local education agency staff; and conducting annual evaluations.
Scaling up the system. Example sub-activities include increasing the number of districts or schools participating in the IADA assessment system over time.

Thinking about the [STATE’S/SYSTEM’s] experience of developing and implementing [NAME OF IADA SYSTEM], identify whether the [STATE/SYSTEM] engaged in the listed activities below at any point up to or through the system’s [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. The system did not have to complete the activity to be considered “engaged in” the activity.

ACTIVITIES	SELECT ONE RESPONSE IN EACH ROW
ACTIVITIES	ENGAGED IN THE ACTIVITY	WAS NOT ENGAGED IN THE ACTIVITY
Developing the initial plan for the IADA assessment	1	0
Preparing for the first operational assessment	1	0
Administering an operational assessment	1	0
Conducting post-administration activities	1	0
Conducting continuous improvement activities	1	0
Scaling up the system	1	0

For which activities did you serve on the TAC for the [NAME OF IADA SYSTEM]? Select an activity if you served on the TAC at any time when the system was engaging in the activity.

[DISPLAY ONLY THOSE ACTIVITIES WHERE RESPONSE TO 1-1 = 1]

ACTIVITIES	I SERVED ON THE TAC DURING THIS ACTIVITY (SELECT ONE RESPONSE IN EACH ROW)
ACTIVITIES	YES	NO
Developing the initial plan for the IADA assessment	1	0
Preparing for the first operational assessment	1	0
Administering an operational assessment	1	0
Conducting post-administration activities	1	0
Conducting continuous improvement activities	1	0
Scaling up the system	1	0

The next portion of the survey asks you to rate, based on your observations, the extent to which the [STATE/SYSTEM] experienced challenges associated with each major assessment development and implementation activity the [STATE/SYSTEM] engaged in. Please keep the following response definitions in mind as you rate the challenges:

Not applicable: the potential challenge was not a possibility for the assessment system. For example, select ‘not applicable’ if the system had not yet reached the stage for the challenge to be addressed or if the potential challenge does not apply given the type of assessments included in the system.
Not a Challenge: the potential challenge was not a topic considered by the TAC for guidance, a recommendation, or an endorsement, either as an agenda item or during the TAC’s discussions about the IADA assessment system.
Minor Challenge: the potential challenge was somewhat problematic; for example, the potential challenge was difficult but did not require substantive discussion or debate for the TAC to provide guidance, a recommendation, or an endorsement.
Major Challenge: the potential challenge was considerably problematic; for example, the potential challenge may have required substantive discussion or debate for the TAC to provide guidance, a recommendation, or an endorsement. (This may have led to the TAC making a recommendation to alter plans for or modify the IADA system.)

These definitions also are available to respondents by hovering over the words that appear in blue text in the web survey.

Section 2. Challenges Related to Developing the Initial Plan for the IADA Assessment System

[DISPLAY SECTION ONLY IF QUESTION 1-1a = 1; OTHERWISE, SKIP TO SECTION 3]

2-1. Based on your observations, to what extent was each of the following a challenge for the [STATE/SYSTEM] while developing the initial plan for the IADA assessment system? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If you observed that the [STATE/SYSTEM] experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	DON’T KNOW
Engaging with families	1	2	3	NA	DK
Engaging with educators	1	2	3	NA	DK
Engaging with other stakeholders such as political leaders or community members	1	2	3	NA	DK
Recruiting districts or schools for the pilot or the initial administration	1	2	3	NA	DK
Developing tools to support the assessment such as an aligned assessment blueprint or standardized or calibrated tools for accurate and reliable scoring (such as rubrics)	1	2	3	NA	DK
Determining a method for evaluating the comparability of IADA results to results from the existing statewide assessment	1	2	3	NA	DK
Developing IADA assessment(s) that align to existing content standards	1	2	3	NA	DK
Scheduling sufficient planning time to support the first administration of operational IADA assessments	1	2	3	NA	DK
Allocating funds (within budget) to meet IADA requirements	1	2	3	NA	DK
Securing additional funds to support the IADA assessment system such as grants	1	2	3	NA	DK
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):

[GO TO 2-2:

IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 3; OR
IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 3]

2-2. Based on your observations, which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 3, DISPLAY ONLY THOSE ROWS FROM 2-1A THROUGH 2-1M WITH A VALUE OF 3.

IF MORE THAN ONE OF 2-1A THROUGH 2-1M = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 2-1A THROUGH 2-1M WITH A VALUE OF 2]

POTENTIAL CHALLENGES		SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Engaging with families		1
Engaging with educators		1
Engaging with other stakeholders such as political leaders or community members		1
Recruiting districts or schools for the pilot or the initial administration		1
Developing tools to support the assessment such as an aligned assessment blueprint or standardized or calibrated tools for accurate and reliable scoring (such as rubrics)		1
Determining a method for evaluating the comparability of IADA results to results from the existing statewide assessment	1
Developing IADA assessment(s) that align to existing content standards	1
Scheduling sufficient planning time to support the first administration of operational IADA assessments	1
Allocating funds (within budget) to meet IADA requirements		1
Securing additional funds to support the IADA assessment system such as grants		1
Other challenge		1
(Specify):
Other challenge		1
(Specify):
Other challenge		1
(Specify):

Section 3. Challenges Related to Preparing for the First Operational Assessment

[DISPLAY SECTION ONLY IF QUESTION 1-1b = 1; OTHERWISE, SKIP TO SECTION 4]

3-1. Based on your observations, to what extent was each of the following a challenge for the [STATE/SYSTEM] while preparing for the first operational assessment? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If you observed that the [STATE/SYSTEM] experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

As a reminder, an operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	DON’T KNOW
Developing or supplementing the item bank or pilot testing the items that determine proficiency or mastery of the state-approved competencies or academic standards	1	2	3	NA	DK
Preparing administration procedures or protocols for general administration.	1	2	3	NA	DK
Developing or deploying professional development, supports, or other resources for teachers or building administrators to score the IADA assessment	1	2	3	NA	DK
Developing or deploying supports or other resources to help parents and students interpret score reports	1	2	3	NA	DK
Developing or deploying supports or other resources to help other stakeholder groups such as political leaders or community members interpret score reports	1	2	3	NA	DK
Meeting specific technical challenges required of the IADA assessment (for example, challenges related to fairness, validity, bias, or sensitivity).	1	2	3	NA	DK
(Specify the technical challenge(s)):
Obtaining sufficient funding to support activities to prepare for the first operational assessment such as professional development	1	2	3	NA	DK
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):					DK
Other challenge	1	2	3	NA	DK
(Specify):

[GO TO 3-2:

IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 3; OR
IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 4]

3-2. Based on your observations, which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 3, DISPLAY ONLY THOSE ROWS FROM 3-1A THROUGH 3-1J WITH A VALUE OF 3.

IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 3-1A THROUGH 3-1J WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Developing or supplementing the item bank or pilot testing the items that determine proficiency or mastery of the state-approved competencies or academic standards	1
Preparing administration procedures or protocols for general administration.	1
Developing or deploying professional development, supports, or other resources for teachers or building administrators to score the IADA assessment	1
Developing or deploying supports or other resources to help parents and students interpret score reports	1
Developing or deploying supports or other resources to help other stakeholder groups such as political leaders or community members interpret score reports	1
Meeting specific technical challenges required of the IADA assessment (for example, challenges related to fairness, validity, bias, or sensitivity)..	1
(Specify the technical challenge(s)):
Obtaining sufficient funding to support activities to prepare for the first operational assessment such as professional development	1

Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 4. Challenges Related to Administering an Operational Assessment

[DISPLAY SECTION ONLY IF QUESTION 1-1c = 1; OTHERWISE, SKIP TO SECTION 5]

4-1. Based on your observations, to what extent was each of the following a challenge for the [STATE/SYSTEM] while administering an operational assessment? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If you observed that the [STATE/SYSTEM] experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	DON’T KNOW
Scheduling or administering the assessment	1	2	3	NA	DK
Ensuring that LEAs inform parents in participating schools about the IADA assessment	1	2	3	NA	DK
Troubleshooting during the administration such as maintaining a help desk, providing hardware or connectivity support, or recovering lost data	1	2	3	NA	DK
Maintaining the security of individual results for each student	1	2	3	NA	DK
Obtaining sufficient funding for the administration of an operational assessment	1	2	3	NA	DK
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):					DK
Other challenge	1	2	3	NA	DK
(Specify):

[GO TO 4-2:

IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 3; OR
IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 5]

4-2. Based on your observations, which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 3, DISPLAY ONLY THOSE ROWS FROM 4-1A THROUGH 4-1H WITH A VALUE OF 3.

IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 4-1A THROUGH 4-1H WITH A VALUE OF 2]

POTENTIAL CHALLENGES		SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Scheduling or administering the assessment		1
Ensuring that LEAs inform parents in participating schools about the IADA assessment	1
Troubleshooting during the administration such as maintaining a help desk, providing hardware or connectivity support, or recovering lost data		1
Maintaining the security of individual results for each student		1
Obtaining sufficient funding for the administration of an operational assessment		1
Other challenge		1
(Specify):
Other challenge		1
(Specify):
Other challenge		1
(Specify):

Section 5. Challenges Related to Conducting Post-Administration Activities

[DISPLAY SECTION ONLY IF QUESTION 1-1d = 1; OTHERWISE, SKIP TO SECTION 6]

5-1. Based on your observations, to what extent was each of the following a challenge for the [STATE/SYSTEM] while conducting post-administration activities? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If you observed that the [STATE/SYSTEM] experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	DON’T KNOW
Collecting or gathering data for performance-based assessments	1	2	3	NA	DK
Ensuring the technical quality of the scores	1	2	3	NA	DK
Producing individual student reports	1	2	3	NA	DK
Applying student results for accountability	1	2	3	NA	DK
Using student results to improve instruction or guide decisions	1	2	3	NA	DK
Determining comparability of IADA results to results from the existing statewide assessment	1	2	3	NA	DK
Determining alignment of IADA assessment to state content standards	1	2	3	NA	DK
Obtaining sufficient funding for post-administration activities	1	2	3	NA	DK
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):

[GO TO 5-2:

IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 3; OR
IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 6]

5-2. Based on your observations, which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 3, DISPLAY ONLY THOSE ROWS FROM 5-1A THROUGH 5-1K WITH A VALUE OF 3.

IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 5-1A THROUGH 5-1K WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Collecting or gathering data for performance-based assessments	1
Ensuring the technical quality of the scores	1
Producing individual student reports	1
Applying student results for accountability	1
Using student results to improve instruction or guide decisions	1
Determining comparability of IADA results to results from the existing statewide assessment	1
Determining alignment of IADA assessment to state content standards	1
Obtaining sufficient funding for post-administration activities	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 6. Challenges Related to Conducting Continuous Improvement Activities

[DISPLAY SECTION ONLY IF QUESTION 1-1e = 1; OTHERWISE, SKIP TO SECTION 7]

6-1. Based on your observations, to what extent was each of the following a challenge for the [STATE/SYSTEM] while conducting continuous improvement activities? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If you observed that the [STATE/SYSTEM] experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	DON’T KNOW
Promoting appropriate data use by educators	1	2	3	NA	DK
Obtaining feedback from families on their satisfaction with the IADA assessment system	1	2	3	NA	DK
Obtaining feedback from educators on their satisfaction with the IADA assessment system	1	2	3	NA	DK
Obtaining feedback from other stakeholders (such as political leaders, community members) on their satisfaction with the IADA assessment system	1	2	3	NA	DK
Conducting an annual evaluation	1	2	3	NA	DK
Obtaining sufficient funding to support continuous improvement activities	1	2	3	NA	DK
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):

[GO TO 6-2:

IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 3; OR
IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 7]

6-2. Based on your observations, which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 3, DISPLAY ONLY THOSE ROWS FROM 6-1A THROUGH 6-1I WITH A VALUE OF 3.

IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 6-1A THROUGH 6-1I WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Promoting appropriate data use by educators	1
Obtaining feedback from families on their satisfaction with the IADA assessment system	1
Obtaining feedback from educators on their satisfaction with the IADA assessment system	1
Obtaining feedback from other stakeholders (such as political leaders, community members) on their satisfaction with the IADA assessment system	1
Conducting an annual evaluation	1
Obtaining sufficient funding to support continuous improvement activities
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 7. Challenges Related to Scaling Up the IADA Assessment System

[DISPLAY SECTION ONLY IF QUESTION 1-1f = 1; OTHERWISE, SKIP TO SECTION 8]

7-1. Based on your observations, to what extent was each of the following a challenge for the [STATE/SYSTEM] regarding scaling up the IADA assessment system? Consider whether it was a challenge at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. If you observed that the [STATE/SYSTEM] experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE	DON’T KNOW
Recruiting districts or schools to participate in the IADA assessment system	1	2	3	NA	DK
Establishing IADA assessment quality or other triggers to signal readiness for statewide implementation	1	2	3	NA	DK
Scaling up the system statewide within the allotted time	1	2	3	NA	DK
Obtaining sufficient funding for scale-up activities	1	2	3	NA	DK
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):
Other challenge	1	2	3	NA	DK
(Specify):

[GO TO 7-2:

IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 3; OR
IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 8]

7-2. Based on your observations, which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 3, DISPLAY ONLY THOSE ROWS FROM 7-1A THROUGH 7-1G WITH A VALUE OF 3.

IF MORE THAN ONE OF 7-1A THROUGH 7-1G = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 7-1A THROUGH 7-1G WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Recruiting districts or schools to participate in the IADA assessment system	1
Establishing IADA assessment quality or other triggers to signal readiness for statewide implementation	1
Scaling up the system statewide within the allotted time	1
Obtaining sufficient funding for scale-up activities	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 8. Most Difficult Challenge and Most Difficult Activity

8-1. In the previous sections, you identified the most difficult challenge for each major assessment development and implementation activity. Based on your observations, which of these challenges was the top challenge for the assessment system as a whole?

SYSTEM’S MOST DIFFICULT CHALLENGE FOR EACH ACTIVITY	SELECT THE TOP CHALLENGE FOR THE SYSTEM (SELECT ONLY ONE)
[IF QUESTION 1-1a = 1 AND [IF ONLY ONE OF 2-1A THROUGH 2-1M = 3, THEN DISPLAY THAT ROW FROM 2-1; ELSE IF ONLY ONE OF 2-1A THROUGH 2-1M = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 2-1; ELSE DISPLAY ROW SELECTED FOR ITEM 2-2]]	1
[IF QUESTION 1-1b = 1 AND [IF ONLY ONE OF 3-1A THROUGH 3-1J = 3, THEN DISPLAY THAT ROW FROM 3-1; ELSE IF ONLY ONE OF 3-1A THROUGH 3-1J = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 3-1; ELSE DISPLAY ROW SELECTED FOR ITEM 3-2]]]	1
[IF QUESTION 1-1c = 1 AND [IF ONLY ONE OF 4-1A THROUGH 4-1H = 3, THEN DISPLAY THAT ROW FROM 4-1; ELSE IF ONLY ONE OF 4-1A THROUGH 4-1H = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 4-1; ELSE DISPLAY ROW SELECTED FOR ITEM 4-2]]	1
[IF QUESTION 1-1d = 1 AND [IF ONLY ONE OF 5-1A THROUGH 5-1K = 3, THEN DISPLAY THAT ROW FROM 5-1; ELSE IF ONLY ONE OF 5-1A THROUGH 5-1K = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 5-1; ELSE DISPLAY ROW SELECTED FOR ITEM 5-2]]	1
[IF QUESTION 1-1e = 1 AND [IF ONLY ONE OF 6-1A THROUGH 6-1I = 3, THEN DISPLAY THAT ROW FROM 6-1; ELSE IF ONLY ONE OF 6-1A THROUGH 6-1I = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 6-1; ELSE DISPLAY ROW SELECTED FOR ITEM 6-2]]	1
[IF QUESTION 1-1f = 1 AND [IF ONLY ONE OF 7-1A THROUGH 7-1G = 3, THEN DISPLAY THAT ROW FROM 7-1; ELSE IF ONLY ONE OF 7-1A THROUGH 7-1G = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 7-1; ELSE DISPLAY ROW SELECTED FOR ITEM 7-2]]	1

8-2. Thinking about the experience developing and implementing [NAME OF IADA SYSTEM] at any point up to or through [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA], to your knowledge, which activity was the most challenging?

[DISPLAY ONLY THOSE ACTIVITIES WHERE RESPONSE TO 1-1 = 1]

ACTIVITIES	SELECT THE MOST CHALLENGING ACTIVITY (SELECT ONLY ONE)
Developing the initial plan for the IADA assessment	1
Preparing for the first operational assessment	1
Administering an operational assessment	1
Conducting post-administration activities	1
Conducting continuous improvement activities	1
Scaling up the system	1

This is the end of the survey. Thank you for your participation!

OMB#: 1850-XXXX

Expiration Date: XX/XX/XXXX

Evaluation of the Innovative Assessment Demonstration Authority Pilot Program:

IADA Technical Advisory Committee Interview Protocol

Summer 2024

Notice of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (IES) (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Participating state assessment system will be identified, but not individual interview participants. All of the information you provide may only be used for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Paperwork Reduction Act of 1995

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-XXXX. The approximate time required to participate in this interview is 60 minutes. Your participation is voluntary. If you have any comments concerning the accuracy of the time estimates or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202-465

Section 1. Introduction and Consent (5 minutes)

Thank you for taking the time to speak with me. I am [NAME] of Westat, a research organization in Rockville, MD. I am joined by my colleague [NAME] taking notes. The U.S. Department of Education’s Institute of Education Sciences (IES) has contracted with Westat to evaluate the Innovative Assessment Demonstration Authority (IADA) pilot program, which, as you know includes the [NAME OF IADA SYSTEM] in [STATE]. (For interviews with technical advisory committee (TAC) members for NH PACE, GMAP through-year assessment, or Navvy add, Although the system is no longer part of the IADA program, we believe there are valuable lessons to be learned from former IADA systems and their technical advisory committee members.)

We greatly appreciate your time and willingness to talk about the [STATE’S/SYSTEM’S] experiences with the IADA pilot program. The interview focuses on the most challenging activities the [STATE/SYSTEM] encountered while implementing [NAME OF IADA SYSTEM] and the [STATE’S/SYSTEM’S] strategies for addressing these challenges. In addition, we will ask you to share the technical advisory committee’s (TAC’s) perspective on what practices worked well, those that did not work well or as planned, and practices that the TAC thought might work well but were unable to try and why.

This interview will last about 60 minutes. Participation in the interview is voluntary. [For interviews with LA, NC, and MA add, Participating (or not participating) in the interview will not affect the state’s IADA demonstration authority.] However, your participation is important because the [STATE’S/SYSTEM’S] experiences will help inform policymakers and other states interested in developing innovative student assessment systems. Westat will summarize the findings in a Congressionally mandated report that may include quotes from TAC members without naming individual respondents. The report also may present detailed findings by assessment system. Your coded responses will be available to authorized researchers through a restricted-use file maintained by IES, but you will not be identified by name in any of these files.

1.0. Let’s start with short introductions. Again, I am [NAME] with Westat. Please introduce yourselves and briefly describe your area of expertise that you bring to the [NAME OF IADA SYSTEM] TAC.

Enter each participant’s name and role

Name	Area of expertise

1.1. Do you agree to participate in the interview?

Yes (Continue)
No (Thank participant and end call)

(Obtain agreement to participate from each participant. Thank and excuse any participant not willing to participate.)

Thank you.

1.2. Do you have any questions for me before we get started? (Note and respond to any questions.)

With your permission, I would like to record the interview. The recording will be transcribed, used to verify our notes, and destroyed after the study. Westat will not share the recording or transcription with anyone outside of the research team. You may ask me to stop recording at any point during the interview.

1.3. Do I have your permission to record the interview?

Yes (Start recording)
No (Do not start recording and only take notes.)

(Ask each participant if they give permission to record the interview. If any refuse, do not start recording and only take notes.)

Thank you.

As I go through the interview, I will invite everyone’s input. However, if there are differing opinions, I’ll ask the group to work towards a consensus, or final answer, once everyone has had the opportunity to share their perspective.

Section 2. Major Assessment Implementation Activity Identified as the Most Challenging (20 minutes)

In this first portion of our interview, I would like to discuss the major development and implementation activity for [NAME OF IADA SYSTEM] the TAC believed was the most challenging and the strategies the [STATE/SYSTEM] used to address the challenges. Prior to this interview, we surveyed TAC members who were participating in the IADA system’s TAC as of [FOR CURRENT SYSTEMS SAY: THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA] to identify which of the six major IADA assessment development and implementation activities was the most challenging. As a reminder, these major activities are:

Developing the initial plan for the IADA assessment,
Preparing for the first operational assessment,
Administering an operational assessment,
Conducting post-administration activities such as ensuring technical quality of scores and using reports,
Conducting continuous improvement activities, and
Scaling up the system.

[Using the information from survey item 8-2 across all TAC members for the system who responded to the survey (not just those who are participating in the interview): identify the activity selected most frequently by TAC members as the most challenging activity (that is, identify the activity with the largest number of responses to survey item 8-2). If there are any ties, determine which activity happened earliest and next earliest in the list of activities bulleted above.]

If only one activity was selected most frequently, say: The results across TAC members who responded to the survey revealed that [ACTIVITY IDENTIFIED MOST FREQUENTLY FOR SURVEY ITEM 8-2 ACROSS TAC MEMBERS] was the activity most commonly selected as the most challenging across the TAC members. Please keep this in mind as we move to the next set of questions. If you did not select this activity as the most challenging, we still appreciate your input during the discussion of the challenges related to this activity.

If there was a tie for the activity selected most frequently say, The results across TAC members who responded to the survey revealed that [EARLIEST ACTIVITY IDENTIFIED MOST FREQUENTLY ACROSS SURVEY ITEM 8-2] and [NEXT EARLIEST ACTIVITY IDENTIFIED MOST FREQUENTLY ACROSS SURVEY ITEM 8-2] were identified most frequently across TAC members as the most challenging activities that happened earliest in the implementation timeline. Because major challenges early on in the assessment system development and implementation process can lead to challenges later on in the process, we will focus the interview on the [EARLIEST ACTIVITY IDENTIFIED MOST FREQUENTLY ACROSS ITEM 8-2]. Please keep this in mind as we move to the next set of questions. If you did not select this activity as the most challenging, we still appreciate your input during the discussion of the challenges related to this activity. Also, we will have time later in the interview to capture information on [NEXT EARLIEST ACTIVITY IDENTIFIED MOST FREQUENTLY ACROSS SURVEY ITEM 8-2].

(If the most common, most challenging activity (i.e., if there was only one) or if the earliest of the tied most common, most challenging activities (was:

Developing the initial plan for the IADA assessment, then go to Section 2.1
Preparing for the first operational IADA assessment, then go to Section 2.2
Administering an operational assessment, then go to Section 2.3
Conducting post-administration activities, then go to Section 2.4
Conducting continuous improvement activities, then go to Section 2.5
Scaling up the system, then go to Section 2.6)

(Throughout the interview, if there is disagreement on a response to any item, give each participant the opportunity to voice their opinions. Take note of agreement and disagreement to facilitate the group to focus on area(s) of disagreement, then say: It appears the group is working to decide whether [POTENTIAL ANSWER A OR POTENTIAL ANSWER B] is a consensus answer to [RESTATE THE QUESTION]. Which of these would be most useful to discuss in a best practice report that can inform other states interested in innovative assessments?)

Section 2.1. Major assessment implementation activity #1: Developing the Initial Plan for the IADA assessment

(Using the responses for survey items 2-1 and 2-2 from all TAC members who responded to the survey:

Identify each TAC member’s top challenge for this activity (i.e., the response to survey item 2-2 unless item 2-2 was skipped because a single top challenge can be identified with survey item 2-1);
For each top challenge, look across all TAC member responses for survey item 2-1 and determine if at least one TAC member identified this top challenge as a major challenge.
1. If yes, ask questions 2.1.1 through 2.1.3 below for this top challenge. (We do this to prioritize for the interview those top challenges that also were identified as a major challenge by at least one TAC member.)
2. If no, ask questions 2.1.1 through 2.1.3 for this top challenge only if no other challenge in survey item 2-1 was identified as a major challenge across TAC members.)

(If condition 2a is met, say: I would like to discuss each top challenge that the TAC identified for activity #1: Developing the initial plan for the IADA assessment that also was considered a major challenge by at least one TAC member.)

(if condition 2b is met, say: I would like to discuss each top challenge that the TAC identified for activity #1: Developing the initial plan for the IADA assessment.)

2.1.1. The surveys identified [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY] as one of the most difficult challenges for this activity. Please describe why it was one of the most difficult challenges.

Probe if needed: Was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of staff capacity to operate two assessment systems?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.1.2. To your knowledge, did the [STATE/SYSTEM] identify a strategy to address the challenge of [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY]?

(If the respondents say ‘no’, ask the probing questions below to verify that they did not identify a strategy to address the challenge, then skip to the next top challenge that was identified under this activity area.

If the respondents say ‘yes’, say: Please describe the strategy. If the [STATE/SYSTEM] used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did the [STATE/SYSTEM] try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.1.3. To your knowledge, how effective was [THE RESPONDENTS’ STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENTS’ STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state/system used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about each top challenge, go to Section 2.7)

Section 2.2. Major assessment implementation activity #2: Preparing for the first operational assessment

(Using the responses for survey items 3-1 and 3-2 from all TAC members who responded to the survey:

Identify each TAC member’s top challenge for this activity (i.e., the response to survey item 3-2 unless item 3-2 was skipped because a single top challenge can be identified with survey item 3-1);
For each top challenge, look across all TAC member responses for survey item 3-1 and determine if at least one TAC member identified this top challenge as a major challenge.

If yes, ask questions 2.2.1 through 2.2.3 below for this top challenge. (We do this to prioritize for the interview those top challenges that also were identified as a major challenge by at least one TAC member.)
If no, ask questions 2.2.1 through 2.2.3 for this top challenge only if no other challenge in survey item 3-1 was identified as a major challenge across TAC members.)

(If condition 2a is met, say: I would like to discuss each top challenge that the TAC identified for activity #2: Preparing for the first operational assessment that also was considered a major challenge by at least one TAC member. As a reminder, an operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards.)

(if condition 2b is met, say: I would like to discuss each top challenge that the TAC identified for activity #2: Preparing for the first operational assessment. As a reminder, an operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards.)

2.2.1. The surveys identified [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY] as one of the most difficult challenges for this activity. Please describe why it was one of the most difficult challenges.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of available time for current staff to prepare for administration)?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.2.2. To your knowledge, did the [STATE/SYSTEM] identify a strategy to address the challenge of [TOP CHALLENGE IDENTIFIED FOR THIS ACTIVITY]?

If the respondents say ‘yes’, say: Please describe the strategy. If the [STATE/SYSTEM] used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did the [STATE/SYSTEM] try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Change the timeline for the first operational assessment?
Leverage existing tools or resources?

2.2.3. To your knowledge, how effective was [THE RESPONDENTS’ STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENTS’ STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state/system used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about each top challenge, go to Section 2.7)

Section 2.3. Major assessment implementation activity #3: Administering an operational assessment

(Using the responses for survey items 4-1 and 4-2 from all TAC members who responded to the survey:

Identify each TAC member’s top challenge for this activity (i.e., the response to survey item 4-2 unless item 4-2 was skipped because a single top challenge can be identified with survey item 4-1);
For each top challenge, look across all TAC member responses for survey item 4-1 and determine if at least one TAC member identified this top challenge as a major challenge.
1. If yes, ask questions 2.3.1 through 2.3.3 below for this top challenge. (We do this to prioritize for the interview those top challenges that also were identified as a major challenge by at least one TAC member.)
2. If no, ask questions 2.3.1 through 2.3.3 for this top challenge only if no other challenge in survey item 4-1 was identified as a major challenge across TAC members.)

(If condition 2a is met, say: I would like to discuss each top challenge that the TAC identified for activity #3: Administering an operational assessment that also was considered a major challenge by at least one TAC member. As a reminder, an operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards.)

(if condition 2b is met, say: I would like to discuss each top challenge that the TAC identified for activity #3: Administering an operational assessment. As a reminder, an operational IADA assessment requires that students in a participating school or district are assessed using all IADA testing components (such as all interims and the summative) within a given grade and subject and that IADA results can be used to determine whether students are proficient against the state’s academic standards.)

2.3.1. The surveys identified [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY] as one of the most difficult challenges for this activity. Please describe why it was one of the most difficult challenges.

Probe if needed, For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of staff capacity to operate two assessment systems?
Lack of public support, for example districts’ or schools’ unwillingness to schedule the assessment?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.3.2. To your knowledge, did the [STATE/SYSTEM] identify a strategy to address the challenge of [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY]?

If the respondents say ‘yes’, say: Please describe the strategy. If the [STATE/SYSTEM] used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did the [STATE/SYSTEM] try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Leverage existing tools or resources?

2.3.3. To your knowledge, how effective was [THE RESPONDENTS’ STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENTS’ STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state/system used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about each top challenge, go to Section 2.7)

Section 2.4. Major assessment implementation activity #4: Conducting Post-administration activities

(Using the responses for survey items 5-1 and 5-2 from all TAC members who responded to the survey:

Identify each TAC member’s top challenge for this activity (i.e., the response to survey item 5-2 unless item 5-2 was skipped because a single top challenge can be identified with survey item 5-1);
For each top challenge, look across all TAC member responses for survey item 5-1 and determine if at least one TAC member identified this top challenge as a major challenge.
1. If yes, ask questions 2.4.1 through 2.4.3 below for this top challenge. (We do this to prioritize for the interview those top challenges that also were identified as a major challenge by at least one TAC member.)
2. If no, ask questions 2.4.1 through 2.4.3 for this top challenge only if no other challenge in survey item 5-1 was identified as a major challenge across TAC members.)

(If condition 2a is met, say: I would like to discuss each top challenge that the TAC identified for activity #4: Conducting post-administration activities that also was considered a major challenge by at least one TAC member.)

(if condition 2b is met, say: I would like to discuss each top challenge that the TAC identified for activity #4: Conducting post-administration activities.)

2.4.1. The surveys identified [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY] as one of the most difficult challenges for this activity. Please describe why it was one of the most difficult challenges.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of knowledge about how to collect performance-based assessment data?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources?

2.4.2. To your knowledge, did the [STATE/SYSTEM] identify a strategy to address the challenge of [TOP CHALLENGE IDENTIFIED FOR THIS ACTIVITY]?

If the respondents say ‘yes’, say: Please describe the strategy. If the [STATE/SYSTEM] used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did the [STATE/SYSTEM] try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.4.3. To your knowledge, how effective was [THE RESPONDENTS’ STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENTS’ STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state/system used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about each top challenge, go to Section 2.7)

Section 2.5. Major assessment implementation activity #5: Conducting continuous improvement activities

(Using the responses for survey items 6-1 and 6-2 from all TAC members who responded to the survey:

Identify each TAC member’s top challenge for this activity (i.e., the response to survey item 6-2 unless item 6-2 was skipped because a single top challenge can be identified with survey item 6-1).
For each top challenge, look across all TAC member responses for survey item 6-1 and determine if at least one TAC member identified this top challenge as a major challenge.

If yes, ask questions 2.5.1 through 2.5.3 below for this top challenge. (We do this to prioritize for the interview those top challenges that also were identified as a major challenge by at least one TAC member.)
If no, ask questions 2.5.1 through 2.5.3 for this top challenge only if no other challenge in survey item 6-1 was identified as a major challenge across TAC members.)

(If condition 2a is met, say: I would like to discuss each top challenge that the TAC identified for activity #5: Conducting continuous improvement activities that also was considered a major challenge by at least one TAC member.)

(if condition 2b is met, say: I would like to discuss each top challenge that the TAC identified for activity #5: Conducting continuous improvement activities.)

2.5.1. The surveys identified [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY] as one of the most difficult challenges for this activity. Please describe why it was one of the most difficult challenges.

Probe if needed: For example, was it a challenge because of a:

Lack of data literacy skills such as accurately interpreting the scores or an understanding of how to use the scores to plan for or target instruction?
Lack of public support, for example unwillingness to participate in feedback opportunities or the annual evaluation?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources to conduct an evaluation?

2.5.2. To your knowledge, did the [STATE/SYSTEM] identify a strategy to address the challenge of [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY]?

If the respondents say ‘yes’, say: Please describe the strategy. If the [STATE/SYSTEM] used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did the [STATE/SYSTEM] try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Leverage existing tools or resources?

2.5.3. To your knowledge, how effective was [THE RESPONDENTS’ STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENTS’ STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state/system used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about each top challenge, go to Section 2.7)

Section 2.6. Major assessment implementation activity #6: Scaling Up the System

(Using the responses for survey items 7-1 and 7-2 from all TAC members who responded to the survey:

Identify each TAC member’s top challenge for this activity (i.e., the response to survey item 7-2 unless item 7-2 was skipped because a single top challenge can be identified with survey item 7-1).
For each top challenge, look across all TAC member responses for survey item 7-1 and determine if at least one TAC member identified this top challenge as a major challenge.
1. If yes, ask questions 2.6.1 through 2.6.3 below for this top challenge. (We do this to prioritize for the interview those top challenges that also were identified as a major challenge by at least one TAC member.)
2. If no, ask questions 2.6.1 through 2.6.3 for this top challenge only if no other challenge in survey item 7-1 was identified as a major challenge across TAC members.)

(If condition 2a is met, say: I would like to discuss each top challenge that the TAC identified for activity #6: Scaling up the system that also was considered a major challenge by at least one TAC member.)

(if condition 2b is met, say: I would like to discuss each top challenge that the TAC identified for activity #6: Scaling up the system.)

2.6.1. The surveys identified [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY] as one of the most difficult challenges for this activity. Please describe why it was one of the most difficult challenges.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of staff capacity to operate two assessment systems?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of funding or other resources such as materials needed to administer the assessment?

2.6.2. To your knowledge, did the [STATE/SYSTEM] identify a strategy to address the challenge of [TOP CHALLENGE IDENTIFIED FOR THE ACTIVITY]?

If the respondents say ‘yes’, say: Please describe the strategy. If the [STATE/SYSTEM] used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did the [STATE/SYSTEM] try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the IADA assessment?
Change aspects of the IADA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.6.3. To your knowledge, how effective was [THE RESPONDENTS’ STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENTS’ STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state/system used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about each top challenge, go to Section 2.7)

Section 2.7 Additional Activity Seen as Most Challenging

(Ask this question only if there was a tie for the most common, most challenging activity as identified at the start of section 2. If so, use the “next earliest activity identified most frequently across survey item 8-2” identified at the start of section 2 for the question below. Otherwise, skip to section 3.)

2.7.1 As noted earlier, the survey results revealed that [NEXT EARLIEST ACTIVITY IDENTIFIED MOST FREQUENTLY ACROSS SURVEY ITEM 8-2] also was an activity with a lot of challenges for the system. What was it about [NEXT EARLIEST ACTIVITY IDENTIFIED MOST FREQUENTLY ACROSS SURVEY ITEM 8-2] that made it challenging?

Section 3.0 What worked well, what did not work as intended, and why? (15 minutes)

In the last section, I asked you to tell me about specific challenges related to a particular activity, but now I would like you to think more broadly about the major assessment implementation activities that the [NAME OF IADA SYSTEM] [FOR CURRENT SYSTEMS SAY: HAS BEEN ENGAGED IN AT ANY POINT UP TO OR THROUGH THE SCHOOL YEAR IN WHICH THE SYSTEM MOST RECENTLY PARTICIPATED IN IADA; FOR FORMER SYSTEMS SAY: WAS ENGAGED IN AT ANY POINT UP TO OR THROUGH THE FINAL SCHOOL YEAR IN WHICH THE SYSTEM PARTICIPATED IN IADA]. Considering all relevant activities, please identify what worked well in terms of supporting the system’s objectives and why. As a reminder, the six major activities are:

Developing the initial plan for the IADA assessment,
Preparation for the first operational IADA assessment,
Administering an operational assessment,
Conducting post-administration activities such as ensuring technical quality of scores and using reports,
Conducting continuous improvement activities, and
Scaling up the IADA assessment statewide.

3.1. Thinking across the activities the system engaged in, based on your observations, what practices worked well? Discuss up to three practices that worked well.

Probe if needed, For example, were there particular practices that contributed to the success of the system such as, the practice of recruiting participating districts and schools?

(As respondents are speaking, match the successful practice with the list of major activities using reference list, and then ask the respondent to confirm the match by saying: It sounds like this practice was related to the [MAJOR ACTIVITY X]. Is that correct? Is there an additional activity you would relate this practice to?)

3.2 How did the [THE RESPONDENTS’ PRACTICE ANSWER] contribute to the success of the system?

(Repeat the question for each of up to three practices identified in the previous question and record answers. Then ask: Which one practice was the most important contributor to progress overall for [NAME OF IADA SYSTEM]?)

3.3 Please think about the IADA system as a whole. Based on your observations, were there aspects of the system’s theory of action or the system’s original design that did not work as intended? For example, were there aspects related to the planned grade(s) or subjects tested or planned method for calculating the annual determination of student proficiency that did not work as intended?

Add if needed: This question is related to the system’s overall design rather than about the specific challenges related to assessment activities discussed above.

3.4 Why did [THE RESPONDENTS’ ASPECT ANSWER] not work as intended?

(Repeat this question for each aspect mentioned and record answers. Then ask: Which one aspect stands out as the most important one that you’d want other states/systems to know about and learn from?)

Section 4.0 Practices that the system wanted to try, but was unable to (15 minutes)

4.1. I would like you to reflect on the objectives of the [NAME OF IADA SYSTEM]. To your knowledge, were there any practices that [NAME OF IADA SYSTEM] wanted to implement to support these objectives, but was unable to? What were these practices that the system wanted to implement, but did not, and why? (If none, go to Section 5.)

Probe if needed, For example, were there barriers that prevented the system from implementing a practice that you felt would have supported the objectives of the [NAME OF IADA SYSTEM]?

(As respondents are speaking, match the practice they wanted to try with the list of major activities using reference list, and then ask the respondent to confirm the match by saying: It sounds like this practice was related to the [MAJOR ACTIVITY X]. Is that correct? Is there an additional activity you would relate this practice to?)

4.2. What prevented [NAME OF IADA SYSTEM] from trying [THE RESPONDENTS’ PRACTICE ANSWER]?

(Repeat this question for each practice mentioned and record answers. Then ask: Which one practice faced the largest barrier to trying it?)

Probe if needed, were there barriers related to:

The political climate?
Pressures from the public?
State legislation or policy?
Funding or other resources?
Technical issues?
The IADA program requirements?
The COVID pandemic?

Section 5.0 Wrap Up and Thank You (5 minutes)

(If not all members of the TAC participated in the interview ask question 5.0; otherwise skip to question 5.1)

5.0. Not all members of the TAC were able to participate in today’s interview. As a group, how confident do you feel that the responses provided today are similar to what we would have heard if we had the entire TAC in the interview?

DK □ Don’t know/Unable to judge

1 □ Not similar

2 □ Somewhat similar

3 □ Very similar

5.1. Is there anything we did not discuss regarding the implementation of the [NAME OF IADA SYSTEM] that you would like to share with me?

Thank you for your time.

B-1

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2024-07-22