Appendix D_CGSA Project Director Instruments 06-12-24

Evaluation of the Innovative Assessment Demonstration Authority Pilot Program Survey Data Collections

Appendix D_CGSA Project Director Instruments 06-12-24

OMB:

Document [docx]

Download: docx | pdf

6-10-24 DRAFT

Appendix D:
CGSA Project Director
Data Collection Instruments

OMB#: 1850-XXXX

Expiration Date: XX/XX/XXXX

Evaluation of the Innovative Assessment Demonstration Authority (IADA) Pilot Program:

Competitive Grants for Student Assessment (CGSA) CGSA Project Director Survey

Summer 2024

Notice of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Participating state assessment systems will be identified, but not individual survey participants. All of the information you provide may only be used for research purposes and may not be disclosed or used in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Paperwork Reduction Act of 1995

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-XXXX. The approximate time required to complete the survey is 20 minutes, including the time to review instructions, search existing data resources, gather the needed data, and complete and review the information collection. Your response to this collection is voluntary. If you have any comments concerning the accuracy of the time estimates or suggestions for improving this survey, please write to: U.S. Department of Education, Washington, DC 20202-4651.

Introduction to the Survey

You are invited to complete this 20-minute survey about the extent to which your state experienced challenges developing and implementing the [NAME OF CGSA SYSTEM]. This survey is part of the Congressionally mandated evaluation of the Innovative Assessment Demonstration Authority (IADA) program. Although you do not currently direct an IADA assessment system, you are being contacted because you are the project director for one of the U.S. Department of Education’s Competitive Grants for Student Assessments (CGSA) funded in 2020, 2022, or both 2020 and 2022. The evaluation team selected your grant because it shares similarities with the types of assessment systems awarded under IADA. Your perspective and experiences developing such an assessment will provide valuable information to other states that may want to develop a new assessment system in the future.

The survey asks you to share how much the CGSA system experienced specific challenges associated with six key stages of assessment development and implementation. These activities include: developing the initial plan for the CGSA assessment, preparing for the first operational assessment, administering an operational assessment, conducting post-administration activities, conducting continuous improvement activities, and scaling up the system.
This survey is intended to capture the perspectives of the CGSA project director, but we realize that developing a new assessment is likely a “team effort.” As needed, please reach out to additional state officials or others to contribute to survey responses; however, submit only one completed survey for the system.
Although voluntary, we strongly encourage you to participate in this survey so the IADA evaluation will include your state’s experiences creating innovative assessments. Participating (or not participating) will not affect your CGSA grant award.
After you complete this survey, the study team will request a follow-up interview to learn more about the major implementation activity identified in the survey as the most challenging. The team will ask about the specific challenges for this activity and proposed or implemented solutions.
Survey findings will be summarized in a IADA Best Practices report required by Congress that will share lessons learned from key stakeholders on the development and implementation of IADA and other innovative assessments. The report may present findings separately by CGSA systems, but will not name individual survey participants. Survey responses will be available to authorized researchers through a restricted-use data file maintained by the U.S. Department of Education’s Institute of Education Sciences (IES). The data file will not include individual survey participant’s names.
IES has contracted with Westat, an independent research organization, to lead the evaluation and administer this survey.

Click the Next Button to
Continue to the Survey

Section 1. Major Assessment Development and Implementation Activities for the State

This section asks you to identify which of six activities (stages) of assessment development and implementation your state engaged in at any point up to or through the school year in which the system most recently participated in CGSA. The six activities to consider are listed below with example sub-activities. Please note that your system design may have had different and/or additional sub-activities.

Developing the initial plan for the CGSA assessment. Example sub-activities include establishing test and item requirements, creating an implementation timeline, providing information to parents and students, and providing training to staff on the system’s purpose and major features.
Preparing for the first operational assessment. We define an operational assessment for CGSA as one that is used to generate student information that is shared with educators and used in some meaningful way (such as contributing to grades, used as a benchmark, reports provided to students or parents, etc.). Field tests used solely to gather information about test quality would not be considered an operational assessment. Example sub-activities include developing initial items, piloting items, ensuring appropriate testing accommodations, developing assessment administration procedures, and conducting professional development for educators.
Administering an operational assessment. Example sub-activities include administering full assessments and determining student proficiencies/competencies.
Conducting post-administration activities. Example sub-activities include ensuring the technical quality of scores, producing student score reports, generating data reports for educators (such as classroom- or school-level results to inform instruction), and identifying schools for additional support in the accountability system.
Conducting continuous improvement activities. Example sub-activities include getting feedback from participating teachers, school leaders, or local education agency staff; and conducting annual evaluations.
Scaling up the system. Example sub-activities include increasing the number of districts or schools participating in the CGSA assessment system over time.

Thinking about the state’s experience developing and implementing [NAME OF CGSA SYSTEM], identify whether the state engaged in the listed activities below at any point up to or through the school year in which the system most recently participated in CGSA. Your state did not have to complete the activity to be considered “engaged in” the activity.

(As a reminder, please reach out to additional state officials or others to contribute to survey responses as needed; however, you will submit only one completed survey for the system.)

ACTIVITIES	SELECT ONE RESPONSE IN EACH ROW
ACTIVITIES	ENGAGED IN THE ACTIVITY	WAS NOT ENGAGED IN THE ACTIVITY
Developing the initial plan for the CGSA assessment	1	0
Preparing for the first operational assessment	1	0
Administering an operational assessment	1	0
Conducting post-administration activities	1	0
Conducting continuous improvement activities	1	0
Scaling up the system	1	0

For which activities were you the CGSA Project Director? Select an activity if you were the CGSA Project Director at any time during the activity.

[DISPLAY ONLY THOSE ACTIVITIES WHERE RESPONSE TO 1-1 = 1]

ACTIVITIES	I WAS THE CGSA PROJECT DIRECTOR DURING THIS ACTIVITY (SELECT ONE RESPONSE IN EACH ROW)
ACTIVITIES	YES	NO
Developing the initial plan for the CGSA assessment	1	0
Preparing for the first operational assessment	1	0
Administering an operational assessment	1	0
Conducting post-administration activities	1	0
Conducting continuous improvement activities	1	0
Scaling up the system	1	0

The next portion of the survey asks you to rate the extent to which the [STATE/SYSTEM] experienced challenges associated with each major assessment development and implementation activity the [STATE/SYSTEM] engaged in. Please keep the following response definitions in mind as you rate the challenges:

Not applicable: the potential challenge was not a possibility for your assessment system. For example, select ‘not applicable’ if your system had not yet reached the stage for the challenge to be addressed or if the potential challenge does not apply given the type of assessments included in your system.
Not a Challenge: the potential challenge was not a concern; for example, the potential challenge did not cause changes to the design or implementation timeline of the system
Minor Challenge: the potential challenge was somewhat problematic; for example, the potential challenge was difficult but did not cause significant setbacks or delays in implementation of the system, and may have led to additional planning time or the reallocation of resources.
Major Challenge: the potential challenge was considerably problematic; for example, the potential challenge may have caused significant setbacks or delays in implementation of the system, led to modifications to the original CGSA system design, or required additional expertise as well as additional planning time or resources.

These definitions also are available to respondents by hovering over the words that appear in blue text in the web survey.

Section 2. Challenges Related to Developing the Initial Plan for the CGSA Assessment System

[DISPLAY SECTION ONLY IF QUESTION 1-1a = 1; OTHERWISE, SKIP TO SECTION 3]

2-1. To what extent was each of the following a challenge for the state while developing the initial plan for the CGSA assessment system? Consider whether it was a challenge at any point up to or through the school year in which the system most recently participated in CGSA. If your state experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE
Engaging with families	1	2	3	NA
Engaging with educators	1	2	3	NA
Engaging with other stakeholders such as political leaders or community members	1	2	3	NA
Recruiting districts or schools for the pilot or the initial administration	1	2	3	NA
Developing tools to support the assessment such as an aligned assessment blueprint or standardized or calibrated tools for accurate and reliable scoring (such as rubrics)	1	2	3	NA
Determining a method for evaluating the comparability of CGSA results to results from the existing statewide assessment	1	2	3	NA
Developing CGSA assessment(s) that align to existing content standards	1	2	3	NA
Scheduling sufficient planning time to support the first operational assessment	1	2	3	NA
Gaining agency staff buy-in for the CGSA assessment system	1	2	3	NA
Building capacity within the agency to implement the CGSA assessment system	1	2	3	NA
Allocating existing funds, beyond the CGSA grant funding, to support the CGSA assessment system	1	2	3	NA
Securing additional funds, beyond the CGSA grant funding, to support the CGSA assessment system such as other grants	1	2	3	NA
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):

[GO TO 2-2:

IF MORE THAN ONE OF 2-1A THROUGH 2-1O = 3; OR
IF MORE THAN ONE OF 2-1A THROUGH 2-1O = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 3]

2-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 2-1A THROUGH 2-1O = 3, DISPLAY ONLY THOSE ROWS FROM 2-1A THROUGH 2-1O WITH A VALUE OF 3.

IF MORE THAN ONE OF 2-1A THROUGH 2-1O = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 2-1A THROUGH 2-1O WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Engaging with families	1
Engaging with educators	1
Engaging with other stakeholders such as political leaders or community members	1
Recruiting districts or schools for the pilot or the initial administration	1
Developing tools to support the assessment such as an aligned assessment blueprint or standardized or calibrated tools for accurate and reliable scoring (such as rubrics)	1
Determining a method for evaluating the comparability of CGSA results to results from the existing statewide assessment	1
Developing CGSA assessment(s) that align to existing content standards	1
Scheduling sufficient planning time to support the first operational assessment	1
Gaining agency staff buy-in for the CGSA assessment system	1
Building capacity within the agency to implement the CGSA assessment system	1
Allocating existing funds, beyond the CGSA grant funding, to support the CGSA assessment system	1
Securing additional funds, beyond the CGSA grant funding, to support the CGSA assessment system such as other grants	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 3. Challenges Related to Preparing for the First Operational Assessment

[DISPLAY SECTION ONLY IF QUESTION 1-1b = 1; OTHERWISE, SKIP TO SECTION 4]

3-1. To what extent was each of the following a challenge for the state while preparing for the first operational assessment? Consider whether it was a challenge at any point up to or through the school year in which the system most recently participated in CGSA. If your state experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

As a reminder, we define an operational assessment for CGSA as one that is used to generate student information that is shared with educators and used in some meaningful way (such as contributing to grades, used as a benchmark, reports provided to students or parents, etc.). Field tests used solely to gather information about test quality would not be considered an operational assessment.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE
Developing or supplementing the item bank or pilot testing the items that determine proficiency or mastery of the state-approved competencies or academic standards	1	2	3	NA
Preparing administration procedures or protocols for general administration.	1	2	3	NA
Developing or deploying professional development, supports, or other resources for teachers or building administrators to score the CGSA assessment	1	2	3	NA
Developing or deploying supports or other resources to help parents and students interpret score reports	1	2	3	NA
Developing or deploying supports or other resources to help other stakeholder groups such as political leaders or community members interpret score reports	1	2	3	NA
Meeting specific technical challenges required of the CGSA assessment (for example, challenges related to fairness, validity, bias, or sensitivity)	1	2	3	NA
(Specify the technical challenge(s)):
Obtaining sufficient funding to support activities to prepare for the first operational assessment such as professional development	1	2	3	NA
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):

[GO TO 3-2:

IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 3; OR
IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 4]

3-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 3, DISPLAY ONLY THOSE ROWS FROM 3-1A THROUGH 3-1J WITH A VALUE OF 3.

IF MORE THAN ONE OF 3-1A THROUGH 3-1J = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 3-1A THROUGH 3-1J WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Developing or supplementing the item bank or pilot testing the items that determine proficiency or mastery of the state-approved competencies or academic standards	1
Preparing administration procedures or protocols for general administration.	1
Developing or deploying professional development, supports, or other resources for teachers or building administrators to score the CGSA assessment	1
Developing or deploying supports or other resources to help parents and students interpret score reports	1
Developing or deploying supports or other resources to help other stakeholder groups such as political leaders or community members interpret score reports	1
Meeting specific technical challenges required of the CGSA assessment (for example, challenges related to fairness, validity, bias, or sensitivity)..	1
(Specify the technical challenge(s)):
Obtaining sufficient funding to support activities to prepare for the first operational assessment such as professional development	1

Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 4. Challenges Related to Administering an Operational Assessment

[DISPLAY SECTION ONLY IF QUESTION 1-1c = 1; OTHERWISE, SKIP TO SECTION 5]

4-1. To what extent was each of the following a challenge for the state while administering an operational assessment? Consider whether it was a challenge at any point up to or through the school year in which the system most recently participated in CGSA. If your state experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE
Scheduling or administering the assessment	1	2	3	NA
Informing parents in participating schools about the CGSA assessment	1	2	3	NA
Troubleshooting during the administration such as maintaining a help desk, providing hardware or connectivity support, recovering lost data	1	2	3	NA
Maintaining the security of individual results for each student	1	2	3	NA
Obtaining sufficient funding for the administration of the operational assessment	1	2	3	NA
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):

[GO TO 4-2:

IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 3; OR
IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 5]

4-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 3, DISPLAY ONLY THOSE ROWS FROM 4-1A THROUGH 4-1H WITH A VALUE OF 3.

IF MORE THAN ONE OF 4-1A THROUGH 4-1H = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 4-1A THROUGH 4-1H WITH A VALUE OF 2]

POTENTIAL CHALLENGES			SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Scheduling or administering the assessment			1
Informing parents in participating schools about the CGSA assessment			1
Troubleshooting during the administration such as maintaining a help desk, providing hardware or connectivity support, recovering lost data			1
	Maintaining the security of individual results for each student	1
Obtaining sufficient funding for the administration of the operational assessment			1
Other challenge			1
(Specify):
Other challenge			1
(Specify):
Other challenge			1
(Specify):

Section 5. Challenges Related to Conducting post-administration Activities

[DISPLAY SECTION ONLY IF QUESTION 1-1d = 1; OTHERWISE, SKIP TO SECTION 6]

5-1. To what extent was each of the following a challenge for the state while conducting post-administration activities? Consider whether it was a challenge at any point up to or through the school year in which the system most recently participated in CGSA. If your state experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE
Collecting or gathering data for performance-based assessments	1	2	3	NA
Ensuring the technical quality of the scores	1	2	3	NA
Producing individual student reports	1	2	3	NA
Applying student results for accountability	1	2	3	NA
Using student results to improve instruction or guide decisions	1	2	3	NA
Determining comparability of CGSA results to results from the existing statewide assessment	1	2	3	NA
Determining alignment of CGSA assessment to state content standards	1	2	3	NA
Obtaining sufficient funding for post-administration activities	1	2	3	NA
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):

[GO TO 5-2:

IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 3; OR
IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 6]

5-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 3, DISPLAY ONLY THOSE ROWS FROM 5-1A THROUGH 5-1K WITH A VALUE OF 3.

IF MORE THAN ONE OF 5-1A THROUGH 5-1K = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 5-1A THROUGH 5-1K WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Collecting or gathering data for performance-based assessments	1
Ensuring the technical quality of the scores	1
Producing individual student reports	1
Applying student results for accountability	1
Using student results to improve instruction or guide decisions	1
Determining comparability of CGSA results to results from the existing statewide assessment	1
Determining alignment of CGSA assessment to state content standards	1
Obtaining sufficient funding for post-administration activities	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 6. Challenges Related to Conducting Continuous Improvement Activities

[DISPLAY SECTION ONLY IF QUESTION 1-1e = 1; OTHERWISE, SKIP TO SECTION 7]

6-1. To what extent was each of the following a challenge for the state while conducting continuous improvement activities? Consider whether it was a challenge at any point up to or through the school year in which the system most recently participated in CGSA. If your state experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE
Promoting appropriate data use by educators	1	2	3	NA
Obtaining feedback from families on their satisfaction with the CGSA assessment system	1	2	3	NA
Obtaining feedback from educators on their satisfaction with the CGSA assessment system	1	2	3	NA
Obtaining feedback from other stakeholders (such as political leaders, community members) on their satisfaction with the CGSA assessment system	1	2	3	NA
Conducting an annual evaluation	1	2	3	NA
Obtaining sufficient funding to support continuous improvement activities	1	2	3	NA
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):

[GO TO 6-2:

IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 3; OR
IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 7]

6-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 3, DISPLAY ONLY THOSE ROWS FROM 6-1A THROUGH 6-1I WITH A VALUE OF 3.

IF MORE THAN ONE OF 6-1A THROUGH 6-1I = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 6-1A THROUGH 6-1I WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Promoting appropriate data use by educators	1
Obtaining feedback from families on their satisfaction with the CGSA assessment system	1
Obtaining feedback from educators on their satisfaction with the CGSA assessment system	1
Obtaining feedback from other stakeholders (such as political leaders, community members) on their satisfaction with the CGSA assessment system	1
Conducting an annual evaluation	1
Obtaining sufficient funding to support continuous improvement activities
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 7. Challenges Related to Scaling Up the CGSA Assessment System

[DISPLAY SECTION ONLY IF QUESTION 1-1f = 1; OTHERWISE, SKIP TO SECTION 8]

7-1. To what extent was each of the following a challenge for the state regarding scaling up the CGSA assessment system? Consider whether it was a challenge at any point up to or through the school year in which the system most recently participated in CGSA. If your state experienced other challenges related to this activity, please identify them and rate the extent to which each was a challenge.

POTENTIAL CHALLENGES	SELECT ONE RESPONSE IN EACH ROW
POTENTIAL CHALLENGES	NOT A CHALLENGE	MINOR CHALLENGE	MAJOR CHALLENGE	NOT APPLICABLE
Recruiting districts or schools to participate in the CGSA assessment system	1	2	3	NA
Establishing CGSA assessment quality or other triggers to signal readiness for statewide implementation	1	2	3	NA
Obtaining sufficient funding for scale-up activities	1	2	3	NA
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):
Other challenge	1	2	3	NA
(Specify):

[GO TO 7-2:

IF MORE THAN ONE OF 7-1A THROUGH 7-1F = 3; OR
IF MORE THAN ONE OF 7-1A THROUGH 7-1F = 2 AND NONE HAVE A VALUE OF 3

OTHERWISE, SKIP TO SECTION 8]

7-2. Which challenge was the most difficult challenge?

[IF MORE THAN ONE OF 7-1A THROUGH 7-1F = 3, DISPLAY ONLY THOSE ROWS FROM 7-1A THROUGH 7-1F WITH A VALUE OF 3.

IF MORE THAN ONE OF 7-1A THROUGH 7-1F = 2 AND NONE HAVE A VALUE OF 3, DISPLAY ONLY THOSE ROWS FROM 7-1A THROUGH 7-1F WITH A VALUE OF 2]

POTENTIAL CHALLENGES	SELECT THE MOST DIFFICULT CHALLENGE (SELECT ONLY ONE)
Recruiting districts or schools to participate in the CGSA assessment system	1
Establishing CGSA assessment quality or other triggers to signal readiness for statewide implementation	1
Obtaining sufficient funding for scale-up activities	1
Other challenge	1
(Specify):
Other challenge	1
(Specify):
Other challenge	1
(Specify):

Section 8. Most Difficult Challenge and Most Difficult Activity

8-1. In the previous sections, you identified the most difficult challenge for each major assessment development and implementation activity. Which of these challenges was the top challenge for your assessment system as a whole?

SYSTEM’S MOST DIFFICULT CHALLENGE FOR EACH ACTIVITY	SELECT THE TOP CHALLENGE FOR THE SYSTEM (SELECT ONLY ONE)
[IF QUESTION 1-1a = 1 AND [IF ONLY ONE OF 2-1A THROUGH 2-1O = 3, THEN DISPLAY THAT ROW FROM 2-1; ELSE IF ONLY ONE OF 2-1A THROUGH 2-1O = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 2-1; ELSE DISPLAY ROW SELECTED FOR ITEM 2-2]]	1
[IF QUESTION 1-1b = 1 AND [IF ONLY ONE OF 3-1A THROUGH 3-1J = 3, THEN DISPLAY THAT ROW FROM 3-1; ELSE IF ONLY ONE OF 3-1A THROUGH 3-1J = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 3-1; ELSE DISPLAY ROW SELECTED FOR ITEM 3-2]]	1
[IF QUESTION 1-1c = 1 AND [IF ONLY ONE OF 4-1A THROUGH 4-1H = 3, THEN DISPLAY THAT ROW FROM 4-1; ELSE IF ONLY ONE OF 4-1A THROUGH 4-1H = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 4-1; ELSE DISPLAY ROW SELECTED FOR ITEM 4-2]]	1
[IF QUESTION 1-1d = 1 AND [IF ONLY ONE OF 5-1A THROUGH 5-1K = 3, THEN DISPLAY THAT ROW FROM 5-1; ELSE IF ONLY ONE OF 5-1A THROUGH 5-1K = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 5-1; ELSE DISPLAY ROW SELECTED FOR ITEM 5-2]]	1
[IF QUESTION 1-1e = 1 AND [IF ONLY ONE OF 6-1A THROUGH 6-1I = 3, THEN DISPLAY THAT ROW FROM 6-1; ELSE IF ONLY ONE OF 6-1A THROUGH 6-1I = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 6-1; ELSE DISPLAY ROW SELECTED FOR ITEM 6-2]]	1
[IF QUESTION 1-1f = 1 AND [IF ONLY ONE OF 7-1A THROUGH 7-1F = 3, THEN DISPLAY THAT ROW FROM 7-1; ELSE IF ONLY ONE OF 7-1A THROUGH 7-1F = 2 AND NONE = 3, THEN DISPLAY THAT ROW FROM 7-1; ELSE DISPLAY ROW SELECTED FOR ITEM 7-2]]	1

8-2. Thinking about the state’s experience developing and implementing [NAME OF CGSA SYSTEM] at any point up to or through the school year in which the system most recently participated in CGSA, which activity was the most challenging?

[DISPLAY ONLY THOSE ACTIVITIES WHERE RESPONSE TO 1-1 = 1]

ACTIVITIES	SELECT THE MOST CHALLENGING ACTIVITY (SELECT ONLY ONE)
Developing the initial plan for the CGSA assessment	1
Preparing for the first operational assessment	1
Administering an operational assessment	1
Conducting post-administration activities	1
Conducting continuous improvement activities	1
Scaling up the system	1

Section 9. Concluding Section

9-1. Thank you for responding to the survey questions. Did additional state personnel or others contribute to the survey responses?

1 □ Yes CONTINUE TO 9-2

0 □ No SKIP TO END OF SURVEY

9-2. Please describe how these other individuals contributed to the CSGA assessment system and list their titles.

Shape1

This is the end of the survey. Thank you for your participation!

OMB#: 1850-XXXX

Expiration Date: XX/XX/XXXX

Evaluation of the Innovative Assessment Demonstration Authority Pilot Program:

Competitive Grants for Student Assessment (CGSA) Project Director Interview Protocol

Summer 2024

Notice of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (IES) (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Participating state assessment system will be identified, but not individual interview participants. All of the information you provide may only be used for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Paperwork Reduction Act of 1995

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 1850-XXXX. The approximate time required to participate in this interview is 60 minutes. Your participation is voluntary. If you have any comments concerning the accuracy of the time estimates or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202-465

Section 1. Introduction and Consent (5 minutes)

Thank you for taking the time to speak with me. I am [NAME] of Westat, a research organization in Rockville, MD. I am joined by my colleague [NAME] taking notes. The U.S. Department of Education’s Institute of Education Sciences (IES) has contracted with Westat to evaluate the Innovative Assessment Demonstration Authority (IADA) pilot program. Although you do not currently direct an IADA assessment system, you are being contacted because you are the project director for one of the U.S. Department of Education’s Competitive Grants for Student Assessments (CGSA). The evaluation team selected your grant because it shares similarities with the types of assessment systems awarded under IADA. Your perspective and experiences developing such an assessment are thus very valuable to the evaluation.

We greatly appreciate your time and willingness to talk about your state’s experiences with the CGSA program. The interview focuses on the most challenging activities the state encountered while implementing the [NAME OF CGSA ASSESSMENT] and your strategies for addressing these challenges. In addition, we will ask you to share what practices worked well, those that did not work well or as planned, and practices that you thought might work well but were unable to try and why.

This interview will last about 60 minutes. Participation in the interview is voluntary. Participating (or not participating) in the interview will not affect your CGSA grant award. However, your state’s participation is important because your experiences will help inform policymakers and other states interested in developing innovative assessments. Westat will summarize the findings in a Congressionally mandated report that may include quotes from CGSA project directors without naming individual respondents. The report also may present detailed findings by assessment system. Your coded responses will be available to authorized researchers through a restricted-use file maintained by IES, but you will not be identified by name in any of these files.

(If the CGSA Project Director is the only interview participant, note that here and skip to question 1.1)

1.0. If anyone is joining the CGSA Project Director on the call, say: Let’s start with short introductions. Again, I am [NAME] with Westat. Please introduce yourselves and briefly describe your role in the [NAME OF CGSA ASSESSMENT].

Enter each participant’s name and role

Name	Role

1.1. Do you agree to participate in the interview?

Yes (Continue)
No (Thank participant and end call)

(If there are multiple participants, obtain agreement to participate from each participant. Thank and excuse any participant not willing to participate.)

Thank you.

1.2. Do you have any questions for me before we get started? (Note and respond to any questions.)

With your permission, I would like to record the interview. The recording will be transcribed, used to verify our notes, and destroyed after the study. Westat will not share the recording or transcription with anyone outside of the research team. You may ask me to stop recording at any point during the interview.

1.3. Do I have your permission to record the interview?

Yes (Start recording)
No (Do not start recording and only take notes.)

(If there are multiple participants, ask each participant if they give permission to record the interview. If any refuse, do not start recording and only take notes.)

Thank you.

(If there are multiple participants, say: As I go through the interview, I will invite everyone’s input. However, if there are differing opinions, I’ll ask the group to work towards a consensus, or final answer, once everyone has had the opportunity to share their perspective.)

Section 2. Major Assessment Implementation Activity Identified as the Most Challenging (20 minutes)

In this first portion of our interview, I would like to discuss the major development and implementation activity for [NAME OF CGSA ASSESSMENT] you believe was the most challenging and the strategies the state used to address the challenges. As you may recall, prior to this interview, we surveyed you to identify which of the six major assessment development and implementation activities was the most challenging. As a reminder, these major activities are:

Developing the initial plan for the CGSA assessment,
Preparing for the first operational assessment,
Administering an operational assessment,
Conducting post-administration activities such as ensuring technical quality of scores and using reports,
Conducting continuous improvement activities, and
Scaling up the system.

The survey response indicates that [ACTIVITY SELECTED AS THE MOST CHALLENGING (RESPONSE TO SURVEY ITEM 8-2)] was the most challenging major activity. Please keep this in mind as we move to the next set of questions.

(If the most challenging activity was:

Developing the initial plan for the CGSA assessment, then go to Section 2.1
Preparing for the first operational assessment, then go to Section 2.2
Administering an operational assessment, then go to Section 2.3
Conducting post-administration activities, then go to Section 2.4
Conducting continuous improvement activities, then go to Section 2.5
Scaling up the system, then go to Section 2.6)

(Throughout the interview, if the interview includes multiple participants and there is disagreement on a response to any item, give each participant the opportunity to voice their opinions. Take note of agreement and disagreement to facilitate the group to focus on area(s) of disagreement, then say: It appears the group is working to decide whether [POTENTIAL ANSWER A OR POTENTIAL ANSWER B] is a consensus answer to [RESTATE THE QUESTION]. Which of these would be most useful to discuss in a best practice report that can inform other states interested in innovative assessments?)

Section 2.1. Major assessment implementation activity #1: Developing the initial plan for the CGSA assessment

(Prepopulate questions 2.1.1 and 2.1.2 with the challenge identified as a major challenge from survey item 2-1. If no challenges were identified as major by the respondent in survey item 2-1, prepopulate questions 2.1.1 – 2.1.2 below with the minor challenge identified as the most difficult challenge in survey item 2-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.1.1 through 2.1.3 below should be repeated for each challenge identified as a major challenge..)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #1: Developing the initial plan for the CGSA assessment.

[If major challenges are displayed, say: Recall that we defined a major challenge as one that was considerably problematic; for example, the potential challenge may have caused significant setbacks or delays in implementation of the system, led to modifications to the original CGSA system design, or required additional expertise as well as additional planning time or resources.

If the most difficult minor challenge is displayed, say: Recall that we defined a minor challenge as one was somewhat problematic; for example, the potential challenge was difficult but did not cause significant setbacks or delays in implementation of the system, and may have led to additional planning time or the reallocation of resources.]

2.1.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER DEVELOPING THE INITIAL PLAN ON SURVEY ITEM 2-1/2-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: Was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of staff capacity to operate two assessment systems?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of sufficient funding or other resources such as materials needed to administer the assessment?

2.1.2. Did your state identify a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER DEVELOPING THE INITIAL PLAN ON SURVEY ITEM 2-1/2-2]?

(If the respondent says ‘no’, ask the probing questions below to verify that they did not identify a strategy to address the challenge, then skip to the next major challenge that was identified under this activity area if applicable.

If the respondent says ‘yes’, say: Please describe the strategy. If your state used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your state try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the CGSA assessment?
Change aspects of the CGSA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.1.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.2. Major assessment implementation activity #2: Preparing for the first operational assessment

(Prepopulate questions 2.2.1 and 2.2.2 with the challenge identified as a major challenge from survey item 3-1. If no challenges were identified as major by the respondent in survey item 3-1, prepopulate questions 2.2.1 – 2.2.2 below with the minor challenge identified as the most difficult challenge in survey item 3-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.2.1 through 2.2.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #2: Preparing for the first operational assessment. As a reminder, we define an operational assessment for CGSA as one that is used to generate student information that is shared with educators and used in some meaningful way (such as contributes to grades, used as a benchmark, reports provided to students or parents, etc.). Field tests used solely to gather information about test quality would not be considered operational.

2.2.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER PREPARING FOR THE FIRST OPERATIONAL ASSESSMENT ON SURVEY ITEM 3-1/3-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of available time for current staff to prepare for administration)?
Change in legislation or policy?
Lack of sufficient funding or other resources such as materials needed to administer the assessment?

2.2.2. Did your state identify a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR] CHALLENGE UNDER PREPARING FOR THE FIRST OPERATIONAL ASSESSMENT ON SURVEY ITEM 3-1/3-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your state used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your state try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the CGSA assessment?
Change aspects of the CGSA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Change the timeline for the first operational assessment?
Leverage existing tools or resources?

2.2.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.3. Major assessment implementation activity #3: Administering an operational assessment

(Prepopulate questions 2.3.1 and 2.3.2 with the challenge identified as a major challenge from survey item 4-1. If no challenges were identified as major by the respondent in survey item 4-1, prepopulate questions 2.3.1 – 2.3.2 below with the minor challenge identified as the most difficult challenge in survey item 4-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.3.1 through 2.3.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #3: Administering an operational assessment. As a reminder, we define an operational assessment for CGSA as one that is used to generate student information that is shared with educators and used in some meaningful way (such as contributes to grades, used as a benchmark, reports provided to students or parents, etc.). Field tests used solely to gather information about test quality would not be considered operational.

2.3.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER ADMINISTERING OPERATIONAL ASSESSMENT ON SURVEY ITEM 4-1/4-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed, For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of staff capacity to operate two assessment systems?
Lack of public support, for example districts’ or schools’ unwillingness to schedule the assessment?
Change in legislation or policy?
Lack of sufficient funding or other resources such as materials needed to administer the assessment?

2.3.2. Did your state identify a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER ADMINISTERING OPERATIONAL ASSESSMENT ON SURVEY ITEM 4-1/4-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your state used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your state try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the CGSA assessment?
Leverage existing tools or resources?

2.3.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.4. Major assessment implementation activity #4: Conducting Post-administration activities

(Prepopulate questions 2.4.1 and 2.4.2 with the challenge identified as a major challenge from survey item 5-1. If no challenges were identified as major by the respondent in survey item 5-1, prepopulate questions 2.4.1 – 2.4.2 below with the minor challenge identified as the most difficult challenge in survey item 5-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.4.1 through 2.4.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #4: Conducting post-administration activities.

2.4.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING POST-ADMINISTRATION ACTIVITIES ON SURVEY ITEM 5-1/5-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of technical knowledge or lack of knowledge about how to collect performance-based assessment data?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of sufficient funding or other resources?

2.4.2. Did your state identify a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING POST-ADMINISTRATION ACTIVITIES ON SURVEY ITEM 5-1/5-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your state used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your state try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the CGSA assessment?
Change aspects of the CGSA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.4.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.5. Major assessment implementation activity #5: Conducting continuous improvement activities

(Prepopulate questions 2.5.1 and 2.5.2 with the challenge identified as a major challenge from survey item 6-1. If no challenges were identified as major by the respondent in survey item 6-1, prepopulate questions 2.5.1 – 2.5.2 below with the minor challenge identified as the most difficult challenge in survey item 6-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.5.1 through 2.5.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #5: Conducting continuous improvement activities.

2.5.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING CONTINUOUS IMPROVEMENT ACTIVITIES ON SURVEY ITEM 6-1/6-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of data literacy skills such as accurately interpreting the scores or an understanding of how to use the scores to plan for or target instruction?
Lack of public support, for example unwillingness to participate in feedback opportunities or the annual evaluation?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of sufficient funding or other resources to conduct an evaluation?

2.5.2. Did your state identify a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER CONDUCTING CONTINUOUS IMPROVEMENT ACTIVITIES ON SURVEY ITEM 6-1/6-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your state used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your state try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the CGSA assessment?
Leverage existing tools or resources?

2.5.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/most difficult minor challenge, go to Section 2.7)

Section 2.6. Major assessment implementation activity #6: Scaling Up the System

(Prepopulate questions 2.6.1 and 2.6.2 with the challenge identified as a major challenge from survey item 7-1. If no challenges were identified as major by the respondent in survey item 7-1, prepopulate questions 2.6.1 – 2.6.2 below with the minor challenge identified as the most difficult challenge in survey item 7-2 and use “most difficult minor challenge” in place of “major” in the interview questions where noted. Questions 2.6.1 through 2.6.3 below should be repeated for each challenge identified as a major challenge.)

I would like to discuss [EACH/THE] challenge that you identified as [MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] within activity #6: Scaling up the system.

2.6.1. You identified [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER SCALING ON SURVEY ITEM 7-1/7-2] as [A MAJOR/THE MOST DIFFICULT MINOR] challenge. Please describe why it was [A MAJOR/THE MOST DIFFICULT MINOR] challenge.

Probe if needed: For example, was it a challenge because of a:

Lack of capacity such as lack of staff capacity to operate two assessment systems?
Lack of public support, for example a resistance to change in assessments?
Lack of political support or a change in political climate such as a change in state leadership?
Change in legislation or policy?
Lack of sufficient funding or other resources such as materials needed to administer the assessment?

2.6.2. Did your state identify a strategy to address the challenge of [CHALLENGE IDENTIFIED AS [A MAJOR/THE MOST DIFFICULT MINOR CHALLENGE] CHALLENGE UNDER SCALING ON SURVEY ITEM 7-1/7-2]?

If the respondent says ‘yes’, say: Please describe the strategy. If your state used more than one strategy to address the challenge, please identify the primary strategy.)

Probe if needed, For example, did your state try to:

Increase capacity such as engaging an external partner?
Improve or increase communications about the CGSA assessment?
Change aspects of the CGSA assessment such as changing the subject of the assessment, the grades tested, or other aspects of the design of the assessment?
Leverage existing tools or resources?

2.6.3. How effective was [THE RESPONDENT’S STRATEGY ANSWER] in addressing the challenge? Would you say it was very effective, somewhat effective, or not effective? What evidence leads you to believe [THE RESPONDENT’S STRATEGY ANSWER] was [ANSWER GIVEN TO EFFECTIVE QUESTION]?

(If the state used more than one strategy, only ask this question for the primary strategy identified and record answers.)

(After asking about all major challenges/the most difficult minor challenge, go to Section 2.7)

Section 2.7 Additional Activity Seen as Most Challenging

(Ask the questions in this section only if the study team’s scoring of the challenges responses in the surveys suggested that another major activity was very challenging for the system; otherwise go to Section 3.)

2.7.1 Your survey responses suggested that [MOST CHALLENGING ACTIVITY BASED ON SCORES] also was an activity with a lot of challenges for your system. That is, the top challenge you reported for system, [CHALLENGE LISTED IN SURVEY ITEM 8-1], did not fall under the [ACTIVITY SELECTED AS THE MOST CHALLENGING (RESPONSE TO SURVEY ITEM 8-2)]. What was it about [ACTIVITY SELECTED AS THE MOST CHALLENGING BASED ON RESPONSE TO SURVEY ITEM 8-2] that made it more challenging than [MOST CHALLENGING ACTIVITY BASED ON SCORES]?

Section 3.0 What worked well, what did not work as intended, and why? (15 minutes)

In the last section, I asked you to tell me about specific challenges related to a particular activity, but now I would like you to think more broadly about the major assessment implementation activities that your CGSA system has been engaged in at any point up to or through the school year in which the system most recently participated in CGSA. Considering all relevant activities, please identify what worked well in terms of supporting the system’s objectives and why. As a reminder, the six major activities are:

Developing the initial plan for the CGSA assessment,
Preparation for the first operational assessment,
Administering an operational assessment,
Conducting post-administration activities such as ensuring technical quality of scores and using reports,
Conducting continuous improvement activities, and

Scaling up the CGSA assessment system

3.1. Thinking across the activities the state engaged in, what practices worked well? Discuss up to three practices that worked well.

Probe if needed, For example, were there particular practices that contributed to the success of the system such as, the practice of recruiting participating districts and schools?

(As respondent is speaking, match the successful practice with the list of major activities using reference list, and then ask the respondent to confirm the match by saying: It sounds like this practice was related to the [MAJOR ACTIVITY X]. Is that correct? Is there an additional activity you would relate this practice to?)

3.2 How did the [THE RESPONDENT’S PRACTICE ANSWER] contribute to the success of the system?

(Repeat the question for each of up to three practices identified in the previous question and record answers. Then ask: Which one practice was the most important contributor to progress overall for [NAME OF CGSA ASSESSMENT]?)

3.3 Please think about the CGSA system as a whole. Were there aspects of the system’s theory of action or the system’s original design that did not work as intended? For example, were there aspects related to the planned grade(s) or subjects tested or planned method for computing student scores or other information (e.g., profiles, competencies) that did not work as intended?

Add if needed: This question is related to the system’s overall design rather than about the specific challenges related to assessment activities discussed above.

3.4 Why did [THE RESPONDENT’S ASPECT ANSWER] not work as intended?

(Repeat this question for each aspect mentioned and record answers. Then ask: Which one aspect stands out as the most important one that you’d want other states to know about and learn from?)

Section 4.0 Practices that the State Wanted to Try, but was unable to (15 minutes)

4.1. I would like you to reflect on the objectives of the [NAME OF CGSA ASSESSMENT]. Were there any practices that the state wanted to implement to support these objectives, but was unable to? What were these practices that the system wanted to implement, but did not, and why? (If none, go to Section 5.)

Probe if needed, For example, were there barriers that prevented the system from implementing a practice that you felt would have supported the objectives of the [NAME OF CGSA ASSESSMENT]?

(As respondent is speaking, match the practice they wanted to try with the list of major activities using reference list, and then ask the respondent to confirm the match by saying, It sounds like this practice was related to the [MAJOR ACTIVITY X]. Is that correct? Is there an additional activity you would relate this practice to?)

4.2. What prevented [NAME OF CGSA ASSESSMENT] from trying [THE RESPONDENT’S PRACTICE ANSWER]?

(Repeat this question for each practice mentioned and record answers. Then ask: Which one practice faced the largest barrier to trying it?)

Probe if needed, were there barriers related to:

The political climate?
Pressures from the public?
State legislation or policy?
Funding or other resources?
Technical issues?
The CGSA program requirements?
The COVID pandemic?

Section 5.0 Wrap Up and Thank You (5 minutes)

5.1. Is there anything we did not discuss regarding the implementation of the [NAME OF CGSA ASSESSMENT] that you would like to share with me?

Thank you for your time.

D-1

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2024-12-24