When STAAR data from the 2022-23 school year was released in September 2023, questions immediately emerged about the dramatic number of students at all grade levels earning a zero score for the Extended Constructed Response (ECR) items on the new reading and writing assessment. For example, 42% of third graders and 46% of fourth graders received a zero for the ECR. As Commissioner Morath remarked in speaking to the State Board of Education August 2023 meeting, it was unclear whether students struggled to read or struggled to type. (Read more at this link)
This spring, TEA revealed that for the first time, an Automated Scoring Engine (ASE) was used as the primary tool to score December STAAR EOC English I and II retests, with only 25% of responses scored by humans. Once again, concerns were raised over the number of zero scores statewide. (Read more at this link). TEA offered educators the opportunity to review the zero-score responses to provide better insights and to respond to concerns about the accuracy of the scoring methodology. Educators shared their feedback from the review of these responses, as well as their insights into other factors that may have contributed to so many students failing on the ECR questions.
The consensus of educators we spoke with was that the change in the writing task itself and in the new scoring rubric were primary factors affecting students’ zero scores. While the December 2023 STAAR assessment has not been released, TEA posted resources to support evidence-based writing in October 2022 and the more detailed Constructed Response Scoring Guides in September 2023 along with the spring 2023 released tests in the Texas Assessment platform. The text-based prompts clearly posed a greater challenge for students. As Al Garcia, Chief Academic Officer in El Paso ISD, noted, “The folks who struggled the most in our district were the 4th and 7th grade teachers. This was not the writing sample they were used to teaching, where strategies like dialogue or colorful language might be effective. This was a response to a stimulus that needed a thesis or argument with evidence that was congruent to that thesis or argument.” Dallas ISD Chief Academic Officer Angie Gaylord concurred that, “We needed to untrain on some of the strategies we had taught before” such as using personal anecdotes and experiences as supporting evidence. Alana Morris, High School Humanities Coordinator in Spring Branch ISD, noted that on the zero score responses she examined, “many students simply summarized the selection and did not respond to the prompt or provide any text evidence to support a thesis.” Thus, changes in the task resulted in many students not yet fully understanding the writing purpose and using strategies that they may have relied upon previously that were no longer effective.
The new rubrics have a more explicit focus on structure and purpose. Separate categories were created for controlling idea/thesis (informational writing) or argument (persuasive writing); organization, including introductions and conclusions; and evidence. These factors were combined into two categories in the previous rubric (see for example the 2018_STAAR_English_I_Scoring_Guide.) The threshold for students to earn at least a one-point score on Development and Organization of Ideas is higher on the new 2023 rubrics: the thesis or argument must be evident in the response, as well as an organizational structure and at least some text evidence. If students do not earn at least a one for this section, their response is also automatically also scored as zero on the Conventions section of the rubric. Tony Perez, Instructional Specialist from Northside ISD, said that “the previous rubrics gave kids more flexibility. The new rubric is more of a tightrope. Students have to answer the question directly or they are in danger of getting a zero.” With the new text-based prompt, students can no longer rely on their own prior knowledge, experiences, or opinions for evidence. They must closely read and examine the passages to select the best evidence from the texts to support their thesis. Tony noted that students must also clearly understand the texts, the author’s purpose, and HOW the author uses details and facts to achieve the purpose. In order to score at the highest level, a three, a response must have a clearly developed thesis or argument, a purposeful structure with an introduction and conclusion, and text evidence along with an explanation of how the evidence supports the claim. For prompts based on two selections, grades 3—5 students may draw evidence from only one selection to earn a 3, but grades 6—EOC II students must cite evidence from BOTH selections to earn a 3. For an argumentative response at grades 8—EOC II, students must also identify and refute counterarguments to earn a score of 3.
While educators still have questions about the use of the ASE for EOC assessments this December, those who reviewed the zero scores felt that for the most part, the responses were accurately scored. Of the 3700 responses reviewed by the Dallas ISD team, only 15 were questioned. Similarly, in Northside ISD, of 1451 responses reviewed, they only challenged 7. Carin Adermann, Northside ISD Director of Testing and Evaluation, remarked, “We went in with rubrics in hand. As I read, I could figure out why most of the responses got a zero. They weren’t responding to what was asked, much less providing text evidence.” However, more technical data is needed to build confidence in the accuracy of the ASE. As Angie Gaylord said, “Reviewing zeros gave us confidence that the ASE was accurate on zeros, but we need more evidence of accuracy and equity for ALL our students. Seeing the responses that were scored above a zero would help us better understand how they calibrate and would help us explain the scores to our community.” Roxanne Ramos, El Paso ISD Accountability and Assessment Coordinator, noted that the number of EL students receiving zero scores on the ECR for English I and II EOC did not show a large variation from their population as a whole.
As more information becomes available about the ECR assessments, professional development will be essential to ensure teachers have the information to fine tune instructional strategies to prepare students for success. When ECR scores are released this fall, TEA also plans to release the STAAR assessments, student responses, and scoring guides. “You learn so much from reading those responses that tells you about the instructional implications for your campus,” said Carin Adermann from Northside. Al Garcia said El Paso ISD has already taken bold steps over the past year to implement HQIM’s and has invested heavily in intensive professional development down to the classroom level. In spring 2024, El Paso ISD implemented district-wide Trend Walks, classroom visits by multi-disciplinary teams that include district and campus leaders from various departments. The teams informally gather indicators of fidelity of implementation and provide feedback to celebrate strengths and support extending learning. In Dallas ISD, Angie Gaylord said that, “this year we implemented HQIM in Reading Language Arts and Math, which created a common language around a shared set of resources for Tier 1 instruction,” but added that in a district with over 9,500 teachers, “when there’s so much change so quickly, we had to accelerate our overall academic strategy to build understanding across the entire system.” Tony Perez commented that despite providing instructional coaching over the last year, he still had questions from master teachers, and he was concerned that newer teachers may not be asking questions but may still be struggling. Dr. Janis Jordan, Chief Academic Officer for Northside ISD, voiced the concerns of many instructional leaders when she said that “we have strong district systems but communicating the information to teachers takes time. The X factor is also the context we are in with teacher shortages, budget cuts, and the stress that creates. More time would be hugely beneficial given the amount of change.”
Texas educators and students have always risen to the challenges posed by changing assessment requirements. As Al Garcia, who served on the STAAR Redesign Committee, summed up, “I think our students are poised this year and have a greater understanding of the cognitive demand and the context of the Extended Constructed Response Items.”
Read more:
On the use of ASE/AI: Is It Fair and Accurate for AI to Grade Standardized Tests? | EdSurge News
Texas Education Agency defends use of computers to grade STAAR (houstonchronicle.com)
How Texas will use AI to grade this year’s STAAR tests | The Texas Tribune
Accessing Early STAAR Results for Spring 2024 (TEA Assessment website)