q
Welcome to moose theme.
Enter any text or widget here.

Celebrating Success: There’s More to the Story

Compton Mills

Blog-Compton-Mills-Robert-ComptonAuthor 1:

Robert H. Compton, Ph.D.

3710 Landmark Drive, Suite 201

Columbia, SC 29204

803-734-8067

In his 16th year in public education, Dr. Compton serves as the Assistant Superintendent of Academic Services for the South Carolina Public Charter School District.

 

 

Courtney MillsAuthor 2:

Courtney Mills, M.Ed.

3710 Landmark Drive, Suite 201

Columbia, SC 29204

803-734-0525

Formerly a math teacher and school administrator, Courtney Mills is the Director of Academic Programs for the South Carolina Public Charter School District.

 

 

This article describes one district’s incentive program that rewards EVAAS Level 5 growth while reviewing sample reports that can help pinpoint areas of need.

 

While the names and some details have been changed, the following tale is far from fictional. It is based on the true story of two public charter schools: one where 87 percent of students are “Ready” according to ACT Aspire (99th percentile in South Carolina) and one where only 22 percent of students are “Ready” (13th percentile).   While the first school may appear far more successful, there is more to the story. Believe it or not, the South Carolina Public Charter School District (SCPCSD) just publicly celebrated the performance of the second school—the one performing at the 13th percentile. How is this possible? Read the story below to find out.

A Tale of Two Charters       

For six years, Excellence Charter School has built a reputation to match its name. Year after year, the school ranks among the highest-performing elementary schools in South Carolina. In fact, due to its proven success and innovative practices, the SCPCSD has encouraged the school’s leaders to replicate into another location in the state so more students can have the opportunity to attend a high quality public charter school.

Progress Charter School has taken a different trajectory since its inception four years ago. This high-poverty rural school opened its doors with a mission to provide a better educational option for families in its community. While it has consistently outperformed the local district where the school physically resides, its overall performance elicits serious concerns. Not only does the school rank in the 13th percentile for all middle schools, but it scored an “F” on its last two federal accountability ratings. If South Carolina had not paused school ratings this year, the school may have faced automatic closure under the charter law.

Defining Success

Traditionally speaking, a school’s success has been measured by one indicator: the percent of students meeting or exceeding state standards. Sadly, based on last year’s test scores, fewer than 50 percent of students in South Carolina can boast this accomplishment. Because so many students in South Carolina are not considered “ready”—and because school turnaround takes years— many states like Tennessee, North Carolina, Ohio, and Pennsylvania use an additional tool to measure success. This tool—which happens to currently be free and accessible to South Carolina school districts—is the Education Value-Added Assessment System, or “EVAAS.” Building on methodology developed at the University of Tennessee at Knoxville, EVAAS enables educators to recognize progress and growth within the year and over time, even when a school is far from reaching the ultimate goal of student proficiency. It also shines a light on what it takes to achieve the world-class knowledge, skills, and life and career characteristics envisioned by the Profile of the South Carolina Graduate.

On December 10, 2015, when the EVAAS data became available to districts, an eye-opening discovery revealed itself. Excellence Charter School, which ranked at the 99th percentile, received an EVAAS Level 1 growth rating. According to SAS—the group that created EVAAS—Level 1 means “least effective: significant evidence that the school’s students made less progress than the growth standard.” On the other hand, Progress Charter School, where only 22 percent of the children are meeting or exceeding state standards, earned the highest possible EVAAS growth rating: Level 5.

Just like many who assess school effectiveness, perhaps we’ve been placing too much emphasis on one data point. While overall proficiency is ultimately the most important measure, how else can we define “success” for the schools in South Carolina who are far from that target of absolute proficiency? And once students have attained the status of “proficient,” are there additional ways to measure their success beyond just maintaining that status? What if there was more to the story of success?

Rewarding Growth

The SCPCSD believes in the powerful value of EVAAS data, and that schools and teachers earning Level 5 growth—which means their students far surpassed what was expected of them—should be recognized and rewarded. At the surprise of our school leaders, during a monthly leaders meeting, four school leaders were asked to stand and be recognized. Before revealing the EVAAS data, we asked everyone in the room what these four schools had in common. No one could guess. Two of the schools represented were well known “high fliers”, but not all of the high performers were standing. The principal at Excellence Charter School was not. Then there were two other unlikely suspects standing, one of which was Dr. Plummer, principal at Progress Charter School. Most of the leaders in the room know about the struggles facing his school. They were stumped.

To break the silence, we finally announced that these four schools’ all earned Level 5 success on EVAAS. Unprompted, many of the school leaders—all of which come from competing charter schools—stood up and clapped. It was at this point that we introduced our inaugural group of the “Hi-FIVE Club.”

ComptonMills Photo 1
Charter school leaders celebrate the success of their schools in achieving EVAAS Level 5 growth. Photo by Laura Bayne.

To celebrate successful growth—and leverage the power of EVAAS—the SCPCSD created the Hi-FIVE (Funding Instructionally Valuable Educators) Incentive Program, where every certified employee in the Level 5 schools who returned from the previous year will receive a one-time stipend. We decided to reward all certified employees of Level 5 schools instead of just rewarding the teachers with an EVAAS score because many teachers would not be eligible based on their grade levels or subject matter taught. Secondly, we believe it takes a community to achieve greatness, and every member of the school community played a part in the school’s overall success.

The celebration will not stop with the school leaders. Over the next month, just like the Publishers Clearinghouse Prize Patrol, district administrators, including the superintendent, will make surprise visits to the four schools’ faculty meetings to break the news to the teachers about their stipends and to let them know how much we appreciate the positive impact they are having on the students.   In addition to the four EVAAS Level 5 schools, an additional seven returning teachers at other charter schools achieved individual Level 5 growth. Those individual teachers will receive a stipend as well.

Compton Mills
District representatives surprise charter school teachers with their incentive stipends and welcome them to the “Hi-FIVE” Club.  Photo by Laura Bayne.

After the Leaders’ Meeting where the Hi-FIVE announcement was made and we celebrated the achievement of the Level 5 schools, Dr. Plummer from Progress Charter School wanted to meet with several members of the district leadership team about his quandary.   “I want you to know that while I’m happy to be recognized as a Level 5 school, I’m not satisfied yet. Our school is still mediocre in that I know we can do better. It’s great that we’re outperforming the district where we are located, but it’s not enough. The students here deserve better.”   He ended with a promise of excellence. “I’m not going to let my staff be complacent and hang our hats on being Level 5 because the truth is, until I can increase our overall performance as measured by proficiency on the State Assessments, my team and I will not stop.”

Using EVAAS

The comparisons between Excellence Charter School and Progress Charter School perfectly illustrate the difference between measuring student achievement and student growth. Traditional measures of school performance focus primarily on absolute achievement—typically the percentage of students who perform at or above some predetermined cut score. While this type of measure provides valuable information about the current demonstrated knowledge of the students in the school, it does not fully take into account how much students have grown. EVAAS does—it uses students’ entire testing history, along with current performance, to provide a more complete picture of whether achievement is moving in the right direction.

ComptonMills Figure 1
Figure 1: Student Growth versus Student Achievement

Consider Figure 1, which depicts the performance trajectory of two different schools. The school above the green line could be Excellence Charter School. All of its students are scoring proficiently, so it would fare well on traditional measures of school performance. But even though they remained above proficient, the trajectory indicates that these students are not maintaining their previously high performance, and this school’s EVAAS score would likely be a Level 1.   On the other hand, the school below the green line could be Progress Charter School. Its students are all below proficient—some years behind—but it is clear to see they are moving in the right direction and catching up to their peers. The difference between achievement and growth is striking when visualized this way.

What can a school or district leader do with this data? The overall composite score of Level 1-5 is good information, but EVAAS provides so much more. School-level data is broken down by teacher, by grade and by subject area over several years so that trends can be identified, and through sophisticated statistical modeling, this is available even when the summative assessment changes each year. As Dr. Plummer discovered, a handful of excellent teachers can greatly impact a school’s composite score. As other leaders probably noticed, just a few ineffective teachers can have the opposite effect. Thanks to EVAAS, all leaders now have this data at their fingertips.

Perhaps the most valuable information contained in EVAAS is found in the diagnostic reports, a sample of which is seen in Figure 2. SAS has divided students into performance groups based on their prior achievement and then measured the growth for each group. In this example, it is apparent that the school achieved significantly more growth with the highest performing students than it did with the lowest performing group. This type of information, when available for every tested grade and subject, can be invaluable in determining what programs are working and where more resources may need to be devoted.

Compton Mills Figure 2
Figure 2: SAS EVAAS School Diagnostic Report

EVAAS reporting can be expanded even further, down to the classroom level. Examine Figure 3, which represents a sample Decision Dashboard Report at Progress Charter School. The numbers 1 through 5 correspond to the students’ prior achievement levels, and the colors indicate the level of growth observed on the most recent assessment. When Dr. Plummer studied these reports for his 4th grade teacher, he observed she made significant progress in ELA with the lower (1-2) and higher (4-5) achieving students, but not with the middle (3) group. The opposite proved true in math, where groups 3-4 achieved the greatest growth. In Science, the higher achieving groups (4-5) showed a lack of growth. This prompted Dr. Plummer to begin a conversation with the teacher about her specific intervention plans in the different subjects.

Compton Mills Figure 3
Figure 3: SAS EVAAS Decision Dashboard Report

Countering Arguments

Not everyone is a proponent of EVAAS, and it does have its limitations. The first argument against its use relates to how it might be applied for accountability—that teachers should not be held fully accountable for any one test or data point, given the range of factors and measures involved in student learning.  This is a valid concern and the reason why EVAAS results should be used in context of a bigger picture. Classroom observations, background details on individual students, and knowledge of the assessment landscape should all be considered when determining how to apply EVAAS results. For instance, back in 2015 and again for 2016, South Carolina adopted new summative assessments for grades 3-8. While the EVAAS statistical method can still be applied to measure growth across new assessments, the fact that a change occurred for which some students may not have been adequately prepared is certainly a factor that must be weighed.

When Dr. Plummer delved into his data, he found the results mostly supported what he already knew about his teachers, and he appreciated having data to confirm the positive work observed in classroom walkthroughs. But it also shined a light on a few areas that needed improvement, such as the results shown in Figure 3. He plans to use this information for targeted professional development with certain teachers. It is this balanced approach that SAS recommends when using EVAAS data—reward success and provide technical assistance and adjustments where improvement is needed.

There is no one measure that provides a complete picture of a student’s, teacher’s, or school’s performance. But we are convinced that this picture cannot be completed without the kind of growth data that EVAAS provides. And what we hope to accomplish with our Hi-FIVE program is a way to reward hard working educators for a job well done, rather than—as some have suggested—using this information in a punitive way. Is EVAAS perfect? No. Will this provide encouragement for all teachers and schools to achieve the same outstanding results? We certainly hope so. And as for Excellence Charter School and Progress Charter School, which one will ultimately produce the most successful graduates? Well, there’s more to the story.