Opening New York City schools for the 2020-2021 school year has involved delays, temporary closures, and policy changes due to the COVID-19 virus, challenges which have been compounded by the social and political stresses that the nation faced in 2020. Yet, despite these challenges, the 2020-2021 school year has also provided a critical opportunity for schools to provide families with both in-person and remote modes of instruction and collect data that will inform how schools can continue to adapt to a changing world. At Classical Charter Schools, a network of four K-8 charter schools in the South Bronx, we have embraced this research opportunity and provided parents with an option to continue learning remotely or attend school in-person to support all our scholars during a historically difficult time. We have relied on dedicated families and staff, thoughtful changes in daily instruction, and detailed data analysis to deliver a consistent education to scholars learning in-person and remotely. And, in tandem, we have leveraged our dedicated Data & Research team to understand how scholar learning is affected by the changes to education that were prompted by the COVID-19 pandemic, using key insights to further direct how we leaned on existing systems and adapted others to best serve our students.
In this article, we examine how Classical is leveraging its data infrastructure, data culture, and trove of historical student data—specifically scholars’ reading level data and scholars’ performance on formal assessments—in the context of its 2020-2021 learning program of families choosing to learn either in-person or remote. We illustrate how custom data solutions and dynamic dashboards have provided the flexibility needed to pivot research in summer slide to understand how COVID-19 is affecting academic achievement, a phenomenon many are calling the COVID-slide. And, finally, we share how our ongoing analysis of student data is helping us plan for a post-pandemic world.
Creating the Conditions to Support In-Person and Remote Learning
Two key prerequisites to effective COVID-slide research are collecting quality data in remote and in-person models and creating the flexible data systems necessary to analyze new types of data. To prepare for reliable data collection and a smooth transition to a partially remote program, we ensured that all scholars had reliable access to technology and were attending class daily. At the beginning of the school closures in March of 2020, Classical provided households with devices such as laptops and tablets based on surveys administered to families to determine need. Classical next created systems for attendance tracking with Power BI dashboards and set clear expectations for attendance, and helped students and families meet these expectations. Teachers took attendance for each class and by the next day deans were able to call families of scholars who were not present by addressing concerns such as technology issues and getting scholars in class the next day. Due to these two preparations, scholar attendance remained on par with that of a typical academic year. Classical’s network-wide attendance was consistently 96% for the remote learning program.
After ensuring scholars could reliably access the remote learning program, Classical took steps to ensure that data collected remotely would be a valid and reliable proxy for scholar learning. Families learning at home received training on our remote testing platforms, which is particularly important for younger scholars who need parent assistance accessing online tools. Further, kindergarten and first grade scholars received 20% more time than in-person scholars to take tests to account for the extra time it takes to navigate the platform. In November, additional testing time was rolled out to all grades, which helped to improve overall test completion online. Lastly, scholars learning remotely were given the option to take tests in-person if the scholar might benefit from this adjustment.
We knew it would be important to monitor the impact of our remote program in real time—at the national level, researchers predicted learning loss in both reading and math, particularly for students in low-income areas and with younger scholars losing the most learning. Fortunately, we had strong foundations to do this monitoring work. Classical has historically relied on data analysis and research to support teachers in planning individualized lessons and school leaders in analyzing trends in scholar performance across our network of schools. To do so, it has utilized Microsoft Power BI dashboards to display and analyze scholars’ academic (and non-academic) data in one place. These dashboards, which were developed and iterated over several years by our own Data & Research team, allow teachers, school leaders, and other staff to identify trends in scholar performance at the individual, class, school level, and help tailor interventions and guide improvement across the network. For example, using its custom Power BI dashboards, Classical researched and ultimately reduced learning loss in reading over the summer by offering a reading-focused summer academy, and tracking scholar reading levels before summer break each year and again after scholars return in August. This data-driven intervention helped reduce the summer slide year-over-year, allowing 87% of scholars to begin the year on grade level in reading in 2019-2020, as compared to 79% of scholars the year before.
Monitoring COVID-Slide with Reading Levels
Reading has always been a focus at Classical, and it has remained an area of careful monitoring throughout the pandemic. At Classical, we use the Fountas and Pinnell program to assess scholars’ reading levels, which allows us to compare data year over year and define grade-level benchmarks. Classical began the 2020-2021 school year on August 19th with all scholars receiving remote instruction. Teachers therefore began testing scholars’ reading levels remotely over Zoom adhering to the Fountas and Pinnell system. Across the network, there was a small drop from June to August in the percentage of scholars on grade level in reading. This was to be expected, considering the known effects of the summer slide; however, Classical’s Data & Research team was also interested in understanding the compounding effects of remote learning the previous spring on expected summer slide reading loss. To investigate this, we examined scholar reading growth beginning in March 2020, when Classical—along with all NYC schools—pivoted to remote learning.
In March 2020, 90% of Classical scholars were on grade level for reading, which was the highest percentage we had seen for March since we began tracking these data in 2008. This finding aligns with the historical trends we have seen at Classical, as we continue to find better ways to support our scholars: over time, we have seen scholars slide less over the summer each year, and we have started the subsequent school year with more and more scholars on grade level in reading (see Figure 1). The high reading achievement that preceded the COVID-19 pandemic was fortuitous, as it set scholars up to end the year at or close to grade level, despite potential learning loss accrued during the adjustment to remote learning. Nevertheless, for the first time since 2015, Classical scholars ended the 2019-2020 year with less than a years’ growth in reading. With this knowledge, Classical leaned on its Summer Learning Academy to mitigate further slide. Importantly, (1) in 2020 the Summer Learning Academy was virtual and (2) we expanded the Summer Learning Academy to more students than usual. As shown in Table 1, scholars slid the same amount over the summer in 2020 as they did the previous summer, which is a testament to a unified effort of teachers, scholars, and families. Despite these efforts, we knew that many scholars would begin the 2020-2021 academic year slightly behind in reading, and the Data & Research team knew that close monitoring of reading growth would continue to be vital.
Since the beginning of the 2020-2021 school year, Classical has continued to monitor scholars’ reading growth. As expected, we are seeing a lower percentage of scholars on grade level than we have in previous years (from an average of 87% to 81% at the start of the academic year). We are also seeing that this drop is happening across all grades and not, as we expected, primarily affecting the youngest of scholars. These insights are being used by Classical’s at-risk team to give extra support to struggling scholars and to create more instructional supports for the spring and summer to make up for lost learning.
Creating a COVID-Slide Dashboard
To understand the COVID-slide, it was not enough for Classical to track the data it always tracked, in the ways it always had. Specifically, with some scholars learning remotely and others learning in-person, there were bound to be differences in academic performance across these two student subgroups—despite all of our efforts to standardize and bring the same level of rigor and supports across the at-home and in-person learning experiences. To monitor these differences, we built a COVID-slide dashboard that compares how scholars are performing in the 2020-2021 year with the previous year’s performance. The dashboard has multiple views, drilling down to focus on performance holistically in a subject, performance on individual standards, and performance by learning location (remote or in-person) and testing location (online or in-school and on paper).
The flexibility of the Power BI dashboard has allowed our instructional team to continuously adapt our learning program. For example, due to the novelty of the remote testing platform, we noticed more scholars with incomplete tests than during a typical school year. To address this issue, we made tests in the younger grades more remote-friendly—for instance, including more oral questions—and began offering in-person testing for remote scholars. Data-informed ongoing adjustments in our schooling approach will lead to more accurate data collected online and, in turn, a clearer picture of scholars’ growth. As the year goes on, the data dashboards will allow for real- and near-time monitoring of performance that will contextualize scholar learning in comparison to prior years and help teachers in goal-setting and communicating with families.
In fact, monitoring to date of Classical’s COVID-slide data has already revealed some interesting trends. In some grades, in-person scholars are performing at levels commensurate with pre-COVID scores on Classical’s internal assessments. We take this to suggest that smaller class sizes for in-person instruction is leading to accelerated learning, which is mitigating the effects of COVID-slide. For example, fourth graders learning in-person have stayed on par with 2019-2020 scores in English Language Arts. In contrast, remote learners are performing consistently lower than historical averages, suggesting consistent remote instruction has been insufficient to compensate for the COVID-slide (see Figure 2). In other grades, remote instruction has improved over time, leading to remote scholars catching up to in-person scholars and performing at the same level. In eighth grade, for instance, remote and in-person learners are performing very similarly in math. However, both groups are performing at significantly lower levels than we would expect, based on Classical’s historical data. We take this to suggest that the difference in quality for both remote and in-person instruction for this subject and this grade band has been effectively eliminated. However, instruction in both cases has not been enough to compensate for the compounding effects of the COVID-slide, such that students will likely need additional supports and learning time to catch up (see Figure 3). Our instructional coaches are continually monitoring these changes to support specific scholars and classes and to identify areas of the curriculum that may require review on a larger scale across an entire grade.
Throughout the rest of the year, we will continue to learn from the granular data we collect on student achievement, both for in-person and remote learning, to improve our programs and minimize the impact of the COVID-slide on Classical scholars. Looking to a post-pandemic future, however, we are actively trying to learn from this difficult year, to see when and where (in what subjects, what grades, for which students) options and additional supports we have implemented this year can help students in years to come—making up for lost learning and creating new learning opportunities with which to enrich the Classical schooling experience.
Next Steps
While we will continue to search for trends in academic performance across our classrooms and schools, we remain aware that important differences in scholars’ experience during this unprecedented time. These differences are leading to a highly individualized picture of achievement for each of our scholar. Some of these differences are due to scholars’ personalities and preferences—while some scholars are struggling to learn remotely, others are flourishing at home. Other differences are due to access, opportunity, and material circumstances. The fact is that learning remotely looks very different for scholars based on the devices they have access to, the reliability of their Wi-Fi, and availability of adults to help with technical issues. Even in-person learning can be very different across school sites, as students attending school in person can be sent home for short periods due to changes in city policy or a rise in COVID-19 infection rates in the greater community. Ultimately, while Classical continues to work to offer all of its scholars the best learning experience possible, and to ensure that none fall behind, these differences in scholar experience remain outside of our control. This makes it all the more urgent for us to study our data, learn from what we are doing, and prepare to turbo-charge student supports once we return to greater stability in a post-pandemic world.
As more data are collected this year, Classical’s Data & Research team will continue to consider what is driving success in the remote program and what can be done in terms of remediation over the summer, to set scholars up for success next fall. We plan to investigate the impact of in-person instructional days and level of engagement for remote learners. We will also examine how factors such as attendance and access to technology affect academic outcomes and identify which common core state standards scholars need support with at the grade level. This continued research is critical to our ability to support scholar’s individualized needs in this uncertain time and will—hopefully—allow us to emerge from this challenging period better prepared than ever to help our scholars learn and excel.
Appendix
Table 1. Annual Reading Growth (in years)
Figure 1. Percent of Scholars on Grade Level in Reading
Figure 2. Fourth Grade English Language Arts Test Average by Learning Location
Figure 3. Eighth Grade Math Test Average by Learning Location
This post was contributed by Ms. Forshee and Ms. Maguire, members of our data & research team. As a non-CMO charter network, we rely on the thoughts, opinions, and innovations of our staff to move our mission forward and provide an excellent academic option to families in the South Bronx. To hear more from our staff, check out the next post! Or, click here to learn more.
