Third grade is a turning point in learning, where students transition from learning to read to reading to learn. Thus, many education systems now focus on early literacy. There’s a lot of work to do: the National Center of Education Statistics measured in 2015 that only 36% of 4th graders nationwide were proficient in reading. Many research findings show that early literacy is very important in determining later-life outcomes. Students who are not at reading proficiency by third grade have been found to be four times more likely to not complete high school. Lower early literacy rates are closely correlated with lower career readiness and higher incarceration rates.
Because of these statistics, Oklahoma passed the Reading Sufficiency Act (RSA) in 2012 with the intention of ensuring statewide third grade reading proficiency. RSA formalizes the definition of third grade reading proficiency and requires each school district to develop a “reading sufficiency plan” to direct intervention programs towards students who fall behind at any point between kindergarten and third grade.
Our team — fellows Monica Alexander, Charlotte Huang, and Maximilian Klein, with technical mentors Ali Vanderveld and Kevin Wilson and project manager Chad Kenney — has partnered with Tulsa Public Schools (TPS) this summer to develop both an early warning indicator and an intervention recommendation system for the students who do fall behind. The goal is to not only predict which students will struggle to reach 3rd grade reading proficiency, but to also help assign the appropriate type of intervention based on the different needs of individual students.
In Tulsa, reading on grade level is defined through two methods: passing the Oklahoma Core Curriculum Test (OCCT) or scoring at or above the 40th percentile on the Northwest Evaluation Association’s Measure Academic Progress (MAP) exam. Sixty-nine percent of Oklahoma third grade students achieve proficiency according to OCCT scores, while only 42% of students in TPS do — a differential is of great concern to TPS.
The MAP assessment is conducted 3 times a year and is a unified standard across grades, so the data can show early elementary testing gains across the student body in Tulsa. Over time, the distribution of scores does increase in variation over time. Additionally, if TPS students are split into “grade level” and “below grade level” based on their final 3rd grade MAP assessment score, the average score trend lines show two different trajectories. Despite starting in a similar range in first grade, students that will have attained reading proficiency by third grade far outgrow students that will not.
Figures 1-2. The distribution of the test scores thins out over time, showing an increase in variance (left). Students who are qualified either as “grade-level” or “not grade-level” have different growth trajectories (right).
In order to understand what factors impact early reading ability, TPS provided a series of student data to the team. From these details, the team will build a model to evaluate the significance of factors that predict 3rd grade MAP reading scores. Once the key factors are identified, the team will evaluate the optimal effectiveness of each intervention program for different student subgroups.
Figure 3. The model will take these predictors as features and predict the target test scores. Furthermore, the team will analyze which of the actionable interventions can be most impactful on those predictors.
However, our conversations with Tulsa Public Schools raised two concerns about early warning indicators (EWI). First of all, these models often make predictions too late to intervene — accurately identifying a struggling student shortly before they take the OCCT in 3rd grade leaves little time for the school to offer extra assistance. To mitigate against predicting too late, the team intends to mostly look at additional longitudinal factors over the student’s entire early elementary span, starting as early as pre-kindergarten. From the data we received, the team can infer additional details such as transfers between schools, MAP score and grade trends, and the timing of transfers, absences, and other disruptions, which may prove to be useful in detecting early signs of falling behind grade level. We’ll test the importance of these new factors when we begin building predictive models on the data.
Secondly, EWI predictions are typically based on characteristics that cannot be directly changed by school systems, such as demographics or attendance. Several previous studies found that attendance strongly influences 3rd grade reading test scores. But despite an intense program to improve attendance rates in Tulsa, the system’s attendance proved hard to budge. Therefore, the team will focus our model on factors with higher potential for impact, such as the intervention programs, in order to influence the predictors.
By the end of the summer, the team plans to develop a model that can identify which students will not be reading on grade level by the end of third grade and make intervention recommendations for various student needs.