A Northwestern University research project to analyze Evanston school data to identify indictors for student success reported its progress this week at a joint meeting of the District 65 and ETHS school boards.
The schools partnered with David Figlio, dean of the school of education and social policy at Northwestern, to collect and analyze school datato identify the factors that contribute to post-secondary success and develop a model that identifies on-track indicators from middle school through high school.
The partnership, called Northwestern Evanston Education Research Alliance, or NEERA, was established last fall, with a goal to “bring teaching, research and service goals of Northwestern University in a mission to be of service to our home towns,” said Figlio.
Both superintendents – Paul Goren of District 65 and Eric Witherspoon of ETHS – cautioned that the project is a work in progress. “It will take time to build the infrastructure,” said Goren.
Figlio said that because of funding from the Lewis-Sebring Family Foundation and Spencer Foundation, “we’ve been able to do in one year what it would have taken five years” to do without the grants, including hiring staff, transferring data securely, and aligning data across the districts," as outlined in the project report.
While initially working in collaboration with the Evanston schools, Figlio added, “We aspire to work with other leading organizations such as the McGaw YMCA.”
“This project is the first of its kind in the country,” Figlio said.
The research goals were determined by the administrations of the two school systems.
Because ETHS has a relatively high graduation rate, Figlio said, at least compared to the Chicago public schools, NU’s other “home town” and to the statewide average, the research team identified success as “college persistence,” defined as completion of five consecutive semesters of college or completion of a two-year or certificate program.
“We will look at key points in a student’s progress through the school system,” explained Lila Goldstein, NEERA research data analyst lead, to determine if there are “factors related to their experience in school that can predict post-secondary success.”
The project started with eighth and ninth graders to see what indicators predict post-secondary outcomes and will then fill in other grades, Goldstein said. The next question is what middle school indicators can predict high school success.
Chart from NEERA analysis.
The first analysis looked at ninth grade data for students in the ETHS graduating classes of 2012-14, with the goal of identifying actionable, trackable factors that correlate with college persistence.
The data shows that 85 percent of ninth graders for the class years 2012-14 graduated and 85 percent of those enrolled in college. Only 64 percent of those who enrolled in college completed five consecutive semesters or a 2-year certificate or associates degree.
Chart from NEERA report.
The data also documented racial disparities in graduation, college enrollment and persistence.
The project analyzes factors that are considered both trackable and actionable to predict outcomes and address students’ needs early.
- progression through school
- course-taking pattern
- standardized test scores
- disciplinary incidents
In addition, the project measures other factors, including race, sex, family, birth country, and home language to ensure that the predictors work for all students.
Goldstein noted that different measures have different levels of predictive power, such as getting an F or not getting an F in 9th grade, or the interaction of different factors, such as absences and grades in English classes.
However, she emphasized, the formula does not tell us how to address a student’s needs and is not deterministic. In addition, she said, “we cannot measure everything that impacts student outcomes.”
Moving forward, Goldstein said, they’ll add grades 10-12, begin to predict high school readiness from grades 6-8, predict other college outcomes, and identify thresholds for flagging students for support.
The research team anticipates that the data analysis will be able to address students’ needs early, inform instruction and practice, provide feedback for program improvement and monitor high level trends.
“I’m grateful for what you’re doing,” said Jonathan Baum, ETHS board member, “but also frustrated,” with the seeming lack of progress on goals, such as the joint literacy goal that was established five years ago. “You said we don’t have all the data yet. What’s missing?”
FIglio responded that the data collection infrastructure has only been in place since February and they needed to harmonize the data between the two districts and sign data sharing agreements.
Based on the work done so far, “Northwestern, the donors and Evanston taxpayers are getting a great deal,” he said.
“Some of the data has never been cleaned before,” he added, explaining that they’ve tightened up some variables and added years of national clearinghouse data, which provides post-secondary education data. “It may take longer to develop formulas for some predictors.”
“What can be done faster,” asked Baum. “Are there combinations of variables that are predictive of what Evanston cares about? Are some years more important than others? Are you looking at that?”
Figlio responded that Northwestern researchers wanted to look at everything to find breakthroughs.
Jude Laude, another ETHS board member noted that there are other factors that affect college persistence.
“What does a successful college student look like? What are the non-academic factors when a student gets on campus, such as availability of academic support, culture shock, resiliency and financial aid? What are the attributes of colleges and university where students succeed?”
Figlio responded that they start from what is administratively available. “Although there may be other factors besides the on-track indicator, we need to collect the data systematically.”
Joseph Hailpern, D65 board member, expressed his frustration that they don’t seem to know the effectiveness of various interventions in early grades, such as what approaches work to help kids who aren’t reading at grade level.
“There seem to be lots of metrics to measure,” he said. “Can we or will you be able to measure success at the early childhood level?”
Figlio responded that there will be a cascading system of on-track indicators that will be synchronized from early childhood through high school.
Sergio Hernandez, District 65 board member, noted that several factors, such as attendance and disciplinary incidents, can be backtracked to early childhood.
“That’s way past reading and math scores.” he said. “If attendance matters starting at pre-school, we can provide supports to get them in school.”
Hailpern asked how this data can be used. “Our faculty K through 5 needs to actively use these factors.”
“We need to know who requires support,” he added, but we also need to know what supports to provide and their effectiveness. “There’s data in the system, but we need to actively identify the experiences our kids are having -- who is getting what support -- and track that as part of the student profile.”
“It’ll take ten years” to see an effect,” he said.
“You’re way too pessimistic,” Figlio said. “It won’t take ten years. If the data is in the system, it’s just elbow grease to clean it up.”
“We can’t stop doing what we know to do,” said Monique Parsons, ETHS board member who is also chief operating officer at McGaw YMCA. “We’re not waiting for Northwestern. Whatever we get we will use at the agencies we’re responsible for.”