This post has been authored by Christian Rennert.
Improving Student Success: Helping Study Planners by Evaluating Study Plans with Partial Orders
In recent work, we’re tackling a critical challenge in higher education: how to help students complete their studies on time. For example, across all study programs offered in 2022 in Germany, nearly 247,000 students received their bachelor’s degree [1]. However, first-time graduates received their degree the same year on average after around 4 years of studies [2]. Understanding why these delays occur and what can be done to address them is vital for improving education systems.
Our recent research focuses on how study plans — blueprints for the sequence of courses that students must take — align with actual student behavior. Using data analysis techniques and partial order alignments from the field of conformance checking, we’ve developed a method to uncover where students deviate from their study plans and how / how much they deviate.
A New Approach that Supports to Understand Study Behavior
We use process models to represent a study plan and compare the resulting study plans to the actual traces students take, which are described by an educational event log. An educational event log contains the course enrollments and completions for each student. By modeling these traces as partial orders — an approach to avoid introducing a strict order when courses are taken in parallel during a semester — we can identify mismatches between the planned and actual course orders.
Figure 1: A proposed framework to obtain aggregated deviation information based on order-based and temporal-based deviations from the present study plan.
Our approach can be better explained using the framework shown in Figure 1. Here’s how it works:
1. Model the Study Plan and Translate the Event Log into Partial Orders:
You can model several study plans and check for the best fitting if there are changes to the study plans in between what may well happen in a university setting. The process model of the study plan describes for each course a range of terms that the exam can be taken in. It does not allow for any courses to be skipped since they are mandatory. Further, each student’s exam-taking behavior must be transformed into a partial order. Therefore, each exam try of a course is mapped to a relative term for the student and then a partial order is created. An actual study plan that we obtained data for is shown in Figure 2. In Figure 3, we show an example educational event log that one can create a partial order.
Figure 2: RWTH’s computer science study plan from 2018 being modeled as a process model.
Figure 3: Translation from an example educational event log to a partial order.
2. Computing the Partial Order Alignments:
To determine deviations in the ordering between the expected ordering in the study plan and the actual ordering of exams and courses for a student, a partial order alignment [3] is computed. Such an alignment is a sequence of synchronous moves between both the trace and the model, log moves, and model moves. Here, you choose the best fitting alignment in case you have several study plans that may be equally possible for a student.
3. Aggregate Information Based on the Alignment and Term Distances Between the Actual and Expected Terms for a Course:
Based on the partial order alignment, we know that each course occurs in the model side either as a model or as a synchronous move. Therefore, there are the following relative positions that a course can have that is happening on the log side:
-
-
- A course occurs synchronously between the process model and the partial order: This means that a student is likely to agree to their exam-taking order with the expected order.
- At an earlier position in the partial order alignment a log move occurs with the course ID and later the mandatory model move: The student is likely to have taken the course earlier than expected.
- At a later position in the partial order alignment a log move occurs with the course ID and earlier the mandatory model move: The student is likely to have taken the course later than expected.
- There is only a model move for a course and no log move: This may be some data quality issue, where a course is missing in the educational event log for the student.
-
Figure 4: Example of the different ordering-based cases for a total order (right) and a partial order (left) and their partial order alignments. Obtaining such different alignments is also the reason why we use partial orders instead of total orders which is beyond the scope of this blogpost.
This derived information from the partial order alignments is then combined with some course-taking distances between the actual and the expected term that the course should be taken in. The distance is calculated in years and for a cohort of students the combination of an order-based relation and the corresponding temporal distance for a course are combined and counted, resulting in the data shown in Table 1 and Table 2.
Table 1: An excerpt from the aggregated result for the investigated students and the 2010 study plan.
Table 2: An excerpt from the aggregated result for the investigated students and the 2018 study plan.
Key Insights from the Research
Using data from RWTH Aachen University, we applied this approach to study plans from 2010 and 2018. Here’s what we found:
- Shifting Courses between expected and actual position: We can detect whether courses are moved backward or forward in the order in which courses should be taken. For example, courses C05 and C12 in Table 1 are moved forward by a smaller fraction of students while most students comply between expected position and expected time. Courses C16 and C18 are courses that occur more often to be taken later in studies and may lead to a longer study duration since they are most often also delayed by at least one year.
- Well-conforming courses: We can check if courses conform well between expectations and actual data. For example, the course C17 in Table 2 is taken from most students in the right order and a lower fraction of students take a course late.
- Adaptations over study plan: While study plans can change over time, this can have an effect on the conformance between students and study plans. Here, we can compare courses C12 and C17 between the aggregated results in Table 1 and Table 2 that belong to the analysis of a 2010 and a 2018 computer science study plan, respectively. While the change improved conformance for course C17, changes to course C12 reduced conformance here.
Possible Things to Come
Our findings highlight the potential for universities to use this methodology to evaluate and refine their study plans systematically. The results derived may also be used to enhance the information within our event logs directly, e.g., by adding a notion for an unconforming activity using what type of non-conformance it is. However, since optimal alignments are not necessarily deterministic, there may be improvements to make towards the reproducibility of each run of the presented framework and its interpretability. Further, we could also analyze the framework’s capability for educational event logs of other degree programs, and we can imagine that the framework can also be used to gain deeper insights into a cohort of students or for other metrics as well. We can also imagine the approach to be applicable to other types of event data that contain relative timings and corresponding process models.
Further Reading
This post is based on the research paper [3] that was accepted for publication and was presented at the EduPM – ICPM 2024 Workshop. Please find the preprint in the references section.
References:
[1] The average study duration of first-degree university graduates in Germany from 2003 to 2023, https://www.statista.com/statistics/584454/bachelor-and-master-degrees-number-universities-germany/, 2024, last access 2025-01-21
[2] Number of Bachelor’s and Master’s degrees in universities in Germany from 2000 to 2023, https://www.statista.com/statistics/584454/bachelor-and-master-degrees-number-universities-germany/, 2023, last access 2025-01-21
[3] Rennert, Christian, Mahsa Pourbafrani, and Wil van der Aalst. “Evaluation of Study Plans using Partial Orders.” arXiv preprint arXiv:2410.03314 (2024).