Learning platforms without historical data: a decision model predicts pupil success better than 28 trained models

Without any data from the target platform, Jev achieves an average AUC of 0.706 in knowledge tracing, outperforming the best deep learning models trained on eight learners, at one hundredth the cost of a reasoning LLM.

Knowledge tracing consists of estimating the probability that a learner will answer the next question correctly from his or her history. Traditional models need many students already registered, so a new course starts without a reliable tool. A study published on arXiv on October 8 tests another way: interrogating a "system 1" model, which directly returns a probability in a single pass, rather than an LLM that generates a response and reasons on several samples.

Cold start advantage

Out of seven datasets, Jev without any data from the platform obtains an average AUC of 0.706, compared to 0.689 for the best of 28 deep models trained on 8 learners and 0.650 for the Thinking-KT approach based on reasoning, which is about a hundred times more expensive in APIs. By adding a few examples and a statistic of similar learners (JevKT), the score rises to 0.722; the advantage remains clear up to 16 learners, remains on average up to 64, and supervised models regain control between 64 and 128.

The gain seems to be specific to Jev: three other LLMs questioned with the same query do worse on the seven games, and the contamination checks do not find any trace of memorized data. For new students, interest plays a role from the first interactions; on the other hand, on new exercises when all the students are known, the deep models keep the advantage.