A new study demonstrates how to retrieve the underlying concepts behind a question. They use a LLM to determine the knowledge components that the question evaluates.

When a student fails a math problem, the teacher intuitively knows that several concepts are at play. But a digital system cannot tell whether the student stumbled on the Pythagorean theorem, on computing a square root, or simply on an addition. Bridging this gap between the richness of pedagogical insight and the poverty of machine-exploitable information is precisely what a team of researchers from ETH Zürich and LMU Munich set out to do.

For an adaptive learning tool to work, it first needs to understand what each question is about. Until now, this annotation work relied entirely on a team of pedagogical experts — a long and costly process. What the authors show is that a chain of three well-crafted prompts is enough for a LLM to decompose a question, extract its key concepts, and organize them in an exploitable way. The authors report that automatically generated annotations are judged of better quality than human annotations in the majority of cases evaluated by experts.

These concepts will serve as a basis to improve the performance of many Knowledge Tracing models.

Methodology

KCQRL is broken down into three modules:

• Module 1 annotates the question from its content with a step-by-step solution and concepts corresponding to each step.
• Module 2 learns question representations by leveraging solution steps and key knowledge components (K.C.) through an adapted learning objective.
• Module 3 integrates the learned question representations into existing Knowledge Tracing models.

Annotating questions

This is the core research in Knowledge Tracing: finding Knowledge Components from a question. This allows a better understanding of the question, and therefore a better localization of a learner’s weaknesses.

The article’s authors use a Large Language Model with the chain of thought technique, which consists of chaining prompts.

They propose several steps:

Generating the answer steps

To understand the complexity of a question, one must know the steps required to answer it. For example, the question A right triangle has one side of 3 cm and another of 4 cm. What is the length of the hypotenuse? — if we only look at the question, a teacher would see Pythagoras, etc. But a tool that only sees the wording might miss important concepts. This is why the authors also generated the answer steps, for example:

• Step 1: Identify that the triangle is right-angled, so the Pythagorean theorem applies
• Step 2: Recall the Pythagorean theorem
• Step 3: Identify that 3 and 4 are the sides of the right angle (not the hypotenuse)
• Step 4: Compute the squares
• Step 5: Perform the addition
• Step 6: Compute a square root

The prompt is as follows:

Your task is to generate clear and concise step by step solutions of the provided Math problem. Please consider the below instructions in your generation.

- You will also be provided with the final answer. When generating the step by step solution, you can leverage those information pieces, but you can also use your own judgment.
- It is important that your generated step by step solution should be understandable as stand-alone, meaning that the student should not need to additionally check final answer or explanation provided.
- Please provide your step-by-step solution as each step in a new line. Don't enumerate the steps. Don't put any bullet points. Separate the solution steps only with one newline \n.

Question: <QUESTION TEXT>
Final Answer: <FINAL ANSWER>
Step by Step Solution:

By decomposing the answer into steps, we can see that behind this question there are concepts around triangles, calculations, etc.

Example:

For this example, ChatGPT responds:

In a right triangle, we use the Pythagorean theorem: c² = a² + b², where c is the hypotenuse.

We substitute the known values: c² = 3² + 4².

We compute the squares: 3² = 9 and 4² = 16.

We add them: c² = 9 + 16 = 25.

We take the square root: c = √25 = 5.

The length of the hypotenuse is therefore 5 cm.

Once the steps have been determined, the authors have the LLM find Knowledge Components for each answer step.

Generating the knowledge components

Now that the LLM has determined the solution steps, it has enough context to provide the underlying knowledge components for this question.

The prompt used is the following:

You will be provided with a Math question, its final answer and its step by step solution. Your task is to provide the concise and comprehensive list of knowledge concepts in the Math curriculum required to correctly answer the questions. Please carefully follow the below instructions:
- Provide multiple knowledge concepts only when it is actually needed.
- Some questions may require a figure, which you won't be provided. As the step-by-step solution is already provided, Use your judgment to infer which knowledge concept(s) might be needed.
- For a small set of solutions, their last step(s) might be missing due to limited token size. Use your judgment based on your input and your ability to infer how the solution would conclude.
- Remember that knowledge concepts should be appropriate for Math curriculum between 1st and 8th grade.
If the annotated step-by-step solution involves more advanced techniques, use your judgment for more simplified alternatives.
Question: <QUESTION TEXT>
Final Answer: <FINAL ANSWER>
Step by Step Solution: <SOLUTION STEPS>

It therefore builds on the result of the previous prompt.

ChatGPT responds:

Recognizing a right triangle Pythagorean theorem (relationship between the sides of a right triangle) Computing the square of a number Addition of whole numbers Understanding and computing a simple square root

At this point, the underlying knowledge components of the question are extracted and we can now manipulate them in various tools. This is precisely what the authors propose, using this data to improve knowledge tracing models.

We then classify the underlying concepts against the answer steps using another prompt that builds on the previous results.

Learning vector representations

The concepts generated in the previous step are textual and cannot therefore be directly processed by algorithms. A vector representation (embeddings) is needed to perform semantic matching and determine questions with similar concepts. Current semantic matching techniques, such as BERT models, are too general and cannot determine similar concepts if they are semantically distant. For example, “square root” and “Pythagoras” — a standard BERT model would see them as distant.

This is why the authors trained a model by contrastive learning. This is a method belonging to self-supervised learning. This learning needs pairs to work, and this is precisely what was built in the previous steps. The concepts associated with the answer steps form pairs that the authors use to train a model capable of bringing together pedagogical concepts. These embeddings allow Knowledge Tracing models to better understand the links between questions. For example, if a student succeeds all questions close to “Pythagoras”, the model can predict they will likely succeed a new question on that topic.

The authors were able to test with more than 15 Knowledge Tracing models. Results show that the performance of the tested models increased considerably.

Conclusion

This study addresses two distinct needs. The first is to determine concepts from a question. The authors are not alone in pursuing this research direction, but they have the merit of doing it with elegance and simplicity — just a chain of three prompts and the LLM determines all the underlying key concepts.

The benefit for a mathematics teacher would be to classify weak concepts. Where learners succeed the least on questions, since we can now have concepts linked to these questions, the system can automatically determine the concepts that posed the most problems. It is not an error detection system — we cannot know if the student made an error on square roots or on applying the Pythagorean theorem. However, we end up with a space of plausible concepts that can be worked on further by proposing exercises or lessons.

The second contribution of this study is to improve the performance of existing Knowledge Tracing models, which allows for better prediction of a learner’s success rate on a given question.

As always, a few limitations are worth noting. The first is that the model was primarily designed for mathematics, even though it can be transferred to domains such as physics, provided the answer to a question can be broken down into several steps. The second concerns the dependency on the LLM used — results may vary between Claude, ChatGPT, a SLM or a local model.

Sources

• The article: KCQRL: Knowledge Component Question Representation Learning for Knowledge Tracing
• Code repository: KCQRL