Reliability of Checking an Answer Given by a Mathematical Expression in Interactive Learning Systems

In this article, we propose a solution for the problem of establishing identities in formal mathematical systems. This problem is known to be formally unsolvable (in fact, this is a corollary of the Gödel’s theorem). And the problem may appear even in computer systems supporting analytic calculations (see an example below). We consider this problem in connection with the development of an interactive educational system where a computer algebra system is used as a computational module and the nonsolvability of the identity problem may result in an inappropriate behavior. A more detailed overview of our interactive educational system is given further in the article. Here we will just point out that the system generates training tasks and performs answer checking, where the human-obtained answer in the form of a mathematical expression is compared with the one obtained by the system. This is where the problem of identity checking appears. In this article, we suggest to reinforce (if necessary) the native algorithm for comparing mathematical expressions by the computer algebra system by an additional pointwise checking procedure. The article provides a detailed investigation of the probability of error which appear in this case.

Roughly speaking, it appears that this probability is much less than any reasonable value. For instance, this value could be of the order of the probability of obtaining a non-unique result when shuffling a deck of $36$ cards (which happens to be about $0.27\cdot 10^{-41}$). More precisely, this is true if we exclude some exotic cases like frequently oscillating functions.

We end up with the following scheme for checking the answer:

• •

  compare two answers using computer algebra system’s standard tools;

• •

  if the result is indefinite, run the pointwise checking procedure in order to improve the result.

The article is divided into two main sections. The first section provides a brief overview of the interactive educational system and the problem of automatic answer checking. In Section 2, we present an algorithm for additional answer checking and provide a detailed analysis of the algorithm error probability.

There are plenty of approaches to the e-Learning systems development: from simple databases of classified theoretical material to complex interactive intelligent systems. Lots of achievements had been reached in the delivery of educational courses; one can say that the key idea of democratic education is now being embodied. In this article, we will consider the problem of training tasks organization. Consider classes on, say, Mathematical Analysis at some first grades in higher school. Students face a number of training tasks to work the appropriate theoretical material out. The main sources for such tasks are schoolbooks. Let us call it the traditional approach to the organization of practical classes. This approach has some major drawbacks:

• •

  Following this approach, one cannot take into account the individual characteristics of each student in a group. This means that the professor provides tasks according to some concept of an average student. But in fact each student needs his own individual amount of tasks to succeed in learning something. While one has to work out say 5 tasks to ensure the complete understanding of the material, another student needs 15 tasks to reach the same result. And this is not up to the professor to develop some individual working plans for each student in a group. This requires too much time and will heavily increase the demands.

• •

  When talking about some kind of home tasks, professors may spend too much time just to check all tasks up. This is a mechanical job but it needs to be done.

• •

  It is nearly impossible to create a unique amount of tasks for each student. No book is large enough to provide that amount of different but at the same time equivalent tasks.

• •

  Student may suffer a lack of information if he encounters an error. The thing is that if you fail you will not get any additional information about that particular problem from the schoolbook.

We claim that the situation can be improved. The main idea here is that the practical part of educational process, and especially training tasks generation, delivery and processing, can be highly automatized. We develop this approach in our Interactive Educational System — a project for educational process support where we aim to improve educational process for exact sciences to make it more effective and less painful.

Our Interactive Educational System is a web application, where we concentrate on solving the problems of task creation and delivery and answer checking providing a high level of automatization. The approach of training task automatization is based on the following three general concepts:

• •

  task templates

• •

  random generation

• •

  automatic answer checking

The concept of task templates prescribes to create some general definitions for some task classes instead of defining each task separately. Such task templates include a number of indefinite parameters, where each parameter takes its values from its own set. These parameters, being defined appropriately, define a concrete task from the task class defined by the template. An important thing to point out is that all concrete tasks that can be retrieved from some task template are equivalent in a sense meant by the author. In other words, it is up to the task author to decide what equivalent tasks should look like. In fact, he makes this decision naturally, behind the scenes, each time he creates a new task template.

To define a task, one needs to specify its problem (the problem formulation), solution (a sequence of steps with some explanations that describes how the task is solved), and answer (which in our case is a mathematical expression). Similarly, to define a task template, one needs to specify its problem, solution, and answer templates. Let us take a look at the problem formulation. It usually includes some few sentences that describe what needs to be done and some special objects. In the case of exact sciences, these objects are usually some mathematical expressions — here and further we consider precisely this case. So problem formulation consists of some natural-language sentences and some mathematical expressions. The situation is similar to the solution, while an answer must be a strict object, i.e., a mathematical expression. In the Interactive Educational System, we use Markdown¹¹1 Markdown is a markup language by John Gruber with plain-text formatting syntax which is easy to read and manage. Official website: http://daringfireball.net/projects/markdown/ language for the text markup and our special language for defining the task dynamic attributes — SmallTask. The SmallTask language was designed specifically to serve the needs of task template definition and provides tools for mathematical expressions declaration, evaluation, and representation.

Given these tools and a special editor, a user can define task templates which the system can further concretize providing additional calculations automatically. Let us take a trivial problem from Mathematical Analysis class. Consider a task on the chain rule. If one wants to teach somebody this rule, he will probably explain it to the student and provide him with some special tasks on the subject. The problem formulation for such task may look as follows:

The solution explanation for this problem may look as follows:

The answer to the problem is $-\sin(x)\cdot\cos(\cos(x))$.

These three declarations (problem, solution, and answer) together define one training task for the chain rule. In fact, to create a new task, we can just replace sine and cosine with some other functions in the problem formulation and perform appropriate changes in solution and answer. One may define the general problem formulation for all these tasks:

Here $f$ and $g$ belong to some function class (i.e., of base elementary functions). Performing appropriate changes in the solution and answer declarations, we end up with a generalization of training tasks on the chain rule which, in fact, is the task template introduced earlier. In the Interactive Educational System, we allow the users to define such task templates and also to specify the way the undefined attributes (such as $f$ and $g$ in the example above) should be defined. For instance, $f$ and $g$ in the example above may be some random base elementary functions (such as $sin$, $cos$, $exp$, $log$, etc.). Given such declaration, the system will automatically replace all occurrences of such attributes by appropriate values retrieved automatically according to their randomization declaration. That is what the random generation concept is about. This is done each time someone calls a task: the system creates the concrete task problem, solution, and answer according to the template and then the user gets a concrete problem formulation while the appropriate solution and answer are stored on the server for further needs. Given the problem formulation, a student tries to solve the problem and after that he is to put his answer into a special field on the page and to confirm it. Then this answer is sent to the server where it is checked against the answer calculated by the system in order to decide whether a student succeeded or not. The appropriate correct solution is sent back to the student in the latter case.

The most challenging thing here is the automatic answer checking. At the same time, this is the key idea of the project. Together with the templating system and randomization, it forms a mechanism which simplifies the process of training tasks creation, delivery, and checking, which is a heavy and essential part of practical classes.

The problem of automatic answer checking is rigidly coupled with the problem of determining whether two expressions are semantically equal. We call two mathematical expressions semantically equal if they both represent the same mathematical object. The concept of semantic equality is opposite of the concept of syntactic equality, which means that two expressions have exactly the same representation. For example, $2^{x}$ and $e^{x\log 2}$ are semantically equal, since they both represent the same object, but they are syntactically different by means of the given representation in mathematical notation. It is obvious that the problem of testing the semantic equality of two expressions is equivalent to the problem of checking whether a particular expression equals zero (since $f_{1}\equiv f_{2}$ if and only if $g=f_{1}-f_{2}\equiv 0$). The latter is known as the Constant Problem ²²2 It is also referred to as the Identity Problem . It was shown (see Richardson (1969), also Laczkovich (2003)) that this problem is undecidable for a class of functions that contains $log2$, $\pi$, $e^{x}$, $\sin(x)$ and $|x|$. This result is known as Richardson’s theorem.

The undecidability of the Constant Problem raises a huge problem in the learning system development. Automatic answer checking is one of the general features of the system but there is still no way to guarantee the correctness of the answer comparison using computer algebra systems’ native algorithms. Although the problem may seem quite seldom or even exotic, it needs to be examined, since its side effects can appear unexpectedly. In our investigation, we have discovered that one of the tested computer algebra systems (CAS) was unable to determine whether $2^{x}$ equals $e^{x\cdot\log(2)}$ or not. That exact problem was fixed in further versions of that CAS, but the general problem still remains open. In order to cope with the issue, we have to impose some limitations on the answers in the system and to provide an algorithm for additional answer comparison, which should increase the probability of correct answer checking.

For most of CAS, the undecidability of the constant problem results in the situation where, in some cases, a system is unable to make a decision whether the two given expressions are equal or not, thus returning nothing instead of True or False. Such result must be treated as the “I don’t know”-answer, which means that we are unable to say for sure whether the user’s answer is correct or not. But still we need to make a decision. There are two trivial ways to do this: always consider such an ambiguous answer as correct or, on the contrary, always consider such an answer as incorrect. The second alternative may lead to a critical misunderstanding and it also violates the student’s rights in a way, namely, one can say the student is unfairly punished. It may force bad ratings among the students, and this is definitely unacceptable. From this point of view, the first alternative seems a bit less harmful, namely, no one is directly hurt because of the answer checking error — most of the time it is not detected at all. However, this is definitely not the best way to solve the problem. A better way is to provide an algorithm that performs additional tests on answers if the CAS is unable to give an acceptable result.

We suggest the following algorithm for additional answer comparison, which runs if it is impossible to determine the answer correctness safely, i.e., the CAS answer is uncertain. Consider task $T$ with an answer that is a mathematical function of a single variable. Let $f_{real}(x)$ be the real answer to $T$ problem obtained by the system, and let $f_{user}(x)$ be a student-calculated answer. The algorithm accepts $f(x)=f_{real}(x)-f_{user}(x)$ and tries to determine whether the equality $f(x)\equiv 0$ is correct. Here is its pseudocode:

The notion of the algorithm is quite simple: it repeatedly picks up a random number $d$ from $\mathcal{U}(A,B)$ until either the maximum number of iterations $m$ is reached or $f(d)\neq 0$, i.e., a non-zero point is found³³3 $\mathcal{U}(A,B)$ is a uniform distribution on $[A,B]$. . The $True$ value is returned in the first case (which corresponds to the positive result of comparison, i.e. $f_{real}=f_{user}$). In the second case, the algorithm returns $False$ which means that $f=f_{real}-f_{user}\not\equiv 0$. There is a special limitation on the selected points: we require them to be different, since there is no need to test the function value twice. For that purpose, a set of used points (called $used$ in the pseudocode above) is stored; on each iteration, we demand that the new point does not belong to this set, and then we populate the set with this new point.

Although the algorithm suggested above is rather simple, it has several advantages. The first thing to mention is that it is “student-friendly”: the only case the algorithm fails is when it returns $True$, while, in fact, the student’s answer is incorrect and, as this will be shown below, this situation is quite seldom. Another thing to notice is that we limit execution time for $f$. The function value computation can be really slow in some definite cases, while we do not really care about each and every concrete value — the only thing we need is a number of points to test the function value at. So if the computations run out of some predefined time limit, we just pick another point instead of that “complicated” point.

The result of Algorithm 0.1 is not exact. In fact, it can fail if the $f_{real}$ and $f_{user}$ functions are too close to each other and the maximum number of points to check is too low. In order to estimate its error probability, the following proof is suggested.