On Philomatics and Psychomatics
for Combining Philosophy and Psychology with Mathematics

Attention mechanism was primarily noticed in cognitive science at the end of 20-th century [59]; however, it was mainly proposed for natural language processing recently. Before the proposal of the attention mechanism in 2017 [60], recurrent neural networks [48] and long short-term memory networks [31] were mostly used for natural language processing. However, attention mechanism changed the research path by using the concept of attention which is a mathematical model for feature weighting. Assume you are looking at the portrait of Mona Lisa. You do not attend to all regions if the picture; rather, you focus on her face more than the river in her background. Likewise, in skimming as a technique for fast reading, you do not attend to all words equally but you focus on important words and ignore the less informative words. This more and less attention to different features of picture or text can be modeled mathematically as weighting [21].

In the attention mechanism [60], the different words of a sentence or a text corpus attend to each other with different weights, where the weights are obtained by the amount of similarity of every word with another word. As a result, the more similar two words are, the more attention they have to each other. Two language models, named BERT (Bidirectional Encoder Representations from Transformers) [11] and GPT (Generative Pre-trained Transformer) [44] have been proposed based on the attention mechanism. In contrast to the previous language models, such as word2vec [37, 38] and GloVe [39], BERT and GPT are based context-aware language models which provide different embedding vectors for a word in different contexts. For example, the word “bank" has different meanings and therefore different embeddings in the sentences “Money is in the bank" and “Some plants grow in bank of rivers".

The explained attention mechanism is closely related to Gottlob Frege’s “context principle" [16, 17] and Willard Quine’s “semantic holism" [42] in analytical philosophy and the philosophy of language. Note that the attention mechanism was proposed for natural language processing in 2017 while the context principle was proposed more than a century ago in 1884–1892. In 1884, Frege stated that “never […] ask for the meaning of a word in isolation, but only in the context of a proposition" [16]. Moreover, in 1892, Frege stated that a word acquires its meaning only within the context of an entire sentence [17].

The context principle or the contextualism [41] states that the meaning of a word is understandable only as a part of a whole, i.e., a sentence or context. This was developed further into the “usage theory of meaning" in the philosophy of later Wittgenstein. In his “Philosophical Investigations", Wittgenstein stated “comprehending a proposition means comprehending a language" [65]. This “usage theory of meaning" replaced the “picture theory of meaning" in his early philosophy which claimed that the meaning of words in a language exist in an abstract world of logic (cf. Wittgenstein’s Tractatus Logico-Philosophicus [64]). After the later Wittgenstein, this theory was further developed and generalized into Quine’s “semantic holism" stating “the unit of measure of empirical meaning is all of science in its globality" [42]. In Quine’s perspective, the meaning of a claim is perceivable only if it is considered in a sequence of propositions and contexts. In this perspective, human’s mind is like a court which considers all the evidences for understanding the meaning of something [9].

Comparing the above explanations in natural language processing and the philosophy of language shows that they have close relation. Attention mechanism, as a weighting scheme in mathematics, is using the ideas of context principle, usage theory of meaning, and semantic holism in the philosophy of language. All of these theories are defining the meaning of every word in the natural language based on the context it appears in.

In metaphysics, "particulars" are the existing objects in the world and "universals" are the properties, such as colors, which are shared between many particulars [34]. There are two approaches regarding the ontology (existence) of universals in the world. Realism states that the universals exist in the world while nominalism claims that there are no existing universals and they are nothing but resembled characteristics between the particulars [1].

One of the first approaches in realism was proposed by Plato’s theory of forms, also called the theory of ideas. According to this theory, everything in this physical world has some true essence, or idea (also called form), in a world of ideas where the things merely imitate the true ideas. In this theory, the physical world is not as true as the timeless, absolute, unchangeable ideas [50]. In his Republics [40], Plato has illustrated his theory by an example where several people are in a cave facing the wall of the cave. The true ideas are outside of the cave in the real world but their shadows appear on the wall. The people in the cave suppose that the shadows are the real things; however, they are just shadows of the real things existing outside of the cave. Some researchers believe that the theory of ideas has had significant impact on the monotheistic religions [14].

The theory of forms is also one of the origins of idealism in metaphysics stating that the reality is entirely a mental construct and only the ideas are real or they are the highest level of reality [26]. Many philosophers developed other variants of idealism; e.g., Berkeley, Kant, and Hegel developed empirical [4], transcendental [33], and absolute [30] idealism, respectively [13].

The theory of forms in philosophy is completely related to the holographic principle in modern physics and string theory. Black holes are the regions in the space-time of Einstein’s field equations where the gravity, i.e., the curvature of space-time, is too huge that nothing including light cannot escape from. This causes an information paradox because the quantum information of the things falling in black hole should not be lost. One possible solution to this paradox is the evaporation of black holes through the Hawking’s radiation [29]. Another possible way to resolve this paradox is the holographic principle [52] which states that the information of falling things into the black hole is encoded as a lower-dimensional copy on the boundary of the black hole. In this way, the information is not lost but its low-dimensional copy is restored. It is noteworthy that this low-dimensional encoding is also related to dimensionality reduction in mathematical machine learning [20]. Physicists have even hypothesized that the current world might just be the encoded hologram of another high-dimensional world [54]; this makes sense because the world has been assumed to be of higher dimensions than four (three dimensions for space and one dimension for time) in the string theory or the M theory [46].

The main idea behind the holographic principle is very similar to the theory of forms by Plato. In the holographic principle, the world is supposed to the be the hologram of a high dimensional world. In Plato’s theory of forms, also, the world is a projection or shadow of a true world of ideas (forms).

There is also a relation between Plato’s theory of forms and object oriented programming [2] in computer programming. In object oriented programming, which is a modular programming suitable for team-working, the programmer defines some classes as modules of code which can perform some tasks. The computer program can make objects or instances from these classes. Every class has several attributes which are initially set for every individual object when the object is created. Moreover, every class has some functions for performing some tasks. These functions are also called methods. One of the methods, named constructor, is for setting the initial values of attributes. The classes can inherit from each other in the way that the inheritor inherits some methods (functions) and attributes from the other class.

As it is obvious, there is a close relation of object oriented programming and the theory of forms. The classes are like the forms or ideas in the world of forms. The objects or instances of classes are like the instances of forms in this phenomenal world. Every class or the form has some attributes and the instances of the class or the form has those attributes consequently. The classes or the forms can also inherit from each other.

An example is the form/class of human. Benyamin and Morteza are two instances/objects of human. The form/class of human has some attributes such as height, weight, color, language, mood, etc. Benyamin and Morteza each have some states/values of these attributes. These attributes are set initially at the time of birth or by the constructor method, when the instance/object is created. However, these attributes might be reset again later to other values. The form/class of human has some functions or methods such as walking, sitting, talking, thinking, etc. At every moment, Benyamin and Morteza perform one or several of these functions. The form/class of human inherits from the form/class of mammal, which itself inherts from the form/class of animal. Some functions (methods) and attributes of the form/class of human are inherited from the form/class of mammal. However, some functions and attributes are specific to the form/class of human. For example, the way of intercourse and giving birth is inherited from mammals but talking in complicated languages and thinking to its own death can be enumerated as the functions specific to human.

The theory of forms is also related to classification and clustering in machine learning. Consider a set of data points, called a dataset. They can be grouped into multiple clusters or classes of data, depending on whether they have labels or not. In both cases of clustering and classification, every cluster or class can be considered as a group. Every group can have a representative. This representative is usually a summary statistics, such as mean, of the group. If the summary statistics is not median, the representative of a group may not be any of the data points in that group. As a result, this representative can be considered as the true form of the group with instances existing in the group.

Another relation of the theory of forms with machine learning can be considered in generative models. In many of the generative models, a latent noise is fed to the model and the model generates some semantic data instance corresponding to that noise [28]. The most fundamental generative model in machine learning is factor analysis [18, 22] and one of the state-of-the-art generative models is generative adversarial networks [27, 23]. In generative models, every latent noise corresponds to a generated data point. As a result, the latent noise can be considered as the true form of the generated instance in the world of forms. It is noteworthy that vector arithmetics can be performed between the latent noise vectors [43]. For example, if the corresponding noise to a facial image without eyeglasses is subtracted from the corresponding noise to a facial image with eyeglasses, the obtained noise vector corresponds to the image of eyeglasses! The corresponding noise to the eyeglasses can be considered as the form of eyeglasses in the world of ideas.