The case for psychometric artificial general intelligence

The roots of artificial intelligence (AI) can be found in the Dartmouth Summer Research Project of 1956 [1], organised with the hope of making a significant advance in programming intelligence into a computer over the course of 2 months. While the problem the participants set out to solve has been found to be vastly more complex than they imagined, in the decades since that summer project the field has made huge advances. AIs can now perform at expert human level, or beyond, in several tasks. Some of the most well-known include the game of Go [2], modern computer games such as StarCraft [3], image classification [4], language translation [5] and medical diagnosis [6]. We may well be on the cusp of an era in which AIs will prove to be better at more tasks than humans. But, if one of these superhuman AIs were to be placed in an environment only slightly different to the one it usually operates in, the strong likelihood is that it would cease to perform its task effectively, if at all [7]. Despite the great deal of skill current AIs have attained, they cannot match the human, or even animal, ability to adapt to almost any environment they find themselves in. Humans can often see a new task performed only once or twice and proceed to be able to repeat the task. In fact, often enough they can perform a task, on the first attempt, without even seeing it performed first (e.g. someone encountering a new device does not always need the instructions or a demonstration to use it correctly). This is the case for an inconceivable number of tasks. No AI we have today displays such adaptability. There are several classes in which AI is usually placed. The most used terms, introduced by Searle [8], are ‘strong’ AI and ‘weak’ AI. The distinction between strong and weak is mainly a philosophical one, concerned with whether an AI is capable of understanding in the same way a human can, or merely act like it does, respectively. The terms ‘narrow’ AI and ‘general’ AI are also used. These classifications refer to the scope of ability of an AI. A ‘narrow’ AI is only capable of performing tasks within a small number of domains (usually only a single domain). This is the level of AI we can currently create. AlphaGo, for example, has only been shown so far to perform extremely well in the domain of board games. ‘General’ AI is an AI that can perform tasks across a huge number of domains, even domains which were unknown at the time of its creation – it displays general intelligence. The closest examples we have to general intelligence are not artificial agents, but ourselves – humans demonstrate general intelligence. An AI demonstrating the same is termed an artificial general intelligence (AGI). In order to compare the level of general intelligence between agents, we must use some sort of benchmark. To paraphrase the eminent physicist Lord Kelvin [9] \sayWhen you can measure a thing, you know something about it; when you cannot measure it, your knowledge is of an unsatisfactory kind.

Several surveys of proposed definitions of AGI, along with tests, benchmarks and frameworks for detecting and measuring it already exist [10, 11, 12]. Here we cover some newer and some older but not widely surveyed ideas.

Mikhaylovskiy [13] propose 6 tests. Their Explanation Test requires an AI to provide an explanation and quantitative characteristics of an empirical phenomenon, given a well-defined scientific theory concerned with said phenomenon. Next, their Problem-Setting task requires an AI to be able to produce a problem which could be given in the first task. The Refutation test asks for an AI to be able to devise an experimental methodology that could determine the best model of some phenomena from a set of competing models. The New Phenomenon Prediction test tasks an AI with producing a new prediction, given some well-defined scientific theory. Their Business Creation test looks for an AI being capable of creating a successful business start-up. Finally, to pass their Theory Creation test, an AI would be required to produce a new and better scientific theory in some research field.

Chollet [14] proposes a benchmark dataset, influenced largely by the field of psychometrics, named the Abstraction and Reasoning Corpus (ARC). Chollet argues that any approach to AGI should be human-centric. Accordingly, the ARC benchmark was created with the over-arching goal of providing a method of comparing any two agents, human or artificial, in a fair way that quantifies what he calls their ‘generalisation power’. By fair, ARC stipulates the prior knowledge that is assumed of an agent. The prior knowledge is a set of core knowledge priors, innate to humans (or acquired very early in life) [15].

Potapov et al [16] propose as a benchmark the Primary School Olympiad Math Tasks (PSOMT), used in America as an educational assessment for young children.

For Bringsjord et al. [17] the crux of intelligence is assumed to be in the ability to be creative. This was the view espoused by Lady Lovelace, and the proposed test is named after her. To pass the Lovelace Test, an artificial agent must be able to consistently ‘surprise’ it’s creators, in the sense that the artifacts the agent produces are not explainable.

A test posited by Bringsjord and Licato [18] consists of a hypothetical room (termed the Piaget-MacGyver Room (PMR), after some of the thinkers that influenced its conception) containing a fixed set of ingredients. An agent, assumed to know what the set of ingredients is before being tested, is said to exhibit general intelligence if, and only if, it can pass any test constructed of the ingredients.

Several ‘practical’ tests are outlined by Goertzel et al [19]. The common theme here is that each test involves completion of some common human task, such as making a coffee or understanding a story. These tasks presumably require intelligence for a human to perform successfully, so as per Minsky’s definition [20] of AI successfully completing such a task would indicate intelligence, but not necessarily general intelligence.

Nilsson’s Employment Test [21] proposes the measuring progress towards human-level machine intelligence (HLMI), rather than general intelligence (hence side-stepping the issue of just what general intelligence is). The metric here is the percentage of all jobs in which people may be employed which can be successfully performed by artificial agents.

There are many factors that are important to consider when evaluating how well some test, framework or benchmark can drive progress in AGI. We will use factors previously considered by Legg and Hutter [11].

• •

  Validity

• •

  Informativeness

• •

  Range

• •

  Generality

• •

  Dynamism

• •

  Bias

• •

  Fundamentalism

• •

  Formality

• •

  Objectiveness

• •

  Completeness of definition

• •

  Universality

• •

  Practicality

In broad terms, the proposals in II can be split into two classes – tests and benchmarks.

The evaluation in the previous section has shown how a psychometric approach to AGI measurement surpasses task-specific approaches in every consideration. While there is no claim that it is the perfect framework to use, it can provide a guiding light towards faster progress in the development of ever more general artificial agents. Attempting to use a task-specific approach is akin to skipping learning how to walk, in the hope that learning to run will teach the simpler skill.

PAGI inspired benchmarks could be incorporated into an online repository. With some standard interface, such a repository could allow researchers and developers to, in real-time or close to real-time, obtain results for the systems they are engaged with across a range of benchmarks. We expect the existence of such a repository would result in a noticeable increase in momentum in the field of AGI, with knock-on effects to the, at present more useful (economically speaking), predominant field of narrow AI and its applications.