Language Models: A Guide for the Perplexed

The first step in building a machine is deciding what we want the machine to do. People who build power plants, transportation devices, or cooking appliances work from a specification that spells out the inputs and outputs of the desired system in great detail. It’s not enough to say that “the power plant must provide electricity to all the homes in its town.” Engineers require a precise statement of how many kiloWatt-hours are to be produced, the budget for building the plant, environmental impacts expected, all the laws regulating the construction of plants that are in effect to guarantee safety, and much more.

To take an example that’s much simpler and more relevant to building an NLP system, consider a computer program (which is a “machine” in a very abstract sense; we’ll also call it a “system”) that sorts a list of names alphabetically. This task sounds simple, and computer science students would likely start thinking about different procedures for sorting lists. There are, however, some details that need to be addressed before we start writing code, such as:

• •

  How will the names be input to the program, and what should the program do with the output? (E.g., will the program run locally on a user’s laptop? Or is there a web interface users will use to type in the input and then see the output in their browser tab? Or will they upload/download files? If so, what is the format for those files?)

• •

  What set of characters will appear in the input, and what rules are we using to order them? (E.g., how do we handle the apostrophe in a name like “O’Donnell”? How should diacritic (accented) characters be handled? What happens if some names are in Latin script and others in Arabic script?)

• •

  Are there constraints on how much memory the program can use, or on how quickly it needs to execute? If the input list is so long that the program will violate those constraints, should the user get a failure message?

These may seem like tedious questions, but the more thoroughly we anticipate the eventual use of the system we’re building, the better we can ensure it will behave as desired across all possible cases.

For the two task examples discussed above (translation and sentiment analysis tasks), we noted that demonstrations (inputs with outputs) would be relatively easy to find for some instances of the tasks. However, data might not always be so easy to come by. The availability of data is a significant issue for two reasons:

• •

  For most NLP applications, and most tasks that aim to approximate those applications, there is no “easy” source of data. (Sentiment analysis for movie reviews is so widely studied, we believe, because the data is unusually easy to find, not because there is especially high demand for automatic number-of-stars prediction.)

• •

  The best known techniques for building systems require access to substantial amounts of extra data to build the system, not just to evaluate the quality of its output.

For almost a decade, and with a small number of exceptions, the dominant approach to building an NLP system for a particular task has been based on machine learning. Machine learning (ML) refers to a body of theoretical and practical knowledge about data-driven methods for solving problems that are prohibitively costly for humans to solve. These methods change over time as new discoveries are made, as different performance requirements are emphasized, and as new hardware becomes available. A huge amount of tutorial content is already available about machine learning methods, with new contributions following fast on the heels of every new research advance. Here, we introduce a few key ideas needed to navigate the current landscape.

The first concept is a parameter. A parameter is like a single knob attached to a system: Turning the knob affects the behavior of the system, including how well it performs on the desired task. To make this concrete, let’s consider an extremely simple system for filtering spam emails. Due to budgetary constraints, this system will have only one parameter. The system works as follows: it scans an incoming email and increments a counter every time it encounters an “off-color” word (e.g., an instance of one of the seven words the comedian George Carlin claimed he wasn’t allowed to say on television). If the count is too high, the email is sent to the spam filter; otherwise, it goes to the inbox. How high is too high? We need a threshold, and we need to set it appropriately. Too high, and nothing will get filtered; too low, and too many messages may go to spam. The threshold is an example of a parameter.

This example neatly divides system-building problem into two separate parts:

1. 1.

   Deciding what parameters the system will have and how they will work. In our spam example, the system and the role of the off-color word threshold parameter are easy to explain. The term architecture (or model architecture, to avoid confusion with hardware architecture) typically refers to the decision about what parameters a model will have. For example, picture a generic-looking black box with lots of knobs on it; the box has a slot on one side for inputs and a slot on the other side for outputs. The “architecture” of that model refers to the number of knobs, how they’re arranged on the box, and how their settings affect what occurs inside the box when it turns an input into an output.

2. 2.

   Setting parameter values. This corresponds to determining what value each individual knob on the box is turned to. While we likely have an intuition about how to set the parameter in the spam example, the value that works the best is probably best determined via experimentation.

We now walk through how ML works in more detail and introduce some components you’ll likely hear about if you follow NLP developments.

The language modeling task is remarkably simple in its definition, in the data it requires, and in its evaluation. Essentially, its goal is to predict the next word in a sequence (the output) given the sequence of preceding words (the input, often called the “context” or “preceding context”). For example, if we ask you to come up with an idea of which word might come next in a sentence in progress—say, “This document is about Natural Language ____”—you’re mentally performing the language modeling task. The real-world application that should come to mind is some variation on an auto-complete function, which at this writing is available in many text messaging, email, and word processing applications.

Language modeling was for several decades a core component in systems for speech recognition and text translation. Recently, it has been deployed for broad-purpose conversational chat, as in the various GPT products from OpenAI, where a sequence of “next” words is predicted as a sequential response to a natural language prompt from a user.

What would make it possible to achieve high accuracy at predicting the next word across many contexts? At a fundamental level, natural language is predictable because it is highly structured. People unconsciously follow many rules when they use language (e.g., English speakers mostly utter verbs that agree with their subjects sometime after those subjects, and they place adjectives before the nouns whose meaning they modify). Also, much of our communication is about predictable, everyday things (consider how frequently you engage in small talk).

As an NLP task, language modeling easily checks the two critical boxes we discussed in section 2: data and evaluation. LMs need only text; every word in a large collection of text naturally comes with the preceding context of words. When we say “only text,” we mean specifically that we don’t need any kind of label to go with pieces of text (like the star ratings used in sentiment analysis tasks, or the human-written translations used in translation tasks). The text itself is comprised of inputs and outputs. Because people produce text and share it in publicly visible forums all the time, the amount of text available (at least in principle, ignoring matters of usage rights) is extremely large. The problem of fresh, previously unseen test data is also neatly solved because new text is created every day, reflecting new events and conversations in the world that are reliably different from those that came before. There is also a relatively non-controversial evaluation of LMs that requires no human expertise or labor, a more technical topic that we return to in section 3.4.

We have thus far established what the language modeling task is. However, we haven’t explained why this task is worth working on. Why do we bother building a model that can predict the next word given the words that have come before? If you already make use of auto-complete systems, you have an initial answer to this question. But there are more reasons.

For many years, NLP researchers and practitioners believed that a good language model was useful only for estimating fluency. To illustrate this, imagine a language model faced with guessing possible continuations for a partial sentence like “The dog ate the ____” or “Later that afternoon, I went to a ____.” As English speakers, we share a pretty strong sense that the following word is likely to be either a noun or part of a descriptor preceding a noun. Likewise, if we have a good language model for this type of English, that model will have needed to implicitly learn those kinds of fluency-related rules to perform the language modeling task well. This is why LMs have historically been incorporated as a component in larger NLP systems, such as machine translation systems; by taking their predictions (at least partially) into account, the larger system is more likely to produce more fluent output.

In more recent years, our understanding of the value of LMs has evolved substantially. In addition to promoting fluency, a sufficiently powerful language model can implicitly learn a variety of world knowledge. Consider continuations to the following partial sentences: “The Declaration of Independence was signed by the Second Continental Congress in the year ____,” or “When the boy received a birthday gift from his friends, he felt ____.” While there are any number of fluent continuations to those sentences—say, “1501” or “that the American Civil War broke out” for the first, or “angry” or “like going to sleep” for the second—you likely thought of “1776” as the continuation for the first sentence and a word like “happy” or “excited” for the second. Why? It is likely because you were engaging your knowledge of facts about history as well as your common sense about how human beings react in certain situations. This implies that to produce those continuations, an LM would need at least a rudimentary version of this information.

NLP researchers got an early glimpse of this argument in Peters et al. (2018). This paper reported that systems that trained an LM first as an early stage of building systems for varied tasks, ranging from determining the answer to a question based on a given paragraph to determining which earlier entity a particular pronoun was referencing, far outperformed their analogous versions that weren’t informed by an LM (as measured by task-specific definitions of quality). This finding led to widespread researcher acceptance of the power of “pretraining” a model to perform language modeling and then “finetuning” it (using its pretrained parameters as a starting point) to perform a non-language-modeling task of interest, which also generally improved end-task performance.

It shouldn’t be too surprising that LMs can perform well at filling in the blanks or answering questions when the correct answers are in the training data. For a new task, it seems that the more similar its inputs and outputs are to examples in the pretraining data, the better the LM will perform on that task.

There are two important caveats to our earlier claim that collecting data for a language model is “easy.” First, because there is a massive amount of text available on the internet which could be downloaded and used to build or evaluate LMs, at least for research purposes, a language model builder must decide which data to include or exclude. Typical sources of data include news articles, books, Wikipedia, and other web text that is likely to be carefully edited to conform to professional writing conventions. Some LMs include more casual text from social media websites or online forums, or more specialized language from scientific texts. While researchers have generally considered training language models on publicly available text data to be covered by fair use doctrine, the relationship between copyright protections and language model practices is not fully settled; we discuss this further in section 6.2.1.

A major decision is whether to filter texts to only certain languages.⁶⁶6The problem of assigning a language identifier to a text (e.g., is it English, Spanish, etc.?) constitutes another family of NLP tasks. It’s a useful exercise to consider how to select the set of language names to use as labels for language identification, e.g.,which dialects of a language are separate from each other and should receive different labels? Depending on the community of users one intends the LM to serve, it may be preferable to filter text on certain topics (e.g., erotica) or text likely to contain offensive content or misinformation. Today’s LM datasets are too large for a person to read in a single lifetime, so automated tools are employed to curate data. The implications of these decisions are a major topic for current research, and we return to them in section 4.4.1.

The other caveat is a more technical one: what counts as a “word”? For languages with writing systems that use whitespace to separate words, like English, this is not a very interesting question. For writing systems with less whitespace between words (e.g., Chinese characters), segmentation into words could be a matter of choosing an arbitrary convention to follow or of adopting one of many competing linguistic theories. Today, LMs are often built on text from more than one natural language as well as programming language code. The dominant approach to defining where every word in the data starts and ends is to apply a fully automated solution to create a vocabulary (set of words the language model will recognize as such) that is extremely robust (i.e., it will always be able to break a text into words according to its definition of words). The approach (Sennrich, Haddow, and Birch 2016) can be summed up quite simply:

• •

  Any single character is a word in the vocabulary. This means that the LM can handle any entirely new sequence of characters by default by treating it as a sequence of single-character words.

• •

  The most frequently occurring two-word sequence is added to the vocabulary as a new word. This rule is applied repeatedly until a target vocabulary size is reached.

This data-driven approach to building a language modeling vocabulary is effective and ensures that common words in the LM’s training data are added to its vocabulary. Other, rarer words will be represented as a sequence of word pieces in the model’s vocabulary (similarly to how you might sound out an unfamiliar word and break it down into pieces you’ve seen before). However, note that a lot depends on the data through the calculation of what two-word sequence is most frequent in that data at each step. Unsurprisingly, if the dataset used to build the vocabulary includes little or no text from a language (or a sub-language), words in that language will get “chopped up” into longer sequences of short vocabulary words (some a single character), which has been shown to affect how well the LM performs on text in that language.

We mentioned earlier that the language modeling task has a straightforward evaluation method. At first, we might think that a “good” language model has a low word error rate: when it guesses the next word in a sequence, it should seldom predict the wrong word. (A “wrong word” here means anything other than the actual next word in the test data sequence.)⁷⁷7We give a formal mathematical definition of word error rate in the appendix.

LMs have generally not used the error rate to evaluate LM quality for two reasons. First, applications sometimes predict a few options for the next word; perhaps it’s just as good to rank the correct next word second or third as it is to rank it first. The error rate could be modified to count as mistakes only the cases where the correct word is ranked below that cutoff. But how long the list should be is a question for application designers and moves the task definition in a more specialized/concrete direction, perhaps unnecessarily. Second, at least earlier in the history of language modeling, most systems weren’t good enough at predicting the next word to have error rates that weren’t extremely high. If all LMs achieve error rates close to one, the error rate measurement isn’t very helpful for comparing them.

The evaluation method that is typically used for LMs avoids both of these issues. This method is known as perplexity, and can be considered a measure of an LM’s “surprise” as expressed through its outputs in next word prediction. Perplexity manages to work around the problems we’ve described by taking advantage of how LMs decide on a next word in practice.

When an LM produces a next word, that next word is in reality a somewhat processed version of that LM’s actual output. What the LM actually produces given some input text is a probability distribution over its vocabulary for which word comes next. In other words, for every possible next word in its vocabulary, the LM generates a number between 0 and 1 representing its estimate of how likely that word is as the continuation for the input text.⁸⁸8Because this is a probability distribution, all those numbers must add up to 1, and in practice, LMs always set their probabilities to numbers strictly greater than 0.

Rather than evaluating an LM based on however an application developer chooses to process those probability distributions into next words (whether by sampling, or by choosing the word with the highest estimated likelihood, or something else), perplexity instead directly evaluates the probability distributions produced by the LM. Given a test set of text, perplexity examines how high the LM’s probabilities are for the true observed next words overall, averaged over each word in the text-in-progress. The higher that LM’s average probability for the true words is, the lower the LM’s perplexity (corresponding to the LM being less “surprised” by the actual continuations of the text).⁹⁹9For those interested, we walk through the mathematics underlying the definition of perplexity in the appendix.

Like any evaluation method, perplexity depends heavily on the test data. In general, the more similar the training and test data, the lower we should expect the text data perplexity to be. And if we accidentally break the cardinal rule and test on data that was included in the training data, we should expect extremely low perplexity (possibly approaching 1, which is the lowest possible value of perplexity, if the model were powerful enough to memorize long sequences it has seen in training).

Finally, it’s worth considering when perplexity seems “too” low. The idea that there is some limit to this predictability, that there is always some uncertainty about what the next word will be, is an old one (Shannon 1951), motivating much reflection on (1) how much uncertainty there actually is, and (2) what very low perplexity on language modeling implies. Some have even suggested that strong language modeling performance is indicative of artificially intelligent behavior. (We return to this question in section 5.)

Given the tools from section 2 and our presentation of the language modeling task, it’s straightforward to describe how today’s best LMs are built:

1. 1.

   Acquire a substantial amount of diverse training data (text), filtering to what you believe will be high quality for your eventual application. Set aside some data as the test data.

2. 2.

   Build a vocabulary from the training data.

3. 3.

   Train a model with learnable parameters to minimize perplexity on the training data using a variant of stochastic gradient descent.

4. 4.

   Evaluate the perplexity of the resulting language model on the test set. In general, it should be very possible to evaluate the LM on another test set because (1) we can check that the new proposed test data doesn’t overlap with the training data, and (2) the vocabulary is designed to allow any new text to be broken into words.

The third step reveals another attractive property of perplexity: it can serve as a loss function because it is differentiable with respect to the model’s parameters.¹⁰¹⁰10In practice, the loss function is usually the logarithm of perplexity, a quantity known as cross-entropy. Note the difference between training set perplexity (calculated using training data) and test set perplexity calculated in the last step.¹¹¹¹11One common question about language models is why they sometimes “hallucinate” information that isn’t true. The fact that next word prediction is the training objective used for these models helps to explain this. The closest an LM comes to encoding a “fact” is through its parameters’ encoding of which kinds of words tend to follow from a partially written sequence. Sometimes, the context an LM is prompted with is sufficient to surface facts from its training data. (Imagine our example from earlier: “The Declaration of Independence was signed by the Second Continental Congress in the year ____.” If an LM fills in the year “1776” after being given the rest of the sentence as context, that fact has been successfully surfaced.) Other times, however, it’s not, and we just get a fluent-sounding next word prediction that’s not actually true, or a “hallucination.”

The preceding process is how some well-known models, like GPT-2, GPT-3, and LLaMA, were built, and it’s the first step to building more recent models like ChatGPT and GPT-4. These newer models have been further trained on additional kinds of data (which is less “easy” to obtain than the text we use for next word prediction). We return to this topic in section 4.3.4.

This is not a history book, but there is one obvious lesson to be learned from the history of NLP: more training data helps make higher quality models. One period of major changes in the field occurred in the late 1980s and 1990s when three trends converged almost concurrently:

1. 1.

   Increasingly large collections of naturalistic, digital text data became easier to access by growing numbers of researchers thanks to the rise of the internet and the world-wide web.

2. 2.

   Researchers shifted from defining rules for solving NLP tasks to using statistical methods that depend on data. This trend came about in part due to interaction with the speech processing community, which began using data-driven methods even earlier.

3. 3.

   Tasks, as we described them above, became more mature and standardized, allowing more rigorous experimental comparisons among methods for building systems. This trend was driven in part by government investment in advancing NLP technology, which in turn created pressure for quantitative measures of progress.

During the 1990s and 2000s, the speed of progress was higher for tasks where the amount of available training data increased the fastest. Examples include topic classification and translation among English, French, German, and a few other languages. New tasks emerged for which data was easy to get, like sentiment analysis for movies and products sold and reviewed online. Meanwhile, progress on tasks where data was more difficult to obtain (such as long text summarization, natural language interfaces to structured databases, or translation for language pairs with less available data) was slower. In particular, progress on NLP for English tasks was faster than for other languages, especially those with relatively little available data.

The recognition that more data tends to help make better systems generates a lot of enthusiasm, but we feel obliged to offer three cautionary notes. First, easily available data for a task doesn’t make that task inherently worth working on. For example, it’s very easy to collect news stories in English. Because the style of many English-language newspapers puts the most important information in the first paragraph, it’s very easy to extract a decent short summary for each story, and we now have a substantial number of demonstrations for an English-language news summarizer. However, if readers of the news already know that the first paragraph of a news story is usually a summary, why build such a system? We should certainly not expect a system built on news summarization task data to carry over well to tasks that require summarizing scientific papers, books, or laws.

The second cautionary note is that the lack of easy data for a task doesn’t mean the task isn’t worth solving. Consider a relatively isolated community of people who have more recently gained access to the internet. If they do not speak any of the dominant languages on the internet, they may be unable to make much use of that access. The relative absence of this community’s language from the web is one reason that NLP technology will lag behind for them. This inequity is one of the drawbacks of data-driven NLP.

The third cautionary note is that data isn’t the only factor in advancing NLP capabilities. We already mentioned evaluation methods. But there are also algorithms and hardware, both of which have changed radically over the history of NLP. We won’t go into great detail on these technical components here, but we note that the suitability of an algorithm or a hardware choice for an NLP task depends heavily on the quality and quantity of training data. People often use the term “scale” to talk about the challenges and opportunities associated with very large training datasets. As early as 1993, researchers were claiming that “more data is always better data” (Church and Mercer 1993). We would add that which algorithms or computers are better for building a system that performs a task depends highly on the availability of appropriate data for that task, whether high or low or in between. And indeed, as it turns out, the second factor we now mention falls into the category of a change in algorithm: a change in model architecture.

Not long ago, students of NLP would be introduced to a wide range of different architectures. One would likely hear about the relative merits of each and learn what particular kinds of problems it was well suited to solve. From year to year, new ones would be added, sometimes replacing those no longer deemed optimal in any setting. Today, these diverse architectures have virtually all been replaced by a single architecture called the transformer, whimsically named after a brand of 1980s robot toys, proposed by Vaswani et al. (2017).

The transformer, a type of neural network, was introduced by researchers at Google for machine translation tasks. Though we won’t go into detail about how it works, its design was inspired by earlier developments in neural networks, and it was primarily optimized to allow the GPU-based simultaneous processing of all parts of even long input texts instead of word-by-word processing. Earlier architectures were largely abandoned¹²¹²12They were not totally abandoned, however, and are still used occasionally when datasets are small. because they didn’t effectively use GPUs and could not process large datasets as quickly.

It didn’t take long for researchers to realize that with the transformer would allow for training models more quickly and/or on more data, as well as training much larger models than other architectures ever allowed. By “larger models,” we mean models with more parameters. These three elements—larger datasets, faster hardware, and larger models—all depend on each other. For example, a larger model could better encode patterns in the training data, but without faster hardware, training such a model may be infeasible. And if the model is trained on an insufficient sample of data, it may not generalize well.¹³¹³13At its extreme, this phenomenon, known as “overfitting,” leads to models that “memorize” what they see in the training data but perform poorly on new data, e.g., the test data. Conversely, a substantial dataset may require a larger model (more parameters) to encode the larger set of discoverable patterns in the data. Indeed, there is a fundamental tradeoff when selecting architectures: too few parameters, and the architecture will be limited in what input-output mappings it can learn, no matter how much training data is used. Too many parameters (i.e., too large a model), and the model might overfit.

The simultaneous, rapid increase in datasets and parameter counts, aided by improved hardware, affected computer vision before affecting NLP. In fact, the term “deep learning” was originally a reference to these larger models (“deep” refers to models with increasing numbers of “layers” in the architecture, where layers are iterations of repeated calculations with different parameters at each round). The “deepening” of transformers applied to the language modeling task led to what are now called “large language models.” “Large” usually refers to the parameter count, but it could also refer to the size of the training dataset.

The models in wide use for NLP today have billions of parameters; older generations of OpenAI models increased from sizes of over a billion parameters with the largest version of GPT-2 to 175 billion parameters with GPT-3. The main drawback is that running their training algorithms on large datasets requires very many GPUs working in parallel for a long time, which in turn requires a lot of energy. From the perspective of improving the quality of generated text (in perplexity but also subjective human judgments), these LLMs represent a major advance.

From a scientific perspective, it’s difficult to assess which of these changes—data size, number of parameters, architecture, etc.—matter the most. Larger models are more data-hungry; over the last few years, models have gone from training on datasets with millions of words to trillions of words. While some work, such as that by Hoffmann et al. (2022), tried to disentangle the impacts of model scale and data scale, the additional influence of yet other factors (like hyperparameters on a training run) complicates efforts to confidently draw conclusions from such research. These experiments require the repeated training of models that are estimated to cost millions of dollars apiece. In addition, it would take far too long to train fairly matched models based on previously popular, pre-transformer architectures (i.e., with similar parameter counts on similar amounts of data to the strongest models of today); this means that it’s impossible to measure how much benefit the transformer offers other than allowing for larger models.

What was the impact of LLMs? In short, they caused language modeling performance to improve dramatically. To see this qualitatively for yourself, try typing out the beginning of a sentence and instruct a language model like ChatGPT to complete that sentence. Chances are, you will immediately see a sentence that reads much more naturally than you saw generated by a simpler autocomplete system at the beginning of this section. Many people have shared this subjective experience of more fluent text generation, and it is backed up by quantitative evaluations like perplexity. However, if that were their only contribution, LLMs probably wouldn’t have entered the public consciousness.

Model capabilities depend directly on the specific data used to train them. The closer a string of text (say, the instructions provided to an LLM) is to the kind of data that the model was trained on (which, for current models, is a large portion of the data on the internet), the better we expect that model to do in mimicking reasonable continuations of that “kind” of language.¹⁶¹⁶16Note that we are not implying that language models are only mimics; characterizing the precise ways in which they merely copy vs. generalize is work still to be done. Conversely, the further the language of some text is from the model’s training data, the less predictable the model’s continuation of that text will be. (In section 5.1, we discuss the implications for choosing which prompts to supply to a model.)

You can test this out. Try instructing a model (for example, ChatGPT) to generate some text (a public awareness statement, perhaps, or a plan for an advertising campaign) about a very specific item X geared towards a specific subpopulation Y, preferably with an X and Y that haven’t famously been paired together.

Grammatically, the answer returned is probably fine. However, if the content of the model’s response seems generic, that’s not too surprising. The amount of text that models like ChatGPT are trained on that could serve as a close example to a particular prompt is typically far greater than that which is relevant for precise ideas specific to whatever personal combination you thought up.

If you speak a language besides English, you’ll likely also notice a worse answer or a more stilted, generic tone if you translate your question into that language and ask it again. And again, this is directly related to the model’s training data: however much text there is relevant to your issue or product on the internet in English, there’s likely less of it in your other language, meaning there was less available to use for training.

In short, yes. Section 4.4 hinted at this, but to be more explicit: the specific wording of the prompt that you supply to an LM significantly affects the model output that you receive. This likely means that you’ll want to experiment with a few different wordings for instructing the model to do something. When you prompt a model, if your input and the correct output are close to sample text the model has encountered in its training data, the model should “respond” (that is, continue the prompt by predicting a sequence of next words) well. Trying different prompt wording means that you’re casting a wider net across patterns that the model has learned about language and giving yourself a better chance of encountering one that the model has an easier time continuing.

To test this out, try rephrasing something you want an LM to do in a few very different ways. Then, try supplying each of these prompts separately to a model like ChatGPT. Chances are that you see some notable differences in the different results that you get!

At first glance, it might seem that a prompt that produces believable model output means there’s nothing left for you to do. However, you should never take model output at face value. Always check for the following important issues.

The emergence of language model products has fueled many conversations, including some that question whether these models might represent a form of “intelligence.” In particular, some have questioned whether we have already begun to develop “artificial general intelligence” (AGI). This idea implies something much bigger than an ability to do tasks with language. What do these discussions imply for potential users of these models?

We believe that these discussions are largely separate from practical concerns. Until now in this document, we’ve mostly chosen used the term “natural language processing” instead of “artificial intelligence.” In part, we have made this choice to scope discussion around technologies for language specifically. However, as language model products are increasingly used in tandem with models of other kinds of data (e.g., images, programming language code, and more), and given access to external software systems (e.g., web search), it’s becoming clear that language models are being used for more than just producing fluent text. In fact, much of the discussion about these systems tends to refer to them as examples of AI (or to refer to individual systems as “AIs”).

A difficulty with the term “AI” is its lack of a clear definition. Most uncontroversially, it functions as a descriptor of several different communities researching or developing systems that, in an unspecified sense, behave “intelligently.” Exactly what we consider intelligent behavior for a system shifts over time as society becomes familiar with techniques. Early computers did arithmetic calculations faster than humans, but were they “intelligent?” And the applications on “smart” phones (at their best) don’t seem as “intelligent” to people who grew up with those capabilities as they did to their first users.

But there’s a deeper problem with the term, which is the notion of “intelligence” itself. Are the capabilities of humans that we consider “intelligent” relevant to the capabilities of existing or hypothetical “AI” systems? The variation in human abilities and behaviors, often used to explain our notions of human intelligence, may be quite different from the variation we see in machine intelligence. In her 2023 keynote at ACL (one of the main NLP research conferences), the psychologist Alison Gopnik noted that in cognitive science, it’s widely understood that “there’s no such thing as general intelligence, natural or artificial,” but rather many different capabilities that cannot all be maximally attained by a single agent (Gopnik 2023).

In that same keynote, Gopnik also mentioned that, in her framing, “cultural technologies” like language models, writing, or libraries can be impactful for a society, but it’s people’s learned use of them that make them impactful, not inherent “intelligence” of the technology itself. This distinction, we believe, echoes a longstanding debate in yet another computing research community, human-computer interaction. There, the debate is framed around the development of “intelligence augmentation” tools, which humans directly manipulate and deeply understand, still taking complete responsibility for their own actions, vs. agents, to which humans delegate tasks (Shneiderman and Maes 1997).

Notwithstanding debates among scholars, some companies like OpenAI and Anthropic state that developing AGI is their ultimate goal. We recommend first that you recognize that “AGI” is not a well-defined scientific concept; for example, there is no agreed-upon test for whether a system has attained AGI. The term should therefore be understood as a marketing device, similar to saying that a detergent makes clothes smell “fresh” or that a car is “luxurious.” Second, we recommend that you assess more concrete claims about models’ specific capabilities using the tools that NLP researchers have developed for this purpose. You should expect no product to “do anything you ask,” and the clear demonstration that it has one capability should never be taken as evidence that it has different or broader capabilities. Third, we emphasize that AGI is not the explicit or implicit goal of all researchers or developers of AI systems. In fact, some are far more excited about tools that augment human abilities than about autonomous agents with abilities that can be compared to those of humans.

We close with an observation. Until the recent advent of tools marketed as “AI,” our experience with intelligence has been primarily with other humans, whose intelligence is a bundle of a wide range of capabilities we take for granted. Language models have, at the very least, linguistic fluency: the text they generate tends to follow naturally from their prompts, perhaps indistinguishably well from humans. But LMs don’t have the whole package of intelligence that we associate with humans. In language models, fluency, for example, seems to have been separated from the rest of the intelligence bundle we find in each other. We should expect this phenomenon to be quite shocking because we haven’t seen it before! And indeed, many of the heated debates around LMs and current AI systems more generally center on this “unbundled” intelligence. Are the systems intelligent? Are they more intelligent than humans? Are they intelligent in the same ways as humans? If the behaviors are in some ways indistinguishable from human behaviors, does it matter that they were obtained or are carried out differently than for humans?

We suspect that these questions will keep philosophers busy for some time to come. For most of us who work directly with the models or use them in our daily lives, there are far more pressing questions to ask. What do I want the language model to do? What do I not want it to do? How successful is it at doing what I want, and how readily can I discover when it fails or trespasses into proscribed behaviors? We hope that our discussion helps you devise your own answers to these questions.

For perspective, let’s consider two past shifts in the field of NLP that happened over the last ten years. The first, in the early 2010s, was a shift from statistical methods—where each parameter fulfilled a specific, understandable (to experts) role in a probabilistic model—to neural networks, where blocks of parameters without a corresponding interpretation were learned via gradient descent. The second shift, around 2018–19, was the general adoption of the transformer architecture we described in section 4.2, which mostly replaced past neural network architectures popular within NLP, and the rise of language model pretraining (as discussed in section 3.2).

Most in the field didn’t anticipate either of those changes, and both faced skepticism. In the 2000s, neural networks were still largely an idea on the margins of NLP that hadn’t yet demonstrated practical use; further, prior to the introduction of the transformer, another, very different structure of neural network¹⁸¹⁸18It was called the LSTM, “long short-term memory” network. was ubiquitous in NLP research, with relatively little discussion about replacing it. Indeed, for longtime observers of NLP, one of the few seeming certainties is of a significant shift in the field every few years—whether in the form of problems studied, resources used, or strategies for developing models. The form this shift takes does not necessarily follow from the dominant themes of the field over the preceding years, making it more “revolutionary” than “evolutionary.” And, as more researchers are entering NLP and more diverse groups collaborate to consider which methods or which applications to focus on next, predicting the direction of these changes becomes even more daunting.

A similar difficulty applies when thinking about long-term real-world impacts of NLP technologies. Even setting aside that we don’t know how NLP technology will develop, determining how a particular technology will be used poses a difficult societal question. Furthermore, NLP systems are being far more widely deployed in commercial applications; this means that model developers are getting far more feedback about them from a wider range of users, but we don’t yet know the effects that deployment and popular attention will have on the field.

Remembering how these models work at a fundamental level—using preceding context to predict the next text, word by word, based on what worked best to mimic demonstrations observed during training—and imagining the kinds of use cases that textual mimicry is best-suited towards will help us all stay grounded and make sense of new developments.

An important conversation about the future of language models centers around possible regulation of these models. This topic encompasses many related discussions: companies’ self-regulation, auditing of models by third parties, restrictions on data collection by private companies (such as those recently instituted by Reddit), and potential government oversight. Given that companies producing these models must already make decisions about how to adjust their models’ behavior, it seems most realistic to consider not whether regulation by some party will occur, but rather which forms of regulation would be beneficial. We will first describe some early attempts at regulating AI and then hypothesize about what future regulations might focus on.

Before doing that, we make one additional point. It’s worth bearing in mind that calls in the public sphere for or against regulation can arise for a variety of different reasons. For example, as Kevin Roose recently wrote for the New York Times, “some skeptics have suggested that A.I. labs are stoking fear out of self-interest, or hyping up A.I.’s destructive potential as a kind of backdoor marketing tactic for their own products. (After all, who wouldn’t be tempted to use a chatbot so powerful that it might wipe out humanity?)”¹⁹¹⁹19See also this opinion article by Bruce Schneier and Nathan Sanders. Past a certain point, discussion of AI regulation can become politically charged, drawing on many complicated variations of societal values. Therefore, similar to when participating in any public discussion, it’s helpful to get in the habit of thinking about why a specific person might be saying what they’re saying given their background and interests, as well as who they’re hoping their comments will influence.

There are a lot of important actions that help move us towards a future where AI systems are developed in beneficial ways. We’ll list a few here.

• •

  If you’re a student interested in AI systems: you can become one of the people helping to decide how these models work. For anyone in this position, you’ll find it useful to study computing, math, statistics, and also fields that reason about society. After all, the question of what we build these systems to do deserves just as much attention as the question of how we build these systems to do it.

• •

  If you’re an expert in something other than AI (e.g., healthcare, a scientific or humanistic field): the people building these models could really benefit from your expertise. Determining how to adapt AI systems to safely assist with problems faced by experts is not something computer scientists can do alone. To make these kinds of models useful for you and your field (and to avoid trying to solve problems that don’t really need solving), model developers need your input and help. As more scientists and engineers enter the growing AI field, it should become easier to find people in your network who are working on the models. Engage with them!

• •

  If you make decisions in a business sphere: you can set a high bar for evaluating possible AI-based systems in your company’s workflow. There’s considerable flashy language about some of these systems. By ignoring that and instead discussing with developers how a particular system was tested, how well that testing relates to your intended use case for it, and what’s missing from those tests, you can help raise overall standards for evaluating AI.

• •

  If you’re a concerned consumer: it’s a huge help for you to assume a thoughtful, reflective distance about LMs and AI news. In recent months, there’s been seemingly nonstop discussion of these topics, and there’s sure to be a lot more coming. Our biggest goal for this document is that it will help to equip you with the knowledge you need to filter the hype and make sense of the substance.

The first important property for a loss function is that it takes into account all the potential good and bad things about outputs when deducting points. The more dissimilar our model’s output given a particular input is from that input’s correct output, the higher the loss function should be. The second important property is that we must be able to deduce, fully automatically and in parallel for all parameters, what adjustments would make the loss function decrease. You may recall from a course on calculus that questions like “How does a small change to an input to a function affect the function’s output?” are related to the concept of differentiation. In sum, we need the loss function to be differentiable with respect to the parameters. (This may be a bit confusing because in calculus, we think about differentiating a function with respect to its inputs. In a mathematical sense, the input is only part of the input to the mathematical function encoded by a neural network; the parameters are also part of its input.) If the loss function has this property, then we can use differentiation to automatically calculate a small change for each parameter that should decrease the loss on a given example.

These two properties—faithfulness to the desired evaluation and differentiability with respect to parameters—conflict because most evaluation scores aren’t differentiable. Bleu scores for translation and error rates for sentiment analysis are stepwise functions (“piecewise constant” in mathematical terms): changing the parameters a tiny bit usually won’t affect these evaluation scores; when it does, it could be a dramatic change. Human judgments also are not differentiable with respect to parameters.

Once we know a differentiable loss function, and with a few additional assumptions, we quickly arrive at the algorithm for stochastic gradient descent (SGD), for setting system parameters. To describe its steps a bit more formally than we did in section 2.3.2:

1. 1.

   Initialize the parameters randomly.

2. 2.

   Take a random sample of the training data (typically 100 to 1000 demonstrations); run each input through the system and calculate the loss and its first derivative with respect to every parameter. (When first derivatives are stacked into a vector, it’s called the gradient.) Keep a running total of the sum of loss values and a running total of the sum of gradients.

3. 3.

   For each parameter, change its value proportional to the corresponding value in the gradient vector. (If the gradient is zero, don’t change that parameter.)

4. 4.

   Go to step 2 if the loss is converging.

Given some test data (some text the language model wasn’t trained on), we can calculate the error rate as follows. Let the words in the test data be denoted by $w_{1}$, $w_{2}$, … , $w_{N}$.

1. 1.

   Set $m=0$; this is the count of mistakes.

2. 2.

   For every word $w_{i}$ in the test data ($i$ is its position):

   1. 1.

      Feed $w_{i}$’s preceding context, which after the first few words will be the sequence $w_{1}$, $w_{2}$, … , $w_{i-1}$, into the language model as input.

   2. 2.

      Let the language model predict the next word; call its prediction $w_{\textrm{pred}}$.

   3. 3.

      If $w_{\textrm{pred}}$ is anything other than $w_{i}$, the language model made an incorrect prediction, so add 1 to $m$.

3. 3.

   The error rate is $m/N$.

Section 3.4 describes underlying properties of how LMs make “decisions” about next words. Here, to prepare for a deeper dive into perplexity, we summarize and build on those properties:

• •

  Based on the context of preceding words, a calculation is made by the neural network that assigns a probability to every word in the vocabulary, that is, every possible choice of what word could come next. These probabilities must always sum to one (that’s part of the definition of a probability distribution), and we also impose a “no zeros” rule: the probability of every vocabulary word must always be at least slightly positive.

• •

  To predict the next word, the model can either (a) choose the one with the highest probability (as assumed in the error rate calculation above) or (b) simulate a draw from the probability distribution, choosing a word at random such that each word’s chance of being drawn is given by its probability. To illustrate, imagine a pub trivia team where individual members have different past success rates of being correct. Approach (a) would correspond to the team always submitting the answer proposed by the trivia-whiz team member whose suggested answers had most often been correct before. Approach (b) would correspond to randomly picking who should answer, with the trivia whiz’s answer being most likely to be chosen, the second-best team member’s answer next most likely, then the third-best team member’s answer, and so on. Note that the most likely outcome from (b) is the same as the outcome from (a), but (b) will sometimes lead to another, lower-probability word.

Whether (a), (b), or some other approach is used when an LM is deployed is an important design decision. In keeping with our earlier rejection of error rate, researchers try to avoid evaluating LMs in a way that makes unnecessary commitments to its eventual use.²⁰²⁰20The technical term for our desired evaluation is “intrinsic” evaluation, meaning that we want a measure of the intrinsic quality of a model, not its performance in some extrinsic setting. Option (b) is interesting because it suggests a workaround to the pitfalls of simply counting mistakes discussed in section 3.4.

In the preceding appendix subsection’s error rate calculation procedure, we could apply option (b) in step 2.2. Suppose we do this not once, but many times for each context/word pair and average the error rate across these random draws. With enough draws, this approach would provide meaningful error rates because we’d expect to get each word right some of the time (no zeros rule). In practice, rather than actually carrying out the random draws, we instead use the LM’s probabilities directly to assign a score for every word in the test data. The results of this approach are that:

• •

  If the language model gave probability 1 to the correct next word, the score for that word would be 1. This can’t happen exactly because the probabilities of all the wrong words have to exceed zero (no zeros rule). But we can get arbitrarily close in principle if the probabilities of all the wrong words get infinitesimally small.

• •

  If the LM gave probability 0 to the correct next word, the score for that word would be 0. But this can’t happen either because of the no zeros rule.

• •

  In general, the greater the probability the LM assigns to the correct next word, even if it’s not the most probable word, the higher the score.

Because of the no zeros rule, the per-word probability scores are always somewhere between 0 and 1.

Given the test data, we can calculate the LM probability for every word given its preceding context. If we took a simple average of these probability scores and subtracted that from 1, we would get something like an error rate (technically, an “expected” error rate under prediction method (b)). What is done in practice is similar in spirit but slightly different: we take the geometric average of the inverses of these probability scores, a value known as (test data) perplexity. The reasons are partly practical (tiny numbers can lead to a problem in numerical calculations, called underflow), partly theoretical, and partly historical. For completeness, here’s the procedure:

1. 1.

   Set $m=0$. (This quantity is no longer a running tally of mistakes.)

2. 2.

   For every word $w_{i}$ in the test data ($i$ is its position):

   1. 1.

      Feed $w_{i}$’s preceding context, which after the first few words will be the sequence $w_{1}$, $w_{2}$, … , $w_{i-1}$, into the language model as input.

   2. 2.

      Let $p$ be the probability that the language model assigns to $w_{i}$ (the correct next word).

   3. 3.

      Add $-\log(p)$ to $m$.

3. 3.

   The perplexity is $\exp(m/N)$.

Though it’s probably not very intuitive from the preceding procedure, perplexity does have some nice intuitive properties:

• •

  If our model perfectly predicted every word in the test data with probability 1, we would get a perplexity of 1.²¹²¹21To see this, note that $-\log(1)=0$, so $m$ stays 0 throughout step 2. Note that $\exp(0/N)=\exp(0)=1$. This can’t happen because (1) there is some fundamental amount of uncertainty in fresh, unseen text data, and (2) some probability mass is reserved for every wrong word, too (no zeros rule). If perplexity comes very close to 1, the cardinal rule that test data must not be used for anything other than the final test, like training, should be carefully verified.

• •

  If our model ever assigned a probability of 0 to some word in the test data, perplexity would go to infinity.²²²²22To see this, note that $\log(0)$ tends toward infinity. This won’t happen because of the no zeros rule.

• •

  Lower perplexity is better.

• •

  The perplexity can be interpreted as an average “branch factor”; in a typical next word prediction instance, how many vocabulary words are “effectively” being considered?