Relational Programming with Foundation Models

These methods combine the complementary benefits of neural learning and symbolic reasoning. They include domain-specific solutions (yi2018nsvqa; mao2019nscl; li2020closed; wang2019satnet; xu2022dont; chen2020nerd; minervini2020ctp) as well as general programming frameworks, such as DeepProbLog (robin2018deepproblog) and Scallop (li2023scallop). These methods typically concern training or fine-tuning neural models in the presence of logical programs, whereas we target building applications atop foundation models with zero-shot or few-shot examples. Another recent work, the STAR framework (rajasekharan2023star) also connects a language model (neural) to an answer set programming reasoner (symbolic). It is conceptually similar to Vieira but only focuses on natural language understanding and does not support probabilistic reasoning.

These models target different modalities and domains (touvron2023llama; openai2023gpt4; alec2021transferablevisualmodels; kirillov2023segment; alec2021transferablevisualmodels). Their reasoning capabilities continue to improve with larger context sizes (ratner2023parallel), smarter data selection (adadi2021survey), and the discovery of new prompting methods, such as chain-of-thought (wei2023chainofthought; kojima2022large), self-consistency (wang2023selfconsistency), and ReAct (yao2023react). Vieira is orthogonal to these techniques and stands to further enhance the robustness and reliability of foundation models in end-to-end AI applications.

There are many efforts that seek to improve the reasoning abilities of language models (LMs) by incorporating external programs and tools (gao2023pal; schick2023toolformer; reiichiro2021webgpt; davis2023testing). For instance, AutoGPT (autogpt) and TaskMatrix.AI (liang2023taskmatrixai) allows black-box LMs to control symbolic reasoning by invoking commands or calling APIs. On the other hand, many works attempt to extract structured information from LMs for downstream tasks (Gupta2022VisProg; beurer2022prompting). Vieira unifies these two strategies for augmenting model capabilities, and extends them into a glue language for composing multi-modal foundation models.

In Vieira, language models like GPT (openai2023gpt4) and LLaMA (touvron2023llama) can be used as basic foreign predicates for text completion:

In this case, gpt is an arity-2 FP that takes in a String as the prompt and produces a String as the response. It uses the model gpt-3.5-turbo by default. To make the interface more relational and structural, we provide an FA:

Here, we declare a relation named population which produces a population number (num) given a location (loc) as input. Notice that structured few-shot examples are provided through the argument examples.

One can directly configure language models to perform semantic parsing. For instance, the semantic parser for the simple Query DSL (partially defined in the Language section) can be declared as follows:

Internally, the language model is expected to generate a fully structured Query in its string form. Then, Vieira attempts to parse the string to construct actual ADT values. In practice, the success of semantic parsing depends heavily on the design of the DSL, involving factors like intuitiveness (e.g., names and arguments of ADT variants) and complexity (e.g., number of possible ADT variants).

Structural relational knowledge available in free-form textual data can be extracted by language models. We introduce a foreign attribute @gpt_extract_relation for this purpose. For instance, the following declared predicate takes in a context and produces (subject, object, relation) triplets:

This attribute differs from the text completion attribute in that it can extract an arbitrary number of facts. The underlying implementation prompts LMs to respond with JSON-formatted strings, allowing structured facts to be parsed.

Textual embeddings are useful in performing tasks such as information retrieval. The following example declares an FP encapsulating a cross-encoder (nogueira2019passage):

In the last line, we compute the cosine-similarity of the encoded embeddings using a soft-join on the variable e. As a result, we obtain a probabilistic fact like 0.9::sim() whose probability encodes the cosine-similarity between the textual embeddings of "cat" and "neko".

Image-text alignment models, such as CLIP (alec2021transferablevisualmodels), can naturally be used as zero-shot image classification models. Fig. 1(b) shows an example usage of the @clip attribute. We also note that dynamically-generated classification labels can be provided to CLIP via a bounded argument in the predicate.

OWL-ViT (minderer2022simple), Segment Anything Model (SAM) (kirillov2023segment), and DSFD (jian2018dsfd) are included in Vieira as image segmentation (IS) and object localization (LOC) models. IS and LOC models can provide many outputs, such as bounding boxes, classified labels, masks, and cropped images. For instance, the OWL-ViT model can be used and configured as follows:

Here, the find_obj predicate takes in an image, and finds image segments containing “human face” or “rocket”. According to the names of the arguments, the model extracts 3 values per segment: ID, label, and cropped image. Note that each produced fact will be associated with a probability, representing the confidence from the model.

Visual generative models such as Stable Diffusion (Rombach_2022_diffusion) and DALL-E (ramesh2021zero) can be regarded as relations as well. The following example shows the declaration of the gen_image predicate, which encapsulates a diffusion model:

As can be seen from the signature, it takes in a String text as input and produces a Tensor image as output. Optional arguments such as the desired image resolution and the number of inference steps can be supplied to dictate the granularity of the generated image.

In this task adapted from BIG-bench (srivastava2023imitation), the model is given a context and asked to compute a date. The questions test the model’s temporal and numerical reasoning skills, as well as its grasp of common knowledge. Unlike BIG-bench where multiple-choice answers are given, we require the model to directly produce its answer in MM/DD/YYYY form.

Our solution leverages GPT-4 (5-shot¹¹1In this work, $k$ in “$k$-shot” means the number of examples provided to the LM component within the full solution. Each example is a ground-truth input-output pair for the LM.) for extracting 3 relations: mentioned dates, duration between date labels, and the target date label. From here, our relational program iterates through durations to compute dates for all date labels. Lastly, the date of the target label is returned as the output.

In this task from BIG-bench, a textual description of pairwise object swaps among people is given, and the model needs to track and derive which object is in a specified person’s possession at the end. There are three difficulty levels depending on the number of objects to track, denoted by $n\in\{3,5,7\}$.

Our solution for tracking shuffled objects relies on GPT-4 (1-shot) to extract 3 relations: initial possessions, swaps, and the target person whose final possessed object is expected as the answer. Our reasoning program iterates through all the swaps starting from the initial state and retrieves the last possessed object associated with the target.

CLUTRR (sinha2019clutrr) is a kinship reasoning dataset of stories which indicate the kinship between characters, and requires the model to infer the relationship between two specified characters. The questions have different difficulty levels based on the length of the reasoning chain, denoted by $k\in\{2\dots 10\}$.

Our solution for kinship reasoning invokes GPT-4 (2-shot) to extract the kinship graph from the context. We also provide an external common-sense knowledge base for rules like “mother’s mother is grandmother”. Our program then uses the rules to derive other kinship relations. Lastly, we retrieve the kinship between the specified pair of people.

This task is drawn from the GSM8K dataset of arithmetic word problems (cobbe2021training). The questions involve grade school math word problems created by human problem writers, and the model is asked to produce a number as the result. Since the output can be fractional, we allow a small delta when comparing the derived result with the ground truth.

Our solution to this task prompts GPT-4 (2-shot) to produce step-by-step expressions, which can contain constants, variables, and simple arithmetic operations. We evaluate all the expressions through a DSL, and the result associated with the goal variable is returned. By focusing the LM’s responsibility solely on semantic parsing, our relational program can then achieve faithful numerical computation via DSL evaluation.

We choose HotpotQA (yang2018hotpotqa), a Wikipedia-based question answering (QA) dataset under the “distractor” setting. Here, the model takes in 2 parts of inputs: 1) a question, and 2) 10 Wikipedia paragraphs as the context for answering the question. Among the 10 Wikipedia pages, at most 2 are relevant to the answer, while the others are distractors.

Our solution is an adaptation of FE2H (li2022fe2h), which is a 2-stage procedure. First, we turn the 10 documents into a vector database by embedding each document. We then use the embedding of the question to retrieve the 2 most related documents, which are then fed to a language model to do QA. In this case, the QA model does not have to process all 10 documents, leading to less distraction.

We use Amazon’s ESCI Product Search dataset (reddy2022shopping). The model is provided with a natural language (NL) query and a list of products (23 products on average). The goal is to rank the products that best match the query. In the dataset, for each pair of query and product, a label among $E$ (exact match), $S$ (substitute), $C$ (complementary), and $I$ (irrelevant) is provided. The metric we use to evaluate the performance is nDCG. The gains are set to be 1.0 for $E$, 0.1 for $S$, 0.01 for $C$, and 0.0 for $I$.

One challenge of this dataset is that many queries contain negative statements. For example, in the query “#1 treadmill without remote”, the “remote” is undesirable. Therefore, instead of computing the embedding of the full query, we decompose the query into positive and negative parts. We then perform semantic search by maximizing the similarity of the positive part while minimizing that of the negative part.

We choose two compositional VQA datasets, GQA (drew2019gqa) and CLEVR (johnson2016clevr). In this task, the model is given an image and a question, and needs to answer the question. For GQA, the majority of questions expect yes/no answers, while CLEVR’s questions demand features like counting and spatial reasoning. We uniformly sample 500 and 480 examples from GQA and CLEVR datasets respectively. Following VQA conventions (kim2021vilt), we use Recall@$k$ where $k\in\{1,3\}$ as the evaluation metrics.

Our solution for GQA is an adaptation of VisProg (Gupta2022VisProg). We create a DSL for invoking vision modules such as ViLT and OWL-ViT, and use GPT-4 for converting questions into programs in this DSL. Our solution for CLEVR is similar, directly replicating the DSL provided by the original work. OWL-ViT and CLIP are used to detect objects and infer attributes, while the spatial relations are directly computed using the bounding box data.

We evaluate on two datasets, VQAR (huang2021scallop) and OFCP. For VQAR, the model is given an image and a programmatic query, and is asked to produce bounding boxes of the queried objects in the image. Our solution composes a relational knowledge base, defining entity names and relationships, with object retrieval (OWL-ViT) and visual QA (ViLT) models.

Online Faces of Celebrities and Politicians (OFCP) is a self-curated dataset of images from Wikimedia Commons among other sources. For this dataset, the model is given an image with a descriptive NL filename, and needs to detect faces relevant to the description and tag them with their names. Our solution obtains a set of possible names from GPT-4 and candidate faces from DSFD. These are provided to CLIP for object classification, after which probabilistic reasoning filters the most relevant face-name pairs.

We adopt the task of image editing from (Gupta2022VisProg). In this task, the instruction for image editing is provided through NL, and can invoke operations such as blurring background, popping color, and overlaying emojis. Due to the absence of an existing dataset, we repurpose the OFCP dataset by introducing 50 NL image editing prompts. Our solution for this task is centered around a DSL for image editing. We incorporate GPT-4 for semantic parsing, DSFD for face detection, and CLIP for entity classification. Modules for image editing operations are implemented as individual foreign functions.

For free-form generation and editing of images, we curate IGP20, a set of 20 prompts for image generation and editing. Instead of using the full prompt, we employ an LM to decompose complex NL instructions into simpler steps. We define a DSL with high-level operators such as generate, reweight, refine, replace, and negate. We use a combination of GPT-4, Prompt-to-Prompt (hertz2022prompttoprompt), and diffusion model (Rombach_2022_diffusion) to implement the semantics of our DSL. We highlight our capability of grounding positive terms from negative phrases, which enables handling prompts like “replace apple with other fruits” (Fig. 3).