A Competence-aware Curriculum for Visual Concepts Learning via Question Answering

Visual question answering (VQA) [39, 56, 48, 29, 17] is a popular task for gauging the capability of visual reasoning systems. Some recent studies [2, 3, 24, 30, 59] focus on learning the neural module networks (NMNs) on the CLEVR dataset. NMNs translate questions into programs, which are further executed over image features to predict answers. The program generator is typically trained on human annotations. Several recent works target on reducing the supervision or increasing the generalization ability to new tasks in NMNs. For example, Johnson et al. [30] replaces the hand-designed syntactic parsers by a learned program generator. Neural-Symbolic VQA [60] explores an object-based visual representation and uses a symbolic executor for inferring the answer. Neural-symbolic concept learner [41] uses a symbolic reasoning process and manually-defined curriculum to bridge the learning of visual concepts, words, and the parsing of questions without explicit annotations. In this paper, we build our model on the neural-symbolic concept learner [41] and learn an adaptive curriculum to select the most profitable training samples.

Learning-by-asking (LBA) [42] proposes an interactive learning framework that allows the model to actively query an oracle and discover an easy-to-hard curriculum. LBA uses the expected accuracy improvement over candidate answers as an informativeness measure to pick questions. However, it is costly to compute the expected accuracy improvement for sampled questions since it requires to process all the questions and images through a VQA model. Moreover, the expected accuracy improvement cannot help to learn which specific component of the question contributes to the performance, especially while learning from the answers with little information such as “yes/no”. In contrast, we select questions by explicitly modeling the difficulty of visual concepts, combined with model competence to infer the difficulty of each question.

The competence-aware curriculum in our work is related to curriculum learning [7, 53, 55, 20, 51, 44, 21, 47] and machine teaching [61, 62, 38, 11, 40, 15, 58]. Curriculum learning is firstly proposed by Bengio et al. [7] and demonstrates that a dataset order from easy instances to hard ones benefits learning process. The measures of hardness in curriculum learning approaches are usually determined by hand-designed heuristics [53, 55, 51, 41]. Graves et al. [20] explore learning signals based on the increase rates in prediction accuracy and network complexity to adjust data distributions along with training. Self-paced learning [33, 27, 28, 51] quantifies the sample hardness by the training loss and formulates curriculum learning as an optimization problem by jointly modeling the sample selection and the learning objective. These hand-designed heuristics are usually task-specific without any generalization ability to other domains.

Machine teaching [61, 62, 38] introduces a teacher model that receives feedback from the student model and guides the learning of the student model accordingly. Zhu et al. [61, 62] assume that the teacher knows the ground-truth model (i.e., the Oracle) beforehand and constructs a minimal training set for the student model. The recent works learning to teach [15, 58] break this strong assumption of the existence of the oracle model and endow the teacher with the capability of learning to teach via a reinforcement learning framework.

Our work explores curriculum learning in visual reasoning, which is highly compositional and more complex than tasks studied before. Different from previous works, our method requires neither hand-designed heuristics nor an extra teacher model. We combine the idea of competence with curriculum learning and propose a novel mIRT model that estimates the concept difficulty and model competence from accumulated model responses.

We briefly describe the neural-symbolic concept learner. It uses a symbolic reasoning process to bridge the learning of visual concepts and the semantic parsing of textual questions without any intermediate annotations except for the final answers. We refer readers to [41, 60] for more details on this model.

Recalling the program execution is fully differentiable w.r.t. the concept embeddings, we learn the concept embeddings by directly maximizing $\log p(A|\hat{l},I;\theta_{e})$ using gradient descent and the gradient $\nabla_{\theta_{e}}\log p(A|\hat{l},I;\theta_{e})$ can be calculated through back-propagation. Since the hard selection of $\hat{l}$ through Monte Carlo sampling is non-differentiable, the gradients of the question parser cannot be computed by back-propagation. Instead we optimize the question parser using the REINFORCE algorithm [57]. The gradient of the reward function $J$ over the parameters of the policy is:

where $r$ denotes the reward. Defining the reward as the log-probability of the correct answer and again, we rewrite the intractable expectation with one Monte Carlo sample $\hat{l}$:

where $b$ is the exponential moving average of $\log p(A|\hat{l},I;\theta_{e})$, serving as a simple baseline to reduce the variance of gradients. Therefore, the update to the question parser at each learning step is simply the gradient of the log-probability of choosing the program, multiplied by the probability of the correct answer using that program.

Item response theory (IRT) [5, 6] was initially created in the fields of educational measurement and psychometrics. It has been widely used to measure the latent abilities of subjects (e.g., human beings, robots or AI models) based on their responses to items (e.g., test questions) with different levels of difficulty. The core idea of IRT is that the probability of a correct response to an item can be modeled by a mathematical function of both individual ability and item characteristics. More formally, if we let $i$ be an individual and $j$ be an item, then the probability that the individual $i$ answers the item $j$ correctly can be modeled by a logistic model as:

where $\theta_{i}$ is the latent ability of the individual $i$ and $a_{j},b_{j},c_{j}$ are the characteristics of the item $j$. The item parameters can be interpreted as changing the shape of the standard logistic function: $a_{j}$ (the discrimination parameter) controls the slope of the curve; $b_{j}$ (the difficulty parameter) is the ability level, it is the point on $\theta_{i}$ where the probability of a correct response is the average of $c_{j}$ (min) and 1 (max), also where the slope is maximized; $c_{j}$ (the guessing parameter) is the asymptotic minimum of this function, which accounts for the effects of guessing on the probability of a correct response for a multi-choice item. Equation 4 is often referred to as the three-parameter logistic (3PL) model since it has three parameters describing the characteristics of items. We refer the readers to [5, 6, 14] for more background and details on IRT.

Traditional IRT is proposed to model the human responses to several hundred items. However, datasets used in machine learning, especially deep neural networks, often consist of hundreds of thousands of samples or even more. It is costly to collect human responses for large datasets, and more importantly, human responses are not distinguishable enough to estimate the sample difficulties since samples in machine learning datasets are usually straightforward for humans. Lalor et al. [34, 35] empirically shows on two NLP tasks that IRT models can be fit using machine responses by comparing item parameters learned from the human responses and the responses from an artificial crowd of thousands of machine learning models.

Similarly, we propose to fit IRT models with accumulated model responses (i.e., the predictions of model snapshots) from the training process. Considering the compositional nature of visual reasoning, we propose a multi-dimensional IRT (mIRT) model to estimate the concept difficulty and model competence (corresponding to the subject ability in original IRT), from which the question difficulty can be further calculated.

Formally, we have $C$ concepts, $M$ model snapshots saved from all time steps, and $N$ questions. Let $\Theta=\{\theta_{ic}\}_{i=1..M}^{c=1...C}$, where $\theta_{ic}$ is the $i$-th snapshot’s competence on the $c$-th concept, and $B=\{b_{c}\}^{c=1...C}$, where $b_{c}$ is the difficulty of the $c$-th concept, $\mathcal{Q}=\{q_{jc}\}^{c=1...C}_{j=1...N}$, where $q_{jc}$ is the number of the $c$-th concept in the $j$-th question and $g_{j}$ is the probability of guessing the correct answer to the $j$-th question, $\mathcal{Z}=\{z_{ij}\}_{i=1...M}^{j=1...N}$, where $z_{ij}\in\{0,1\}$ be the response of the $i$-th snapshot to the $j$-th question ($1$ if the model answers the question correctly and $0$ otherwise). The probability that the snapshot $i$ can correctly recognize the concept $c$ is formulated by a logistic function:

Then the probability that the snapshot $i$ answers the question $j$ correctly is calculated as:

The probability that the snapshot $i$ answers the question $j$ incorrectly is:

The total data likelihood is:

This formulation is also referred to as conjunctive multi-dimensional IRT [49, 50].

The goal of fitting an IRT model on observed responses is to estimate the latent subject abilities and item parameters. In traditional IRT, the item parameters are usually estimated by Marginal Maximum Likelihood (MML) via an Expectation-Maximization (EM) algorithm [9], where the subject ability parameters are randomly sampled from a normal distribution and marginalized out. Once the item parameters are estimated, the subject abilities are scored by maximum a posterior (MAP) estimation based on their responses to items. However, the EM algorithm is not computational efficient on large datasets. One feasible way for scaling up is to perform variational Bayesian inference on IRT [43, 35]. The posterior probability of the parameters in mIRT can be written as:

where $p(\Theta),p(B)$ are the priors distribution of $\Theta$ and $B$. The integral over the parameter space in Eq 9 is intractable. Therefore, we approximate it by a factorized variational distribution on top of an independence assumption of $\Theta$ and $B$:

where $\pi_{ic}^{\theta}$ and $\pi_{c}^{b}$ denote Gaussian distributions for model competences and concept difficulties, respectively. We adopt the Kullback-Leibler divergence (KL-divergence) to measure the distance of $p$ from $q$, which is defined as:

where $p(\Theta,B|\mathcal{Z})$ is still intractable. We can further decompose the KL-divergence as:

In other words, we also have:

As the log evidence $\log p(\mathcal{Z})$ is fixed with respect to $q$, maximizing the final term $\mathcal{L}(q)$ minimizes the KL divergence of $q$ from $p$. And since $q(\Theta,B)$ is a parametric distribution we can sample from, we can use Monte Carlo sampling to estimate this quantity. Since the KL-divergence is non-negative, $\mathcal{L}(q)$ is an evidence lower bound (ELBO) of $\log p(\mathcal{Z})$. By maximizing the ELBO with an Adam optimizer [31] in Pyro [8], we can estimate the parameters in mIRT.

The proposed model can estimate the question difficulty for the current model competence without looking at the ground-truth images and answers. It facilitates the active selection for future training samples. More specifically, we can easily calculate the probability that the model answers a given question correctly from Eq. 5 and Eq. 6 (without guessing) using estimated $\Theta$ and $b$. This probability serves as an indicator of the question difficulty for the learner in each stage. The higher the probability, the easier the question. To select appropriate training samples, we rank the questions and filter out the hardest questions by setting a probability lower bound (LB) and the easiest questions by a probability upper bound (UB). Algorithm 1 summarizes the overall training process. We will discuss the influence of LB and UB on the learning process in Section 4.5.

Please refer to the supplementary materials for detailed model settings and learning techniques during training.

Figure 3 shows the accuracies of the model variants at different timesteps on the training set (left) and validation set (right). Notably, the proposed NSCL-mIRT converges almost 2 times faster than NSCL-Fixed and 3 times faster than NSCL (i.e., 400k v.s. 800k v.s. 1200k). Although NSCL-mIRT spends extra time to estimate the parameters of the mIRT model, such time cost is negligible compared to other time spent in training (less than 1%). From Table 2, we can see that NSCL-mIRT consistently outperforms FilM-LBA at various iterations, which demonstrates the preeminence of mIRT in building an adaptive curriculum.

Besides, NSCL-mIRT consumes less than 300k unique questions for training when it converges. It indicates that NSCL-mIRT saves about 60% of the training data, which largely eases the data redundancy problems. It provides a promising direction for designing a data-efficient curriculum and helping current data-hungry deep learning models save time and money cost during data annotation and model training.

Moreover, NSCL-mIRT obtains even higher accuracy than NSCL and NSCL-Fixed. This indicates that the adaptive curriculum built by the multi-dimensional IRT model not only remarkably increases the speed of convergence and reduces the data consumption during the training process, but also leads to better performance, which also verifies the hypothesis made by Bengio et al. [7].

The estimated concept difficulty and model competence after converging is shown in Figure 4 for studying the performance of the mIRT model. Several critical observations are: (1) The spatial relations (i.e., left/right/front/behind) are the easiest concepts. It satisfies our intuition since the model only needs to exploit the object positions to determine their spatial relations without dealing with appearance. The spatial relations are learned during the late stages since they appear more frequently in complex questions to connect multiple concepts. (2) Colors are the most difficult concepts. The model needs to capture the subtle differences in the appearance of objects to distinguish eight different colors. (3) The model competence scores surpass the concept difficulty scores for all the concepts. This result corresponds to the nearly perfect accuracy ($>99\%$) on all questions and concepts.

Figure 5 shows the estimation of the model competence for each attribute type at various iterations. We can observe that model competence consistently increases throughout the training. Figure 5 shows the estimations of the concept difficulty at different learning steps. As the training progresses, the estimations become more stable with smaller variance since more model responses are accumulated.

We apply the count-based concept evaluation metric proposed in [41] to measure the performance of the concept learner, which evaluates the visual concepts on synthetic questions with a single concept such as “How many red objects are there?” Table 2 presents the results by comparing with several state-of-the-art methods, which includes methods based on neural module network with programs (IEP [29]) and neural attentions without programs (MAC [24]). Our model achieves nearly perfect performance across visual concepts and outperforms all other approaches. This means the model can learn visual concepts better with an adaptive curriculum. Our model can also be applied to the VQA. Table 4 summarizes the VQA accuracy on the CLEVR validation split. Our approach achieves comparable performance with state-of-the-art methods.

The question selection strategy is controlled by two hyper-parameters: the lower bound (LB) and upper bound (UB). We conduct experiments by learning with different LBs and UBs, and Table 4 shows the VQA accuracy at various iterations. It reveals that the proper lower bound can effectively filter out too hard questions and accelerate the learning at the early stage of the training, as shown in the first three rows. Similarly, a proper upper bound helps to filter out too easy questions at the late stage of the training when the model has learned most concepts. Please refer to the supplementary material for the visualization of selected questions at various iterations.