The Power of Randomization
Distributed Submodular Maximization on Massive Datasets¹¹1The authors are listed alphabetically.

A set function $f:2^{V}\rightarrow{\mathbb{R}}_{\geq 0}$ on a ground set $V$ is submodular if $f(A)+f(B)\geq f(A\cap B)+f(A\cup B)$ for any two sets $A,B\subseteq V$. Several problems of interest can be modeled as maximizing a submodular objective function subject to certain constraints:

where $\mathcal{C}\subseteq 2^{V}$ is the family of feasible solutions. Indeed, the general meta-problem of optimizing a constrained submodular function captures a wide variety of problems in machine learning applications, including exemplar clustering, document summarization, sensor placement, image segmentation, maximum entropy sampling, and feature selection problems.

At the same time, in many of these applications, the amount of data that is collected is quite large and it is growing at a very fast pace. For example, the wide deployment of sensors has led to the collection of large amounts of measurements of the physical world. Similarly, medical data and human activity data are being captured and stored at an ever increasing rate and level of detail. This data is often high-dimensional and complex, and it needs to be stored and processed in a distributed fashion.

In these settings, it is apparent that the classical algorithmic approaches are no longer suitable and new algorithmic insights are needed in order to cope with these challenges. The algorithmic challenges stem from the following competing demands imposed by huge datasets: the computations need to process the data that is distributed across several machines using a minimal amount of communication and synchronization across the machines, and at the same time deliver solutions that are competitive with the centralized solution on the entire dataset.

The main question driving the current work is whether these competing goals can be reconciled. More precisely, can we deliver very good approximate solutions with minimal communication overhead? Perhaps surprisingly, the answer is yes; there is a very simple distributed greedy algorithm that is embarrassingly parallel and it achieves provable, constant factor, worst-case approximation guarantees. Our algorithm can be easily implemented in a parallel model of computation such as MapReduce [2].

Before describing our general algorithm, let us recall the standard greedy algorithm, Greedy, shown in Algorithm 1. The algorithm takes as input $\langle V,\mathcal{I},f\rangle$, where $V$ is a set of elements, $\mathcal{I}\subseteq 2^{V}$ is a hereditary constraint, represented as a membership oracle for $\mathcal{I}$, and $f:2^{V}\to{\mathbb{R}}_{\geq 0}$ is a non-negative submodular function, represented as a value oracle. Given $\langle V,\mathcal{I},f\rangle$, Greedy iteratively constructs a solution $S\in\mathcal{I}$ by choosing at each step the element maximizing the marginal increase of $f$. For some $A\subseteq V$, we let $\textsc{Greedy}(A)$ denote the set $S\in\mathcal{I}$ produced by the greedy algorithm that considers only elements from $A$.

The greedy algorithm satisfies the following property:

Algorithm. We now describe our general, randomized distributed algorithm, RandGreeDi, shown in Algorithm 2. Suppose we have $m$ machines. Our algorithm runs in two rounds. In the first round, we randomly distribute the elements of the ground set $V$ to the machines, assigning each element to a machine chosen independently and uniformly at random. On each machine $i$, we execute $\textsc{Greedy}(V_{i})$ to select a feasible subset $S_{i}$ of the elements on that machine. In the second round, we place all of these selected subsets on a single machine, and run some algorithm Alg on this machine in order to select a final solution $T$. We return whichever is better: the final solution $T$ or the best solution amongst all the $S_{i}$ from the first phase.

Analysis. We devote the rest of this section to the analysis of the RandGreeDi algorithm. Fix $\langle V,\mathcal{I},f\rangle$, where $\mathcal{I}\subseteq 2^{V}$ is a hereditary constraint, and $f:2^{V}\to{\mathbb{R}}_{\geq 0}$ is any non-negative, monotone submodular function. Suppose that Greedy is an $\alpha$-approximation and Alg is a $\beta$-approximation for the associated constrained monotone submodular maximization problem of the form (1). Let $n=|V|$ and suppose that $\mathrm{OPT}=\arg\max_{A\in\mathcal{I}}f(A)$ is a feasible set maximizing $f$.

Let $\mathcal{V}(1/m)$ denote the distribution over random subsets of $V$ where each element is included independently with probability $1/m$. Let ${\mathbf{p}}\in[0,1]^{n}$ be the following vector. For each element $e\in V$, we have

Our main theorem follows from the next two lemmas, which characterize the quality of the best solution from the first round and that of the solution from the second round, respectively. Recall that $f^{-}$ is the Lovász extension of $f$.

Combining Lemma 4 and Lemma 3 gives us our main theorem.

If we use the standard greedy algorithm for Alg, we obtain the following simplified corollary of Theorem 5.

We consider the problem of maximizing a non-monotone submodular function subject to a hereditary constraint. Our approach is a slight modification of the randomized, distributed greedy algorithm described in Section 3, and it builds on the work of [4]. Again, we show how to combine the standard Greedy algorithm, together with any algorithm Alg for the non-monotone case in order to obtain a randomized, distributed algorithm for the non-monotone submodular maximization.

Algorithm. Our modified algorithm, NMRandGreeDi, works as follows. As in the monotone case, in the first round we distribute the elements of $V$ uniformly at random amongst the $m$ machines. Then, we run the standard greedy algorithm twice to obtain two disjoint solutions $S_{i}^{1}$ and $S_{i}^{2}$ on each machine. Specifically, each machine first runs Greedy on $V_{i}$ to obtain a solution $S_{i}^{1}$, then runs Greedy on $V_{i}\setminus S_{i}^{1}$ to obtain a disjoint solution $S_{i}^{2}$. In the second round, both of these solutions are sent to a single machine, which runs Alg on $S=\bigcup_{i}(S_{i}^{1}\cup S_{i}^{2})$ to produce a solution $T$. The best solution amongst $T$ and all of the solutions $S_{i}^{1}$ and $S_{i}^{2}$ is then returned.

Analysis. We devote the rest of this section to the analysis of the algorithm. In the following, we assume that we are working with an instance $\langle V,\mathcal{I},f\rangle$ of non-negative, non-monotone submodular maximization for which the Greedy algorithm has the following property:

The standard analysis of the Greedy algorithm shows that ($\mathrm{GP}$) is satisfied with constant $\alpha$ for hereditary constraints such as matroids, knapsacks, and $p$-systems (see Table 4).

The analysis is similar to the approach from the previous section. We define $\mathcal{V}(1/m)$ as before. We modify the definition of the vector ${\mathbf{p}}$ as follows. For each element $e\in V$, we have

We can now combine Lemmas 8 and 7 to obtain our main result for non-monotone submodular maximization.

We experimentally evaluate and compare the following distributed algorithms for maximizing a monotone submodular function subject to a cardinality constraint: the RandGreeDi algorithm described in Section 3, the deterministic GreeDi algorithm of [10], and the Sample&Prune algorithm of [8]. We run these algorithms in several scenarios and we evaluate their performance relative to the centralized Greedy solution on the entire dataset.

Exemplar based clustering. Our experimental setup is similar to that of [10]. Our goal is to find a representative set of objects from a dataset by solving a $k$-medoid problem [7] that aims to minimize the sum of pairwise dissimilarities between the chosen objects and the entire dataset. Let $V$ denote the set of objects in the dataset and let $d:V\times V\rightarrow{\mathbb{R}}$ be a dissimilarity function; we assume that $d$ is symmetric, that is, $d(i,j)=d(j,i)$ for each pair $i,j$. Let $L:2^{V}\rightarrow{\mathbb{R}}$ be the function such that $L(A)={1\over\left|V\right|}\sum_{v\in V}\min_{a\in A}d(a,v)$ for each set $A\subseteq V$. We can turn the problem of minimizing $L$ into the problem of maximizing a monotone submodular function $f$ by introducing an auxiliary element $v_{0}$ and by defining $f(S)=L(\left\{v_{0}\right\})-L(S\cup\left\{v_{0}\right\})$ for each set $S\subseteq V$.

Tiny Images experiments: In our experiments, we used a subset of the Tiny Images dataset consisting of $32\times 32$ RGB images [12], each represented as $3,072$ dimensional vector. We subtracted from each vector the mean value and normalized the result, to obtain a collection of $3,072$-dimensional vectors of unit norm. We considered the distance function $d(x,y)=\|x-y\|^{2}$ for every pair $x,y$ of vectors. We used the zero vector as the auxiliary element $v_{0}$ in the definition of $f$.

In our smaller experiments, we used 10,000 tiny images, and compared the utility of each algorithm to that of the centralized greedy. The results are summarized in Figures 1(c) and 1(f).

In our large scale experiments, we used one million tiny images, and $m=100$ machines. In the first round of the distributed algorithm, each machine ran the Greedy algorithm to maximize a restricted objective function $f$, which is based on the average dissimilarity $L$ taken over only those images assigned to that machine. Similarly, in the second round, the final machine maximized an objective function $f$ based on the total dissimilarity of all those images it received . We also considered a variant similar to that described by [10], in which 10,000 additional random images from the original dataset were added to the final machine. The results are summarized in Figure 1(i).

Remark on the function evaluation. In decomposable cases such as exemplar clustering, the function is a sum of distances over all points in the dataset. By concentration results such as Chernoff bounds, the sum can be approximated additively with high probability by sampling a few points and using the (scaled) empirical sum. The random subset each machine receives can readily serve as the samples for the above approximation. Thus the random partition is useful for for evaluating the function in a distributed fashion, in addition to its algorithmic benefits.

Maximum Coverage experiments. We ran several experiments using instances of the Maximum Coverage problem. In the Maximum Coverage problem, we are given a collection $\mathcal{C}\subseteq 2^{V}$ of subsets of a ground set $V$ and an integer $k$, and the goal is to select $k$ of the subsets in $\mathcal{C}$ that cover as many elements as possible.

Kosarak and accidents datasets⁵⁵5The data is available at http://fimi.ua.ac.be/data/.: We evaluated and compared the algorithms on the datasets used by Kumar et al. [8]. In both cases, we computed the optimal centralized solution using CPLEX, and calculated the actual performance ratio attained by the algorithms. The results are summarized in Figures 1(a), 1(d), 1(b), 1(e).

Synthetic hard instances: We generated a synthetic dataset with hard instances for the deterministic GreeDi. We describe the instances in Section B. We ran the GreeDi algorithm with a worst-case partition of the data. The results are summarized in Figure 1(h).

Finding diverse yet relevant items. We evaluated our NMRandGreeDi algorithm on the following instance of non-monotone submodular maximization subject to a cardinality constraint. We used the objective function of Lin and Bilmes [9]: $f(A)=\sum_{i\in V}\sum_{j\in A}s_{ij}-\lambda\sum_{i,j\in A}s_{ij}$, where $\lambda$ is a redundancy parameter and $\left\{s_{ij}\right\}_{ij}$ is a similarity matrix. We generated an $n\times n$ similarity matrix with random entries $s_{ij}\in\mathcal{U}(0,100)$ and we set $\lambda=n/k$. The results are summarized in Figure 1(g).

Matroid constraints. In order to evaluate our algorithm on a matroid constraint, we considered the following variant of maximum coverage: we are given a space containing several demand points and $n$ facilities (e.g. wireless access points or sensors). Each facility can operate in one of $r$ modes, each with a distinct coverage profile. The goal is to find a subset of at most $k$ facilities to activate, along with a single mode for each activated facility, so that the total number of demand points covered is maximized. In our experiment, we placed 250,000 demand points in a grid in the unit square, together with a grid of $n$ facilities. We modeled coverage profiles as ellipses centered at each facility with major axes of length $0.1\ell$, minor axes of length $0.1/\ell$ rotated by $\rho$ where $\ell\in\mathcal{N}(3,\frac{1}{3})$ and $\rho\in\mathcal{U}(0,2\pi)$ are chosen randomly for each ellipse. We performed two series of experiments. In the first, there were $n=900$ facilities, each with $r=5$ coverage profiles, while in the second there were $n=100$ facilities, each with $r=100$ coverage profiles.

The resulting problem instances were represented as ground set comprising a list of ellipses, each with a designated facility, together with a partition matroid constraint ensuring that at most one ellipse per facility was chosen. As in our large-scale exemplar-based clustering experiments, we considered 3 approaches for assigning ellipses to machines: assigning consecutive blocks of ellipses to each machine, assigning ellipses to machines in round-robin fashion, and assigning ellipses to machines uniformly at random. The results are summarized in Figures 1(j) and 1(k); in these plots, GreeDi(rr) and GreeDi(block) denote the results of GreeDi when we assign the ellipses to machines deterministically in a round-robin fashion and in consecutive blocks, respectively.

In general, our experiments show that random and round robin are the best allocation strategies. One explanation for this phenomenon is that both of these strategies ensure that each machine receives a few elements from several distinct partitions in the first round. This allows each machine to return a solution containing several elements.

Acknowledgements. We thank Moran Feldman for suggesting a modification to our original analysis that led to the simpler and stronger analysis included in this version of the paper.