Astronomy in the Cloud: Using MapReduce for Image Coaddition

Of the 100,000 FITS files in our experimental dataset, only a small number will ultimately contribute to any one query’s coadd. If we can exclude some of the irrelevant files from consideration before running MapReduce we may improve the overall job’s performance. Therefore, one of the most straightforward ways to improve efficiency is to filter the input and not process the entire dataset. Filtering can be accomplished by considering the dataset’s directory structure and file name convention and subsequently building a partially exclusive glob pattern (similar to a regular expression) when specifying the input file set for MapReduce.

Our prefilter works in two ways. Fig. 3 shows a schematic of the SDSS camera which contains an array of 30 independent CCDs. These 30 CCDs are divided into five rows corresponding to five filters and six columns corresponding to six parallel nonoverlapping strips of the sky. In the Stripe 82 dataset, these six strips correspond to parallel strips in declination. Any individual FITS file in the dataset corresponds to an image captured by one of these 30 CCDs. Since a query explicitly indicates a coadd of a specified bandpass, we can immediately reduce our input dataset by a factor of five by eliminating from consideration those FITS files captured in one of the four other bandpass filters. Furthermore, the dataset is structured in such a way that is relatively straightforward to filter by CCD column, i.e., by spatial location along the declination axis (see Fig. 6). We perform this secondary filter by determining which of the six columns the query’s sky-bounds overlap. For a very small query we might therefore achieve a further six-fold reduction in the input dataset, although for larger queries this reduction is usually on the order of 1.5 at best (elimination of two of the six total columns).

The following diagram shows how the SDSS Stripe 82 database is organized. As an example, let us assume we wish to construct a filter which accepts strips 2, 3, and 4, and bandpass ’g’. One possible glob⁴⁴4There are multiple possible ways of constructing a glob which satisfies the filter requirement. For example, the second ’[234]’ in the example shown is redundant and could be replaced with a ’*’. would be /SDSSDB_ROOT/*/*/corr/[234]/fpC-*-[g][234]-*.fit. The diagram below, of the directory hierarchy of the SDSS database, illustrates how this glob pattern can be used to exclude the unnecessary files when building the input set for MapReduce.

1. SDSSDB_ROOT

   • 5902 $\leftarrow$ SDSS Stripe 82 run id

     • 40 $\leftarrow$ rerun id

       • corr $\leftarrow$ constant directory name (correlated images)

         1. 1 $\leftarrow$ strip (camera column 1-6)

            • fpC-005902-u1-0690.fit $\leftarrow$ FITS file

            • fpC-005902-g1-0690.fit

            • fpC-005902-u1-0691.fit

            • fpC-005902-g1-0691.fit

         2. 2

            • fpC-005902-u2-0690.fit

            • fpC-005902-g2-0690.fit

            • fpC-005902-u2-0691.fit

            • fpC-005902-g2-0691.fit

         3. 3

         4. 4

         5. 5

         6. 6

We ran our experiments on two query sizes. Table 1 (second row) and Fig. 7 (first bar in each set) show the results of our first round of experiments with the larger query’s results in the left column of the table and left plot group. Note that the spatial filter occurs on only one spatial axis, not two as would be required for a perfect filter. Given a column that passes the prefilter, all FITS files in the dataset that were captured in that column will be considered by the mappers. This is unfortunate since many of those FITS files will still not overlap the query bounds in right ascension. They will be detected via their empty intersections and be discarded by the mappers, but these false positives represent wasted effort. We discuss this inefficiency and methods for handling it in section 4.1.4, but first we investigate where the prefiltered method focused its time.

Fig. 8 shows a breakdown of the prefiltered job’s overall running time for the larger query into the salient stages. The last bar represents the total running time, 42 minutes, the same measurement indicated in the first bar of Fig. 7. A call to main() involves a preliminary setup stage in the driver followed by a single call runJob() which encapsulates the MapReduce process. Thus, the bar in the fourth region is the sum of the bars in the first and third regions. Likewise, the call to runJob() runs the MapReduce process and has been further broken down into the components shown in the second region. Thus, the bar in the third region is the sum of the bars in the second region. We observe that the dominating step of the overall process is Construct File Splits. This indicates that the Hadoop coadd process spent most of its time locating the FITS files on HDFS and preparing them as input to the mappers. The cost associated with this step results from a set of serial remote procedure calls (RPCs) between the client and the cluster. The actual image-processing computations are represented only in the two shortest bars labeled Mapper Done and Reducer Done indicating that Hadoop spent a small proportion of its total time on the the fundamental task. This observation suggests that there is substantial inefficiency in the associated method, namely in the RPCs involved. The next section describes one method for alleviating this problem.

We never explicitly measured the performance without prefiltering but an estimate is easy to calculate. Given that the running times with prefiltering for our two experimental query sizes were 26 and 42 minutes respectively and given that the running time was dominated by the serial RPC bottleneck and given that the prefilter reduced the number of input files by a factor of 7.5 (about 13,000 files out of the 100,000 total passed the prefilter), we can estimate that without prefiltering, the process would have taken approximately 194 and 315 minutes respectively (Table 1, first row). This estimate assumes a linear relationship between the number of files to be found on HDFS and the required time, which is justified given that the number of RPCs is constant per file and the amount of time required per RPC is constant.

While Hadoop is designed to process very large datasets, it performs better on a dataset of a given size if the total number of files is relatively small. In other words, given two datasets of identical data with one dataset stored in a large number of small files and the other stored in a small number of large files, Hadoop will generally perform better on the latter dataset. While there are multiple contributing factors to this variation in behavior, one significant factor is the number of remote procedure calls (RPCs) required in order to initialize the MapReduce job, as illustrated in the previous section. For each file stored on HDFS, the client machine must perform multiple RPCs to the HDFS namenode to locate the file and notify the MapReduce initialization routines of its location. This stage of the job is performed in serial and suffers primarily from network latency. For example, in our experimental database we processed 100,000 files. Each file had to be independently located on HDFS prior to running the MapReduce job and this location was performed serially over set of 100,000 files. Consequently, this step alone could take nearly five hours on our setup. Even in the prefiltered method described above, this step still took about 30 minutes (see Fig. 8).

The above is a well-known limitation of the Hadoop system (White, 2009) and Hadoop includes an API and a set of data structures specifically targeted at alleviating it. The relevant structure is a sequence file. A sequence file contains an arbitrary number of key/value pairs where a key is used to indicate (and locate) a specific file and the value is the file in question. In this way, individual files can be retrieved from a sequence file in a random-access fashion. In effect, a sequence file is a concatenation of smaller files bundled with an index.

We use sequence files by grouping many individual FITS files into a single sequence file, indexed simply by the FITS filename itself. Thus, we can retrieve a specific FITS file rapidly while simultaneously reducing the number of actual files on disk from the number of FITS files to the number of resulting sequence files. As a first strategy, we randomly grouped FITS files into sequence files (thus producing unstructured sequence files), as illustrated in Fig. 9 (top). Table 1 and Fig. 7 show the results of using this approach in row three of the table and bar two of each plot group. As the results show, this approach improved upon the prefiltered method (described above) by a factor of five and theoretically improved upon the most naive approach by a factor of approximately 38. It should be noted that we never implemented the most naive method and so have no running times for it. Consequently, the prefiltered method as applied directly to the FITS database should be taken as the baseline for performance comparisons.

Note that unstructured sequence files outperform prefiltering of the direct FITS file dataset by a factor of five. Table 1 and Fig. 7 illustrate this fact. What is interesting is that this five-fold improvement occurs despite the fact that in the prefiltered case we actually reduced the input of 100,000 FITS files down to about 13,000 files while in the sequence file case we could not perform any prefiltering. In the unstructured sequence file case, many more FITS files were processed relative to the prefiltered FITS file case. Despite this inefficiency on part of the sequence file method, it still significantly outperforms the prefiltering method due to its vast reduction in the number of actual files on disk, i.e., the conversion of the input representation from a set of FITS files to a set of sequence files.

FITS files can be assigned to sequence files in any way the programmer chooses. In the previous section, we considered the worst case performance by using an unstructured sequence file database. The resulting sequence file database could not be pruned in any meaningful fashion at runtime, e.g., the prefiltering mechanism described in section 4.1.1, and therefore had to be fully read and processed during a MapReduce job. Results of using such a sequence file database were presented in the previous section.

However, it is also worth considering a more rationally motivated sequence file structure since doing so might permit pruning, i.e., filtering as demonstrated above. We refer once again to Fig. 3 which shows the SDSS camera’s CCD layout. This time, we use the SDSS camera’s CCD arrangement not to define a prefiltering method on FITS files, but rather to impose a corresponding structure on the sequence file database. We define 30 distinct sequence file types, one corresponding to each CCD of the camera (see Figs. 3 and 9). Thus, a given sequence file contains only FITS files originating from one of the original 30 CCDs. If the sequence files are named in a way which reflects the glob filter described previously, we can then filter entire sequence files in the same way that we previously prefiltered FITS files. This method of filtering permits us to eliminate entire sequence files from consideration prior to MapReduce on the basis of bandpass or column coverage of the query. We anticipated a corresponding improvement in performance resulting from the reduction of wasted effort spent considering irrelevant FITS files in the mappers.

Table 1 (row four) and Fig. 7 (third bar in each set) show the results of using structured sequence files and prefiltering in concert. We observe a further reduction in running time over unstructured sequence files by a factor of two on the larger query. The savings in computation relative to unstructured sequence files resulted from the elimination of much wasted effort in the mapper stage considering and discarding FITS files either whose bandpass did not match the query or whose sky-bounds did not overlap the query bounds.

Both prefiltering methods suffer from false positives resulting from the fact that the spatial filter occurs on only one spatial axis. Likewise, the non-prefiltered method performs no prefiltering at all. Therefore, in all three methods, effort is still being wasted considering irrelevant FITS files. The prefiltering methods processed 13,000 FITS files in the mappers while the non-prefiltered method processed the entire dataset of 100,000 FITS files. However, the number of FITS files that actually contributed to the final coadd was only 3885. In the next section we describe how we used a SQL database and query prior to running MapReduce to eliminate this problem.

All three methods described previously (prefiltered FITS files, unstructured sequence files, and prefiltered structured sequence files) process irrelevant FITS files in the mappers. Toward the goal of alleviating this inefficiency, we devised a new method of prefiltering. This method consists of building a SQL database of the FITS files outside of HDFS and outside of Hadoop in general. The actual image data is not stored in the SQL database — only the bandpass filter and sky-bounds of each FITS file are stored, along with the necessary HDFS file reference data necessary to locate the FITS file within the sequence file database, i.e., its assigned sequence file and its offset within the sequence file. Running a job using this method consists of first performing a SQL query to retrieve the subset of FITS files that are relevant to the user’s query, and from the SQL result constructing a set of HDFS file splits⁵⁵5Although the concept of a file split is complicated when one considers the full ramifications of storing files on a distributed file system, it suffices for our discussion to define a file split simply as the necessary metadata to locate and retrieve a FITS file from within a host sequence file. to specify the associated FITS files within the sequence file database (See Fig. 10). The full set of file splits then comprises the input to MapReduce.

The consequence of using SQL to determine the relevant FITS files and sending only those FITS files to MapReduce is that this method does not suffer from false positives as described above. Thus, the mappers waste no time considering (and discarding) irrevelant FITS files since every FITS file received by a mapper is guaranteed to contribute to the final coadd. The intention is clearly to reduce the mapper running time as a result. Table 2 shows the number of FITS files read as input to MapReduce for each of the six experimental methods. Note that prefiltering is imperfect, i.e., it suffers from false positives and accepts FITS files which are ultimately irrelevant to the coaddition task. However, the SQL methods only process the relevant files.

Fig. 11 shows the results of the new method. We have removed the prefiltered FITS method from the plot (shown in Fig. 7) so that we may concentrate our attention on the faster methods, namely, the two sequence file methods previously described (the first two bars shown in Fig. 11), and two new methods, one each for performing SQL against the unstructured sequence file database (the third bar) and for performing SQL against the structured sequence file database (the fourth bar).

Several patterns are observed in the plot. First, note that it is only meaningful to compare SQL vs. nonSQL for a given sequence file database, either structured or unstructured. Thus, the appropriate comparisons to make are between bars one and three or between bars two and four. We observe that in such comparisons, the SQL method does successfully outperform the nonSQL method in most cases, but to a lower degree than we hoped for when we implemented it, i.e., bar three shows only a small improvement over bar one and bar four shows virtually no improvement over bar two.

Let us consider the results indicated by bars two and four for the larger query. The noteworthy observation is that while SQL shows some benefit, that benefit is lower than originally anticipated given the 3.5x difference in mapper input records (FITS files), 13335 vs. 3885. 13335 is the number of FITS files read in by the prefiltered sequence file method (of which 3885 were relevant and the rest discarded) while 3885 is the number of FITS files read in by the SQL method (all of which were relevant). Therefore, we can conclude that in this case the cost of considering and discarding numerous irrelevant FITS files was negligible and likewise that the additional complexity imposed by supporting and using an external SQL database offers no benefit.

Let us consider the results indicated by bars one and three for the larger query. Again, we note little improvement in performance given the variance of the confidence intervals. This is surprising given the dramatic difference in mapper input records, a full 26 fold difference. The first bar represents non-prefiltered unstructured sequence files, thus 100,058 input records while the third bar represents a SQL method, thus 3885 input records. The conclusion is similar in this case, that the cost of discarding irrelevant files is low.

We might then ask: why is there a two-fold difference in performance between the two SQL methods considering that they both processed exactly the same amount of data? To answer this question we must investigate not merely the number of mapper input records but the number of mapper objects launched by Hadoop. These two values are rarely equal because a mapper can be reused to process multiple input records. In the SQL-unstructured-sequence-file method, the 3885 input FITS files were processed by 1714 mapper objects (about two FITS files each) while in the SQL-structured-sequence-file method, 338 mapper objects were used (about eleven FITS files each). This discrepancy is due to the way we assign FITS files to each mapper object. For each mapper object, we assign FITS files in the same HDFS block. Due to the replication of blocks across HDFS, copies of each block will be stored on multiple hosts. When possible, Hadoop will schedule mapper objects to run on one of the hosts where that mapper’s input blocks are located so that the files can be efficiently accessed from local disks. Clearly, this approach generates a variable number of mapper objects depending on the structure of the input sequence file. In the case of unstructured sequence files, the FITS files relevant to a given query are scattered throughout the sequence file database whereas in the case of structured sequence files, the relevant FITS files are more tightly packed. Thus, greater data locality is achieved in the assignment of FITS files to mappers in the structured case and fewer mapper objects are required. To complete our understanding of the difference in running times we must further consider the maximum number of mapper objects that the cluster can sustain simultaneously. Since a mapper represents a conceptual unit of computation, it corresponds to — or directly relies upon — a processor core in order to operate. Therefore, the number of simultaneously sustainable mapper objects for a Hadoop job is limited by the number of cores available on the entire cluster ⁶⁶6Ignoring for the moment additional factors such as distribution of core assignments between mapper and reducer objects and shared usage of the cluster by concurrent MapReduce jobs unrelated to our application.. In our case, this value is about 800. We can now see one fundamental problem with the unstructured case: not all of the 1714 mappers could run simultaneously; some could not begin processing until after others had completed their own processing. This limitation was not true in the case of structured sequence files where only 338 slots were required and they could all run simultaneously. However, one might actually predict superior performance from the unstructured case for the simple fact that it benefits from 800x parallelism while the structured case only benefits from 338x parallelism. We theorize that the explanation for why such behavior was not observed lies in the nonnegligible startup cost of launching a mapper object, i.e., there is a genuine benefit in reusing mappers and there must be some tipping point in this tradeoff where the benefit of additional parallelism is outweighed by the cost of creating mappers for brief computational needs.