The DESI Survey Validation: Results from Visual Inspection of Bright Galaxies, Luminous Red Galaxies, and Emission Line Galaxies

The Dark Energy Spectroscopic Instrument (DESI, Levi et al., 2013) survey is a stage-IV experiment for probing the nature of dark energy. It will obtain tens of millions of spectra of stars, galaxies, and quasars and use the 3D positions of extragalactic sources to detect the signals of baryon acoustic oscillations (e.g., Eisenstein et al., 2005) as a function of redshift. By doing so, DESI will measure the expansion rate of the Universe across cosmic time and constrain the equation of state of dark energy. In addition, DESI will probe the growth of cosmic structure via redshift-space distortion and constrain the summed neutrino masses. All the measurements and constraints will be obtained with unprecedented precision (DESI Collaboration et al., 2016a, b).

To achieve its scientific goals, DESI has been designed to observe four types of extragalactic sources:

• •

  Bright galaxies (bright galaxy survey, BGS) (Hahn et al., 2022; Ruiz-Macias et al., 2020), a galaxy sample whose redshifts can be obtained during the bright time of the observations together with the Milky Way stars (Cooper et al., 2022; Allende Prieto et al., 2020). The BGS consists of two selections, (1) BGS bright — a magnitude limit sample with $r<19.5$, which is similar to the depth of the Galaxy And Mass Assembly (GAMA) survey (Driver et al., 2011) and about 2 magnitude deeper than the main galaxy sample from the Sloan Digital Sky Survey (Strauss et al., 2002), and (2) BGS faint — a sample of fainter $19.5<r<20.175$ galaxies with a color-selection to achieve high redshift efficiency. The designed redshift coverage of BGS is from $z\sim 0$ to $z\sim 0.5$.

• •

  Luminous red galaxies (LRGs) (Zhou et al., 2022, 2020), a population of massive passive galaxies selected based on their colors and brightness. The DESI LRG sample has a significantly higher target density than that of any previous LRG surveys, including the SDSS LRG survey (Eisenstein et al., 2001), Baryon Oscillation Spectroscopic Survey (BOSS) (Reid et al., 2016) and extended Baryon Oscillation Spectroscopic Survey (eBOSS) (Prakash et al., 2016), and also probes higher redshifts with a designed redshift coverage from $z\sim 0.4$ to $z\sim 1.1$.

• •

  Emission line galaxies (ELGs) (Raichoor et al., 2022, 2020), a color-magnitude selected population of star-forming galaxies that emit [O II]$\lambda\lambda 3726,3729$ strong lines. The designed redshift coverage of ELGs is from $z\sim 0.6$ to $z\sim 1.6$. This emission-line galaxy population has been revealed by deep spectroscopic surveys across a few square degrees, e.g. DEEP2 survey (Newman et al., 2013) and has been used as a tracer of the large-scale structure, e.g. the WiggleZ Dark Energy Survey (Blake et al., 2011) and eBOSS ELG survey (Raichoor et al., 2021). The DESI ELG survey will offer a sample for cosmological measurements with 10 times higher target density by observing galaxies up to one magnitude fainter than the eBOSS ELG sample (Raichoor et al., 2022).

• •

  Quasars (QSOs) (Chaussidon et al., 2022; Yèche et al., 2020), a quasar population selected by a Random Forests algorithm based on the color and magnitude information of the sources. The DESI QSO sample is designed to cover a wide range of redshifts from the local Universe to $z\sim 3$. Quasars with $z<2$ will be used primarily as tracers of the matter distribution, while quasars with $z>2$ will be used primarily as background light to probe the matter distribution traced by the Lyman $\alpha$ forest in the foreground. DESI aims to obtain approximately three million quasar spectra and quadruple the number of known quasars obtained from previous surveys (e.g. eBOSS, Dawson et al., 2016; Lyke et al., 2020).

Together, these sources will enable DESI to map the matter density distribution of the Universe across 10 billion years of cosmic time from the local Universe to $z\sim 2.5$. However, to meet the requirements of the DESI design, the sources are selected from a novel parameter space. For example, DESI is the first large-scale survey to observe a high density of ELGs with $1.1<z<1.6$. Similarly, our BGS and LRG surveys will cover fainter objects that have not been extensively observed in previous surveys.

In order to maximize the efficiency of obtaining redshifts of extragalactic sources covering the desired redshift ranges, a set of Survey Validation (SV) observations (DESI Collaboration et al., 2023) were designed to cover a wider range of parameter space with longer exposure times than the Main Survey. This SV dataset is used to explore and define the final target selections for the DESI Main Survey.

To support such analyses, one of the key components of the DESI SV is to construct catalogs of the DESI targets with robust redshift identifications. While an automatic data pipeline (Guy et al., 2022; Bailey et al., 2023) has been developed to reduce the raw data and obtain the best-fit redshifts of spectra, it is crucial to have an additional validation of the dataset which enables us to (1) assess the quality of data; (2) assess the precision and accuracy of the redshift measurements from the pipeline; and (3) offer feedback for improving the performance of the data pipeline, especially during the early phase of the survey. To this end, we have visually inspected (VI) more than 15,000 DESI SV spectra and produced VI catalogs of BGS, LRGs, ELGs and QSOs observed in the DESI SV program. Visual inspection of survey datasets has been demonstrated to be essential during the development of a survey (e.g., Newman et al., 2013; Comparat et al., 2016; Pâris et al., 2018).

In this work, we describe the procedure that we adopt for visually inspecting DESI spectra. We make use of the VI catalogs to explore the target selections, and to validate the design of the DESI Main Survey. Given that galaxy and quasar spectra have different properties, in this work we focus on the VI catalogs of galaxies, including BGS, LRGs and ELGs. The QSO VI catalog and results are described in Alexander et al. (2022).

The structure of the paper is as follows. The procedure of VI is described in Section 2. The data and the VI catalogs are described in Section 3. We show our analysis for validating the DESI design in Section 4 and demonstrate how this VI process assists the development of the DESI pipeline in Section 5. In Section 6, we show some spectra of rare galaxy types identified during the VI process and highlight scientific projects that can be done with the incoming DESI datasets. We summarize in Section 7.

The main goals of visual inspection of DESI spectra are (1) to construct catalogs of different targets with redshift and source-type identifications verified by inspectors and (2) to assess the performance of the data pipeline and the redshift identifications from the Redrock algorithm (Bailey et al., 2023). This VI step enables us to have additional redshift classification information for each spectrum that does not entirely rely on the data pipeline and may be less sensitive to artifacts and systematic features in the data.

The general flow of the VI procedure is as follows:

1. 1.

   Each spectrum has at least two inspectors.

2. 2.

   For each spectrum, inspectors report four key features:

   1. (a)

      Redshift;

   2. (b)

      Quality of the redshift with 5 discrete values from 0 to 4 with 4 being the highest quality and 0 being the poorest quality;

   3. (c)

      Type of the source, i.e. star, galaxy, or quasar;

   4. (d)

      Any artifacts or systematic features in the spectrum;

   5. (e)

      Plus any extra comments, e.g., there are two objects in the spectrum.

3. 3.

   The outputs from the inspectors are compared. For each spectrum, the multiple VI reports are automatically merged into the final VI results, if the outputs of all the inspectors satisfy the following conditions:

   1. (a)

      The difference of the redshifts, $dz$ from inspectors $i$ and $j$, is within $0.0033$ ($990\rm\,km/s$),

      $$dz_{\rm VI}=\frac{|z_{{\rm VI,}i}-z_{{\rm VI,}j}|}{(1+z_{{\rm VI,}i})}<0.0033;$$

      (1)

   2. (b)

      the difference of the quality values is within 1, i.e. $|{\rm VI\ quality}_{i}-{\rm VI\ quality}_{j}|\leq 1$;

   3. (c)

      The VI spectype from all of the inspectors is the same;

   The final redshift is the mean of the redshifts, the final quality value is the mean of the quality values, and the final source type (best VI type) is the spectype identified by all of the inspectors.

4. 4.

   If any of the above conditions is not met, a merger inspects the spectrum again and determines the final VI results. Note that this merger might occasionally be one of the inspectors.

One of the key steps in the VI procedure is to assess the quality of the VI redshift. The criteria for the quality are:

• •

  Quality 4: confident classification with two or more secure features, e.g., spectra with multiple emission lines and absorption lines;

• •

  Quality 3: probable classification with at least one secure spectral feature and continuum or many weak spectral features, e.g., spectra with a strong emission line feature and weak Balmer series absorption lines;

• •

  Quality 2: possible classification with one strong spectral feature but unsure what it is;

• •

  Quality 1: unlikely classification with some signal but features are unidentified;

• •

  Quality 0: nothing there, no signal.

Figures 1, 2, and 3 show example BGS, LRGs, and ELGs spectra respectively, of each of these different qualities.

Given the above definitions, we consider the final VI redshift with an overall VI quality $\geq 2.5$ as a robust VI redshift. Sources with an overall VI quality $<2.5$ typically indicate that the corresponding spectra are too shallow to detect spectral features or the sources are at redshift regions with spectral features beyond the wavelength coverage of the DESI spectrograph, e.g. galaxies at $z>1.62$ where the [O II]$\lambda\lambda 3726,3729$ doublet is beyond the wavelength coverage of the instrument.

Finally, we summarize our dedicated tool, Prospect, for performing the VI and our efforts of recruiting and training visual inspectors in the Appendix.

The data used in this work was obtained by the DESI instrumentation with a 3.2 degree diameter field of view and 5020 robotic fiber positioners installed on the 4-meter Mayall Telescope at Kitt Peak National Observatory (Abareshi et al., 2022; Silber et al., 2023; Miller et al., 2022). The DESI spectrograph covers the optical wavelength region from 3600 $\rm\AA$ to 9800 $\rm\AA$ with a spectral resolution ranging from $\sim 2000$ in the shortest wavelength channel to $\sim 5000$ in the longest wavelength channel. In addition, the DESI targets are selected based on the photometric properties from the DESI Legacy Imaging Surveys (Schlegel et al., 2023; Zou et al., 2017; Dey et al., 2019). The target selection procedure is summarized in Myers et al. (2022). The pipelines used to automatically assign fibers to targets and used to plan and optimize the observational procedure are summarized in Raichoor et al. (2023) and Schlafly et al. (2023).

During the DESI SV observations, several fields were selected to have approximately $10$ times longer exposure time than the designed nominal exposure time of the DESI Main Survey (DESI Collaboration et al., 2023). This is for detecting spectral features that might not be detected with the normal exposure time and for validating the exposure time of the DESI Main Survey. Because the observational conditions, such as airmass and galactic extinction of the field, vary, the exposure time for each observation was calibrated into the effective exposure time, $\rm T_{eff}$, that corresponds to the exposure time with airmass 1, zero galactic extinction, $1^{\prime\prime}.1$ seeing (full width half maximum), and zenith dark sky (DESI Collaboration et al., 2023). In addition, to explore the target selection schemes that maximize the efficiency of the survey, galaxy targets were selected from a wider magnitude and color space during the SV observations than the range planned for the Main Survey.

To support these analyses, we have conducted two visual inspection campaigns. During the first one, we visually inspected $\sim 17,000$ deep co-added spectra, which were based on an early version of the data reduction pipeline¹¹1Software release 20.12, including desispec 0.36.0 and redrock 0.14.4, from SV observations, including BGS, LRG, and ELG targets. Through the visual inspection, we identified some spectral features due to instrument effects, calibration issues, or sky residuals which will be discussed in Section 5. These features prevented the inspectors from recovering the true redshifts of the sources. Therefore, in the second campaign, we performed the visual inspection for sources with low VI quality values identified during the first one but with the spectra reduced by the latest version²²2Software release 22.2, including desispec 0.51.13 and redrock 0.15.4 of the data reduction pipeline. The sources included in the second campaign are

1. 1.

   all SV BGS galaxies with VI quality$<=2.5$,

2. 2.

   all SV LRG galaxies with VI quality$<=2.5$, and

3. 3.

   all ELGs in the Main Survey selection with VI quality$<=2.5$.

Note that we only included ELGs that are in the Main Survey selection because the main goal of the analysis is to use the VI results to test the Main Survey target selection and performance. With this second campaign, we recovered a few percent of redshifts that were identified with low quality previously. In the final VI catalogs, we adopted the second campaign VI results for sources included in the second campaign and the first campaign VI results for the rest. The basic information of the VI fields and the statistics are summarized in Table 1 and 2. In the following, we describe the VI properties of each target type.

We use the VI galaxy catalogs obtained from DESI spectra with long exposure times to validate the performance of the algorithm, Redrock (Bailey et al., 2023), that DESI adopts for recovering the redshifts of galaxies at the designed nominal exposure times, i.e. effective exposure time $\sim 1000$s for ELGs (Raichoor et al., 2022) and LRGs (Zhou et al., 2022) and $\sim 180$s for bright galaxies (Hahn et al., 2022). For those galaxies with deep co-added spectra, we use all the data from individual exposures to produce several spectra with similar exposure times to the designed exposure time of the Main Survey. For ELGs and LRGs, we use spectra with $\rm 800<T_{eff}<1200s$ and for BGS, we use spectra with $\rm 160<T_{eff}<200s$. In this way, one galaxy target has multiple redshift measurements from the Redrock algorithm, and we can then compare the redshift measurements from the pipeline with the VI redshift to assess the performance of the DESI observation and data pipeline.

There are two key quantities from Redrock used in this data assessment:

• •

  Redrock redshift: the best-fit redshift of the spectrum from Redrock,

• •

  $\Delta\chi^{2}$: the difference between the $\chi^{2}$ value of the 2nd best-fit model and the $\chi^{2}$ value of the 1st best-fit model, $\chi^{2}_{2nd}-\chi^{2}_{1st}$. A larger $\Delta\chi^{2}$ means that the 1st best-fit model better describes the data than the 2nd best-fit model. The $\Delta\chi^{2}$ parameter is an important parameter for identifying robust redshifts as described below.

Visually inspecting the DESI spectra is a crucial step in identifying unforeseen problems in the dataset and to further help the development and improvement of the data pipeline. To demonstrate the importance of VI, here we list two major issues which were identified by VI:

• •

  Artificial excess of flux in z-band: While visually inspecting spectra processed by an early version of the pipeline, we identified that 1-2% of BGS spectra showed a non-physical increase in flux in the z-band, as shown in the left panel of Figure 10. This non-physical feature led to erroneous estimates of the redshift using the Redrock algorithm as it simulates a strong $\rm 4000\AA$ break. After investigation, we identified that this feature is due to the contamination of flux from adjacent fibers targeting bright stars. The data pipeline was updated to properly model this effect (Guy et al., 2022). The right panel of Figure 10 shows the spectra of the same targets as shown in the left panel but processed by the latest version of the pipeline. This pipeline improvement reduces the catastrophic failure rate of BGS redshifts by a factor of three.

• •

  Spectra affected by a CCD issue: During the VI process, we identified spectra with unexpected spectral features around $8000\rm\,\AA$ as shown in Figure 11, causing Redrock to fit incorrect redshifts. About 0.5% of spectra reduced by an early version of the DESI pipeline have such features. We find that these problematic spectra tend to be obtained from a certain set of fibers and are due to a region of CCD with issues, which were not identified and properly masked. The spectra affected by this problem are now masked by the data pipeline and will not be used in the redshift catalogs (Guy et al., 2022).

These results demonstrate that the visual inspection process is crucial for identifying unexpected problematic features which can affect the quality of the data product. Without VI in the early development of the project, it would be difficult to find such problems in the dataset.

In addition to revealing nonphysical spectra, the visual inspection process also helped us identify unexpected sources that live in the same parameter spaces as the DESI targets. Here we show two types of rare sources that were identified during the VI process:

• •

  Candidates of Lyman $\alpha$ emitters: We find about 60 spectra that show one strong, asymmetric emission line feature at about 4000-5000 $\rm\AA$ in observed wavelengths. These spectra are consistent with the spectra of Lyman $\alpha$ emitters, a star-forming galaxy population that produces a strong Lyman $\alpha$ emission line (e.g., Leclercq et al., 2017; Ouchi et al., 2020). Figure 12 shows examples of these spectra. This detection demonstrates that it is possible to use DESI to detect Lyman $\alpha$ emitters and potentially use them as mass tracers at $z>2$ for probing structure formation (e.g., Hill et al., 2008).

• •

  Spectra with two redshifts: We also identify about 200 spectra comprised of two objects at two distinct redshifts, examples of which are shown in Figure 13. The purple dashed lines correspond to the spectral features of one galaxy and the blue dashed lines correspond to the features of another galaxy. These sources are lensing system candidates (e.g., Bolton et al., 2008; Brownstein et al., 2012; Talbot et al., 2018). If confirmed, these sources can be used to probe the stellar mass and dark matter mass around foreground galaxies, especially for the halos of star-forming galaxies at $z>1$ which are rarely explored via strong lensing.

We note that for most of the Lyman $\alpha$ emission spectra, Redrock fails to obtain the correct redshifts due to the fact that this unusual spectral feature is not included in the templates. We plan to add these VI identified spectra of potential Lyman $\alpha$ emitters into the Redrock templates. By doing so, future versions of Redrock may automatically identify similar spectra. For the spectra with two redshifts identified by VI, Redrock reports one of the redshifts that has dominated spectral features.

During the Survey Validation of the DESI survey, we visually inspected approximately 17,000 DESI spectra of galaxy targets and thereby compiled VI redshift catalogs. These catalogs enable us to perform various statistical measurements for characterizing the performance of the DESI observational system. Here we summarize our main results:

• •

  We show that the redshifts of $97\%$ of BGS, $99\%$ of LRGs and $76\%$ of ELGs with long-exposure spectra observed in the SV observations can be identified from the VI process. These VI redshift catalogs are used to help define the selections for the Main Survey. In the Main Survey selections, the redshifts of $>99\%$ of BGS and LRGs can be visually identified and the redshifts of $\sim 85\%$ of LOP ELGs and $\sim 99\%$ of VLO ELGs can be visually identified.

• •

  We use the VI redshift catalogs to characterize the performance of the DESI Main Survey design and the Redrock algorithm. The results show that with the current selected criteria combining $\Delta\chi^{2}$ and other parameter values, the DESI survey can obtain samples of BGS, LRGs, and ELGs with purity $>99\%$.

• •

  Utilizing the spectra of the same targets obtained from multiple independent exposures, we quantify the precision of the redshift measurements from the Redrock algorithm. The results show that the precision of the redshift measurements of BGS and ELGs is about 10 km/s and that of LRGs is about 40 km/s.

• •

  We empirically test the accuracy of the redshift measurements from the Redrock algorithm by comparing the Redrock redshifts to the redshift measurements of the DEEP2 survey, using galaxies observed in both surveys. The results show that the systematic redshift differences from the two surveys are less than 10 km/s for BGS, LRGs, and ELGs.

• •

  Via the VI process, we identify spectra with non-physical features. These identifications help us reveal and fix issues in both the software and hardware of the early DESI data pipeline that were not anticipated. In addition, we find spectra of rare astronomical objects, including candidates of Lyman $\alpha$ emitters and spectra with two redshifts as potentially strong lensing candidates. These spectra illustrate the scientific potential offered by the DESI dataset.

These results illustrate the utility and necessity of performing systematic visual inspections of astronomical datasets, especially during the early phase of sky surveys. As demonstrated in this work, the VI redshift information is essential for testing the redshift measurements from the pipeline and for identifying both spurious features in the reduced spectra and unexpected astronomical phenomena.

These VI catalogs will have a lasting impact. They will be used to perform the same statistical measurements every time the DESI pipeline is upgraded throughout the whole DESI operation. By doing so, we can confirm the stability of the DESI pipeline and assess the improvement quantitatively. The catalogs can also be used to train novel machine learning models of detecting and identifying spectral features. Finally, with this experience of visually inspecting DESI spectra, we recommend that upcoming spectroscopic surveys, such as PFS (Takada et al., 2014), SDSS-V (Kollmeier et al., 2017), 4MOST (de Jong et al., 2016) and MOONS (Cirasuolo & MOONS Consortium, 2016), deploy similar VI efforts during the survey operations.

All data points shown in the figures are available at Zenodo https://doi.org/10.5281/zenodo.6618765.

TWL is supported by the Ministry of Science and Technology (MOST 111-2112-M-002-015-MY3), the Ministry of Education, Taiwan (MOE Yushan Young Scholar grant NTU-110VV007 and NTU-110VV007-2), National Taiwan University research grants (NTU-CC-111L894806, NTU-111L7318, and NTU-112L7302), and NSF grant AST-1911140. DMA acknowledges the Science Technology and Facilities Council (STFC) for support through grant code ST/T000244/1.

This research is supported by the Director, Office of Science, Office of High Energy Physics of the U.S. Department of Energy under Contract No. DE–AC02–05CH11231, and by the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility under the same contract; additional support for DESI is provided by the U.S. National Science Foundation, Division of Astronomical Sciences under Contract No. AST-0950945 to the NSF’s National Optical-Infrared Astronomy Research Laboratory; the Science and Technologies Facilities Council of the United Kingdom; the Gordon and Betty Moore Foundation; the Heising-Simons Foundation; the French Alternative Energies and Atomic Energy Commission (CEA); the National Council of Science and Technology of Mexico (CONACYT); the Ministry of Science and Innovation of Spain (MICINN), and by the DESI Member Institutions: https://www.desi.lbl.gov/collaborating-institutions.

The authors are honored to be permitted to conduct scientific research on Iolkam Du’ag (Kitt Peak), a mountain with particular significance to the Tohono O’odham Nation.

The visual inspection is a collaborative effort contributed by DESI members. Here we summarize the key tasks for performing and coordinating the VI effort.

VI tool — A dedicated tool, Prospect³³3https://github.com/desihub/prospect, was developed to produce HTML pages containing the spectra and Redrock products to be inspected. Individual HTML pages typically include 50 spectra and are directly accessible by collaboration members from any web browser. Figure 14 shows an example of the HTML page. The main panel displays an individual spectrum, which can be smoothed by the inspector with a variable width Gaussian kernel. The associated noise at each wavelength, as derived from the processing pipeline, can be overlaid, as well as models matching the best fits from Redrock, and sets of emission and/or absorption lines. The associated imaging from the Legacy Survey (Dey et al., 2019) is displayed next to the spectrum. Inspectors can view relevant information from Redrock catalogs, including the ZWARN flag⁴⁴4https://github.com/desihub/redrock/blob/master/py/redrock/zwarning.py, which indicates whether or not the spectrum has issues found by the pipeline, and the best fit $\chi^{2}$ value with respect to second best fit. They can dynamically adjust the redshift for the displayed model and lines. The bottom left panel of the VI pages allows the inspectors to report relevant information, which is saved into an ASCII file for each set of 50 spectra.

Recruiting and training — To perform the VI, we recruited volunteers from the DESI collaboration. More than 50 DESI members had contributed to the visual inspection of galaxy spectra. In order to obtain consistent results from each inspector, we prepared materials for training the inspectors, including text documents with instructions and videos of experts performing VI and explaining the decision for each spectrum. Meetings were organized for the training as well. All inspectors were required to go through the materials before conducting the VI. After finishing the materials, they were asked to inspect 100 spectra for a given target type of their interest. Their results were then compared to the results from VI experts. If the inconsistent rates between the two results are lower than $2\%$ for BGS, and $10\%$ for ELGs and LRGs, the volunteers can start to perform VI after they reviewed the inconsistent results and agreed with the results. If the inconsistent rates were not lower than the thresholds, before starting the VI, the inspectors were asked to go through the training materials again and they would need to make sure that they understood the inconsistency. Each spectrum was at least inspected by two DESI members and one of the inspectors was an experienced astronomer who had worked on the specific target type. The results from the two inspectors were merged by the VI coordinators into the final VI results and the spectra with inconsistent results were resolved by the merger as described in the main text.