A multispeaker dataset of raw and reconstructed speech production real-time MRI video and 3D volumetric images^††thanks: Article submitted to Nature Scientific Data.

Seventy-five healthy subjects (40 females and 35 males; 49 native and 26 non-native American English speakers; age 18-59 years) were included in this study. Each participant filled out a questionnaire on basic demographic information including birthplace, cities raised and lived, and first and second languages. Demographics are summarized in Table LABEL:tab1. All subjects had normal speech, hearing, and reading abilities, and reported no known physical or neurological abnormalities. All participants were cognizant of the nature of the study, provided written informed consent, and were scanned under a protocol approved by the Institutional Review Board of the University of Southern California (USC). The data were collected at the Los Angeles County – USC Medical Center between January 24, 2016 and February 24, 2019.

Three types of MRI data were recorded for each subject: (i) dynamic, real-time MRI of the vocal tract’s mid-sagittal slice at 83 frames per second during production of a comprehensive set of scripted and spontaneous speech material, averaging 17 minutes per subject, along with synchronized audio; (ii) static, 3D volumetric images of the vocal tract, captured during sustained production of sounds from the full set of American English vowels and continuant consonants, 7 seconds each; (iii) T2-weighted volumetric images at rest position, capturing fine detail anatomical characteristics of the vocal tract (See Figure 1).

All data were collected using a commercial 1.5 Tesla MRI scanner (Signa Excite, GE Healthcare, Waukesha, WI) with gradients capable of 40 mT/m amplitude and 150 mT/m/ms slew rate. A custom 8‐channel upper airway receiver coil array [21], with four elements on each side of the subject’s cheeks, was used for signal reception. Compared to commercially available coils that are designed for neurovascular or carotid artery imaging, this custom coil has been shown to provide 2-fold to 6-fold higher signal-to-noise-ratio (SNR) efficiency in upper airway vocal tract regions of interests including tongue, lips, velum, epiglottis, and glottis. The subjects, while imaged, were presented with scripts of the experimental stimuli via a projector-mirror setup [22]. Acoustic audio data were recorded inside the scanner using commercial fiber-optic microphones (Optoacoustics Inc., Yehuda, Israel) concurrently with the RT-MRI data acquisition using a custom recording setup [23].

The stimuli were designed to efficiently capture salient, static and dynamic, and articulatory and morphological aspects of speech production of American English in a single 90-minute scan session.

Table 2 lists the speech stimuli used for the RT-MRI data collection of the first 45-minute sub scan session. Each individual task was designed to be performed within 30 seconds at a normal speaking rate, however the actual recordings varied in duration depending on the length of the task and the natural speaking rate of the individual. The stimulus set contained material to elicit both scripted speech and spontaneous speech. The scripted speech tasks were consonant production in symmetric vowel-consonant-vowel context, vowels /V/ produced between the consonants /b/ and /t/, i.e., in /bVt/ contexts, four phonologically rich sentences [24], and three reading passages commonly used in speech evaluation and linguistic studies [25, 26, 27]. Scripted instructions to produce several gestures were also included. These gestures were clenching, wide opening of mouth, yawning, swallowing, slow production of the sequence /i/-/a/-/u/-/i/, tracing of the palate with the tongue tip, and singing “la” at their highest and lowest pitches. The stimuli in the scripted speech were repeated twice. The spontaneous speech tasks were to describe the content and context of five photographs (Supplemental Figure 1) and to answer five open-ended questions (e.g., “What is your favorite music?”). The full scripts and questions are provided in Table 3.

Table 4 lists the speech stimuli used for the 3D volumetric data collection of a 30-minute part of the scan session. Each individual task was designed for a subject to sustain vowels, consonant sounds, or to maintain postures for 7 seconds. The stimulus set contained consonant production in symmetric vowel-consonant-vowel context, vowels /V/ produced between the consonants /b/ (or /p/) and /t/, and production of several postures. All of the stimuli were repeated twice.

RT-MRI acquisition was performed using a 13-interleaf spiral-out spoiled gradient-echo pulse sequence [28]. This is an efficient scheme for sampling MR measurements, each with the different initial angle being interleaved by the bit-reversed order in time [29]. The 13 spiral interleaves, when collected together, fulfil the Nyquist sampling rate. Imaging parameters were: repetition time = 6.004 ms, echo time = 0.8 ms, field-of-view (FOV) = 200 $\times$ 200 mm², slice thickness = 6 mm, spatial resolution = 2.4 $\times$ 2.4 mm² (84 $\times$ 84 pixels), receiver bandwidth = $\pm$ 125 kHz, flip angle = 15^∘. Imaging was performed in the mid-sagittal plane, which was prescribed using a real‐time interactive imaging platform [30] (RT-Hawk, Heart Vista, Los Altos, CA). Real-time visualization was implemented within the custom platform by using a sliding window gridding reconstruction [31] to ensure subject’s compliance with stimuli and to detect substantial head movement.

Acquisition was divided into 20-40 second task intervals presented in Table 2, each followed by a pause of the same duration so as to allow enough a brief break for the subject prior to the next task and to avoid gradient heating.

The dataset provides one specific image reconstruction that has been widely used in speech production research [21, 28]. This image reconstruction involves optimizing the following cost function:

where A represents the encoding matrix that models for the non-uniform fast Fourier transform and the coil sensitivity encoding, m is the dynamic image time series to be reconstructed, d is the acquired multiple-coil raw data, $\lambda$ is the regularization parameter, $\bigtriangledown_{t}$ is the temporal finite difference operator, and $\|\cdot\|_{2}^{2}$ and $\|\cdot\|_{1}$ are l₂ and l₁ norms, respectively. The coil sensitivity maps were estimated using the Walsh method [32] from temporally combined coil images. The regularization term encourages voxels to be piecewise constant along time. This regularization has been successfully applied to speech RT-MRI [21, 33, 34] and a variety of other dynamic MRI applications where the primary features of interest are moving tissue boundaries [35, 36, 37].

Our reference implementation solves Equation (1) using nonlinear conjugate gradient algorithm with Fletcher-Reeves updates and backtracking line search [38]. The algorithm was terminated either at 150 maximum iterations or the step size fell below $<$ 1e-5 during line search. Reference images were reconstructed using 2 spiral arms per time frame, resulting in a temporal resolution of 83.28 frames per second. This temporal resolution is enough to capture important articulator motions1, but reconstruction at different temporal resolutions is also possible with the provided dataset by adjusting the number of spiral arms per time frame. The algorithm was implemented in both MATLAB (The MathWorks, Inc., Natick, MA) and Python (Python Software Foundation, https://www.python.org/). The provided image reconstruction was performed in MATLAB 2019b on a Xeon E5-2640 v4 2.4GHz CPU (Intel, Santa Clara, CA) and a Tesla P100 GPU (Nvidia, Santa Clara, CA). Reconstruction time was 160.69 $\pm$ 1.56 ms per frame. Parameter selection of $\lambda$ is described in the technical validation section below.

One main microphone was positioned $\sim$ 0.5 inch away from the subject’s mouth. The microphone signal was sampled at 100 kHz each. The data were recorded on a laptop computer using a National Instruments NI-DAQ 6036E PCMCIA card. The audio sample clock was hardware-synchronized to the MRI scanner’s 10 MHz master clock. The audio recording was started and stopped using the trigger pulse signal from the scanner. The real-time audio data acquisition routine was written in MATLAB. Audio was first low-pass filtered and decimated to a sampling frequency of 20 kHz. The recorded audio was then enhanced using a normalized least-mean-square noise cancellation method [23] and was aligned with the reconstructed MRI video sequence to aid linguistic analysis.

An accelerated 3D gradient-echo sequence with Cartesian sparse sampling was implemented to provide static high-resolution images of the full vocal tract during sustained sounds or postures. Imaging parameters were: repetition time = 3.8 ms, spatial resolution = 1.25 $\times$ 1.25 $\times$ 1.25 mm3, FOV = 200 $\times$ 200 $\times$ 100 mm3 (respectively in the anterior-posterior, superior-inferior, and left-right directions), image matrix size = 160 $\times$ 160 $\times$ 80, and flip angle = 5^∘. The central portion (40 $\times$ 20) of the k_y-k_z space was fully sampled to estimate the coil sensitivities from the data itself. The outer portion of the k_y-k_z space was sampled using a sparse Poisson-Disc sampling pattern, which together resulted in 7-fold net acceleration, and a total scan time of 7 seconds.

Data were acquired while the subjects sustained for 7 seconds a sound from the full set of American English vowels and continuant consonants (Table 4). The stimuli were presented to the subject via a projector-mirror setup, and upon hearing a “GO” signal given by the scanner operator, a subject started to sustain the sound or posture; this was followed by the operator triggering the acquisition manually. Subjects sustained the postures as long as they could hear the scanner operating. A recovery time of 5-10 seconds was given to the subject between the stimuli.

Image reconstruction was performed off-line by a sparse-SENSE constrained reconstruction similar to the optimization problem shown in Equation (1). In contrast to Equation (1) , isotropic spatial total variation constraints were used [28, 39]. The reconstruction was achieved using the open-source Berkeley Advanced Reconstruction Toolbox (BART) [40]; the image reconstruction was performed in MATLAB using GPU acceleration.

Fast spin echo-based sequence was performed to provide high-resolution T2-weighted images with fine detail anatomical characteristics of the vocal tract at rest position. Imaging was run to obtain full sweeps of the vocal tract in the axial, coronal, and sagittal orientations, for each of which the number of slices ranges from 29 to 70, depending on the size of the vocal tract. Imaging parameters were: repetition time = 4600 ms, echo time = 120 - 122 ms, slice thickness = 3 mm, in-plane FOV = 300 $\times$ 300 mm², in-plane spatial resolution = 0.5859 $\times$ 0.5859 mm², number of averages = 1, echo train length = 25, and scan time = approximately 3.5 minutes per orientation.

Raw RT-MRI data are provided in the vendor-agnostic MRD format (previously known as ISMRMRD) [42, 43], which stores k-space MRI measurements, k-space location tables, and sampling density compensation weights. Parameters for the acquisition sequence are contained in the file header information. In addition, this dataset includes reconstructed image data for each subject and task in HDF5 format, audio files in WAV format, and videos in MPEG-4 format.

For each subject, RT-MRI raw data is contained in the subfolder raw/ (e.g., sub001/2drt/raw/), reconstructed image data in recon/, co-recorded audio (after noise cancellation) in audio/, and reconstructed speech videos with aligned audio in video/. Table 5 summarizes the data structure and naming conventions for this dataset.

3D volumetric MRI data contains reconstructed image data and imaging parameters in MAT format (MATLAB, The MathWorks, Inc., Natick, MA) in the subfolder recon/ (e.g., sub001/3d/recon/) and a mid-sagittal slice image in PNG format in the subfolder snapshot/.

T2-weighted image data from axial, coronal, and sagittal orientations are stored in Digital Imaging and Communications in Medicine (DICOM) format in the subfolder dicom/ (e.g., sub001/t2w/dicom/).

The imaging DICOM files were de-identified using the Clinical Trial Processor (CTP) developed by the Radiological Society of North America (RSNA) [44]. Specifically, data anonymization was completed using a command line tool developed in the Java programming language [45]. All images followed the standardized DICOM format and some of the attributes were removed or modified to preserve privacy of the subjects, specifically: PatientName was modified to follow the subXYZ pattern, and the original study dates were shifted by the same offset for all subjects.

We provide presentation slides that contain the experimental stimuli including scripts and pictures used for the visualization to the subjects, in PPT format in Stimuli.ppt. Demographic information for each subject is contained in XLSX format in Subjects.xlsx. This meta file contains sex, race, age (at the time of scan), height (cm), weight (kg), origin, birthplace, cities raised and lived, L1 (first language), L2 (second language, if any), L3 (third language, if any), and the first language and birthplace of each subject’s parents.

Additionally, meta information for each subject and RT-MRI task is contained in JSON format in metafile_public_$<$timestamp$>$.json. For each subject, we include the following demographic information: 1) L1 (first language), 2) L2 (second language, if any), 3) sex (M/F), 4) age (years at the time of scan). Further, visual and audio quality assessment scores are provided using a 5-level Likert scale for each subject stratified into categories: off-resonance blurring (1, very severe; 2, severe; 3, moderate; 4, mild; 5, none), video SNR (1, poor; 2, fair; 3, good; 4, very good; 5, excellent), aliasing artifacts (1, very severe; 2, severe; 3, moderate; 4, mild; 5, none), and audio SNR (1, poor; 2, fair; 3, good; 4, very good; 5, excellent). Specifically, an MRI expert with 6 years of experience in reconstructing and reading speech RT-MRI examined all the RT-MRI videos reconstructed for each subject and assessed and scored their visual and audio quality. For each task of the individual subjects, the information about task’s index, name, existence of file, and notes taken during data inspection by inspectors are also contained in the meta file.

Four inspectors manually performed qualitative data inspection for the datasets. After reconstruction, all images were converted from HDF5 format to MPEG-4 format, at which time co-recorded audio (WAV format) was integrated. All qualitative data inspection was performed manually based on MPEG-4 format videos. Included in the dataset were files that met all of the following criteria:

1. 1.

   The video (MPEG-4) exists (no data handling failure).

2. 2.

   The audio recording (WAV) exists (no data handling failure).

3. 3.

   The video and audio are synchronized (based on inspection by a human).

After the inspection, 75 subjects were included in this dataset. It should be noted that although three subjects (sub18, sub74, and sub75) present severe radio-frequency-interference artifacts that were later determined to be from a leak in the MRI radio-frequency cage, we included those three subjects in this dataset in the hopes of potentially facilitating development of a digital radio-frequency interference correction method. Files that did not exist (criteria 1) were annotated for each task and subject in the meta file
(i.e., metafile_public_$<$timestamp$>$.json).

Figure 2 shows representative examples of the data quality from three subjects that are included in this dataset (sub35, sub41, and sub58). Note that all 75 subjects exhibited acceptable visualization of all soft tissue articulators. However, two types of artifacts were commonly observed:

1. 1.

   Blurring artifact due to off-resonance (green arrows, Figure 2c): This artifact appears as blurring or signal loss predominantly adjacent to air-tissue boundaries that surround soft tissue articulators. It is induced in spiral imaging by rapid changes of local magnetic susceptibility between the air and tissue. This artifact correction is still an active research area, including methods for a simple zeroth order frequency demodulation to advanced model-based [14, 46, 47] and data-driven approaches [15, 48]. At present, we perform zeroth order correction during data acquisition as part of our routine protocol.

2. 2.

   Ringing artifact due to aliasing (yellow arrows, Figure 2c): This artifact appears as an arc-like pattern centered at the bottom-right corner outside the FOV. It is caused by a combination of gradient non-linearity and non-ideal readout low-pass filter when a strong signal source appears outside the unaliasing FOV. This artifact does not overlap with the articulator surfaces that are most important in the study of speech production. However, avoiding and/or correcting this artifact would improve overall image quality and potentially enable the use of a smaller FOV.

Figure 3 shows representative examples of the diverse speech stimuli that are included in this dataset. The intensity vs. time profiles visualize the first 20 seconds of four representative stimuli from sub35, shown in Figure 2a. These examples show the variety of patterns and speed of movement of the soft tissue articulators observed within a speaker as a function of the speech stimuli performed. Figure 4 contains a histogram of the speaking rate in read sentences [49]. The overall mean speaking rate was 149.2 ± 31.2 words per minute. The statistics are calculated from the read “shibboleth,” “rainbow,” “grandfather[1-2],” and “northwind[1-2]” stimuli for all subjects. These selected stimuli are composed of read full sentences.

Figure 5 illustrates variability within the same speech stimuli across different speakers. The image time profiles correspond to the period of producing the first sentence “She had your dark suit in greasy wash water all year” in the stimulus “shibboleth” from sub35 and sub41. Although both speakers share the critical articulatory events (see green arrows), the timing and pattern vary depending on the subject.

We performed a parameter sweep and qualitative evaluation on a subset of the data to select a regularization parameter for the provided reconstructions. Ten stimuli from ten different subjects were randomly selected. The regularization parameter $\lambda$ was swept in the range between 0.008C and 1C in a logarithmic scale. Here, C represents the maximum intensity of the zero-filled reconstruction of the acquired data. Figure 6 shows a representative example of the impact of $\lambda$ on the reconstructed image quality. A small $\lambda$ (= 0.008C) exhibits high noise level in the reconstruction (top, Figure 6). A higher $\lambda$ (= 0.8C) reduces the noise level but exhibits unrealistic temporal smoothing as shown in the intensity vs. time profiles (yellow arrows, Figure 6). The optimal parameter 0.08C was selected by consensus among four experts in the area of MRI image reconstruction and/or speech production RT-MRI. Once the $\lambda$ was optimized for the subset of the data, reconstruction was performed for all datasets. We have empirically observed that the choice of $\lambda$ appears robust across all of the datasets.