VideoSet: A Large-Scale Compressed Video Quality Dataset Based on JND Measurement

The VideoSet consists of 220 source video clips, each of which has a duration of 5 seconds. We show thumbnails images for 30 representative video clips in Fig 2. The source video clips were collected from publicly available datasets in [6, 7, 8]. The original sequences have multiple spatial resolutions (i.e., $4096\times 2160$, $4096\times 1714$, $3840\times 2160$), frame rates (i.e., 60, 30, 24) and color formats (i.e., YUV444p, YUV422p, YUV420p). We pay special attention to the selection of these source video clips to avoid redundancy and enrich diversity of selected contents.

After content selection, we process each 5-second video clip to ensure that they are in similar format. Their formats are summarized in Table 1, where the first column shows the names of the source video material of longer duration and the second column indicates the number of video clips selected from each source material. The third, fourth and fifth columns describe the frame rate, the spatial resolution and the pixel format, respectively. They are further explained below.

• •

  Frame Rate. The frame rate affects the perceptual quality of certain contents significantly [9]. Contents of a higher frame rate (e.g. 60fps) demand a more powerful CPU and a larger memory to avoid impairments in playback. For this reason, if the original frame rate is 60fps, we convert it from 60fps to 30fps to ensure smooth playback in a typical environment. If the original frame rate is not greater than 30fps, no frame rate conversion is needed.

• •

  Spatial Resolution. The aspect ratio of most commonly used display resolutions for web users is $16:9$. For inconsistent aspect ratios, we scale them to $16:9$ by padding black horizontal bars above and below the active video window. As a result, all video clips are of the same spatial resolution – $3840\times 2160$.

• •

  Pixel Format. We down-sample the trimmed spatial resolution $3840\times 2160$ (2160p) to four lower resolutions. They are: $1920\times 1080$ (1080p), $1280\times 720$ (720p), $960\times 540$ (540p) and $640\times 360$ (360p) for the subjective test in building the VideoSet. In the spatial down-sampling process, the lanczos interpolation [10] is used to keep a good compromise between low and high frequencies components. Also, the $4:2:0$ chroma sampling is adopted for maximum compatibility.

It is worthwhile to point out that 1080p and 720p are two most dominant video formats on the web nowadays while 540p and 360p are included to capture the viewing experience on tablets or mobile phones. After the above-mentioned processing, we obtain 880 uncompressed sequences in total.

We use the H.264/AVC [5] high profile to encode each of the 880 sequences, and choose the constant quantization parameter (CQP) as the primary bit rate control method. The adaptive QP adjustment is reduced to the minimum amount since our primary goal is to understand a direct relationship between the quantization parameter and perceptual quality. The encoding recipe is included in the read-me file of the released dataset.

The QP values under our close inspection are between $[8,47]$. It is unlikely to observe any perceptual difference between the source and coded clips with a QP value smaller than $8$. Furthermore, coded video clips with a QP value larger than $47$ will not be able to offer acceptable quality. On the other hand, it is ideal to examine the full QP range; namely, $[0,51]$, in the subjective test since the JND measure is dependent on the anchor video that serves as a fixed reference.

To find a practical solution, we adopt the following modified scheme. The reference is losslessly encoded and referred to as $QP=0$. We use the source $QP=0$ to substitute all sequences with a QP value smaller than 8. Similarly, sequences with a QP larger value than 47 are substituted by that with $QP=47$. The modification has no influence on the subjective test result. This will become transparent when we describe the JND search procedure in Sec. 3.2. By including the source and all coded video clips, there are $220\times 4\times 52=45,760$ video clips in the VideoSet.

The subjective test was conducted in six universities in the city of Shenzhen in China. There were 58 stations dedicated to the subjective test. Each station offered a controlled non-distracting laboratory environment. The viewing distance was set as recommended in ITU-R BT.2022. The background chromaticity and luminance were set up as an environment of a common office/laboratory. We did not conduct monitor calibration among different test stations, yet the monitors were adjusted to a comfortable setting to test subjects. On one hand, the uncalibrated monitors provided a natural platform to capture the practical viewing experience in our daily life. On the other hand, the monitors used in the subjective test were profiled for completeness. Monitor profiling results are given in Fig. 3 and summarized in Table 2. As shown in Table 2, most stations comply with ITU recommendations.

We indexed each video clip with a content ID and a resolution ID and partitioned 880 video clips into 58 packages. Each package contains 14 or 15 sequence sets of a content/resolution ID pair, and each sequence set contains one source video clip and its all coded video clips. One subject can complete one JND point search for one package in one test session. The duration of one test session was around 35 minutes with a 5-minute break in the middle. Video sequences were displayed in their native resolution without scaling on the monitor. The color of the inactive screen was set to light gray.

We randomly recruited around 800 students to participate in the subjective test. A brief training session was given to each subject before a test session starts. In the training session, we used different video clips to show quality degradation of coded video contents. The scenario of our intended application; namely, the daily video streaming experience, was explained. Any question from the subject about the subjective test was also answered.

In the subjective test, each subject compares the quality of two clips displayed one after another, and determines whether these two sequences are noticeably different or not. The subject should choose either ‘YES’ or ‘NO’ to proceed. The subject has an option to ask to play the two sequences one more time. The comparison pair is updated based on the response.

One aggressive binary search procedure was described in [3] to speed up the JND search process. At the first comparison, the procedure asked a subject whether there would be any noticeable difference between $QP=0$ and $QP=25$. If a subject made an unconfident decision of ‘YES’ at the first comparison, the test procedure would exclude interval $QP=[26,51]$ in the next comparison. Although the subjects selects ‘Noticeable Difference’ in all comparisons afterwards, the final JND location would stay at $QP=25$. It could not belong to $QP=[26,51]$ any longer. A similar problem arose if a subject made an unconfident decision of ‘NO’ at the first comparison.

To fix this problem, we adopt a more robust binary search procedure in our current subjective test. Instead of eliminating the entire left or right half interval, only one quarter of the original interval at the farthest location with respect to the region of interest is dropped in the new test procedure. Thus, if a subject made an unconfident decision of ‘YES’ at the first comparison, the test procedure will remove interval $QP=[39,51]$ so that the updated interval is $QP=[0,38]$. The new binary search procedure allows a buffer even if a wrong decision is made. The comparison points may oscillate around the final JND position but still converge to it. The new binary search procedure is proved to be more robust than the previous binary search procedure at the cost of a slightly increased number of comparisons (i.e., from 6 comparisons in the previous procedure to 8 comparisons in the new procedure).

Let $x_{n}\in{[0,51]}$ be the QP used to encode a source sequence. We use $x_{s}$ and $x_{e}$ as the start and the end QP values of a search interval, $[x_{s},x_{e}]$, at a certain round. Since $x_{s}<x_{e}$, the quality of the coded video clip with $QP=x_{s}$ is better than that with $QP=x_{e}$. We use $x_{a}$ to denote the QP value of the anchor video clip. It is fixed in the entire binary search procedure until the JND point is found. The QP value, $x_{c}$, of the comparison video is updated within $[x_{s},x_{e}]$. One round of the binary search procedure is described in Algorithm 1.

The global JND search algorithm is stated below.

• •

  Initialization. We set $x_{s}=0$ and $x_{e}=51$.

• •

  Search range update. If $x_{a}$ and $x_{c}$ exhibit a noticeable quality difference, update $x_{r}$ to the third quartile of the range. Otherwise, update $x_{l}$ to the first quartile of the range. The ceiling and the floor integer-rounded operations, denoted by $\left\lfloor*\right\rfloor$ and $\left\lceil*\right\rceil$, are used in the update process as shown in the Algorithm of the one round of the JND search procedure.

• •

  Comparison video update. The QP value of the comparison video clip is set to the middle point of the range under evaluation with the integer-rounded operation.

• •

  Termination. There are two termination cases. First, if $x_{c}-x_{l}\leq 1$ and the comparison result is ‘Noticeable Difference’, then search process is terminated and $x_{c}$ is set to the JND point. Second, if $x_{r}-x_{c}\leq 1$ and the comparison result is ‘Unnoticeable Difference’, the process is terminated and the JND is the latest $x_{c}$ when the comparison result was ‘Noticeable Difference’.

The JND location depends on the characteristics of the underlying video content, the visual discriminant power of a subject and the viewing environment. Each JND point can be modeled as a random variable with respect to a group of test subjects. We search and report three JND points for each video clip in the VideoSet. It will be argued in Sec. 5 that the acquisition of three JND values are sufficient for practical applications.

For a coded video clip set, the same anchor video is used for all test subjects. The anchor video selection procedure is given below. We plot the histogram of the current JND point collected from all subjects and then set the QP value at its first quartile as the anchor video in the search of the next JND point. For this QP value, $75\%$ of test subjects cannot notice a difference. We select this value rather than the median value, where $50\%$ of test subjects cannot see a difference, so as to set up a higher bar for the next JND point. The first JND point search is conducted for QP belonging to $[0,51]$. Let $x_{N}$ be the QP value of the $N^{th}$ JND point for a given sequence. The QP search range for $(N+1)^{th}$ JND is $[x_{N},51]$.

As described in Sec. 2.2, video clips are encoded with $QP=[8,47]$ while $QP=0$ denotes the source video without any quality degradation. The QP range is further extended to $[0,51]$ by substituting video of $QP=[1,7]$ with video of $QP=0$, and video of $QP=[48,51]$ with video of $QP=47$. With this substitution, the video for $QP=[1,7]$ is actually lossless, and no JND point should lie in this range. If a JND sample of a subject comes to this interval, the subject is treated as an outlier. All collected samples from this subject are removed.

The ITU-R BT 1788 document provides a statistical procedure on subject screening. It examines score consistency of a subject against all subjects in a test session, where the scores typically range from 1 to 5 denoting from the poorest to the best quality levels. This is achieved by evaluating the correlation coefficient between the scores of a particular subject with the mean scores of all subjects for the whole test session, where the Pearson correlation or the Spearman rank correlation is compared against a pre-selected threshold. However, this procedure does not apply to the collected JND data properly since our JND data is the QP value of the coded video that meets the just noticeable difference criterion.

Alternatively, we adopt the z-scores consistency check. Let $\boldsymbol{x}_{n}^{m}$ be the samples obtained from subject $m$ on a video sequence set with video index $n$, where $m=1,2,\dots,M$ and $n=1,2,\dots,N$. For subject $m$, we can form a vector of his/her associated samples as

Its mean and standard deviation (SD) vectors against all subjects can be written as

Then, the z-scores vector of subject $m$ is defined as

The quantity, $z^{m}_{n}$, indicates the distance between the raw score and the population mean in the SD unit for subject $m$ and video clip $n$. The dispersion of the z-score vector shows consistency of an individual subject with respect to the majority.

Both the range and the SD of the z-score vector, $\boldsymbol{z}^{m}$, are used as the dispersion metrics. They are defined as

respectively. A larger dispersion indicates that the corresponding subject gives inconsistent evaluation results in the test. A subject is identified as an outlier if the associated range and SD values are both large. An example is shown in Fig. 4. We provide the boxplot of z-scores for all 32 subjects in Fig. 4(a) and the corresponding dispersion plot in Fig. 4(b). The horizontal and the vertical axes of Fig. 4(b) are the range and the SD metrics, respectively.

For this particular test example, subjects #8, #9 and #32 are detected as outliers because some of their JND samples have $QP=1$. Subjects #4, #7 and #27 are removed since their range and SD are both large. For subject #15, the SD value is small yet the range is large due to one sample. We remove that sample and keep others.

Besides unreliable subjects, we consider outlying samples for a given test content. This may be caused by the impact of the unique characteristics of different video contents on the perceived quality of an individual. Here, we use the Grubbs’ test [12] to detect and remove outliers. It detects one outlier at a time. If one sample is declared as an outlier, it is removed from the dataset, and the test is repeated until no outliers are detected.

We use $\boldsymbol{s}=(s_{1},s_{2},\dots,s_{N})$ to denote a set of raw samples collected for one test sequence. The test statistics is the largest absolute deviation of a sample from the sample mean in the SD unit. Mathematically, the test statistics can be expressed as

At a given significant level denoted by $\alpha$, a sample is declared as an outlier if

where $t_{\alpha/(2N),N-2}^{2}$ is the upper critical value of the t-distribution with $N-2$ degrees of freedom. In our subjective test, the sample size is around $N=30$ after removing unreliable subjects and outlying samples. We set the significance level at $\alpha=0.05$ as a common scientific practice. Then, a sample is identified as an outlier if its distance to the sample mean is larger than the $2.9085$ SD unit.

Each JND point is a random variable. We would like to check whether it can be approximated by a Gaussian random variable [3] after outlier removal. The $\beta_{2}$ test was suggested in ITU-R BT.500 to test whether a collected set of samples is normal or not. It calculates the kurtosis coefficient of the data samples and asserts that the distribution is Gaussian if the kurtosis is between 2 and 4.

Here, we adopt the Jarque-Bera test [13] to conduct the normality test. It is a two-sided goodness-of-fit test for normality of observations with unknown parameters. Its test statistic is defined as

where $n$ is the sample size, $s$ is the sample skewness and $k$ is the sample kurtosis. The test rejects the null hypothesis if the statistic $JB$ in Eq. (8) is larger than the precomputed critical value at a given significance level, $\alpha$. This critical value can be interpreted as the probability of rejecting the null hypothesis given that it is true.

We show the percentage of sequences passing normality test in Table 3. It is clear from the table that a great majority of JND points do follow the Gaussian distribution after the post-processing procedure.