psc2code: Denoising Code Extraction from Programming Screencasts

To build a reliable deep learning model, we need to label sufficient training data. Although developers may use different system environments and tools in programming screencasts, the software applications and tools are used for the same purpose (e.g., code development, document editing, web browsing) and often share some common visual features. Therefore, it is feasible to label a small amount of frames that contain typical software applications commonly used in programming screencasts for model training.

In this work, we randomly selected 50 videos from the dataset of programming screencasts we collected (see Table 2 for the summary of the dataset). This dataset has 23 playlists of 1142 programming video tutorials from YouTube. The 50 selected videos contain at least one video from each playlist. Eight selected videos come from the playlists (P2, P12, P19 and P20) in which the developers do not use Eclipse as their IDE. These 50 selected videos contain in total 5188 informative frames after removing non-informative ones following the steps in Section 3.1.

To label these 5188 informative frames, we developed a web application that can show the informative frames of a programming screencast one by one. Annotators can mark a frame as invalid frame or valid code frame by selecting a radio button. Identifying whether a frame is a valid code frame or not is a straightforward task for human. Ott et al. reported that a junior student usually spend 5 seconds to label an image (Ott et al., 2018a). In this study, the first and second authors labeled the frames independently. Both annotators are senior developers and have more than five years Java programming experience. Each annotator spent approximately 10 hours to label all 5188 frames. We use Fleiss Kappa⁷⁷7Fleiss Kappa of [0.01, 0.20], (0.20, 0.40], (0.40, 0.60], (0.60, 0.80] and (0.80, 1] is considered as slight, fair, moderate, substantial, and almost perfect agreement, respectively. (Fleiss, 1971) to measure the agreement between the two annotators. The Kappa value is 0.98, which indicates almost perfect agreement between the two annotators. There are a small number of frames that the two annotators disagree with each other. For example, one annotator sometimes does not consider the frames with a tiny popup window such as the parameter hint when using functions (see Fig. 5) as noisy-code frames. For such frames, the two annotators discuss to determine the final label. Table 1 presents the results of the labeled frames. There are 1,864 invalid frames and 3,324 valid code frames, respectively.

We randomly divide the labeled frames into two parts: 90% as the training data to train the CNN-based image classifier and 10% as the testing data to evaluate the trained model. We use 10-fold cross validation in model training and testing. The CNN model requires the input images having a fixed size, but the video frames from different screencasts often have different resolutions. Therefore, we rescale all frames to 300$\times$300 pixels. We follow the approach used in the study of Ott et al. (Ott et al., 2018a) that leverages a VGG network to predict whether a frame contains source code or not. A VGG network consists of multiple convolutional layers in succession followed by a max pooling layer for downsampling. It has been shown to perform well in identifying source code frames in programming screencast (Ott et al., 2018a).

We use Keras⁸⁸8https://keras.io/ to implement our deep learning model. We set the maximum number of training iterations as 200. We use accuracy as the metric to evaluate the trained model on the test data. For the training of the CNN network we follow the appraoch of Ott et al. (Ott et al., 2018a), i.e., use the default trained VGG model in Keras and only train the top layer of this model. We run the model on a machine with Intel Core i7 CPU, 64GB memory, and one NVidia 1080Ti GPU with 16 GB of memory. Finally, we obtain a CNN-based image classifier that achieves a score of 0.973 in accuracy on the testing data, which shows a very good performance on predicting whether a frame is a valid code frame or not. We use this trained model to predict whether an unlabeled frame in our dataset of 1142 programming screencasts is a valid code frame or not.

We use the Canny edge detector (Canny, 1986) to extract the edge map of a frame. Probabilistic Hough transform (Matas et al., 2000) is then used to get the horizontal and vertical lines, which are likely to be the boundaries of the sub-windows in the frame. We filter the very short lines (less than 60 pixels in this work), which are unlikely to be sub-window boundaries. Fig. 6 (b) and Fig. 6 (e) show the resulting horizontal and vertical lines for the frames in Fig. 6 (a) and Fig. 6 (d), respectively. We can see that the detected horizontal and vertical lines are noisy. To reduce noisy horizontal (or vertical) lines, we use the density-based clustering algorithm DBSCAN (Ester et al., 1996) to cluster the close-by horizontal (or vertical) lines based on their geometric distance and overlap. Each line cluster is then represented by the longest line in the cluster. By this step, we remove some close-by horizontal (or vertical) lines, which can reduce the complexity of sub-window boundary detection.

Although clustering close-by lines reduce candidate boundary lines, there can still be many candidate boundary lines left which may complicate the detection of sub-window boundaries. One observation we have for programming screencasts is that the developers do not frequently change the window layout during the recording of screencast. For example, Fig. 6 (a) and Fig. 6 (b) show the two main window layouts in a programming video tutorial in our dataset of programming screencasts. Fig. 6 (b) has a smaller code editor but a larger console output to inspect the execution results. Note that the frames with the same window layout may have different horizontal and vertical lines, for example, due to presence/absence of code highlights, scrollbars, etc. But the lines shared by the majority of the frames with the same layout are usually the boundaries of sub-windows.

To detect the frames with the same window layout, we cluster the frames based on detected horizontal and vertical lines in the frames. Let $L=(h_{1},h_{2},...,h_{m},v_{1},v_{2},...,v_{n})$ be the set of the representative $m$ horizontal lines and $n$ vertical lines in a frame after clustering close-by lines. Each line can then be assigned a unique index in $L$ and referred to as $L[i]$. A frame $f$ can be denoted as a line vector $V(f)$, which is defined as follows:

where $ind(l,f)=1$ if the frame $f$ contains the line $l$, and $ind(l,f)=0$ otherwise. We then use the DBSCAN clustering algorithm to cluster the frames in a programming screencast based on the distance between their line vectors. This step results in some clusters of frames, each of which represents a distinct window layout. For each cluster of frames, we keep only the lines shared by the majority frames in the cluster. In this way, we can remove noisy candidate boundary lines that appears in only some frames but not others. Fig. 6 (c) and Fig. 6 (f) show the resulting boundary lines based on the analysis of the common lines in the frames with the same window layout. We can see that many noisy lines such as those from different code highlights are successfully removed.

Based on the clear boundary lines after the above two steps of denoising candidate boundary lines, we detect sub-windows by forming rectangles with the boundary lines. When several candidate rectangles overlap, we keep the smallest rectangle as the sub-window boundary. This allows us to crop the main content region of the sub-windows but ignore window decorators such as scrollbars, headers and/or rulers. We observe that code editor window in the IDE usually occupies the largest area in the programming screencasts in our dataset. Therefore, we select the detected sub-window with the largest rectangle area as the code region. The detected code regions for the two frames in Fig. 6 (a) and Fig. 6 (d) are highlighted in red box in the figures. Note that one can also develop image classification method such as the method proposed in Section 3.2 to distinguish code-editor sub-windows from non-code-editor sub-windows. However, we take a simpler heuristic-based method in this work, because it is more efficient than deep learning model and is sufficient for our experimental screencasts.

In this study, our targeted programming screencasts are live coding video tutorials where tutorial authors demonstrate how to write a program in IDEs (e.g., Eclipse, Intellij). We focus on Java programming in this study but it is easy to extend our approach to other programming languages. Since YouTube has a large number of programming video tutorials and also provide YouTube Data APIs⁹⁹9https://developers.google.com/youtube/v3/ to access and search videos easily, we build our dataset of programming video tutorials based on YouTube videos.

We used the query “Java tutorial” to search video playlists using YouTube Data API. From the search results, we considered the top-50 YouTube playlists ranked by the playlists’ popularity. However, we did not use all these 50 playlists because some tutorial authors did not live coding in IDEs but use other tools (e.g., Power Point slides) to explain programming concepts and code, or the videos in the playlist are not screencast videos. Finally, we used 23 playlists in this study, which are listed in Table 2. For each playlist, we downloaded its videos at the maximum available resolution and the corresponding audio transcripts using pytube¹⁰¹⁰10https://github.com/nficano/pytube. We further found that not all downloaded videos include writing and editing source code. For example, some video tutorials just introduce how to install JDK and IDEs. We removed such videos and got in total 1,142 videos as our dataset for the experiments¹¹¹¹11All videos can be found: https://github.com/baolingfeng/psc2code.

As shown in Table 2, our dataset is diverse in terms of programming knowledge covered, development tools used and video statistics. Many playlists are for Java beginners. But there are several playlists that provide advanced Java programming knowledge, such as multithreading (P6), chess game (P12), and 2D game programming (P15). Among the 23 playlists, the majority of tutorial authors use Eclipse as their IDE, while some tutorial authors use other IDEs including NetBeans (P19, P20), Intellij IDEA (P12), and Notepad++ (P2). The duration of most videos is 5 to 10 minutes except those in the playlist P5 that have the duration of more than one hour. This is because each video in P5 covers many concepts while other playlists usually cover only one main concept in one video. Most of videos have the resolution of 720p (1280$\times$720) except for the videos in the playlist P1, P11 and P22 that have the resolution 360p (480$\times$360).

We applied psc2code to extract source code from these 1,142 programming videos. After performing the step of reducing redundant frames (Section 3.1), the number of informative frames left is not very large. For example, the videos in the playlist P5 are of long duration but there are only 335 informative frames left per video on average. After applying the CNN-based image classifier to remove non-code and noisy-code frames (Section 3.2), more frames are removed. As shown in Table 2, about 62% of informative frames are identified as valid code frames on average across the 23 playlists. psc2code identifies code regions in these valid code frames, and finally OCRs the source code from the code regions and corrects the OCRed source code.

We conduct a set of experiments to evaluate the performance of psc2code, aiming to answer the following research questions:

There are 1924 invalid frames and 2904 valid code frames as predicted by the trained image classifier, respectively (see Table 3). The ratio of invalid frames versus valid code frames in the examined 46 videos is similar to that of the 50 programming videos for model training (see Table 1). The two annotators who label the frames for the training data verify the classification results of the 4828 frames in the 46 videos. We also use Fleiss Kappa to measure the agreement between the two annotators. The Kappa value is 0.97, which indicates almost perfect agreement between the two annotators. For the classification results that the two annotators disagree, they discuss to reach an agreement.

For each frame, there can be 4 possible outcomes predicted by the image classifier: a valid code frame is classified as valid (true positive TP); an invalid frame is classified as valid (false positive FP); a valid code frame is classified as invalid (false negative FN); an invalid frame is classified as invalid (true negative TN). For each playlist, we construct a confusion matrix based on these four possible outcomes, then calculate the accuracy, precision, recall and F1-score. We also merge all the confusion matrixes of all playlists and compute the overall metrics for the performance of the image classifier.

• •

  Accuracy: the number of correctly classified frames (both valid and invalid) over the total number of frames, i.e, $Acc=\frac{TP+TN}{TP+FP+FN+TN}$.

• •

  Precision on valid frames: the proportion of frames that are correctly predicted as valid among all frames predicted as valid, i.e., $P(V)=\frac{TP}{TP+FP}$.

• •

  Recall on valid frames: the proportion of valid frames that are correctly predicted, i.e., $R(V)=\frac{TP}{TP+FN}$.

• •

  Precision on invalid frames: the proportion of frames that are correctly predicted as invalid among all frames predicted as invalid, i.e., $P(IV)=\frac{TN}{TN+FN}$.

• •

  Recall on invalid frames: the proportion of invalid frames that are correctly predicted, i.e., $R(IV)=\frac{TN}{TN+FP}$.

• •

  F1-score: a summary measure that combines both precision and recall - it evaluates if an increase in precision (recall) outweighs a reduction in recall (precision). For F1-score of valid frames, it is $F(V)=\frac{2\times P(V)\times R(V)}{P(V)+R(V)}$. For F1-score of invalid frames, it is $F(IV)=\frac{2\times P(IV)\times R(IV)}{P(IV)+R(IV)}$.

In terms of accuracy, is often larger than 0.9, which shows a high performance of the ps2code image classifier of psc2code. For example, for the playlist P1, our model can identify most of valid and invalid frames. However, the accuracy of our approach on some cases is not very good. For example, for the playlist P11, the accuracy is only 0.38. In this playlist, the NetBeans IDE is used and the image classifier fails to recognize many frames with code completion popups as noisy-code frames. This may be because of the limited number of training data that we have for NetBeans IDE.

In our frame classification task, we care more about the performance on valid code frames. On the other hand, misclassifying too many valid code frames as invalid may result in information loss in the OCRed code for the video. In terms of recall of valid code frames, the overall recall is 0.85. For 14 out the 23 plalists, the recall is greater than 0.9 for 14 playlists. In two cases (e.g., P7 and P17), the recall is equal to 1. However, for the playlist P11, the recall is very low, i.e., 0.13, which might again be caused by the NetBeans IDE used in its programming videos and the limited number of training data which uses the NetBeans IDE. For the remaining playlists, we find that for the most of misclassified valid frames, there are “nearby” valid code frames (i.e., the timestamps of the frames are close to those of the misclassified frames) that have the same or similar code content. We also find that anthoer reason for misclassified frames is the intermediate code. For example, one misclassified code frame contains a line of code System.out.println(""), which is the intermediate code of System.out.println("Copy of list :") in a later frame. Overall, the information loss resulting from misclassifying valid code frames as invalid is acceptable.

On the other hand, misclassifying too many invalid frames as valid may result in much noise in the OCRed code. In terms of precision on valid frames, the overall precision is 0.91 with 16 out of 23 playlists have precision scores above 0.9. Playlist 17 had the lowest precision with 0.66. We find that there are many frames in which the console window overlaps with the code editor, thus our image classifier misclassifies them because this type of invalid frames does not appear in the training data. Overall, the negative impact of misclassifying invalid frames as valid on the subsequent processing steps is minor, but more training data can further improve the model’s precision on valid code frames.

We use the approach of Alahmadi et al. (Alahmadi et al., 2018) as our baseline. Their approach leverages the You Only Look Once (YOLO) neural network (Redmon et al., 2016), which can predict the location of code fragments in programming video tutorials directly. Given a frame, the baseline approach can not only predict whether it is valid or not, but also can identify the location of code fragments for valid frames. We use the labeled frames from Section 3.2.1 to train a model for the baseline and apply the trained model on these 4,828 frames.

Additionally, we use the Intersection over Union (IoU) metric (Zitnick and Dollár, 2014), which is also used in the Alahmadi et al.’s work, to measure the accuracy of a predicted code bounding box for a frame. IoU divides the area of overlap between the bounding boxes of the prediction and ground truth by the area of the union of the bounding boxes of the prediction and ground truth. Given a frame, we use the same IoU computation as the Alahmadi et al.’s work, which is as follows:

• •

  If a model predicts it as an invalid frame and it is correct, then $IoU=1$.

• •

  If the prediction result of a model is incorrect, then $IoU=0$.

• •

  If a model predicts it as a valid frame and it is correct, then $IoU=\frac{A_{gt}\cap A_{pred}}{A_{gt}\cup A_{pred}}$, where $A_{gt}$ and $A_{pred}$ are the area of the ground truth and the predicted bounding boxes respectively.

We compute the average of IoU for each playlist and an overall IoU on all the frames.

To generate the ground truth of the bounding boxes that contain code, we provide a web application that show the frames with the predicted code area and allow the annotators to adjust the bounding box. When two annotators were labeling a valid frame, they were also required to adjust the bounding box of its predicted code area. When the IoU of the bounding boxes labeled by two annotators for a frame is larger than 95%, which indicates most of the two bounding boxes overlap, we think that the two annotators reach an agreement. Since the ground truth of the bounding boxes that contain code is usually the code editor in the IDE (see Figure 6(a) and (d)), the two annotators reach an agreement easily for most of valid frames. A small number of disagreement cases were caused by the margin of the code editor. Most of the disagreement cases are due to the scroll bars in the code editor window. After a discussion, the two annotators include the scroll bars in the annotated code area for all the cases. Compared with the whole code editor windows, the areas of scroll bars are small. Thus, our results and findings would not be affected by much.

The low IoUs of the baseline is caused by its incorrect predicted results. Moreover, even if we ignore the incorrect cases, the overall IoU is still 0.76, which is smaller than that of our approach. To get more insight, we look into the valid code frames with bounding boxes identified by the baseline. We found that the bounding boxes identified by the baseline usually did not cover the whole area of the code editor and missed some parts of the code area, which result in information loss. Figure 8 shows an example identified by the baseline. We can see that the bounding boxes of the two frames identified by the baseline miss some code content.

Although YOLO is a powerful object detection model for generic object detection, but we think there are some limitations of object detection based methods for our code extraction task. First, the boundary features of strokes, characters, words and text lines can confuse the YOLO model for accurately detecting code window boundaries (see Fig. 8 for example). Second, YOLO uses a set of anchor boxes (with pre-defined sizes and aspect ratios) to locate the potential object regions. This works fine for natural objects (e.g., person, dog, car), but it is fundamentally limited for locating code content which can have arbitrary sizes and aspect ratios. Third, YOLO uses CNN to extract image features. Due to the CNN’s spatial downsampling nature, it can only produce an approximate bounding box around the object (with some background and even parts of other objects in the box). This is fine for detecting natural objects as long as the boxes cover a large region of the object. However, such approximate bounding boxes will not satisfy the much higher accuracy requirement of locating code regions, as we do not want to miss any actual code fragments and do not want to include any non-code regions

As shown in Table 5, there are many words with OCR errors in the OCRed source code. Overall, the ratio of incorrect words and correct words is about 1:3. Among all incorrect words, our approach makes corrections for half of them. For the incorrect words that our approach does not attempt to correct, many of them are partial words while the developer is still typing the whole word, and some are meaningless words resulting from the noise in the frame. Among the incorrect words that are corrected by our approach, most of them (88%) are truly corrected. Many words are falsely corrected when there are multiple similar words in the code such as similar variables obj1, obj2, obj3. Overall, our approach can truly correct about half of incorrect words (46%), which can significantly improve the quality of the OCRed source code by reducing the ratio of incorrect words and correct words from 1:3 to 1:6.

In this study, we build a programming video search engine based on the source code of the 1142 programming videos extracted by psc2code. For each programming video in our dataset, we aggregate the source code of all its valid code frames as a document. The source-code documents of all 1142 videos constitutes a source-code corpus. We refer to this source-code corpus as the denoisied corpus as opposed to the noisy source-code corpus produced by the baseline method described below. We then compute a TF-IDF (Term Frequency and Inverse Document Frequency) vector for each video in which each vector element is the TF-IDF score of a token in the source-code document extracted from the video. Given a query, the search engine finds relevant programming videos by matching the keywords in the query with the tokens in the source code of programming videos. The returned videos are sorted by the descending order of the sum of the the TF-IDF scores of the matched source-code tokens in the videos. For each returned programming video, the search engine also returns the valid code frames that contain any keyword(s) in the query.

To study the impact of the denoising features of psc2code on the downstream programming video search, we build two baseline methods to extract source code from programming videos:

• •

  The first baseline method (which we refer to as baseline1) does not use the denoising features of psc2code. To implement this baseline, we follow the same strategy as psc2code in the removal of redundant frames and detection of code regions. However, we do not remove the noisy-code frames and non-code frames. For this baseline, we use Google Vision API to extract the source code and check if the detected subimages have code or not by identifying whether there exist Java keywords in the extracted text; this is the same method used by Kandarp and Guon (Khandwala and Guo, 2018). In this way, the baseline removes non-code frames. Different from psc2code, baseline1 does not remove noisy-code frames, nor does it fix the errors in the OCRed source code. baseline1 uses the two heuristics of Kandarp and Guon (Khandwala and Guo, 2018) to remove line numbers and Unicode errors.

• •

  We use the full-fledged CodeMotion (Khandwala and Guo, 2018) as the second baseline method, which is referred to as baseline2. We obtain the source code of CodeMotion from CodeMotion’s first author’s Github repository.

For each baseline method, we obtain a noisy source-code corpus for the 1142 programming videos in our dataset which we use for the comparison of the search results quality against the denoised source-code corpus created by psc2code.

The majority of videos in our video corpus introduce basic concepts in Java programming. In a typical playlist of a Java programming tutorial (e.g., the playlist P1), the creator usually first introduces syntax of Java and the concept of object-oriented programming (e.g., the java.lang.Object class), then he/she writes code to introduce some common used classes, such as String, Collections, File; Finally, some advanced knowledge (e.g., multi-thread and GUI programming) may be introduced.

Based on the topics discussed in the programming video tutorials in our dataset, we select the common used classes and APIs in our video corpus to design 20 queries (see Table 7), which cover three categories of programming knowledge, including basic Java APIs (e.g., string operations, file reading/writing, Java collection usages), GUI programming (e.g., GUI components, events and listeners), and multi-threading programming (e.g., thread wait/sleep). For these queries, there are reasonable numbers of programming videos in our dataset. This allows us to investigate the impact of the denoised source code by psc2code on the quality of the search results, compared with the noisy source code extracted by the baseline.

In this study, only if all the keywords in a query can be found in at least one valid code frame of a video, the returned video is considered as truly relevant to the query. To verify whether the returned videos are truly relevant to a query (i.e., the video truly demonstrates the usage of the class and API referred to in the query), the first two authors manually and carefully check the top-20 videos in the search results for the 20 search queries.

To compare the search results quality on the denoised source-code corpus by psc2code and the noisy source-code corpus by the baseline, we use several well-known evaluation metrics: precision@k, MAP@k (Mean Average Precision (Baeza-Yates et al., 1999)), and MRR@k (Mean Reciprocal Rank (Baeza-Yates et al., 1999)), which are commonly used in past studies involving building recommendation systems (Rao and Kak, 2011; Tamrawi et al., 2011; Zhou et al., 2012; Xia et al., 2015; Xia and Lo, 2017).

For each query $Q_{i}$, let $V_{i}$ be the number of videos that are truly relevant to the query in the top-k videos returned by a video search engine over a source-code corpus extracted from a set of programming videos. The precision@k is the ratio of $V_{i}$ over $k$, i.e. $\frac{V_{i}}{k}$.

MAP considers the order in which the returned videos are presented in the search results. For a single query, its average precision (AP) is defined as the mean of the precision values obtained for different sets of top-j videos that are returned before each relevant video in the search results, which is computed as:

where $n$ is the number of returned videos, $Rel(j)=1$ indicates that the video at position $j$ is relevant, otherwise $Rel(j)=0$, and $P(j)$ is the precision at the given cut-off position $j$. Then, for the $m$ queries, the MAP is the mean of $AP$, i.e., $\frac{\sum_{i=1}^{m}{AP_{i}}}{m}$.

Different from MAP that considers all relevant videos in the search results, MRR considers only the first relevant video. Given a query, its reciprocal rank is the multiplicative inverse of the rank of the first relevant video in a ranked list of videos. For the $m$ queries, MRR is the average of their reciprocal ranks, which is computed as:

where $rank(q)$ refers to the position of the first relevant video in the ranked list returned by the search engine.

We compute the precision@k, MAP@k and MRR@k metrics for the top-5, top-10 and top-20 search results (i.e., k=5, 10 and 20). Table 7 presents the results of precision@k for each query and Table 8 presents the results of MAP@k and MRR@k for the 20 queries. We can observe that the quality of the search results for the denoised source-code corpus extracted by our psc2code is much higher compared to the search results we obtained on the noisy source-code corpus extracted from the two baselines.

The average precision@5, precision@10 and precision@20 of our approach over the 20 queries are 0.93, 0.81, and 0.63, respectively. The average precision@5, precision@10 and precision@20 of baseline1 are only 0.53, 0.50, and 0.46, respectively, while the average precision@5, precision@10 and precision@20 of baseline2 are only 0.69, 0.56, and 0.46, respectively. The precision@5 of our approach is 1 for 16 out of the 20 queries. That is, for these 16 videos, the top-5 videos returned based on our denoised source-code corpus are all relevant to the search query for these 16 queries. For the other four queries (“Data getTime”, “StringBuffer insert”, “List indexOf” and “thread wait”), some of the top-5 returned videos are not relevant to the query. The reason is that the number of programming videos in our dataset is limited (i.e., 1142). As such, usually only a small number of videos contain the searched APIs. For example, we check the source code of all the videos in our dataset and find that only two programming videos mention the API StringBuffer.insert. In fact, these two videos are returned in the top-5 results for the query “StringBuffer insert”. But as there are only these two truly relevant videos in the whole dataset, the precision@5 is only 0.4 for this query.

We note that both our approach and the baselines have high precision in the search results for some queries, e.g., “event getSource” and “thread sleep”. This is because APIs relevant to these queries are always used together in the source code of the programming screencasts. For such cases, searching programming videos based on the source code extracted by the baseline may achieve acceptable precision. However, the precision of the baseline is generally not satisfactory for most of the queries. The precision@5 based on the noisy source-code corpus by the baseline is 1 only for three queries. For the query “ArrayList isEmpty”, there are no relevant videos returned in the top-5 search results (i.e., precision@5=0). This is because when a developer uses the class ArrayList, the API “isEmpty” frequently appears in the completion suggestion popups but is not actually used in the real code. This noise in the extracted code subsequently leads to less accurate programming video search.

All the MAPs and MRRs of our approach are equal to or close to 1, which indicates that the search engine can rank the truly relevant videos in the dataset at the top of the search results. For some queries, when $k$ increases, the top-k precision of our approach decreases. This is because our video dataset may only have a small number of videos relevant to a query and those ranked lower in the search results are mostly irrelevant to the query. But when the dataset has a large number of relevant programming videos for a query (e.g., “list sort”, “thread sleep”), the precision@10 or even the precision@20 can still be 1, which means that a large number of relevant programming videos (if any) can be found and ranked in the top search results. In contrast, the MAPs and MRRs of the baselines are much lower than those of our approach. For some queries (e.g., “Object NullException”), it is interesting to note that the top-k precision of the baseline increases when $k$ increases. This is because the truly relevant videos may be ranked below the irrelevant videos in the search results due to the noise source code extracted by the baseline.

Summary: the search engine based on the denoised source-code corpus extracted by our approach can recommend programming videos more accurately than the two baseline methods. In the future, we plan to add more information of the programming videos such as their textual description, audio, etc to improve the search results, which is similar to CodeTube (Ponzanelli et al., 2016) and VT-Revolution (Bao et al., 2018).

We developed a web application prototype that leverages the source code of existing programming video tutorials on YouTube extracted by psc2code to enhance the navigation and exploration of programming videos. Figure 9 shows the screenshots of this prototype tool, which supports the following interaction-enhancing features:

We conducted a user study to evaluate the applicability of our prototype tool for enhancing the developers’ interaction with programming videos.

To answer these questions, participants in our user study have to watch, explore and summarize the source code written in the video and the process of writing the code. In particular, participants are asked to summarize the opened and viewed Java class files for each video (V1/V2/V3-Q1). In addition, they are also asked to identify some specific content in the videos, for example, the files that contain the main entry in the first video (V1-Q2), the subclass of the class Bank in the second video (V2-Q2), and the return value of different functions (V2-Q3/Q4/Q5). As only the first video contains some complex APIs (i.e., APIs for GUI programming), we design two questions for the first video that ask participants to identify which APIs are used to set the size of JFrame and Canvas (V1-Q4/Q5).

A unique property of programming videos is to demonstrate the process of completing a programming task. For example, at the beginning of the third video (V3), the tutorial author declares a function getHeight(int x), which assigns a value to the field height and returns the value of height. Then the author adds a function setHeight(int height) to set the value of height and revises the getHeight() by removing the parameter int x and the assignment statement for the field height. Finally, the author makes setHeight(int) set the value of height with the input parameter only if the input parameter is less than 10, otherwise it sets the value of height as 0. Fig. 10 and Fig. 11 present the initial and the revised source code of the function getHeight and setHeight, respectively. For the third video, we designed four questions (V3-Q2/Q3/Q4/Q5) that require participants to properly understand the process (i.e., sequence of steps ) in a programming video in order to answer the questions correctly.

The first author developed standard answers to the questionnaires, which were further validated by the second and third author. A small pilot study with three developers (one for each tutorial) was conducted to test the suitability and difficulty of the tutorials and questionnaires. The complete questionnaires and their answers can be found in the website of our prototype tool¹²¹²12http://baolingfeng.xyz:5000.

We adopted between-subject design in our user study. All the participants are required to complete the questionnaires for the three programming videos. We divided 10 participants into two groups: the experimental group whose participants use the prototype of psc2code-enhanced video player, and the control group whose participants use a regular video player. Each group has 3 junior participants and two senior participants.

After submitting the questionnaire, the participants in the two groups are asked to rate the usefulness of the corresponding tool (psc2code-enhanced video player or the regular video player) they use for navigating and exploring the information in the programming video. All the scores rated by participants are on a 5-points likert scale (1 being the worst, 5 being the best). The participants can also write some suggestions or feedbacks in free text for the tool they use. We mark the the correctness of the answers against the ground-truth answers, which is built when designing the questionnaires. The questionnaire website can calculate the completion time for each participant automatically.

Table 10, Table 11 and Table 12 present the results of the user study for the three programming videos respectively. As shown in these tables, the average completion time of the participants using psc2code-enhanced video player on the three videos are all less than that of the participants using the regular video player. Furthermore, the average answer correctness of the participants using psc2code-enhanced video player is all greater than that of the participants using the regular video player. Since the programming tasks in the three programming videos and the questions about the information in the videos are not very complicated, the difference between the correctness of the participants using the two tools is not very large, except for the workflow related questions of the third video. This suggests that it is difficult to navigate through the workflow and find the workflow-related information in the programming videos using the regular video player. In contrast, psc2code-enhanced video player can help video watchers navigate and explore the workflow information more efficiently and accurately.

All the participants in the experimental group agree that psc2code-enhanced video player can help them navigate and explore the programming videos and learn the knowledge in the video tutorial effectively. One example of positive feedback is “I can navigate the video by searching a specific code element, which help me find the answer easily.” Some participants also give some feedback on the drawbacks of our prototype tool, for example, “some code is not complete (such as missing the bracket ’}’ at the end of the line)” and “there is a bit synchronization latency between the video playing and the update of the file content view”. But they also mention that these drawbacks have no significant impact on understanding the video tutorial. In contrast, the ratings for the regular video player are not high. A participant complains that “I have to watch the video very carefully because I do not want to miss some important information. Otherwise, I may have a headache to find what I miss and where it is in the video”.

CodeMotion (Khandwala and Guo, 2018) is a notable work, which is similar to our application for enhancing programming tutorial interaction. First, CodeMotion uses image processing techniques to identify potential code segments within frames. Then, it extracts text from segments using OCR and removes segments that don’t look like source code. Finally, it detects code edits by computing differences between consecutive frames and splits a video into a set of time intervals based on chunks of related code edits, which allows users to view different phases in the video. CodeMotion also has a web-based interface, in which a programming video are split into multiple mini-videos that is corresponding to the code edit intervals.

However, we think CodeMotion has several weaknesses, which are as follows:

• •

  It doesn’t reduce non-informative frames. CodeMotion extracts the first frame of each second and apply its segmentation algorithm and OCR on all extracted frames.

• •

  It doesn’t remove noisy-code frames. CodeMotion filters segments that is not likely to have source code by identifying keywords of programming languages in the extracted text. But it can only remove the non-code segments. The noisy-code frames will affect the effectiveness of the segmentation algorithm and introduce errors in its follow-up steps. For example, Figure 12 presents an example of segments of a frame from V1 generated by CodeMotion. As shown in this figure, due to the popup windows, CodeMotion detects two segments and infers that these two segments contain source code.

• •

  It doesn’t correct errors in the OCRed source code. CodeMotion only eliminates left-aligned text that looks like line numbers and converts accented characters or Unicode variants into their closest unaccented ASCII versions. However, many other OCR errors are not corrected. For example, among the OCR results of the video V2, there are many errors in the OCRed source code such as “package” $\rightarrow$ “erackage”, “int” $\rightarrow$ “bnt”, etc.

We use the code of CodeMotion¹³¹³13Their code is host in GitHub (https://github.com/kandarpksk/codemotion-las2018), but is not well documented and maintained. to process the three videos in our user study. Table 13 shows the number of frames processed by CodeMotion and psc2Code. In this table, the column Total is the number of frames processed by the segmentation algorithm and OCR technique of CodeMotion; the column Filtered is the number of frames after CodeMotion removes segments that are not likely to have source code; the column #valid and #invalid are the number of valid and invalid frames generated by CodeMotion and psc2code, respectively. As shown, CodeMotion applied their segmentation algorithm and OCR technique on much more frames than our approach psc2code. Only a small number of frames that don’t have source code are removed by CodeMotion. Among these remaining frames, there are many invalid frames.

The implementation of CodeMotion has a web-based interface, which presents a programming video as multiple mini-videos that is corresponding to the code edit intervals detected by it. Each mini-video corresponds a code edit interval, which is identified when: 1) the programming language of the detected code changes, or 2) the inter-frame diff shows more than 70% of the lines differing. Since only Java is used in the three videos used in the study, the code edit intervals would be identified when the inter-frame diff is large. However, the noisy-code frames have a big impact on the detection of the code edit intervals. For example, CodeMotion identified 8 code edit intervals for the video V1, but there were three code edit intervals that were caused by a popup window when running the program. Additionally, due to the errors in the OCRed source code, we find that it is difficult for users to use CodeMotion to navigate and understand the videos. Comparing to CodeMotion, our tool psc2code detected three Java files from the video and showed them in a file content view, which is easier to navigate and understand. Thus, we believe that our tool have better user experience that CodeMotion.