Boilerplate Detection via Semantic Classification of TextBlocks

We present an algorithm called Search-and-Combine Segmentation (SCS) to generate a sequence of text blocks in two phases: the search phase and the combine phase. In the search phase, SCS traverses the DOM-tree of a given HTML file using depth-first search (DFS) and identifies leaf nodes that contain text. On each path during the search phase, SCS removes text-formatting tags and the corresponding closing tags, but not the enclosed text. In the combine phase, starting from the first text block, SCS recursively compares the current text block with the next text block and merge them into a new block if they are siblings and with the same presentation style. Merging text blocks provides needed semantic information for more accurate classification and helps to prevent vanishing gradients during training.

We observe that using semantics contained in the text enclosed in a text block can help identify boilerplate. For the text enclosed inside a text block, if “The US president …” is in one block and “limited time 20% off” in another, then it is easier to tell that the former text block is part of the main content and the latter a boilerplate according to their meanings. To bring in semantics to a text block, we replace each tag with the underlying word or phrase to produce a “sentence” of words. For example, $\langle$p$\rangle$ is replaced by “paragraph”, $\langle$div$\rangle$ by “division”, and $\langle$h1$\rangle$ by “primary headline”. Similarly, class names, especially in HTML files that follow conventional naming rules for understanding and reusing codes are often formed by words or meaningful word abbreviations. Fig. 2 is an example of the DOM-tree for an ABC news webpage, where the main text—the darkened part with the news text omitted—has a sequence of class names like, after text cleansing, “story feed item scrollspy container article flex story article content story”. For the blocks around the main content, we can see class names with words of “header”, “footer”, and “sponsored headline”.

We remove hyphens, dashes, and other delimiters from class names, separate camel-case words, discard numbers, and expand abbreviations to their original words. A class name that cannot be converted to a word string (note that this is rare) is left as is. We recommend keeping these class names to prevent information loss. Another thing worth mentioning is that the ID attributes may be added to the class-name sequence to form an ID-class-name sequence, as used by CSS for manipulating the element with a specific ID. ID-class-name sequences are treated in the same way as class-name sequences.

Fig. 3 depicts an example of converting tag sequences and class-name sequences into string of words for two neighboring text blocks.

Figure 4 depicts the architecture of SemText. The preprocessing component is responsible for generating text blocks from a given HTML file. The word-level-feature-extraction component is a depth-wise CNN model that takes text blocks as input and generates a feature map for each text block. The block-level-sequence-labeling component is a Bi-LSTM-CRF model that labels each feature map as “main” or “boilerplate”.

We construct a CNN encoder to extract word-level features, motivated by Kim’s one-layer CNN model with 1-dimensional convolution [19]. An input $x_{t}$ has three word strings with at most $n$ words in each string, denoted by $x_{t}=[x_{t1},x_{t2},x_{t3}]$, where each word is a $k$-d embedding vector. If $x_{ti}$ ($i=1,2,3$) has less than $n$ words, then fill in the rest of the word slots by 0-vectors. Thus, each $x_{t,i}$ becomes an $n\times k$ matrix $\bm{M}_{i}=[w_{i1},w_{i2},\ldots,w_{in}]^{T}\in\mathbb{R}^{n\times k}$, where $i$ is the feature map number and $w_{ij}$ a $k$-d vertical vector.

Unlike regular CNNs that apply convolution filters to all feature maps to form a single result, our CNN model uses different feature maps to represent different aspects of a text block not necessarily having any cross-feature-map relation with others, and uses different convolution filters on different feature maps. Thus, we perform depthwise 1-d convolution instead of combining them. An added benefit of doings so is that it significantly reduces the number of parameters. In particular, our depthwise 1-d convolution first splits the input to form different feature maps, then carries out convolution operations on each feature map $\bm{M_{i}}$ with a filter $\bm{W}_{i}\in\mathbb{R}^{v\times k}$, where $v$ is the filter width. Let $\bm{M}_{i(j:j+v-1)}$ denote a filter-size block that contains $m$ word-embedding vectors, then the corresponding feature map for the current window is $c_{ij}=\bm{W}_{i}^{T}\bm{M}_{j:j+v-1}+b_{ij},$ where $b_{ij}$ denotes a bias factor. Sliding the filter in the vertical direction, we obtain a feature map as follows: $c_{i}=[c_{i1},c_{i2},c_{i3},\ldots,c_{i(n-v+1)}].$ Convoluting each feature map with the corresponding filter, and then perform max-pooling over each feature map that extracts the largest number in the map, resulting in a 3-tuple of numerical values for $x_{t}$. ;

Repeat the above procedure for $\ell$ different filters with different kernel sizes to produce $\ell$ 3-tuples of numerical values. Group them according to feature maps and flatten the results into a 1-d output, denoted by $\hat{x}_{t}$, to produce the final feature map of $x_{t}$. Figure 4 illustrates the data flow of this process in the word-level feature extraction module.

At the block-level, SemText uses the Bi-LSTM-CRF [20] architecture to capture context for the block sequence. To learn a contextual representation $\hat{h}_{t}$ and a context dependency $\hat{c}_{t}$, the feature map $\hat{x}_{t}$ generated by the CNN model at time $t$, as well as the previous block information $\hat{h}_{t-1}$ and $\hat{c}_{t-1}$, are fed to a long-short-term memory (LSTM) unit [21]. LSTM units are connected forwardly, thus previous block information can be captured, which may impact the representation and output for the current input block. LSTM is designed to resolve the gradient vanishing problem that can be encountered when training a traditional recurrent-neural-network model, allowing long-term dependencies being captured. It uses a set of gates to decide when a certain dependence should be remembered, forgotten, or outputted.

LSTM, however, has a forward-only direction capable of capturing the past context only. To make use of forward and backward information, bidirectional LSTM (Bi-LSTM) [22] puts two LSTM together, one in the forward direction and one in the backward direction. Thus, the contextual representation of the $t$-th text block $x_{t}$ can be rewritten as $\hat{h}_{t}=[\overrightarrow{h_{t}};\overleftarrow{h_{t}}]$. To make effective use of past and future prediction results, a CRF layer is added to the output of Bi-LSTM for jointly decoding the best chain of labels. Let $H$ denote the contextual representation generated by Bi-LSTM through time, that is, $H=[\hat{h}_{1},\hat{h}_{2},\ldots,\hat{h}_{m}].$ The CRF layer maximizes conditional probability $p(Y|H)=\exp({\mbox{score}(H,Y)})/\sum_{Y^{\prime}}\exp({\mbox{score}(H,Y^{\prime})})$ to find the highest-scored result over all possible label sequences, where $Y^{\prime}$ represent a possible sequence and $\mbox{score}(H,Y)=\sum_{i=1}^{m}\log s(y_{i},H)+\log t(y_{i-1},y_{i},H)$ is a log potential function with $s$ being an emission potential and $t$ a transition potential. The emission potential is defined by a fully-connected layer, transferring the output of Bi-RNN at timestamp $t$ to a score. The transition potential denote a transition probability from the previous tag $y_{i-1}$ to the current tag $y_{i}$. Adding the CRF layer is helpful to cultivate strong correlations that exist between labels of adjacent neighborhoods.

The CleanEval dataset [2] is a cross-domain benchmark, consisting of 55 webpages for training (development) and 676 webpages for evaluation. The “clean text file” for each webpage does not align properly between text blocks of the two files, and so cannot be used directly for analysing HTML files at the tag level. We adopt Vogels et al.’s method [17] to make proper alignment. The latest GoogleTrends-2017 dataset [3] published in 2020, consists of 180 documents randomly sampled from a larger pool of websites retrieved from the top Google queries with the corresponding main text. It is customary to use 80 webpages for training and the remaining 100 webpages for testing when using GoogleTrends-2017.

These two datasets, however, are insufficient to train a neural network. It is also more desirable to have a training dataset mixing of past and contemporary webpages from a wider range of domains. To this end, we combine the following three published type-1 datasets: (1) the dataset annotated by Uzun E. et al. [23], consisting of 1,170 webpages from 573 domains published from 2008 to 2011; (2) the dataset annotated by Peters M. and Lecocq D. [24], consisting of 1,381 webpages collected in late 2012 from RSS feeds, news articles, and blogs; (3) the TECO Benchmark Suite, presented by Alarte J. [25], consisting of webpages collected from 130 domains during the period from 2013 to 2020. TECO is a multilingual dataset made for testing language independent features and we remove non-English and non-type-1 webpages from it. We train SemText on this combined dataset, then fine-tune it using the 55 training webpages in CleanEval and 80 webpages in GoogleTreands-2017. This allows us to demonstrate the robustness of SemText on webpages of different structures across a wider spectrum.

We also evaluate SemText on 250 type-2 webpages consisting of 70 pages sampled from the TECO Forum dataset with non-English pages removed¹¹1This dataset is available at https://github.com/dreamlegends/Semtext., and 180 webpages collected from nine large Q&A sites including Quora and Yahoo Answer, with main content extracted by custom-built spiders.

Let $n=50$ be the maximum number of words for each word-string component (without stopwords). For each text block $x$, we translate HTML tags in $\text{tag}(x)$ using the W3C HTML5 reference²²2https://dev.w3.org/html5/html-author/. Class names in class($x$) are processed as described in Section III-B. While we have not seen any text block $x$ with tag$(x)$ or class$(x)$ containing more than 50 words, it is common for text$(x)$ to exceed the 50-word limit and so truncation is used to trim it down to 50. With an average of 10 to 15 words per sentence in the main text (with stopwords removed), 50 words can cover 4–5 sentences, sufficient for representing the semantics of a text block.

Let $m=85$ be the upper bound for the number of text blocks produced by SCS, which is adequate to cover 90% of news articles and would not cause excessive computation. If SCS produces more than 85 text blocks for an HTML file, we divide them into sub-sequences in the same order as evenly as possible so that the number of text blocks in each sub-sequence is the largest possible bounded above by 85.

We use kernels of sizes of 3, 5, and 7 for the word-level CNN encoder, with the corresponding numbers of filters being 128, 128, and 256, for a total number of filters $\ell=512$. This setting covers consecutive words in almost every aspect to capture the essential meaning of a text block. There are 512 hidden units for a single Bi-LSTM. Parameter optimization is performed using stochastic gradient descent (SGD) with a batch size of 64.

We divide at random the combined dataset with a 75-25 split into a training set and a validation set, and choose the checkpoint with the highest F1 score on the validation set. Our model is implemented on Pytorch and trained on a single NVIDIA GeForce GTX 2080Ti GPU.

We compare SemText with BoilerNet [3], Web2Text [17], and BoilerPipe [13], the best models so far, under the measures of average precision and recall. To provide fair comparison, we train and test these models as we train SemText using the same combined dataset and fine-tune them using the same development data of CleanEval and GoodTrends (except BoilerPipe that is not built for fine-tuning). We name the models trained this way as, respectively, BoilerNet-C, Web2Text-C, and BoilerPipe-C. We also train these models using the original development data of CleanEval to obtain, respectively, three models named BoilerNet-1, Web2Text-1, and BoilerPipe-1; and using the original development data of GoogleTrends to obtain, respectively, three models named BoilerNet-2, Web2Text-2, and BoilderPipe-2.

We test all models on the same test data of CleanEval and GoogleTrends. Evaluation is carried out at the text-block level and blocks are treated equally with the same weight. Table I shows the evaluation results, where 1$|$2 means that the results of the corresponding model version 1 (e.g., BoilerPipe-1) are under the CleanEval column and version 2 (e.g., BoilerPipe-2) are under the GoogleTrends column. The numbers in boldface are the highest.

Indications of these results are summarized below:

1. 1.

   SemText outperforms the previous models under each category except that SemText and BoilerNet-C have the same recall on GoogleTrends.

2. 2.

   BoilerNet-C is substantially better than BoilerPipe-C and Web2Text-C on both datasets.

3. 3.

   BoilerPipe-C and Web2Text-C on GoogleTrends are substantially worse than themselves on CleanEval. Likewise, BoilerPipe-2 and Web2Text-2 are substantially worse than BoilerPipe-1 and Web2Text-1, respectively.

4. 4.

   Web2Text-1 is slightly better than BoilerNet-1, and Web2Text-2 is substantially lower than BoilerNet-2.

5. 5.

   The results of SemText and BoilerNet are consistent on both datasets, but the results of BoilerPipe and Web2Text are not.

A possible cause of inconsistency of BoilerPipe and Web2Text across different datasets is that some of their handcrafted features are sensible to webpage structure. For example, both web structure and tag usage have changed gradually in the past ten years, and placing ads from a sidebar to the main content area can increase the link density and decrease the text density, making the text-density feature fail on contemporary webpages. Tag TD, which was widely used before 2009, was replaced by tag DIV in the web evolution. Thus, BoilerPipe and Web2Text would provide anticipated performance only when they are trained and tested on datasets from the same period. BoilerNet, on the other hand, does benefit from the combined dataset, which indicates that neural network models can indeed provide a better and more robust representation. The combined dataset also provides a large vocabulary size to alleviate the Out-of-Vocabulary problem mentioned in the last paragraph of Section II. Since each tag is treated as a token in the BoilerNet, the change of tag usage may still affect the model. SemText mitigates this drawback by taking advantage of the semantics registered inside the tags.

SemText also performs reasonably well on type-2 webpages without any training on type-2 webpages and is substantially better than BoilerNet-C, which in turn is substantially better than Web2Text-C and BoilerPipc-C (see the upper part of Table II).

Next, we retrain all models by including type-2 webpages in the training dataset and evaluate how well they perform on type-2 pages. To this end, we randomly select 50 webpages from the type-2 dataset and add them to the combined dataset of type-1 pages. The remaining 200 type-2 webpages are used for evaluation. We obtain SemText-1, BoilerNet-C1, BoilerPipe-C1, and Web2Text-C1, with the evaluation results shown in the lower part of Table II. It can be seen that all models have improved performance with the same performance ranking as before. In particular, under the F1 measure, SemText-1 is 8.87% higher than BoilerNet-C1. While SemText-1 and BoilerNet-C1 have the same recall, SemText-1’s precision is much higher. As shown in Fig 5, BoilerNet-C1 mistakenly labels the question list in the sidebar and part of the text in the header as the main content, and SemText-1 successfully removes the side bar and the header completely. We observe that the mislabeled text is similar to the main content. The improvement of SemText can be attributed to adding class info (e.g. question list, sidebar) to the representation of the question-list text blocks in the side bar. BoilerPipe has extremely high precision with extremely low recall because BoilerPipe only extracts the first block of the main content.

Let SemText_TXT denote the baseline of SemText with only text sequence, Semtext_TAG = SemText_TXT plus tag sequence, and SemText_CLS = SemText_TXT plus class sequence. Evaluation results are shown in Table III.

We can see that SemText_TXT has the lowest F1 score, while SemText_TAG and SemText_CLS each improves the F1 score over SemTexT_TXT, indicating that adding a tag sequence and a class sequence for a text block each contributes to the improvement of the baseline model. A possible cause of the lowest precision score of SemText_TXT is that it tends to mislabel text blocks that are similar to the main content. On the other hand, SEMText_TXT has a higher recall, which is expected. Adding a tag sequence and a class sequence both contribute to a higher precision, as indented; but they also slightly decrease the recall scores.

In summary, SemText consistently achieves substantially higher F1 scores across type-1 and type-2 webpages over all the evaluated models. Moreover, all models, after trained, run about the same time on the evaluation data. In particular, it takes an average of 38 ms for SemText to extract the main text on a single NVIDIA GeForce GTX 2080Ti GPU, with an average of 683 DOM nodes per page in the evaluation data.