On Automatic Parsing of Log Records

Modern software solutions consist of a stack of hardware and software components [1, 2]. A failure of a component may lead to an outage, affecting the solution’s customers. It is crucial to quickly detect such a failure and identify the potential root cause of the problem to create a fix. Detection of the root cause is a daunting task, taking up to 40% of the time needed to fix the problem [3]. Finding the root causes is done by examining the logs emitted by the solution’s components [3, 4, 5, 6]. Additionally, these logs may be used in the detection of cyberattacks [1] or auditing compliance with policies [1, 7].

There exist tools that do automatic log examinations and speed up diagnostics [3, 4, 5, 6]. Unfortunately, components have individual log formats, while analysis tools prefer the logs in a unified format [2]. Thus, maintainers need to manually create individual converters for every log format [8]. Given that a solution may have thousands of components, this becomes laborious [2, 9]. If one can create a tool that parses logs automatically, this tool will speed up defect detection and repair, allowing developers to focus on creating new functionality and reducing maintenance efforts.

There exists a significant body of literature on recognizing formats. Researchers worked on detecting field type¹¹1Note that it is relatively easy to detect some patterns, e.g., dates, using basic RegEx. But a basic RegEx may have challenges understanding if a date belongs to the log record’s timestamp field or if the date is part of a log message itself, which implies that developers will have to code up additional rules. in a structured record set [10], developing parsers rapidly and incrementally [11], extracting specific information from a log string [12, 13, 8, 7], and detecting log formats by comparing multiple similar log strings [14, 9, 15]. But can we individually parse a complete raw log string (so that we can convert it to a universal format) without comparing it to other log records emitted by the same component?

This leads us to the following research question (RQ): How can we automatically parse an individual raw log string? To answer our RQ, we reduce parsing to a machine translation (MT) task. That is, we need to create an Oracle that, given a raw log string as input, will produce a string of tokens, showing which particular field a character in the input string belongs to. A toy example of the input and output is given in Figure 1.

How should an Oracle look? In the field of MT for natural languages, the best models are currently based on deep neural networks (DNNs) following an encoder-decoder architecture that implements sequence-to-sequence learning. We will discuss the details of the models in Section 2.

DNNs require large volumes of data for training. Thus, we create a tool to generate synthetic logs mimicking real ones. We will further discuss synthetic data generation and real logs (used for validation of the models) in Section 3. We will then discuss the setup of our experiments in Section 4, followed by a comparison of the performance of our models in Section 5 and a summary of our findings in Section 6. Finally, we will provide details for accessing the tool and the logs in Section 7.

We chose DNN architectures, popular for machine translation of natural language. First, we compared recurrent neural networks using gated recurrent units (GRU) [16] against those using long short-term memory (LSTM) [17] cells. Our preliminary experiments showed that LSTM-based translators outperformed the GRU-based ones (hinting that extra memory-related gates present in the LSTM cell but absent in the GRU cell may be necessary for our task). Thus, we focused on LSTM-based translators and converged on three architectures.

The first one, deemed $M_{C}$, is based on the classic neural MT architecture akin to [18] with LSTM-based encoder and decoder layers. The second one, deemed $M_{L}$, is similar to $M_{C}$ but added the attention layer as per Bahdanau et al. [19]. The third one, deemed $M_{S}$, is based on the seq2seq architecture [20] with the LSTM layers and Luong et al. attention mechanism [21]. During our initial tests we evaluated $M_{S}$ with and without beam search decoding [22]. We ended up using $M_{S}$ with regular inference as initial tests found beam search evaluation to be inferior for our datasets.

In this work, we focus on log formats of the Apache HTTP server [23]. The ubiquitousness of this product makes it a good candidate for our tests.

We will discuss our generator tool, designed to create synthetic Apache log records in Section 3.1. Then we will discuss the real-world datasets in Section 3.2, followed by the description of the synthetic datasets used for training the Oracle in Section 3.3.

We want to assess the generalizability of our approach. To do this, we create three groups of experiments where a model is trained on Training Datasets $T_{(\cdot)}$ (described in Section 3.3). We train the neural networks for up to 300 epochs with a mini-batch size of 64 log records. Optimization is done using the Adam algorithm⁵⁵5We used learning rate of $0.001$, $\beta_{1}=0.9$, $\beta_{2}=0.999$, and $\epsilon=10^{-7}$. [32]. Each dataset is randomly split into 90% training data and 10% validation data during training. The best model, preserved for further evaluation, is the one with the lowest cross-entropy loss for $M_{C}$ and $M_{L}$ and sparse categorical cross-entropy loss for $M_{S}$. Translation is done at the character-level, i.e., we do not do any log record prepossessing.

We have tuned the models’ hyper-parameters. For $M_{C}$ and $M_{L}$, we have experimented with different number of cells in these layers, namely $\{256,512,1024\}$. This was done to assess the degree of ‘freedom’ required to recognize different log formats. We also varied the dropout rates in the encoder and decoders layer, setting them to $\{0.0,0.2,0.4,0.6,0.8\}$. This was done to regularize the model and reduce overtraining.

We evaluate the performance of the models on three real-world log files (discussed in Section 3.2). The performance is measured using the Levenshtein (a.k.a. the edit) distance [33], which computes the number of changes needed to transform a string returned by the Oracle into an expected (ground truth) string. We report the absolute and relative Levenshtein distances, deemed $D_{A}$ and $D_{R}$, respectively. For a given log record, $D_{R}$ is computed by dividing $D_{A}$ by the log record length. $D_{A}$ and $D_{R}$ lie within $[0,\infty)$ range: the closer to $0$ — the better.

Our RQ was: How can we automatically parse an individual raw log string? To answer the RQ, we reformulated parsing as an MT task and explored three MT architectures. We showed that the models based on the classic LSTM MT architecture are the most promising. Moreover, the model trained on randomly generated log records was able to partially parse the log records in a format that it did not see in training: 50% of log records per evaluation dataset were parsed with the $D_{R}\leq 28\%$ and 90% of records with the $D_{R}\leq 43\%$. This implies that parsing of the individual strings using MT is possible.

This work is of interest to researchers as it provides novel insights into the log parsing field. The quality of our models’ translation is not adequate for the practitioner at this stage, but we hope that this work will inspire others to explore various MT approaches and improve upon our results.

To aid these researchers, we created a tool for generating synthetic logs in Apache format used for creating training data for the models. We also found and parsed three real-world validation logs for models’ evaluation.

The tool source code repository is [34]. The version of the tool used in this paper is stored at [35]. The training and validation datasets, $T_{(\cdot)}$ and $V_{(\cdot)}$, are available at [36].