DefectChecker: Automated Smart Contract Defect Detection by Analyzing EVM Bytecode

(1) JVM bytecode has a fixed stack depth under different control-flow paths. The execution of JVM cannot reach the same program point with different stack sizes [17]. There are no such constraints for EVM bytecode, which greatly increases the difficulty of identifying the control-flow constructs in EVM bytecode. For example, for a simple recursive code “function f(int a)f(a);”. The code will be compiled in EVM as:

There are two blocks; two block identifiers are pushed in the same block (block 1) and will be read by the same instruction (JUMP). The difference between the JVM and EVM is that the JVM creates a new frame [18] with a new operand stack for each method call, whereas the EVM just has one global operand stack. (A frame is used to store data and partial results, as well as to perform dynamic linking, return values for methods, and dispatch exceptions.)

(2) JVM bytecode has a clearly defined set of targets for each jump [19]. In contrast, the jump target for EVM bytecode is read from the EVM stack. When a conditional jump is used, the target will be affected by the second stack item. For example, in Figure 2, the jump target of JUMPI (ID 140) is read from previous instruction PUSH and will be affected by the second stack item, i.e., ISZERO(GT(10, num)) (details see Section 3.3). If the second item refers to a true value. The jump target is 148; otherwise, the target is 141. The unconditional jump target is also read from the top of the EVM stack. For example, the jump target of JUMP (ID 147) in Figure 2 is also read from the previous instruction PUSH. Therefore, we need to symbolically execute the EVM bytecode to construct the control-flow edges.

(3) JVM bytecode has well-defined method invocation and return instructions [17]. In contrast, EVM bytecode uses jumps to perform its intra-contract function calls. In this case, to resolve an intra-contract function call, we need to inspect the top stack element to determine the jump target. For example, there are two functions A and B. Function A contains three blocks, e.g., A1, A2, A3; function B contains two blocks, e.g., B1, B2. The code on block A2 calls function B. In EVM bytecode, there is no defined method invocation and return instructions. Instead, the code pushes the return address to the stack; the arguments and jump target (block identifier of B1) need to be identified through bytecode. To return, the code pops the caller’s block identifier (A3) and jumps to execute the block. Thus, the execution sequences are A1, A2, B1, B2, A3. The identifiers of B1 and A3 should be obtained from bytecode through symbolic execution.

Our previous work [11] defined 20 contract defects for smart contracts. We divided these contract defects into five “impact” levels; among these contract defects, 11 belong to impact level one (most serious) to three (low seriousness) that might lead to unwanted behaviors. The definition of these 11 contract defects is given in Table I. In this paper, we propose DefectChecker, a symbolic execution tool to detect eight of these impact level one to three contract defects. DefectChecker does not detect contract defects belonging to levels 4 and 5, as these contract defects will not affect the normal running of the smart contracts according to the definition. For example, Unspecified Compiler Version is one of the level 5 smart contract defects. The removal of the contract defects requires the developer of the contract to use a specific compiler like 0.4.25. This contract defect will not affect the normal running of the contract and will only pose a threat for code reuse in the future. This kind of contract defect is also difficult to detect at the bytecode level as much semantic information is lost after compilation.

However, please note that in this work, we do not consider three of the contract defects that belong to impact level 1 to 3 – Unmatched Type Assignment, Hard Code Address and Misleading Data Location, as they are not easy to detect at bytecode level. Our analysis shows that they appear 22, 84, and 1 times among 587 smart contacts, respectively. EVM will remove or add some information when compiling smart contracts to bytecode, which may cover up these taints on the source contract code. For Hard Code Address, the bytecode we obtain from the blockchain does not contain information on the construct function, while we found most Hard Code Address errors appear in construct functions. To detect Unmatched Type Assignment, we need to know the maximum loop iterations, which is usually read from storage, and is not easy to obtain the value through static analysis. For example, for a loop “for(uint8 i = 0; i $\textless$ num; i++)”, the data range of uint8 is from 0 to 255. Thus, if num is larger than 255, the loop will overflow. However, num is usually a storage variable which is read from storage or depends on an external input. Thus, it is difficult to detect this through bytecode analysis. Misleading Data Location is also not easy to detect from bytecode. In Solidity programming, storage in Solidity is not dynamically allocated and the type of struct, array or mapping are maintained on the storage. Thus, these three types created inside a function can point to the storage slot 0 by default, which can lead to potential bugs. However, we cannot know whether the point on slot 0 is correct or a mistake made by EVM.

Figure 1 depicts an overview architecture of the DefectChecker approach. There are four components of DefectChecker, i.e., Inputter, CFG Builder, Feature Detector, and Defect Identifier.

The left part of the figure is the Inputter, and users can feed bytecode as input. Solidity source code is also allowed, but it needs to be compiled into bytecode. Bytecode is then disassembled into opcodes by utilizing API provides by Geth [21]. Then, DefectChecker splits opcode into several basic blocks and symbolically executes instructions in each block. After that, DefectChecker generates the CFG (control flow graph) of a smart contract and records all stack events. During symbolic execution, Feature Detector detects three features (i.e., Money Call, Loop Block and Payable Function), all concepts introduced below. Based on this information, Defect Identifier uses eight different rules to identify the contract defects on smart contracts.

Detecting contract defects by bytecode is very important for smart contracts on Ethereum. All the bytecode of smart contracts are stored on the blockchain, but only less than 1% of smart contracts have opened their source code [22]. Smart contracts usually call other contracts, but the callee contracts may not open their source code for inspection. In such a case, the caller smart contracts can only detect whether the callee contract is secure through their bytecode.

A basic block is a straight-line code sequence with no branches in except to the entry and no branches out except at the exit [23]. We first split the opcode into several blocks and give a type of the block according to its exit type. The exit type can be determined by the last instruction on a block. If the last instruction is JUMP or JUMPI, the block type is unconditional or conditional, respectively. If the last instruction is a terminal instruction (STOP, REVERT and RETURN), the block type is terminal. Some blocks belong to none of these three types, we call their block type as fall. In summary, we consider four types of blocks: unconditional, conditional, fall, and terminal.

Unlike other stack-based machines, e.g., JVM where Java bytecode has a clearly-defined set of targets for every jump, the jump position of EVM bytecode needs to be calculated during symbolic execution. Thus, DefectChecker needs to symbolically execute each single EVM instruction one at a time to obtain the CFG for smart contracts. EVM is a stack-based machine – when executing an instruction, it reads several symbolic states from the top of the EVM stack and put the symbolic result back to the EVM stack. During the symbolic execution, we can obtain the jump relations between blocks. There are three types of block according to the jump behaviors, i.e., conditional jump, unconditional jump and fall execution. Stack Event records all symbolic states on the EVM stack after the execution of each instruction.

Figure 2 is an example of the symbolic execution of the code in Listing 9. There are 4 blocks in this figure, and each block contains several instructions. The instructions in block 1 represent the code if(num >10). The block 2 and block 3 put the value (0 or 1) to the EVM stack, respectively. The instructions in block 4 are used to return the value(0 or 1) to the environment. The left-most number in each line indicates instructions’ index ID, and the center part is the instruction that needs to be executed. All the instructions will execute sequentially according to their index ID. If the instruction is ‘PUSH’, the right-most part will have a value that pushes into EVM stack. There is a Program Counter (PC) that records the ID that being executed at the current time. The PC starts from ID 0 in block 1, and EVM executes this instruction.

The example shown in Figure 2 is a part of the code of a contract, so the PC starts from index ID 130 in block 1. Before EVM executes the instruction JUMPDEST, there is a symbol num in the EVM stack. The symbol num represents the input value of the function (L1 of Listing 9). JUMPDEST marks a valid destination for jumps; it does not read or push any values. So the PC points to ID 131, and EVM pushes a value 0 to EVM stack. Then, ‘10’ is pushed into EVM stack and PC point to 135. DUP3 duplicates the 3rd stack item. Therefore, the symbol num is pushed into EVM stack. GT reads two values from the EVM stack. If the first value (at the top of the stack) is greater than the second value, than EVM push 1 into the stack; otherwise, 0 is pushed. We use a symbol GT(a, num) to represent the result and push the result into the EVM stack. Then, ISZERO reads a value from the top of the EVM stack. ISZERO reads one value from EVM. If the value equal to zero, then we push 1 into stack; otherwise, we push 0. We use a symbol ISZERO(GT(a, num)) to represent the result and push the result into the EVM stack. JUMPI (ID 140) reads two values from the stack, the first value represents the jump position ‘148’, and the second value is a conditional expression. If the result of the conditional expression is “1” (true), the the PC jumps to the index ID 148, which indicates the start position of block 2. Otherwise, if the result is “0” (false), the EVM falls to execute the following index ID 141(the start position of Block 3).

Since the result of ISZERO(GT(a, num)) can be “0” or “1”, this symbolic execution can generate two paths, i.e., Block 1 $\rightarrow$ Block 2 and Block 1$\rightarrow$ Block 3.

We first assume the result of ISZERO(GT(a, num)) is “1” and the path is Block 1 ->Bock 2. In this case, the PC points to the ID 148. The jump type of this path is conditional jump. After executing the instructions on ID 148-152, the EVM falls to execute block 4. The jump type from block 2 to block 4 is fall. When executing the first instruction of the block 4, the EVM stack holds two values, i.e., num and 0. Block 4 then returns the value 0 to the environment and uses instruction STOP to finish the execution.

We then assume the result of ISZERO(GT(a, num)) is “0” and the path is Block 1 ->Bock 3. In this case, the PC points to the ID 141. The jump type of this path is fall execution. JUMP refers to an unconditional jump; it reads one value from the top of the stack. The value reads by JUMP in ID 147 is ‘153’. After executing the instructions on ID 141-147, the EVM then jumps to execute block 4. The jump type from block 3 to block 4 is an unconditional jump. When executing the first instruction of the block 4, the EVM stack holds two values, i.e., num and 1. Block 4 then returns the value 1 to the environment and uses instruction STOP to finish the execution.

When executing a conditional jump, we should determine the satisfiability of the conditional expression, which is typically realized by invoking an SMT (satisfiability modulo theories) solver [24], e.g., Z3 [25]. If the SMT solver cannot find a solution, we consider the corresponding program path as infeasible. Therefore, symbolic execution can be used to discover dead code. However, there may be little dead code in EVM bytecode, because the compiler can eliminate dead code during the compilation of smart contracts. To accelerate our analysis, we consider the conditional expression, which is equal to “0” as unsatisfiable and all other conditional expressions as satisfiable, without checking their satisfiability.

To detect contract defects at the bytecode level, we need to identify some specific behaviors from their opcodes. In this part, we introduce three features that we use when detecting contract defects.

Table II describes the information required to detect each kind of contract defect. To detect TSD and UEC, DefectChecker only needs symbolic states computed by symbolic execution, as we only need to check whether ORIGIN and CALL instructions are read by EQ and ISZERO instruction, respectively. DefectChecker only needs control flow information to detect BID, as we only need to check whether the conditional expression contains block related instructions, e.g., ”BLOCKHASH”.

To detect the other 5 contract defects, DefectChecker needs both control flow information and symbolic states. In the previous subsection, we introduce three features detected by the feature detector, i.e., Money Call, Loop Block, and Payable function. Money Call needs symbolic states, so to detect it, DefectChecker needs check the values on the EVM stack. Loop Block and Payable function require control flow information, as they both need CFGs to locate the loop and the start of the function, respectively. NC, DuEI, GC, and Reentrancy all need to detect Money Call. DuEI and NC also need to detect Loop Block; GC needs to detect Payable function. To detect Reentrancy, DefectChecker needs to travel all the paths that contain the Gas-Unlimited-Money Call, which needs the help of the CFG. To detect SBE, DefectChecker needs to check whether the BALANCE instruction is read by the EQ instruction in the conditional expressions, which needs both control flow information and symbolic state.

Below we describe the detailed patterns that we use to determine whether a smart contract contains one or more of the contract defects.

All experiments were performed on a PC running Mac OS 10.14.4 and equipped with an Intel i7 6-core CPU and 16 GB of memory. We use Solidity 0.4.25 as the compiler to compile source code into bytecode, and use EVM 1.8.14 to disassemble the bytecode to its opcodes.

The dataset we used to evaluate DefectChecker was released in our previous work [11]. We first crawled all 17,013 open sourced smart contracts from Etherscan. Then, we randomly selected 600 smart contracts from these contracts. We found 13 smart contracts do not contain any contents. Thus, we removed them from our dataset. Finally, we obtained 587 smart contracts from Etherscan. These contracts have 231,098 lines of the code and more than 4 million Ethers in their balance.

Table III shows some key features of the dataset, i.e., lines of code, number of functions in the contracts, number of instructions in the contracts, cyclomatic complexity [28] and Ethers hold by the contracts. Cyclomatic complexity is a software metric that indicates the complexity of a program, and it is computed by analyzing the control flow graph. The formulation to compute it is: E - N + 2P. E is the number of edges on CFG; N is the number of nodes on CFG and P is the number of connected components on CFG. Since CFG is a connected graph, so P always equal to 1, and the formulation can be simplified as: E - N + 2.

The simplest contract in our dataset only contains one constructor function with 7 instructions and a cyclomatic complexity of 1. The contract with the highest cyclomatic complexity has 11,696 instructions and 2,004 lines of code. The richest contract in our dataset holds 1.5 million Ethers, while the poorest contract has no Ethers in its balance.

Two authors of our previous work manually labeled the dataset. They both have three years of experience working on smart-contract-based development and research, and took part in the process of defining contract defects. Thus, they have a very good understanding of the smart contract programming and contract defects introduced in this paper. They first manually labeled the dataset independently. Then, they discussed the disagreements after completing the labeling process and gave the final results. Their overall Kappa value [29] was 0.71, which shows a substantial agreement between them.

In this work, we developed a tool named DefectChecker to detect eight contract defects with severity impact levels 1-3. The numbers of each type of contract defect in our dataset are shown in Table IV. This shows that Block Info Dependency is the most frequent contract defect in our dataset, while Transaction State Dependency and Strict Balance Equality are the least popular. Their numbers are 42, 5, and 5, respectively. DefectChecker aims at Solidity version 0.4.0+, which is the most widely used version at the time of writing this paper [30]. However, some smart contracts are designed for Solidity version 0.2.0+ and 0.3.0+. Thus, we removed eight smart contracts and used the remaining 579 smart contracts as our ground truth.

Among the six tools we introduced in Table V, only Zeus open sourced their dataset. However, Zeus still has four kinds of defects which are not included in their dataset. Also, the Zeus authors did not provide the detail of how to built their dataset. Their paper only mentioned that “they manually validated each result” without providing any details, e.g., the number of people who labeled the dataset, and whether they are professional smart contract developers or not. Thus, we did not use these datasets.

There are seven measurements obtained from our experiments: True Positive (TP), True Negative (TN), False Positive (FP), False Negative (FN), Precision (P), Recall (R) and F-Measure (F). TP indicates the results which correctly predict a contract defect in a smart contract. TN indicates the results which correctly predict a smart contract does not have a defect. FP and FN indicate the results which incorrectly predict that a smart contract contains and does not contain a contract defect. $\mathit{Precision}$, $\mathit{Recall}$, and $\mathit{F\mathchar 45\relax Measure}$ can be calculated as:

Table IV summarizes the results of applying DefectChecker to our previous work’s dataset. The first column is the contract defects that need to be detected. The second column is the number of contract defects in our dataset (ground truth). The remaining seven columns are used to measure the performance of DefectChecker. Below, we discuss the analysis of each contract defect.

(1). Transaction State Dependency. DefectChecker detects 5 smart contracts containing this contract defect among 579 smart contracts with 0 false positives and negatives.

(2). DoS Under External Influence. DefectChecker detects 13 smart contracts that have this contract defect among 579 smart contracts with 7 false positives and 0 negatives. The 7 errors are due to the error identification of a loop.

In our detection method, we first split the bytecode into several blocks. Then, symbolic execution is used to find the edge between blocks. We traverse the path of CFG by using DFS. If there is a block that has been visited, we regard this block as the start of the loop (See Section 3.4.2). Since we regard all the paths are reachable, thus we only flag whether two blocks have an edge. This mechanism leads to false positives in detecting loops.

In Listing 10, all the L9, L10, and L11 hold a single block, respectively, and function sub() holds several blocks. EVM first executes the block of line 9, then executes the blocks of function sub() in line 2. After the execution of blocks of line 10, line 11, respectively, the blocks of function sub() will be executed again. Therefore, when traversing the CFG by using DFS, we can find that there is a cycle (fun sub()$\rightarrow$L10$\rightarrow$L11 $\rightarrow$fun sub()). Since we regard all the paths are reachable, we cannot know that the blocks of function sub() cannot jump the block of L10, after executing the block of L11.

This kind of false positive can be addressed if we execute the loop continuously. Using a loop “for(int i = 0; i <100; i++)” as an example; we need to record the state of variable i, and check whether the expression (i <100) is satisfied or not. If we prove the loop can execute continuously, we can confirm it is a real loop not the error we show in Listing 10. However, we need the assistance of an SMT solver to execute the loop, and executing the loop continuously is also time consuming. Thus, we believe the advantages of removing the use of an SMT solver in our approach outweighs the disadvantages.

(3). Strict Balance Equality. DefectChecker detects 4 smart contracts that contain Strict Balance Equality with 0 false positives and 1 false negative. The cause of the error is that the contract defect related to several functions. For example, the contract in listing 11 uses a global variable balance to represent the contract’s balance. Callers first call function getBalance to obtain the balance. The balance will then be checked in Line 5. To detect this contract defect, we need to know that the global variable balance represents the contract balance. Therefore, the contract defect can only be detected when we know users will first invoke getBalance() and then call DefectFunction(). However, it is not easy to detect this contract defect at the bytecode level, as the two operations (i.e., balance == 1 eth and balance = this.balance) are in two independent functions, and we do not know the calling sequence.

(4). Reentrancy. DefectChecker detects 16 smart contracts that contain Reentrancy, with 6 false positives and 2 false negatives. The false positives are because of error-money-call detection. A smart contract contains Reentrancy must have a Gas-Unlimited-Money-Call. To detect it, we first need to check whether the gas limits set are larger than 2,300 gas and the transfer amount is larger than 0. However, in some examples, these two values are represented by complicated symbolic expressions. Some expressions also contain values that read from storage (read by SLOAD). Thus their specific values can not be determined by static analysis. Therefore, DefectChecker failed to detect them. When DefectChecker encounters complicated symbolic expressions, the default value is larger than 2,300 gas and larger than 0, this leads to false positives. When detecting this contract defect, we need to check whether the Slot ID read by SLOAD instruction still holds when executing CALL instruction. Some Slot IDs are also represented by complicated symbolic expressions. DefectChecker failed to detect whether they are equal, which leads to reporting false negatives.

When detecting Money-Call, we use Gas-Limited-Money-Call as default, if we cannot figure out the exact value of the gas limit symbolically. We also conduct another experiment, which uses Gas-Limited-Money-Call as the default. However, DefectChecker failed to detect any Reentrancy default. The reason is that the Gas-Limited-Money-Call usually is easy to detect, as address.transfer(), address.send() will put a specific value “2300” to the EVM stack. Thus, we just need to detect the specific value. However, the gas limit of Gas-Unlimited-Call is not easy to detect, as it usually uses a complicated expression to represent the gas. Since address.call.value() will not change the gas cost. In most situations, this method will not lead to an out-of-gas error. This is the reason why we use Gas-Unlimited-Call as our default.

(5). Nested Call. DefectChecker detects that 11 smart contracts contain a Nested Call defect. Among these 11 smart contracts, we have 2 false positives and 4 false negatives. The cause of the false positives is also the error identification of the loop, which is the same with DoS Under External Influence. The false negatives are because of the complicated data structure. When detecting this contract defect, the first step is to know whether the loop iterations are related to the array’s length. We use the SLOAD instruction related pattern to obtain the loop iterations, as described in Section 3.5.5. However, as shown in Listing 12, self is a structure and its length is obtained through an external function. Since external functions can be designed in different ways, it is challenging to design a pattern to detect it.

(6). Greedy Contract. DefectChecker detects 6 Greedy Contracts, with 0 false positives and negatives.

(7). Unchecked External Call. DefectChecker reports 23 contracts have this kind of contract defect, with 3 false positives and 2 false negatives. We analyzed the false positive examples and find that these contracts use the return value of send() as function’s return value and check the return value in other functions. For example, addr.send() as shown in listing 13 is the return value of function Example, and the value is checked in the callee programs. The false negatives are because the defect happens in a constructor function, while the bytecode of the constructor function is not contained in runtime bytecode. Therefore, we missed it. However, the contract defects in the constructor function will not harm the deployed contracts, as the constructor function will only be executed once when deploying the contracts to the blockchain.

(8). Block Info Dependency. DefectChecker detects 41 smart contracts contain this contract defects, with 0 false positives and 1 false negative. The cause of the false negative is similar to the one with Strict Balance Equality. The defect contract uses a global variable to represent block information and uses this global variable in other functions, which causes the contract defect to be detected.

In our previous work, we investigated whether there are existing tools that can detect some of the contract defects we have defined. We first collected all the papers from top Security and SE conferences/journals, i.e., CCS, S&P, USENIX Security, NDSS, ACSAC, ASE, FSE, ICSE, TSE, TIFS, and TOSEM from 2016 to 2019. Then, we only retain the papers whose titles have the key words “smart contract”, “Ethereum” or “blockchain”. After that, we manually read the abstract to verify their relevance. Finally, we found only four papers that are related to smart contract defects, i.e., Oyente [20], Maian [31], Zeus [35], and ContractFuzzer [34].

To enlarge our baseline methods, we use the same method as proposed by Kitchenham et al. [36]. We first read the references of these 4 relevant papers, and tried to find whether there are existing tools that can detect the defined contract defects. If there is a relevant paper, we read its references repeatedly, until no new paper can be found. In this way we also found two other tools, i.e., Securify [32] and Mythril [33].

Table V shows the input and contract defects that can be detected by these tools. The last column shows the number of the defects can be detected by these tools except the mentioned 8 contract defects. As we know, the bytecode of smart contract on Ethereum are visible to everyone, but only less than 1% of the smart contracts open up their source code [22]. Therefore, detecting contract defects from the bytecode level is very important. To make the comparison fair, we select Oyente, MAIAN, Securify and Mythril as our baseline tools, since they can detect contract defects at the bytecode level, the same as DefectChecker. However, we found that Maian has not been updated to support the latest Ethereum environment and so we could not run MAIAN on our dataset. For example, they use methods provided by web3 [37] to obtain contracts’ information on Ethereum. However, the methods they used have been removed and did not support the current version of Ethereum that we used. In addition, DefectChecker gets 100% F-Measure when detecting Greedy Contract. In this case, we do not compare with MAIAN, and choose Oyente, Securify and Mythril as our baseline tools.

Oyente detects three kinds of security-related vulnerabilities for smart contracts. These three kinds of security-related vulnerabilities are the same as our Unchecked External Calls, Block Info Dependency and Reentrancy. Mythril [33] is a tool developed by ConsenSys, which is a leading global blockchain technology company. They find security problems from online posts or news, which is similar to our previous work [11]. Our previous work analyzed the posts from StackExchange posts and defined 20 contract defects. Mythril can detect 6 contract defects as shown in Table VII. Securify is a smart contract security analyzer that takes EVM bytecode as input. It first decompiles EVM bytecode and analyzes the semantic facts of the decompiled code. In our study, Securify uses several security patterns to detect related vulnerabilities. Securify can detect Reentrancy and Unchecked External Call, which can also be detected by DefectChecker.

Table VI shows the results of running Oyente on our previous dataset [11]. The F-score of Oyente in detecting RE, UEC, and BID are 3.7%, 68.1%, and 37.3%, respectively, while the numbers for DefectChecker are 71.4%, 88.9%, and 98.8%, respectively. We found that Oyente only considers BLOCKHASH instructions when detecting Block Info Dependency, while there are many other instructions, e.g. NUMBER (NUMBER instruction is used to get block’s number), that can lead to this contract defect. Besides, Oyente also has many false positives when detecting Reentrancy. The reason is that they do not distinguish between send(), transfer() and call() functions at the bytecode level, while send() and transfer() will limit gas to 2300 unit, which cannot cause Reentrancy. In addition, the most important reason for these errors is code coverage. Code coverage means the percentage of instructions executed. The average code coverage for Oyente is 18.9%, while the number for DefectChecker is 77.1%. Low code coverage means only a small part of the code can be analyzed for contract defect occurrence, which can lead to a large number of false positives and negatives. There are three reasons that lead to the low coverage of Oyente compared to DefectChecker. First, Oyente checks whether a path can be reached, while DefectChecker assumes that all the paths are reachable. Oyente also only optimizes for Solidity Version 0.4.19, but there is a wide version coverage in our dataset. Finally, the jump positions of some unconditional jump might not be easy to find. To be specific, the jump position might be a result of a complicated expression. Thus both Oyente and DefectChecker can fail to detect these unconditional jumps, and it is the reason why DefectChecker misses some blocks.

Table VII shows the results of Mythril. Mythril fails to detect Transaction State Dependency and Strict Balance Equality in our dataset. In addition, its results contain many false positives, especially in detecting Reentrancy and DoS Under External Influence. We found that Mythril is similar to Oyente - it fails to distinguish between call() with transfer() and send(), which will not lead to Reentrancy. Besides, Mythril failed to distinguish loop related patterns, which lead to errors when detecting loop related defects, e.g., DoS Under External Influence or Nest Call.

Table VIII presents the results of Securify. Securify can detect two common defects with DefectChecker, i.e., Reentrancy and Unchecked External Call. All the DefectChecker, Oyente, Mythril, and Securify can detect these two defects. The performance of Securify in testing Reentrancy (4.9%) is better than Oyente (3.7%), and similar to Mythril (4.9%), but much worse than DefectChecker (71.4%). In terms of detecting Unchecked External Call, the F-score of Securify (62.5%) is a little bit worse than Oyente (68.1%) and much better than Mythril. DefectChecker still get the best F-score, which receives 88.9% in detecting Unchecked External Call

To compare the results between all four tools, we add a comparison of F-measure in Table IX, which shows that DefectChecker obtains the best F-measure of all four tools.

We also calculate the overall precision, recall, and F-measure of all four tools on the whole experimental dataset. Using overall-precision as the example, the overall result is calculated by $\frac{\sum_{i=1}^{n}p_{c_{i}}\times|c_{i}|}{\sum_{i=1}^{n}|c_{i}|}$, in which $p_{c_{i}}$ is the precision of the contract defect $i$, $|c_{i}|$ is the number of contract defect $i$ in the whole dataset. The results are given in Table X, which clearly shows that DefectChecker obtains the best results in detecting contract defects.

Table XI shows the time consumption results of each tool. The second column of the table gives the average time consumption to test a smart contract for each tool. The speed of DefectChecker is the fastest in these four tools. It only needs 0.15s to analyze one smart contract. Oyente and Securify have similar running times. Oyente needs 18.48s to analyze one smart contract, and the time for Securify is 21.55s. Mythril is the slowest tool; it needs 103.55s to analyze one smart contract. The maximum time to analyze a smart contract of DefectChecker is 2.42s, while the time for Oyente, Securify, and Mythril are 1096.32s, 1203.99s and 2480.26s, respectively. The simplest smart contract in our dataset only contains 7 lines with a single constructor function. DefectChecker needs 0.04s to analyze it, while the time for Oyente, Securify, and Mythril are 0.28s, 0.37s and 1.58s, respectively. DefectChecker also has the smallest Standard Deviation value among these four tools, which shows that DefectChecker has the most stable speed in analyzing a smart contract.

In conclusion, the efficiency of these four tools is in order: DefectChecker >Oyente >Securify >Mythril.

Internal Validity. We used a dataset released in our previous work [11] as the ground truth to evaluate DefectChecker. Since the people who developed DefectChecker are the same as the people who labeled the dataset, it is likely that their familiarity with the dataset might lead to potential optimization or omissions when developing DefectChecker. We tried to use the datasets of the baseline tools to evaluate DefectChecker. However, we failed to find the dataset. Luu et al. run Oyente on 19,366 contracts. They only manually check the correctness of some examples, instead of using a complete dataset to evaluate Oyente. We can only find some false positive and true positive values on their paper. Securify uses a complete dataset which consists of 100 smart contracts. However, they do not open their dataset to the public. Mythril is a tool from industry. They even do not have an evaluation section in their technical papers. Thus, we had to build our own dataset. To reduce the influence of our dataset, we first wrote a few demo smart contracts when developing DefectChecker and used these to conduct small-scale testing of our proposed tool. Then, we conducted large-scale testing by using real world bytecode we crawled from the Ethereum blockchain. The dataset is the same as that we introduced in Section 5. During this large-scale testing, we randomly choose a set of smart contracts that can find their source code. We use these smart contracts to improve the performance and patterns that are used to detect contract defects. We admit that the familiarity with the ground truth dataset might lead to a bias, but the methods we used to develop DefectChecker can reduce this influence.

External Validity. The dataset we used to evaluate DefectChecker is based on manual analysis, which may contain false positives and negatives. To address this problem, we double-checked the results and used them to update the dataset when we found some mistakes. Another threat is that Solidity is a fast-growing programming language. There are nine versions released in 2018, which may add or modify any features of the previous version. DefectChecker is designed based on Solidity version 0.4.0+, which is the most popular version in the time of writing the paper [30]. In the future, more smart contracts may use higher versions, which may make our tool unable to work.

To identify whether contract defects are actually prevalent in a large-scale, real-world dataset, we crawled bytecode from Ethereum blockchain by 2019.01 and obtained 183,706 distinct bytecode. Since some smart contract versions are not supported by DefectChecker, and so we removed them from our experimental dataset. Finally, we ran DefectChecker on 165,621 distinct smart contract bytecode. All these bytecode are runtime bytecode. Runtime bytecode does not contain information on their constructor function. It is the default bytecode stored on the Ethereum.

We ran DefectChecker on 165,621 smart contract bytecode. The detailed results are given in Table XII, which aims to show the frequency of each defect on Ethereum. Since DefectChecker only identifies whether a contract contains a defect or not, if the same kind of defects appears multiple times in a smart contract, we only count it once in Table XII. The second column of the table shows how many contracts contain related defects, and the last column gives the percentage of how many contracts contain the defect. If a contract contains multiple defects, all of the defects are counted.

Unchecked External Calls is the most frequent contract defect in the Ethereum, and about 7.5% of real world smart contracts contain this defect. There are about 3.1% of smart contracts that contain Block Info Dependency, which is the second most popular contract defect on the blockchain. Strict Balance Equality is the rarest of our contract defects. DefectChecker only detects 390 smart contracts that have this contract defect. The percentage of Nested Call is also less than 1%, with 1,043 (0.6%) smart contracts having this kind of contract defect. The percentage of Transaction State Dependency and DoS Under External Influence are similar on Ethereum, at about 1.0% and 1.3%, respectively. There are 3,139 greedy contracts on the Ethereum, and 3,892 smart contracts containing the Reentrancy problem, which can lead to serious security problems.

We found that there are 16 smart contracts that contain 4 kinds of contract defects, which are thus the most defective contracts. The number of smart contracts that contain 3 kinds of contract defects is 539, and 3,520 smart contracts contain 2 kinds of contract defects. About 25,815 smart contracts contain at least one kind of defect, which means that about 15.9% smart contracts on Ethereum contain some kinds of defects, as reported by our DefectChecker.

We utilized cyclomatic complexity [28] and the number of instructions to conduct a further analysis. We computed the cyclomatic complexity and number of instructions for contracts in our dataset. We found that the average cyclomatic complexity of smart contracts in Ethereum is 21.3, and the average number of instructions are 2,342.6. Figure 3 shows the relationship between the number of the contract defects that contained in smart contracts and the number of instructions & cyclomatic complexity. The x-axis means the number of contract defects in a smart contract. The left y-axis is the number of x, and the right y-axis is the number of cyclomatic complexity. The two lines have a similar trend.

The number of instructions is proportional to the length of a contracts’ code, which can show the contracts’ complexity at the code level. The number of cyclomatic complexity indicated the complexity of a program. We performed a generalized linear regression with the Poisson error distribution model provided by R [38] to analyze the relationship between the number of defects with instructions, and the number of defects with cyclomatic complexity. In our model, we use the number of instructions and cyclomatic complexity to predict the number of defects, respectively. Since both the correlation coefficients are positive (0.001 with std. error = 0.0009 and 0.023 with std. error = 0.0179, respectively), it shows that the more complex a contract is, the higher is its probability to contain defects. We calculated the correlation level between these two complexity measures using the Pearson correlation method [39] at a 5% significance level. The statistical test shows that the correlation coefficient is 0.702 with $p-value<0.05$. These correlation results imply that the number of instructions and cyclomatic complexity is correlated, and we can use only one of them as a predictor.

DefectChecker found some real-world attacks / financial loss from our large-scale testing on the full Ethereum dataset. In this subsection, we give two examples to show the importance of detecting such contract defects.

There is a loop in the function sendReward(), and the loop iterations are increased with the length of investors[]. However, the contract does not limit its loop iterations. As we know, sending Ethers is expensive as it needs a large amount of gas consumption, and the contract sends Ethers to the contract users in Line 14. So, the gas consumption of executing sendReward() will increase in the length of investors[]. When we check the transaction of the contract, we can find that the contract can work normally at first, as the total gas consumption of sendReward() does not exceed its maximum gas limitation at that time. However, with the increase of the length of investors[], the total gas cost increases rapidly. The gas cost then eventually exceeds the gas limitation, and leads to an out of gas error. Even worse, since the length of investors[] cannot be reduced, once the error happens, the sendReward() cannot be called anymore, which means all the Ethers in the balance are locked forever. Figure 4 shows the detail of a failed transaction. It is clear that when a user calls sendReward(), the out-of-gas error happens.

It should be noticed that although the financial loss in the real world example is caused by Nested Call, the contract shown in Listing 14 also has another contract defect, namely DoS Under External Influence. This contract defect can also lead to the lock of Ethers. Specifically, if the needPay (Line 14) is a contract address, the maximum Gas Limit will be restricted to 2300 gas units, which is not enough to transfer Ethers. Thus, an out-of-gas error will happen in Line 14, and the Ether transfer cannot succeed.

Figure 5 shows an attacking transaction which was launched by an attacking contract. The address of the attacking contract starts with 0xdefbe, and the address of the victim contract starts with 0xbabfe. The attack happens three times on block 4919015, 4919567, and 4919662, respectively. First, the attacking contract sent 1 Ether to the victim contract. Then, the victim contract returned back Ethers to the attack contract. From these 3 attacks, the attacking contract stole about 5 Ethers from the victim contract, which were worth about $1,200 at the time of writing the paper. We only show one example in Figure 5. Actually, the victim contract was attacked by multiple attacking contracts, so the financial loss was far more than 5 Ethers.

Internal Validity. The dataset we used was crawled from Ethereum, which contains different Solidity versions. DefectChecker only supports versions higher than 0.4.0+, and about 20,000 contracts had to be removed from our dataset, which may influence the overall results. However, the bytecode we removed is from many years ago, since the first version of 0.4.0+ was released on Sept. 2016. Even though there are many contract defects in the removed bytecode, these do not represent current smart contract usage.

Another key threat is that we used our DefectChecker to get the results, but DefectChecker also reports false positives and negatives, as shown in the previous section. However, DefectChecker is the most accurate and efficient tool that detects contract defects in the bytecode level, as we also demonstrated in the previous section. Therefore, we believe the results and our conclusions from it are reasonable.

External Validity. There are more than 1,000 smart contracts being deployed to Ethereum every day [40]. Many guidance and security detection tools [41, 31] are released to the public, which can help to improve the quality of smart contracts. In this case, the contract defects in smart contracts may decrease, which may lead to different results to what we found and reported in this section.