Architecture, Dataflow and Physical Design Implications of 3D-ICs for DNN-Accelerators

The architecture of our 3D systolic array is shown in Fig. 1. The MACs are connected to direct neighbors; horizontally via wires and vertically via TSVs/MIVs. The whole array calculates a matrix multiplication. Matrices can enter from top and left, shown in purple and orange. The matrix multiplication is parallelized in the third dimension by splitting it into partial sums. Thus, each layer is entered by two corresponding parts of the input matrices. The MAC units in the layer generate a partial sum. At the end, each pile of stacked MACs accumulates the data; then, the bottom layer returns the output matrix. (Other schemes would not require vertical communication, and our system was equivalent to a scaled-out accelerator.) This is shown in blue in Fig. 1 for one exemplary MAC pile. Only minor modifications to the MAC unit in comparison to a 2D array are necessary: One MUX, the accumulate control signal (partial summing across layers) and the vertical links are added.

Please note that we connect each pair of adjacent MACs with a TSV/MIV array between layers. In general, this is a over-provision of vertical interconnects that induces an area overhead (especially for TSVs) and reduces the chip’s yield. However, we chose this deliberately, as it is a worst-case approximation for 3D DNN-accelerators. Many methods that reduce the area overhead and increase the yield by limiting the number of vertical links exist in the literature (e.g., [12]). We do not discuss these here, as it is outside of the scope of this paper and existing approaches can be applied.

The 3D accelerator talks to the external memory using a memory controller connected in one of the array’s layers. The requested data coming from outside of the chip is distributed to the layers in a fashion similar to one in [7], in which the small systolic arrays are distributed in 3D among dies [7, Fig. 7]. The 3D array profits of a one to two orders of magnitude smaller wire delay for these vertical interconnects [7, Fig. 8].

Data are stored intermediately before being fed into the array in scratchpad memory (cf. [13, Fig. 1]). There are two options: Either, there is SRAM on one tier that interconnects to all tiers; or each tier has dedicated SRAM. Both are possible due to the small wire delay for 3D [7]. As the size and architecture of the scratchpad memory has a vast influence on the performance, this has already been optimized for 2D, e.g., [13]. Those findings can also be applied here. Hence, the architecture and the parameters of scratchpad memory are outside of the scope of this paper.

To summarize, we do not claim any contribution in the memory system for a 3D systolic array and refer to the existing architectures proposed in the recent literature. For main memory, [7] proved a performance advantage of 3D-ICs. For scratchpad memory, the findings of 2D-ICs can be applied, as each tier has dedicated memory.

The mapping of operands plays an important role in determining the performance of a workloads on an accelerator. Chen et al. [1] describe the various mapping strategies in DNN accelerators and introduce a naming convention used here. [13] shows that out of the various strategies, three ones lend themselves for efficient mapping of computation on systolic arrays. These are output stationary (OS), weight stationary (WS) and input stationary (IS). In the following paragraphs we briefly describe these in the context of mapping a General Matrix Matrix Multiplication (GEMM) of two operand matrices $A^{(M\times K)}$ and $B^{(K\times N)}$, and discuss the implications when mapped onto a 3D systolic architecture.

WS and IS dataflows. In the WS dataflow the elements of the matrix $B$ are first stored to the local memory of the MAC units, such that each column of the array gets the element corresponding the a specific col of the matrix B (given sufficient MACs). Once the elements are stored, one element of a row of the transposed matrix of $A$ is fed in from the left side of the array in each cycle. Each MAC then multiplies the incoming element with its stored operand, and forwards incoming value to the MAC unit on its right. The generated product is then added with the incoming sum from the top edge, and the partial sum is sent to the element on the bottom row. Thus, the elements corresponding to each column of the output are generated, within one column of the array, by reducing across the rows. To summarize this strategy, the dimension $N$ is mapped spatially along the columns, the dimension $K$ is also mapped spatially along the rows. However, the dimension $M$ is temporally mapped.

The IS dataflow is similar to WS, but the mapping of matrices $A$ and $B$ are interchanged. The elements of each row in matrix $A$ are first stored into the local memory of MACs along array columns. The columns of matrix $B$ are then fed in from the left, one column per cycle, and the multiplication and reduction takes place similar to the case in WS. Thus, the dimension $M$ is mapped spatially along the array columns. The dimension $K$ is also mapped spatially along the array rows, while the dimension $N$ is mapped temporally.

OS dataflow. The elements of matrix $B$ are streamed from the top edge of the array, such the each column of this matrix send to a particular array column; while the elements of matrix $A$ are streamed from the left edge of the array such that each array row receives elements from the corresponding row of the operand. The partial sums are generated in each MAC and are reduced locally, producing the output matrix. Fig. 4 depicts the schematics of this mapping. To summarize, the dimensions $M$ and $N$ are mapped along the array’s spatial dimensions. The dimension $K$ is mapped along the temporal dimension.

Exploiting the third dimension. The analysis above shows that the temporal dimension contributes to increase in runtime and therefore impedes performance (given an optimal 2D mapping). Adding a third spatial dimension can alleviate this bottleneck. In the context of WS and IS dataflow, this translates to mapping the $M$ and $N$ dimension in the ‘new’ spatial dimension. E.g., if we have a 3D stacked architecture, of 2 planar arrays; in WS dataflow, half of the rows in matrix $A$ would be used in the ‘top’ tier array, while the other half in the ‘bottom’ tier array. In case of IS dataflow, the mapping would be similar, but the roles of matrix $A$ and $B$ would be interchanged. Please note that there is no communication between the arrays on the different tiers. This is identical to a distributed array, and such acceleration lends itself into the well studied model parallelism approach [13].

The OS dataflow, however, is an interesting one for 3D. Tthe dimension $K$ will be mapped to the third spatial dimension, therefore leading to reductions to be performed across the tiers. Fig. 4 depicts the computation flow equivalent to a single MAC on a 2D array, working with OS strategy on the proposed 3D setting. We refer to this new strategy as “distributed output stationary (dOS)". In Fig. 4 we show a schematic of a 2-tiered 3D array employing dOS dataflow. The mapping and reduction across various tiers lead to interesting architectural trade-offs to improve performance. The combination of inplace and cross-tier reduction means that naïvely increasing the number of tiers will lead to increased reduction time and will hamper performance (see Sec. IV-A). In the rest of this paper, we focus on the dOS dataflow and study the performance and implementation aspects in a 3D systolic array setting. We do not dive into details for the WS and IS dataflows as the existing literature on model parallelism provides detailed analysis for these cases [1].

In [13], an analytical performance model has been proposed: For a 2D systolic array with $R$ rows and $C$ columns (i.e. $\mathcal{N}=RC$ MACs) and workload matrices of dimensions $M$, $N$ and $K$, [13, Eq. (4)] gives the calculation time:

$\displaystyle\tau_{2D}=(2R+C+T-2)\lceil\nicefrac{{M}}{{R}}\rceil\lceil\nicefrac{{N}}{{C}}\rceil$

(1)

Given large matrix sizes such that the given 2D array could not map the entire computation at once, serialization is required. The dimension $M$ will be mapped across the rows, if in case the number of rows is insufficient. The entire mapping requires $\lceil\nicefrac{{M}}{{R}}\rceil$ steps. Similarly dimension $N$ is mapped across the columns, leading to $\lceil\nicefrac{{N}}{{C}}\rceil$ steps to complete the mapping. The total number of serial steps required is given by $\lceil\nicefrac{{M}}{{R}}\rceil\lceil\nicefrac{{N}}{{C}}\rceil$.

In each step of the serialization it takes ($R+C-2$) cycles to fill the entire array, since the elements of IFMAP and Filter matrices are fed simultaneously. In OS dataflow the computation for each OFMAP pixel is done in-place within a MAC unit, thus it requires $K$ cycles to generate one OFMAP pixel. Since multiple MAC units are running in parallel, the latency of computation can be hidden for all but one MAC unit, which gets the data at the end. This MAC takes another $K$ cycles after the array is filled. Once all the computation is finished, it takes another $R$ cycles to remove all the generated outputs from the array. The term, ($2R+C+T-2$) therefore indicates the runtime for a single serial step or fold. The authors also propose an optimization method to find the optimal array sizes for a given workload that minimizes this runtime.

The given formula naturally extends to a third dimension. Using the OS dataflow for 3D, the work among tiers is split up in $K$-dimension, i.e. along $K$ in the workload. The workload is not split up along $M$ and $N$. Thus, each of the $\ell$ tiers works on the partial sums with an input workload matrix dimension of $M$, $N$ and $T/\ell$. At the end, the partial sums are accumulated; this requires $\ell-1$ additions. This yields the following formula for the runtime of a 3D systolic array with $R^{\prime}$ rows per tier and $C^{\prime}$ columns per tier:

$\displaystyle\tau_{3D}=(2R^{\prime}+C^{\prime}+(\nicefrac{{K}}{{\ell}}+\ell-1)-2)\left\lceil\nicefrac{{M}}{{R^{\prime}}}\right\rceil\left\lceil\nicefrac{{N}}{{C^{\prime}}}\right\rceil$

(2)

Please note that the constraint for the MAC count changed as the 3D array has $\mathcal{N}=\ell R^{\prime}C^{\prime}$ MACs. Thus, the method from [13] can be applied to optimize the array dimensions for all tiers for the workload by changing the objective function to Eq. 2 and using $\mathcal{N}/\ell$ MACs and a workload size of $M$, $N$ and $K/\ell$. To generate the distributed OS dataflow, each tier has the same array dimensions.

We compare the performance of 3D and 2D using Eq. 1 and Eq. 2. We assume an identical number of MACs that are evenly split up among tiers. (Eq. 1 holds with $\mathcal{N}=RC$ and Eq. 2 holds with $\lfloor\mathcal{N}/\ell\rfloor=R^{\prime}C^{\prime}$.) We round down to avoid resource over-provision.

We compare the power of a 3D-IC with TSVs or MIVs against a 2D-IC. TSVs have a very high capacitance of about 10fF [20], while MIV only have about 0.2fF capacitance [21].

We found that a static power analysis is insufficient. The reason lies in the special dataflow of a 3D-array, in which the horizontal links are heavily utilized while the vertical links are only used for partial sum accumulation. Hence, the switching activities of horizontal and vertical links vary.

We conduct post-synthesis power analysis for a 3-layer IC in 15 nm node with an example workload of $M,N$=$128$ and $K$=$300$ using Synopsys^® PrimeTime PX.

The power consumption of an array with 16384 MACs per layer is shown in Tab. II (excluding power from data transmission from memory), along with the difference in power consumption vs. a 2D-IC with a similar total number of MACs (49284 MACs). We find that TSV-based 3D-IC requires 5.39% less power than a 2D-IC and a MIV-based 3D-IC requires 2.21% less power. As expected, MIVs are more frugal than TSVs.

3D-ICs do draw less power than 2D-ICs because of the special properties of the dataflow. This demonstrates the relevance of dynamic power analysis for 3D systolic arrays.

As thermal performance is one of the most urgent issues of 3D integration [3], we conduct a thermal analysis with HotSpot 6.0 [15]. We chose a three-layer 3D-IC with 4096, 16384 and 65536 MACs per layer and a workload of $M,N=128$ and $K=300$. The respective 2D-IC has as 12321, 49284 and 197136 MACs, which is approximately the MAC count of the 3D case.

The results are shown in Fig. 9 as a boxplot. For 3D, we split the data into the layer near the heatsink (bottom) and the rest (middle). The temperature variability comes from different switching activities and cooler MACs at the borders of the IC as of their fewer neighbors.

3D and 2D ICs get hotter for larger MAC counts. Furthermore, 3D ICs get hotter than 2D ICs. The TSV-based and the MIV-based 3D-ICs are not exceeding their thermal budget. This is a promising finding for 3D-ICs practically used for DNN-accelerators.

The MIV-based IC is hotter than the TSV-based IC. This is counter-intuitive due to the difference in parasitics of TSVs and MIVs. The reason lies in the vast number of vertical links. The large TSVs increase area, enhance heat dissipation and reduces the temperature. In a real system, one would apply TSV-saving schemes to improve area and yield, which will increase the temperature above the level of MIV-based ICs.

We implement the 2D and 3D arrays in a 15 nm node with 8b inputs and 16b outputs for 1 GHz clock frequency. We take TSV area plus keep-out-zone (KOZ) from [20] and MIV area from [22].

The TSV-based 3D-IC is larger than the 2D array from additional area for logic, TSVs and KOZs. Monolithic integration only adds a few percent overhead vs. 2D, as no KOZs are required.

We plot the runtime per chip area to evaluate the area-impled trade-offs. This is plotted in Fig. 9 normalized against 2D for different tier counts for one exemplary given workload from Resnet50. Based on our previous discussion for runtime, we chose a workload that yields a performance benefit for 3D ($M$=$64$, $N$=$147$, $K$=$12100$). The results are shown in Fig. 9 for a TSV-based (orange) and for a MIV-based (purple) 3D-IC.

For 4096 and 32768 MACs, the performance per area of the 3D-IC is worse by up to 75% than the 2D-IC. For 266144 MACs, the area per performance is improved for more than 4 layers by 1.27$\times$ to 2.83$\times$ (cf. 9). This finding underlines again that 3D integration is useful for large MAC counts.

We took a worst-case approach to the TSV count, as we provide a dedicated TSV array connecting each pair of MACs. If we apply TSV-reduction architectures (cf. Sec. III-A), TSV-based 3D-ICs will come off better.

MIV-based 3D-IC enable a better performance per area than TSVs: While the performance per area for 4096 MACs is similar to 2D-ICs, MIV-3D-ICs improve performance per area by up to 7.9$\times$ for larger MAC counts. The general trend is that higher MAC counts and number of tiers improve the performance advantage of 3D vs. 2D.

Two tiers with face-to-face bonding can be manufactured at time of writing this paper. For this, 3D integration allows for 1.19$\times$ to 1.97$\times$ better performance per area.