Ditto: Accelerating Diffusion Model via Temporal Value Similarity

Diffusion models have recently achieved superior performance in various image generation tasks [73, 13, 92]. Inspired by the natural process of diffusion, diffusion models generate images by employing its reverse process [89]. Fig. 1 shows the image generation method in the diffusion model which comprises of forward and reverse diffusion process. It first executes the forward diffusion process, which involves iteratively injecting noise into the original image. Then, the reverse diffusion process generates an image by recurrently removing noise from the image.

Diffusion models utilize a neural network as a denoising model composed of sequentially connected blocks (group of layers) to reduce noise in reverse process [81, 60, 68]. Originally, the denoising model is composed of ResNet Blocks and Attention Blocks as shown in Fig. 2. However, recently, various types of denoising models have been employed in diffusion models, each composed of different types of blocks [68, 66, 57]. For instance, when using conditional techniques (IMG and SDM in Table. I), the Attention Block is replaced by a Conditional Latent Diffusion Transformer Block, resulting in a more complex structure. Moreover, diffusion transformers (DiT and Latte in Table. I) that use only transformer blocks without ResNet Blocks have also emerged [66, 57]. Since there is a wide variety of denoising models with different block structures, each model exhibits distinct layer dependencies and computation flow. During the reverse process, the diffusion model uses the same network and weights for each time step, recursively feeding output from the previous time step ($T_{t}$) as the input for the current step ($T_{t-1}$) [81, 60, 29, 68]. Through the process, diffusion models achieve higher image quality and diversity than previous image generation DNNs, such as GAN [20, 40] and VAE[42, 31].

Despite their advantages, diffusion models incur significant computational overheads in the reverse diffusion process due to their iterative characteristics and the high computational demands of the denoising model [39, 81, 51, 9]. Moreover, the recursive feedback mechanism prevents the parallelization of time steps, leading to long execution times [81, 44] and high arithmetic intensity [17].

Since the diffusion models employ an identical denoising model and its weight for their entire time steps, a continuous adjustment to inputs would generate similarity of the data in layer operations. Thus, we assume that each data element within activations would exhibit a high degree of value similarity between consecutive time steps. To validate our assumption, we conduct detailed analyses, focusing on the similarity between adjacent time steps. In Fig. 3, the similarity within input activation of two layers (e.g., conv-in and up.0.0.skip) is measured through cosine similarity, which is widely used for measuring similarity between multi-dimensional data. The analysis reveals that the similarity of activations exceeds 0.94 across these layers at various time steps (e.g., from time step 25 to 24, and 2 to 1).

We further measure the temporal similarity of all layers for every adjacent time step in various diffusion models, as shown in Fig. 3(b). The details of the diffusion model benchmark in the analysis are provided in Table. I. Our analyses demonstrate that the average cosine similarity in each model consistently surpasses 0.947, with an average similarity of 0.983 across various diffusion models. We additionally measure the spatial similarity of layers, as previous research [58] leverages the spatial similarity inside the layer of computational imaging DNNs. The results show that the diffusion models present a spatial similarity of 0.31 on average, which is lower than the temporal similarity. Since the temporal similarity originates from the iterative process, the inherent characteristic of the diffusion models, the value similarity would exist in all diffusion models.

As high value similarity indicates a minimal difference in values, the range of the temporal differences tends to be narrower compared to the original activations. Note that a reduced value range can improve computational efficiency [46, 47, 58], we first conduct an experiment to examine the value ranges of both the original activations and the differences that can be obtained by subtracting each data element between consecutive time steps. Fig. 4(a) presents the experimental results on two layers (conv-in and up.0.0.skip) of the diffusion model. In the conv-in layer, our analysis reveals that the average value range of the original activations is 4.73, while the average range of the difference is merely 0.23. Similarly, the original activations value range is 21.88, and the difference range is 4.83 on average in the up.0.0.skip layer.

These narrower value ranges occur not only in specific time steps but across all time steps, showing the consistency of the narrower value range. To verify these characteristics across various diffusion models, we conduct further experiments, comparing the average value range of the original activations and temporal differences in all layers, as shown in Fig. 4(b). Using the same models as in Fig. 3(b), we calculate the average value range across all time steps. The experiment results reveal that the value range of temporal differences presents an 8.96$\times$ narrower than the original activations on average. Specifically, the value range of differences exposes up to 25.02$\times$ narrower in the DDPM, and at least 2.44$\times$ narrower in CHUR. These results suggest that high temporal similarity reduces the range of differences between time steps, offering an opportunity to improve the computational performance of the diffusion models [47, 58].

Based on the above observation, we find out that a narrow value range in differences would be advantageous in the quantized diffusion model. Since quantization compresses data into lower bit-width based on the value range of the data, a narrower value range in temporal differences could further reduce the bit-width required for operations [38, 2, 11]. To explore the potential benefit, we define the bit-width requirement as the minimum number of bits required to represent the value of the data. With this term, we compare the bit-width requirement of differences of consecutive time steps with the original data in the quantized diffusion model. Moreover, we also compare the bit-width requirement in the case of leveraging the spatial similarity inside layers. For this case, the method of Diffy [58], the state-of-the-art difference processing accelerator exploiting high spatial similarity in computational imaging DNNs, is adopted. Originally, Diffy targets only the spatial similarity of sliding convolution windows in convolution layers. However, as the diffusion models consist of various types of layers [68, 81, 60, 66, 97], we modify the Diffy method to support the similarity across the row dimension of input activation in fully connected layers and attention layers.

Fig. 5 shows our analysis results of the bit-width requirement in various quantized diffusion models. In the analysis, the average bit-width requirement is measured for all data elements in diffusion models, quantized with simple dynamic quantization with 8-bit activation and weight. The results show that zero temporal differences, indicating no change in values between time steps, constitute 44.48% of the total temporal differences on average. Since similar values are quantized into the same value [24], our results indicate that most values between adjacent time steps are quantized to the same value owing to high temporal value similarity, resulting in a zero differences. On the other hand, the original activations only exhibit zero value in quantization when the values are inherently zero or close to zero, thus, temporal differences show a 26.12% higher ratio of zero than activations. Moreover, due to the relatively low spatial similarity, a method leveraging temporal value similarity achieved an 18.04% higher ratio of zero values compared to the spatial difference method.

We also find out that the values with lower bit-width requirements take a large portion of the temporal differences. In the figure, the temporal differences that require a bit-width within 4 bits account for 51.52%, even excluding zero-value differences. Including zero temporal differences, those requiring 4-bit or fewer account for an average of 96.01% of the total data elements in temporal differences. These results indicate that only 3.99% of the temporal differences require more than 4-bit for representation, which exhibits significant contrast to the original activations and spatial differences, where 42.28% and 25.58% require more than 4-bit.

Our analysis reveals that a significant portion of the temporal differences between consecutive time steps can be represented with reduced bit-width compared to the original activations in the quantized diffusion model. Note that a lower bit-width reduces computational intensity [7, 94], reduced bit-width and a high portion of zero in differences can improve the computational efficiency of the diffusion model.

To verify our assumption, we analyze the relative number of BOPs [5, 50] for a single time step of diffusion models utilizing temporal differences compared to the original quantized model and the spatial difference method as shown in Fig. 6(a). The experiment utilizes our analysis results in Fig. 5. In the figure, the temporal difference approach can achieve 53.3% and 23.1% fewer BOPs on average compared to the original models and the spatial difference method. Especially, DDPM and CHUR achieve 68.8% and 71.5% fewer BOPs due to a higher proportion of zero temporal differences compared to other methods. These models exhibit 41.41% and 35.53% more zero values than the original activations and spatial difference method. We also examine whether this BOPs reduction occurs at every time step. As shown in Fig. 6(b), the last few steps achieved a relatively lower BOPs reduction because much denoising is required to generate the image in the final time steps. However, even in these steps, a lower BOPs is obtained compared to the original activations, and overall, consistent BOPs reduction is achieved across most of the time steps. Consequently, the performance of the diffusion model can be boosted through reduced bit-width and zero skipping by utilizing temporal differences for all time steps.

Moreover, we observe that in cross attention where context is used as input, the values of the context remain unchanged across different time steps (second column in conditional latent diffusion model transformer block in Fig. 2). Therefore, $K^{{}^{\prime}}$ and $V^{{}^{\prime}}$ do not change with varying time steps in the layer. With the observation, the Ditto algorithm treats $K^{{}^{\prime}}$ and $V^{{}^{\prime}}$ as weight in $Q^{{}^{\prime}}\times K^{{}^{\prime}}$ and $P^{{}^{\prime}}\times V^{{}^{\prime}}$, applying the same difference processing approach used in conventional linear layers.

However, there are several challenges in applying the difference processing algorithm to the entire process of diffusion models. First, since non-linear functions require original data to ensure numerical equivalence, the denoising model often needs the original data during execution. Second, linear layer operations require additional memory accesses to obtain the linear layer input from the previous time step in order to calculate differences. Therefore, some layers would be converted into memory-intensive operations due to the increased memory accesses and reduced computational intensity, even though diffusion models are compute-intensive networks [17]. In our analyses, temporal difference processing incurs 2.75$\times$ more memory accesses on average than original activation processing, as shown in Fig. 8. Previous work, Cambricon-D [43], also addressed this issue by modifying non-linear functions such as SiLU and Group Normalization to reduce memory overhead. However, as shown in Fig. 2, various diffusion models utilize a range of non-linear functions such as GeLU, Softmax, and Layer Normalization, limiting the effectiveness of their mechanism, particularly in models that do not use ResNet blocks, such as diffusion transformers.

To mitigate memory overheads in various diffusion models, we propose the Ditto execution flow optimization (Defo). It automatically determines whether to perform each linear layer operation using difference processing, adjusting the execution flow for layers. Fig. 9 shows a detailed execution process of the diffusion model with the difference processing and Defo. In static time, Defo applies a computing graph analysis to find all non-linear functions and check the dependency of layers. Based on the information, it modifies the difference processing algorithm by applying difference calculation and summation only before and after non-linear functions.

Even with the bypassing method, the issue of increased memory access may not be fully resolved. Therefore, it performs execution flow optimization at runtime to maximize the performance of the diffusion models. Defo stores the cycle count of each layer during the first time step ($Cycle_{act.}$), which operates with the original activations. In the second time step, it dynamically determines the efficient execution type of each layer by comparing the cycle stored from the first step with the cycle using the difference processing algorithm ($Cycle_{diff.}$) which is determined by the number of zero and lower bit-width temporal differences. If $Cycle^{L_{i}}_{act.}$ is larger than $Cycle^{L_{i}}_{diff.}$, Defo set $Exe^{L_{i}}_{diff.}$ as True, enabling the layer to be executed using the difference processing method.

As we observed in Fig. 6(b), the ratio of BOPs reduction is consistent across time steps. Based on the observation, Defo applies the execution flow of each layer determined in the second time step to all subsequent time steps. In our evaluation, it successfully selects a more efficient execution flow with 92% accuracy by using only first and second time step information, demonstrating the effectiveness and feasibility of our approach (see Fig. 17). For subsequent time steps, all layers operate with the execution type determined in the second time step. As in the second time step, difference calculation and summation are dynamically bypassed using layer dependency information, further reducing the memory overhead of the temporal difference processing method.

To further boost the performance of diffusion models, we introduce an additional optimization for layers that execute with the original activations. In hardware utilizing temporal differences, they could leverage spatial differences with minor hardware modification. Therefore, the Ditto algorithm is optimized to leverage spatial difference processing in the first time step and for layers determined by Defo to be executed with the original activations (defined as Defo⁺). Defo⁺ calculates the spatial difference between the input data sequences and utilizes the difference in the same way as temporal difference processing. As shown in Fig. 6(a), the spatial difference processing results in a reduction of BOPs compared to original activation processing, while it is higher than the temporal difference processing. Furthermore, since utilizing spatial difference processing calculates the difference within the intra-tensor, it does not require additional operations with the input and output of previous time steps, and thus does not incur memory access overhead. Consequently, the computation reduction achieved through spatial differences enhances performance compared to using the original activations.

Fig. 10 presents the overall architecture of the Ditto hardware. It consists of four main components: Encoding Unit, Compute Unit, Vector Processing Unit, and Defo Unit. Encoding Unit is a specialized hardware unit that calculates data differences and reorders the data elements to exploit the dynamic sparsity of Ditto. It first computes the differences between consecutive steps, and classifies zero value, low bit-width, and full bit-width differences. After classification, it reorders the data elements to skip zero value differences in Compute Unit and notates the full bit-width differences.

With reordered data, Compute Unit, a core unit of the proposed hardware, executes the actual layer operations of the diffusion model. We design Compute Unit as an adder tree based Multiply and Accumulate (MAC) unit that supports two types of bit-width operation, a full bit-width (8-bit) and a low bit-width (4-bit) operation. Since a large portion of the temporal differences is represented in 4-bit data, the bit-width of the baseline multiplier is set to multiplier between 4-bit activation and weights, supporting 8-bit activation by utilizing two multipliers and shifting logic. By leveraging reordered data by Encoding Unit, Compute Unit can skip zero and exploit the benefits of reduced bit-width while supporting full bit-width operation, ensuring numerical equivalent results with original operations. Vector Processing Unit operates the other special functions without linear layers such as non-linear functions. It also executes quantization and dequantization which is essential for quantized DNN [24, 91, 41, 48]. Also, it supports summation for the temporal difference processing.

The three components are designed as a pipelined architecture, a common technique for accelerators [37, 4, 22, 34], to maximize throughput and minimize latency. We set the frequency of all components as the same, and the number of each component to support the maximum throughput of the Compute Unit executing in a low bit-width activation (4-bit). Since the Ditto algorithm requires selective utilization of the three components, we design Defo Unit as a control unit, supporting the dynamic execution flow of the Defo. It stores cycles of layers with original activations at the first time step and determines the type of execution for each layer.

For the simple design of Compute Unit, we design Encoding Unit to eliminate zero differences and notate the full bit-width data during the reordering process. This design allows Compute Unit to execute the reordered data elements without zero differences while also handling full bit-width data efficiently. To support the functionality, Encoding Unit utilizes reordering logic to align data elements with the results of the classification The reordering logic receives the 2-bit signal from the classification process and determines whether to skip the data element or enqueue it into a data queue. By skipping the zero differences, Encoding Unit ensures the Compute Unit only executes data elements that require actual execution.

During the reordering process, Encoding Unit also reorganizes the data elements and notates those requiring full bit-width operations. In the Compute Unit, a single multiplier is designed to multiply low bit-width data (i.e., 4-bit) with weight, while two multipliers are used for full bit-width data (i.e., 8-bit). To support the design, Encoding Unit divides the data elements into high and low bit parts and enqueues them separately. Then, it notates the data as full bit-width data to apply shift operation to the higher part. Since Compute Unit supports shift operation per two multipliers for area efficiency, Encoding Unit reorders the data to align the high bit part with the multiplier with shift operations. As the order of accumulation is independent in the matrix multiplication, the high and low parts do not have to be accumulated in the first stage of the adder tree. Therefore, it only needs to reorder the two parts of the data to ensure the higher part is directed to the multiplier with a shifter. Consequently, it sends four 4-bit data with 2-bit flag for high bit part to a single adder tree unit of Compute Unit. Note that Encoding Unit consists of simple logic, it can be implemented with low area overhead and high throughput. Moreover, as subtraction and comparison can be combined into one cycle and queuing also can be done in one cycle, it can achieve low latency for its operation.

As shown in the figure, the PE receives the data and the metadata for higher part of full bit-width data from the Encoding Unit. For 4-bit data, multiplication between activation and weight is straightforwardly executed through the multiplier, and then results are accumulated via an adder tree. Besides, for 8-bit data, the PE utilizes two multipliers to process each part of the data, the higher 4-bit and lower 4-bit parts, separately. After multiplication, it recognizes the higher 4-bit through the metadata and applies shift operation to the result of a higher 4-bit part and then accumulates with an adder tree. As previously mentioned, those two parts of the data do not have to be accumulated in the adder tree at once, they are accumulated through a partial sum register within PE.

To support the functionality of the Defo Unit, the table requires entries for each layer. Based on our evaluation, the maximum number of layers of the diffusion model is 347, so we design it to have 512 entries to align a power of 2. Additionally, according to our evaluation, first time step and second time step cycle can be represented with 16-bit. Therefore, each entry is designed to have 33-bit, which includes 16-bit for the first time step cycle, 16-bit for the second time step cycle, and 1-bit for the later time step decision. Since the Defo unit is a control unit that only determines the type of execution path, it does not need to be scaled for throughput, unlike other hardware components. Therefore, it consumes only 0.01% of the total hardware area.

To enable execution of the Ditto hardware, the CPU first sends the weights and input data to the DRAM in the Ditto hardware, along with the layer instructions to the Defo Unit. Since the DRAM and buffer memory store full bit-width (8-bit) data, we set the main interconnect between the DRAM, buffer memory, Encoder Unit, and Vector Processing Unit to operate on 8-bit data units. Once all the data is stored, the Defo Unit initiates the operations of each unit based on the layer instructions.

For communication during layer operations, we design the interconnect connected to the Compute Unit to support different bit-width compared to the main interconnect. The Encoder Unit sends difference and weight to the Compute Unit in sets of four 4-bit and four 8-bit data units, while the Compute Unit transfers accumulated data in 32-bit units to the Vector Processing Unit. These sets of data are aligned with the inputs and outputs of the processing units in the Compute Unit, ensuring that they remain unchanged whether the operation requires 4-bit or 8-bit data. After the Vector Processing Unit applies quantization to the output of the activation functions, it stores the resulting 8-bit data in the current activation buffer, and the hardware begins processing the next layer.

For the evaluation, we utilize seven diffusion models, shown in Table. I: one pixel-space unconditional diffusion model (DDPM), two latent-space unconditional diffusion models (BED, CHUR), two latent-space conditional diffusion models (IMG, SDM), and two diffusion transformer (DiT, Latte). We apply quantization to those models except for diffusion transformers with state-of-the-art quantization methods, Q-Diffusion [50]. Since this method requires an offline calibration process to determine the scaling factor, we perform calibration on offline based on their repository. In the similar way, dynamic quantization is adapted to two diffusion transformer models (DiT, Latte).

Table. II presents our evaluation on the accuracy of the diffusion model with the Ditto algorithm. We employ various evaluation metrics commonly used in image generation tasks: Frechet Inception Distance (FID) [28], Inception Score (IS) [75], and CLIP Score (CS) [27]. As shown in the table, the Ditto algorithm preserves the accuracy of all diffusion models compared to baseline FP32 models.

The Ditto hardware is evaluated using an open-source cycle-accurate simulator from Sparse-DySta [14]. The simulator adopts hook functions provided by PyTorch [65] to utilize dynamic sparsity in real DNN workloads. We modify this functionality to detect zero value, low bit-width, and full bit-width differences using actual input activation data in the diffusion model. We design Tensor Core [36] as the baseline hardware in the simulator and extends to our hardware design. Additionally, it is modified to support spatial difference processing instead of the original activation execution and integrated into the simulator (defined as Ditto⁺ hardware). To measure the area and power, Synopsys Design Compiler with a FreePDK 45nm library [84] is used to evaluate the core unit and CACTI [59] to measure the area and energy consumption of memory modules.

For evaluating GPU system, we select an NVIDIA A100 GPU [63]. We adopt the performance measurement method of the previous work [87] to compare GPU with proposed hardware. Additionally, three baseline hardware accelerator designs are selected. As a quantized model is utilized in our evaluation, an integer MAC unit-based Tensor Core-like unit is used as the baseline (defined as ITC). We additionally compare with hardware that employs difference processing methods: Diffy [58] and Cambricon-D [43]. Diffy utilizes spatial differences between sliding windows in convolution operations. Since the diffusion model also uses fully connected and attention layers, we extend the Diffy method to support difference processing along the row dimension of input activations in fully connected and attention layers for a fair comparison. Cambricon-D exploits temporal differences in diffusion models. It also modifies the diffusion model using a software technique called sign-mask data flow, which bypasses difference calculation and summation for non-linear functions, such as SiLU and Group Normalization, to reduce the memory overhead of the temporal difference processing mechanism. Unlike Ditto, it uses outlier PE designs to support full bit-width operations. Also, it does not check layer dependency for non-linear functions and process attention layers with full bit-width operations. For a fair comparison, our dependency check technique and difference processing mechanism of attention layers integrate into the Cambricon-D.

All baseline and Ditto hardware is designed to execute 8-bit activation and 8-bit weight (A8W8) quantized models, as this configuration preserves accuracy [50, 26, 77]. We adjust the number of PE for iso-area comparison as shown in Table III. We fix the SRAM size and frequency to the same configuration across the hardware designs. We also set the frequency of all components in the Ditto hardware (e.g., Encoding Unit, Vector Processing Unit, Defo Unit) to the same frequency as the PEs, as described in Section V-A.

To evaluate the performance of Ditto hardware, we first compare it with other hardware designs in terms of speedup and energy consumption, as shown in Fig. 13. In the speedup evaluation, all hardware accelerator design achieves high performance over the GPU, due to the utilizing dedicated hardware design. Compared to ITC, the Ditto hardware obtains 1.5$\times$ speedup on average, achieving the highest speedup across the other difference processing based hardware. Moreover, the Ditto⁺ hardware exhibits 1.06$\times$ faster results compared to the Ditto hardware. This result aligns with the analysis in Section. III-B, spatial difference processing obtains potential speedup over the original activation execution. Diffy also exploits spatial differences. but shows 24% lower performance compared to the Ditto hardware. Consequently, exploiting temporal difference processing is essential in diffusion models.

The Ditto hardware also shows a 1.56$\times$ speedup compared to Cambricon-D. While Cambricon-D exploits temporal difference processing, their design requires full bit-width operation through outlier PEs. Consequently, with the same area budget, the Ditto hardware can accommodate more PEs to handle reduced bit-width operations and achieves additional speedup through zero skipping, resulting in better performance. Additionally, while Cambricon-D mitigates memory overhead from temporal difference processing by utilizing sign-mask data flow for SiLU and Group Normalization, it cannot be applied to diffusion models using other non-linear functions such as GeLU, Softmax, and Layer Normalization (Fig. 2). However, our Ditto hardware can fully reduce memory overhead by automatically determining the optimal execution flow for each layer in all diffusion models through Defo.

We also evaluate the energy consumption of the Ditto hardware with other hardware designs. Similar to speedup experiments, all of the dedicated hardware achieves lower energy consumption over the GPU. The Ditto and Ditto⁺ hardware achieves 17.74% and 22.92% energy saving over ITC, which is larger than other difference processing based hardware. These energy savings are due to reduced execution times, which lower the energy consumption of Compute Unit by exploiting both dynamic sparsity and bit-width. Diffy also shows similar results with speedup, achieving 14.3% energy saving over ITC. However, Cambricon-D exhibits higher energy consumption than ITC on average. The result comes from a few benchmarks, such as BED, CHUR, and SDM, showing notably higher energy consumption. In these benchmarks, significant memory overhead occurs by the temporal difference processing algorithm. This is due to the large input and output sizes of the linear layers that non-linear functions cannot be resolved by sign-mask data flow. While the Ditto hardware also faces the overheads as shown in Fig. 16, it mitigates these overheads through Defo. As a result, the Ditto hardware achieves 43.24% energy savings compared to Cambricon-D.

Additionally, we evaluate the overhead of the additional components in the Ditto hardware. Since we adopt a fully pipelined architecture in the accelerator design, the Ditto hardware overlaps the Encoding Unit, the Vector Processing Unit, and the Defo Unit with the execution of the Compute Unit. As a result, the latency overheads of the Encoding Unit, the Vector Processing Unit, and the Defo Unit only account for 0.1%, 0.17%, and 0.1% of the total latency, respectively. The energy consumption of these units accounts for only 2.23%, 2.9%, and 0.0001%, respectively in the Ditto hardware.

As temporal difference processing increases memory accesses, we evaluate memory access of hardware designs that leverage temporal differences. In Fig. 14, we find out that the memory accesses in all hardware designs are higher than the baseline ITC. Cambricon-D incurs 1.95$\times$ more memory accesses than ITC, and Ditto and Ditto⁺ show 1.56$\times$ and 1.36$\times$ more accesses, respectively. The Defo algorithm automatically reduces memory overheads of memory-intensive layers, while Cambricon-D only reduces specific layers. As a result, Ditto and Ditto⁺ achieve fewer memory accesses than Cambricon-D and demonstrate greater generality.

In order to further demonstrate the advantages of our hardware over other designs utilizing temporal differences, we conducted a detailed comparison between the Ditto and Cambricon-D, which also leverages temporal differences for diffusion models. Since the software optimization of Cambricon-D and Ditto can be applied to each other, we apply the software techniques of the Ditto algorithm to Cambricon-D and sign-mask data flow of Cambricon-D to the Ditto hardware, as shown in Fig. 15. In the figure, Cambricon-D achieves a 1.16$\times$ speedup when all Ditto algorithm techniques are applied and the Ditto and Ditto⁺ hardware achieve 1.068$\times$ and 1.055$\times$ speedup through the sign-mask data flow. These results indicate that both hardware can benefit from the software techniques of the others since those are complementary.

However, all of the Cambricon-D design shows lower performance than the Ditto hardware due to limitations in outlier PEs based design, performing original activation execution with a smaller number of PEs. Consequently, even with the Defo technique, the cycles for original activation execution are too high, causing memory overhead reduction to be offset by compute overhead. To effectively address memory overhead in temporal difference processing, a design like Ditto hardware, which dynamically selects bit-width, is more efficient.

Several design space explorations are conducted to analyze the effectiveness of the Ditto algorithm and hardware. Firstly, we examine various design choices in Ditto to identify the contribution of each technique, as shown in Fig. 16. The accelerators that benefit from the Ditto algorithm can be categorized into two types. One type of accelerator leverages dynamic sparsity (defined as DS) like sparse accelerator [88, 18, 87]. Others utilize dynamic bit-width (defined as DB) like Bit Fusion [78] or DRQ [82]. As DS does not support dynamic bit-width, they require multipliers that support the multiplication of 8-bit data and weights. Besides, DB is designed to multiply 4-bit data with weight, since DB supports dynamic bit-width. Then, we categorize the software technique of the Ditto algorithm, as attention difference, Ditto (with Defo), and Ditto⁺ (with Defo⁺).

In the figure, DB and DS exhibit more cycle counts than the baseline ITC. While DS and DB can improve their performance in computation cycles, they involve high memory stall cycles due to the temporal difference processing. Combining DB and DS, and applying attention differences can reserve performance improvement over the baseline, but they also suffer from high memory stall cycles, limiting their advantages. Besides, Ditto and Ditto⁺ can effectively decrease memory stall cycles by applying Defo. As a result, Ditto shows slightly higher compute cycles than DB&DS&Attn. design, but 39.24% lower memory stall cycles, achieving 18.32% performance improvement.

To assess the impact of Defo on changes in execution type, a further analysis of Defo is conducted as shown in Fig. 17. The top part of the figure shows the portion of layers that are changed back to their original activation execution through Defo. In the default Defo, it changes 14.4% of the layer into the original activation execution on average. Besides, it changes 38.29% of the layer in Defo⁺. Since performance improvements occur in the first time steps by utilizing spatial differences, the threshold to change the execution flow of each layer is reduced. In particular, 81.6% of the layer change execution flows into spatial difference processing in the Latte. Since Latte is a video generation task, there is spatial similarity between frames, which means that using spatial differences techniques can be beneficial. However, other models show that 63% of the layers do not change their execution flow even with the Defo⁺. As a result, temporal difference processing is essential to accelerate various diffusion models.

To evaluate the feasibility of the Defo, we also measure the accuracy of Defo at the bottom of the figure. The accuracy is measured by assessing whether the execution flow determined by Defo for each layer is indeed the optimal execution flow. In the figure, Defo and Defo⁺ achieve high accuracy of 92% and 88.11%, even if the execution flow is fixed the execution flow at the second time step. Moreover, we observe that layers where Defo incorrectly determines the optimal execution flow result in only minimal performance degradation, as these layers are on the borderline of the threshold. To demonstrate the observation, we compare the ideal design with our hardware as shown in Fig. 18. In the figure, Ideal-Ditto and ideal-Ditto⁺ represent designs that Defo and Defo⁺ have 100% accuracy. Therefore, the ideal design always determines the optimal execution flow of each layer for entire time steps. While the Defo mechanism determines the execution flow in the second time step, Ditto and Ditto⁺ obtains 98.8% and 95.8% performance of the ideal design, indicating the feasibility and effectiveness.

Through our analyses on Fig. 6(b), we find out that temporal difference processing leads to consistent BOPs reduction compared to original activation processing across all time steps in diffusion models. However, some future models with high temporal similarity may exhibit dynamic temporal similarity across the time domain, causing BOPs reduction to vary dynamically. To explore how Ditto can support the case, we adjust the value distribution of our benchmark to make the execution type threshold dynamic and examine the impact. Moreover, we additionally design the modified Defo algorithm, dynamically determining the execution type of each layer (referred to as Dynamic-Ditto). We design the Dynamic-Ditto to change the type of processing only from difference processing to original activation processing, since it cannot get the cycle of difference processing during execution with original activation.

In Fig. 19, the accuracy of Defo shows a slight decline of 7% compared to the original benchmark shown in Fig. 17. This is because both Ditto and Dynamic-Ditto predict the benefits of difference processing based on a past time step, while the cycle reduction in the current time step changes dynamically. However, Ditto and Dynamic-Ditto achieve 98.03% and 98.18% performance of the ideal-Ditto, as the speedup in Defo primarily comes from addressing a few important layers with high memory overhead, which are relatively easy to predict. Additionally, Dynamic-Ditto slightly outperforms the original Ditto due to the dynamic Defo algorithm, which better adapts to varying conditions.