Auxiliary Features-Guided Super Resolution for Monte Carlo Rendering

Monte Carlo rendering algorithms are now widely used to generate photo realistic computer graphics images for applications such as visual effects, video games, and computer animations. These algorithms generate a pixel’s color by integrating over all the light paths arriving at a single point [CPC84]. To rendering a high-quality image, a large number of rays need to be cast for each pixel, which makes Monte Carol rendering a slow process.

A great amount of effort has been devoted to speeding up Monte Carlo rendering. The core idea is to reduce the number of rays for each pixel. For instance, numerous denoising algorithms are now available to reconstruct a high-quality image from a rendering produced at a low sampling rate. Such Monte Carlo denoising algorithms often use auxiliary features generated by a rendering algorithm to help denoise the noisy rendering result. The recent deep neural network-based denoising algorithms can now generate very high-quality images at a fairly low sampling rate [BVM*17, CKS*17, KKR18, GLA*19].

Monte Carlo rendering can also be sped up by reducing the number of pixels to render. For example, pixels from the frames that have already been rendered can be warped to generate frames in-between existing frames to increase the frame rate [BDM*21] or to generate future frames to reduce latency [GFL*21]. Another approach is to only render one pixel for a block of neighboring pixels to further reduce the total number of pixels to render. This can be implemented by first rendering a low-resolution image and then applying super resolution to increase its resolution [XNC*20, HLM*21]. As super resolution is a fundamentally ill-posed problem, it alone often cannot recover high-frequency details from only the low-resolution rendering. To address this problem, Hou et al.render a high-resolution rendering with a low sampling rate and use that together with the high-resolution auxiliary features to help super resolve the low-resolution rendering rendered at a high sampling rate. While this method produces a high-quality result, it needs to render the high-resolution image at a low sampling rate, which still takes a considerable amount of time [HLM*21].

Can we only use the fast-to-obtain high-resolution auxiliary features without the high-resolution-low-sample rendering to effectively assist super resolution of the corresponding low-resolution rendering? If so, we can further speed up Monte Carlo rendering. We are encouraged by the recent work on neural frame synthesis that showed fast-to-obtain auxiliary features of the target frames can greatly help interpolate or extrapolate the target frames [BDM*21, GFL*21]. On the other hand, Hou et al.showed that using a wide range of auxiliary features and the high-resolution-low-sample rendering help super resolution more than only using a subset of auxiliary features within their own deep neural network-based super resolution framework [HLM*21]. Therefore, if we only use a small number of fast-to-compute auxiliary features, we need to have a better super resolution method.

This paper presents a Cross-modality Residual Densely-connected Swin Transformer (XRDS) for super resolution of a Monte Carlo rendering guided by its auxiliary features. For the seek of speed, we only use two auxiliary features: albedo and normal. To effectively use these features, we design a super resolution network based on Swin Transformer that recently has been shown powerful for a wide variety of computer vision tasks. Our Transformer network has two branches, one for the low resolution rendering and the other for the auxiliary features. Such two branches are designed to perform cross-modality fusion to effectively use auxiliary features to assist super resolution of the low-resolution rendering. While the auxiliary feature branch consists of convolutional blocks, the branch for the low-resolution rendering consists of a sequence of residual densely-connected Swin Transformer blocks to extract effective features. The features from the two branches are combined together using a cross-modality fusion module and are finally used to generate the high-resolution high-quality rendering.

This paper contributes to Monte Carlo rendering as follows. First, we present the first super resolution approach to Monte Carlo rendering that only uses fast-to-compute high-resolution auxiliary features to enable high-quality upsampling of a low-resolution rendering. Second, we design a dedicated Cross-modality Swin Transformer-based super resolution network that can learn to effectively combine high-resolution auxiliary features with the corresponding low-resolution rendering to generate the final high-resolution high-quality image. Third, our experiments show that our method outperforms super-resolution and denoising methods in producing high-quality renderings.

This section briefly discusses relevant work to our paper, including Monte Carlo denoising, super resolution, and vision Transformers.

Recently, deep neural network-based denoising methods have shown impressive performance. These methods learn to reconstruct high-quality renderings from small number of samples. In their seminal work, Kalantari et al.estimated optimal filter parameters using a multi-layer perceptron neural network [KBS15]. Bako et al.estimated spatially adaptive kernels for denoising in a convolutional manner [BVM*17]. Vogels et al.extended the concept of kernel prediction methods to temporal denoising [VRM*18]. With asymmetric loss functions, their method could produce high-quality results for a sequence of frames. Chaitanya et al.developed a recurrent autoencoder to denoise a sequence of frames while maintaining temporal stability [CKS*17]. Xu et al.developed an adversarial approach to Monte Carlo rendering denoising that can greatly reduce artifacts such as blurs and unfaithful details from denoising results [XZW*19]. Gharbi et al.developed a kernel splatting network that reconstructs the final image by splatting samples to pixels according to the estimated splatting kernels [GLA*19]. Munkberg et al.proposed to filter auxiliary layers of individual samples [MH20]. Their method works well on outliers and complex visibility. Hasselgren et al.proposed a neural spatial-temporal sampling method for Monte Carlo video denoising [HMS*20]. Their method first estimates the sampling map from the temporal reprojection and auxiliary features and then denoises the resulting image generated using the sampling map to produce high-quality results. Zheng et al.proposed an ensemble denoising technique that learns to combine multiple denoiser together [ZZXY21]. Yu et al.designed a transformer-based neural network for Monte Carlo denoising [YNL*21]. Their network consists multi-scale feature extractor and a self-attention module and achieved promising results. Unlike these denoising methods, our method explores an orthogonal approach that speeds up Monte Carlo rendering by reducing the number of pixels to render via super-resolution.

Super resolution has recently been explored to speed up rendering. Xiao et al.designed a super resolution network that takes both the low-resolution rendering and the corresponding low-resolution auxiliary features as input and outputs a high-resolution frame. They leveraged neighboring frames to further improve super resolution quality [XNC*20]. While their method was designed for a rasterization-based renderering engine, in principal, it can be applied to Monte Carlo rendering. Thomas et al.combined super resolution and Monte Carlo denoising for videos. Their network takes a low-resolution rendering as well as a warped previous frame as input and produces a high-resolution frame [TLP*22]. However, super resolution is essentially an ill-posed problem and cannot fully recover missing high-frequency visual details from only the low-resolution input. To address this problem, Hou et al.developed a super resolution approach based on multiple-resolution sampling. Their method first renders a low-resolution image at a high-sampling rate and a high-resolution image at a low sampling rate. Their method then exploits the high-resolution noisy image to recover high frequency visual details [HLM*21]. While their method generates high-quality renderings, it needs to render the high resolution noisy image and auxiliary features, which takes a considerable amount of time. Different from the above methods, our method obtains high-frequency information from only fast-to-compute high-resolution auxiliary features as inspired by recent interpolation and extrapolation methods that use fast-to-compute auxiliary features of the target frames to help generate the target frames [GFL*21, BDM*21].

This paper proposes a super resolution method guided by the fast-to-compute auxiliary features to speed up the Monte Carlo rendering. Our method takes a low resolution rendering $I_{LR}$ and its high-resolution fast-to-compute auxiliary features $A$ as input, and outputs the corresponding high-quality high-resolution result $I_{SR}$. The high-resolution auxiliary features provide the essential high-frequency information for the super-resolution.

Different from the previous work [HLM*21], which leverages a wide range of auxiliary features, our method only employs the auxiliary features that can be computed very fast [BDM*21], including albedo and normal. On the one hand, although our method doesn’t leverage the shading layers, albedo and normal could provide a lot of high-frequency information, e.g., the texture of the material, which is essential for super-resolution. As we will show, it can help improve the super-resolution results. On the other hand, albedo and normal can be computed fast [BDM*21]. It not only reduces the rendering time but also enables us to render these high-resolution layers at a relatively higher sampling rate, which typically contains fewer artifacts, such as aliasing.

We design a cross-modality transformer network to effectively fuse two categories of visual input, namely the low-resolution rendering and its corresponding high-resolution auxiliary features, to recover visual details. Figure 1 shows the architecture of our network. It contains two parallel branches, one for the low-resolution rendering and the other for the corresponding high-resolution auxiliary features.

We then obtain the downsampled features $\{D_{i}\}_{i=0}^{N-1}$ with a group of deshuffle layers [HLM*21], which is able to downscale the feature while keeping the high frequency information.

where $f_{DSF}(\cdot)$ indicates the deshuffle layer.

We design RDST by combining the ideas of the Residual Densely-connected Network (RDN) [ZTK*18] and Swin Transformer [LLC*21]. We are specifically inspired by SwinIR [LCS*21] that explores Swin Transformers for image restoration tasks. It replaces traditional convolutional layers with Swin layers in residual blocks, allowing for the learning of more descriptive features and delivering impressive results. Taking inspiration from RDN [ZTK*18], we introduce RDST, where the convolution layers in densely-connected blocks are replaced with Swin layers. As shown in Figure 3, RDST consists of a sequence of densely-connected Swin Transformer blocks and a local feature fusion block. For the densely-connected Swin Transformer blocks, we shift the windows. We also use a local skip connection to fuse the features from the shallow layer.

We implement our network in PyTorch. We train our super resolution network on examples of size $256\times 256$. We select Adam [KB14] with $\beta_{1}=0.9$, $\beta_{2}=0.999$ as the optimizer. The learning rate is set to 0.0001. We train the network for 400 epochs with a mini-batch size of 16 for our $4\times$ super resolution models, and we fine-tune our other models using the $4\times$ pretrained weights. It takes about one week to train a single model using 4 Nvidia A40 GPUs. We adopt the BCR dataset [HLM*21] as the training dataset. BCR dataset contains 2449 images from 1463 scenes rendered by Blender Cycles. Following MSSPL [HLM*21], we use 2126 images from 1283 scenes for training, 193 images from 76 scenes for validation, and 130 images from 104 scenes for testing.

We evaluate our network by quantitatively and qualitatively comparing them with state-of-the-art image super resolution methods and the Monte Carlo denoising methods on the BCR dataset [HLM*21] and the Gharbi dataset [GLA*19]. We also conduct the ablation study to examine our method. Following [HLM*21], we adopt Relative Mean Square Error (RelMSE) and PSNR to evaluate our methods in the scene linear color space and the sRGB space, respectively. Please refer to the supplementary material for an interactive demo that provides more results.

The fusion for the high-reflection parts is challenging. Our method produces high-frequency visual details by two means: 1) train a neural network to learn to recover high-frequency information from low-resolution input and 2) use high-frequency information from the high-resolution albedo and normal maps. Our neural network can learn to produce visual details for many examples. However, super resolution from a low-resolution input alone is necessarily an ill-posed problem. In the high-reflection parts of the scene, such as the example shown in Figure 8, when the high-resolution normal and albedo map cannot, by its nature, provide high-frequency details in those regions, our method may fail.

Compared to CNN-based methods, our method is slow. However, compared to another Transformer-based method [YNL*21], our method uses less peak memory (0.89Gb vs 30.56Gb) and is faster (1.0s vs 2.5s) when producing a 1024x1024 image using an Nvidia A40. Research on fast transformers has been advancing quickly recently. Patro et al. [PA23] offer an extensive review of efficient vision transformers. Through the advancement of effective token mixing strategies and efficient MLP layers, vision transformers can be significantly accelerated [LWZ*22, GHW*22, YPL*22]. For example, both CMT [GHW*22] and WaveViT [YPL*22] outperform EfficientNet [TL19] while maintaining a lower computational complexity. Moreover, several transformer hardware accelerators have been introduced to expedite Transformer networks, such as SwiftTron [MDC*23]. We believe that our method can benefit from the quick advance of research on Transformer.

In this paper, we specifically explored albedo and normal as quick-to-compute auxiliary features. However, we acknowledge that other auxiliary features, such as a whitted ray-traced layer, could offer valuable high-frequency information and be generated fast. Incorporating such a layer can potentially improve the performance of our method. Unfortunately, the BCR dataset doesn’t contain such layers. We plan to explore this in our future research.

This paper explored high-resolution fast-to-compute auxiliary features to guide super resolution of Monte Carlo renderings. We developed a dedicated cross-modality Transformer network to fuse high-resolution fast-to-compute auxiliary features with the corresponding low-resolution rendering. We designed a Transformer-based cross-modality module to fuse the features from two modalities. We also developed a Residual Densely-connected Swin Transformer block to learn more representative features. Experimental results indicate that our proposed method surpasses existing state-of-the-art super-resolution and denoising techniques in producing high-quality images.