Differentiable Rendering for
Synthetic Aperture Radar Imagery

The survey of SAR simulators in [4] distinguishes two main categories of SAR simulation systems:

1. (a)

   SAR image simulators, which directly produce focused SAR images.

2. (b)

   SAR raw signal simulators, which simulate raw sensor measurements.

Category (a) simulators include examples such as RaySAR [5], CohRaS^® [6], and SARviz [7]. A comparison of these and assessment of their limitations is provided in [8]. Of these, SARviz is the only simulator that uses rasterization, while the others involve a form of ray tracing. RaySAR is the only project that is open source, but also has the most limited shading effects, relying on Phong shading [9].

Simulators of category (b) are usually based on physical optics approximations [10] and enhanced further with the shooting and bouncing rays (SBR) method pioneered in [11]. Instead of producing images directly, these physics-based simulators simulate raw sensor measurements, which the user must process themselves to coherently form images. These methods are the most realistic, but also computationally intensive. Examples in this category include Xpatch [12, 13], CASpatch [14], FACETS [15], SigmaHat [16], FEKO [17], POFACETS [18], and SARCASTIC [19]. Most simulators in this category are closed-source commercial products and do not explicitly support differentiable optimization. Of these, POFACETS and SARCASTIC are the only projects that are open source, but lack some advanced features such as SBR.

Our proposed differentiable rendering approach can be categorized as a SAR image simulator, though the data used to evaluate it was produced by a SAR raw signal simulator. Importantly, the simulators above are not suitable for solving inverse problems as they are not easily differentiable, and are therefore limited to black-box optimization.

Neural rendering is an umbrella term referring to a broad field of methods that fuse machine learning models with techniques or domain-knowledge from computer-graphics. A beneficial property of techniques in this family is that they are inherently differentiable; as they are composed of neural network building blocks, the resulting models can be easily backpropagated through. While they have been successful, it is worth noting that most neural rendering methods aim to manipulate existing images, rather than create images from raw scene representations [20]. Such techniques are thus more akin to sophisticated pixel shaders, and not renderers in the formal sense. The survey in [21] describes these collectively as “2D Neural Rendering” methods, and distinguishes them from an emerging paradigm it calls “3D Neural Rendering.”

Methods of this new paradigm, such as [22], learn to represent a scene in 3D, using a differentiable rendering algorithm to facilitate training a neural network to represent the scene. Methods of this branch of neural rendering are consequently best described as neural scene representations, and are yet another application of differentiable rendering. Cases such as [22] use volume rendering techniques, which are naturally differentiable while also memory inefficient.

Besides volumetric techniques, most differentiable rendering work in the literature has focused on rasterization-based methods from computer graphics [2]. Rasterization is computationally-efficient, but not naturally differentiable. However, smoothed variants have emerged [23, 24, 25], which have made rasterization-based methods viable. Physics-based differentiable rendering methods which model global light transport effects exist as well [26, 27], but are substantially slower. These methods are the most accurate, but the hypothetical improvements in rendering accuracy may not be worth the increased computational cost.

Use of neural and differentiable rendering methods in the context of SAR has so far been limited. Several works have proposed using CGANs for image-to-image translation from SAR to EO imagery [28, 29, 30]. A more related example is [31], which attempts to improve the realism of the RaySAR SAR image simulator [5] by training a deep neural network to estimate realistic SAR scattering effects. This falls under the “2D Neural Rendering” paradigm and is similar in principal to our use of a CGAN for the Neural Shader.

Other instances of differentiable rendering for SAR include [32], which is an academic report of an earlier iteration of this work, and very recently [33], which proposes a scheme for 3D object reconstruction from SAR images using differentiably-rendered silhouettes. Although the latter’s proposed scheme is capable of outputting simulated SAR images in tandem with the silhouettes, the simulated SAR images do not support differentiation, and thus the scheme is not well-suited for inverse problems such as 3D object reconstruction.

SAR’s unique imaging geometry makes adapting methods in differentiable rendering designed for EO imagery non-trivial. For rasterization-based methods, imaging geometry is controlled primarily through the coordinate transformation sequence. Consequently, understanding the geometry differences between these imaging modalities is necessary to design a coordinate transformation sequence.

In EO imagery, distance from the sensor affects an object’s perceived size. This results in the “oblique aerial” imaging geometry, seen in Figure 4. In computer graphics, this effect is known as “perspective,” and is typically accounted for via a “perspective projection” during the conversion to Normalized Device Coordinates (NDC) [41]. However in SAR imagery, distance from the sensor determines which range-bin reflected energy funnels into, and thus affects an object’s perceived position in range.

This leads to a natural “slant-plane” imaging geometry for points on the ground. SAR images may be projected into the “ground-plane” as part of image formation, nominally mapping image pixels to equally spaced points along the ground. Distance to the sensor is also influenced by an object’s height. Consequently, imaging objects of variable height will yield sampling distortions known as “foreshortening,” “elongation,” and “layover,” as well as SAR’s signature self-shadowing effect. These phenomena are described in detail by [35, 42] and graphically depicted in Figure 5.

The goal of the coordinate transformation stage is to transform mesh coordinates appropriately to get the desired SAR ground-plane imaging geometry, including foreshortening/elongation/layover effects. Coordinate Transformation sequences must start in world space coordinates and end in NDC. The primary sequence is described in Table 1 and illustrated in Figure 6. If the ground-plane projection were to be omitted, the result would be a SAR slant-plane imaging geometry instead. An equivalent EO “oblique aerial” imaging geometry would require omitting both the range transform and ground-plane projection, as well as changing the orthographic projection to a perspective projection.

The rasterization method is based on the soft rasterizer of [25]. The method in [25] blends facet contributions for pixels near edges to mitigate the discontinuities that would arise if trying to backpropagate through a standard rasterization operator. Soft rasterization will yield slightly distorted images but allows backpropagating more useful gradients. Examples of artifacts from soft rasterization are shown in Figure 7.

Rasterizing with our custom coordinate transform sequence produces the fragments necessary to shade the first two feature maps. The first feature map is a basic silhouette of the object. This can be produced by making a mask out of fragments that have no corresponding facets. The second feature we want is pixel-wise surface-normal information, relative to the direction of the sensor and illumination. For monostatic imagery, this is given as:

where $\hat{q}$ is the direction of the illumination, $\hat{n}$ is the per-pixel surface-normal of the object, and $\cdot$ denotes the inner-product.

The last feature map to shade is an alpha channel. Deriving the alpha channel is complicated by SAR’s self-shadowing effect. To address this, we rasterize a reference view from the perspective of the radar, which is used to determine which facets should be visible in its main beam. Any facet that is not visible from this reference viewpoint is masked out from contributing in the main view when calculating the alpha channel feature map. An example of all three shaded feature maps is shown in Figure 8.

The feature map inputs used in training involve minor data augmentation via differing values for blending radius $\sigma$ (0, 1e-5, 1e-4, 1e-3), like those seen in Figure 7. This adds diversity to the training data and should help the CGAN be more robust to distortions caused by soft rasterization. As described in Section 3, the SAR images are augmented via additive (clutter-like) and multiplicative (speckle-like) types of noise.

The CGAN is trained for 200 epochs with early stopping based on performance with the validation subset. Both the generator and discriminator are trained with the AdamW optimizer [44]. AdamW is a slight improvement over the original Adam optimizer [45] by making a correction to the implementation of weight decay. Learning rates start at 2e-4 and decay linearly every epoch to reach 2e-5 by the end of training.

Figure 9 shows example images generated by the fully trained CGAN. Although the most important results are those for the test object, we include predictions from objects in the validation and training sets for additional comparison. Empirically, the predicted and truth images appear very similar, with the most apparent difference being in the sidelobe/glint effects.

The train, validation and test object sets are compared in terms of the loss produced when generating images of each with the CGAN, in Table 2. The $L_{cGAN}$ and $L_{L1}$ losses in Table 2 are similar to those found in [3], except that the weighting for $L_{cGAN}$ is decreased from 1 to 0.5. $L_{cGAN}$ is a measure of how well the generator fools the discriminator, while $L_{L1}$ is the scaled L1 distance between predicted and truth images.

We also include an energy-normalized metric, $L_{L2}^{*}$, to account for bias due to varying object sizes. The sum-squared-error of the residual is normalized by the energy of the label image, and this ratio is converted to a dB scale. The metric is defined in equation 4, where norms are over pixels.

When using the energy-normalized metric $L_{L2}^{*}$, we can see that loss is lowest on the training set as expected, with an increase of +0.48 dB loss on the validation set, and an additional +2.07 dB loss on the test set. This is the pattern of losses we would expect, which suggests the CGAN suffers slightly from both overfitting to the training data and over-tuning to the validation data.

This subsection details losses used for the 3D reconstruction experiments. The total loss is aggregated from five sources, with weights that were tuned based on reconstruction performance using the validation object, and is given as:

The first term uses pixel-wise mean-squared-error to compare the high-fidelity simulated SAR images (at known aspect angles) to output predictions of the differentiable SAR renderer (at the same aspect angles). Despite the notorious fickleness of SAR speckles, we found this loss still works reasonably well for demonstration purposes and is simple. The other loss terms are for regularization of the mesh shape and are described next. The regularization terms are primarily intended to prevent the mesh from becoming tangled or trapped in a poor local minima while fitting.

The 3D object reconstruction is done in two levels, to reconstruct the mesh in a coarse-to-fine manner. Both levels use the stochastic gradient descent optimizer with a learning rate of 1.0, momentum of 0.9, and dampening of 0.9. Each level updates the mesh iteratively as follows:

1. 1.

   A label SAR image of the target object and its aspect angles are chosen.

2. 2.

   The estimated mesh is rendered via our renderer at the same aspect angles.

3. 3.

   The total (regularized) loss is computed by equation 5, using the prediction rendered by the model and the chosen label SAR image.

4. 4.

   The mesh is updated based on backpropagated gradients from the loss.

During the first level of mesh-fitting, the estimated mesh is initialized with a large dome shape of 42 vertices and 80 triangle faces, as seen in Figure 11. We choose a low vertex count to help the optimization loop avoid local minima early on. The soft rasterizer blending radius is $\sigma$=1e-3. The mesh at this level is updated for 1080 iterations using a batch size of two.

After the first level of mesh-fitting, the mesh estimate is up-sampled by inserting new vertices in the middle of each edge, resulting in a mesh with 162 vertices and 320 triangle faces. This up-sampled mesh initializes the second level. The soft rasterizer blending radius is also lowered to $\sigma$=2.5e-4. The mesh is updated for 540 iterations using a batch size of four. The larger batch size of the second level reduces variance of the gradients, helping the mesh fine-tune.

For quantitative evaluation, we consider two main metrics, as well as translation-invariant versions of each. The first metric is voxelized intersection over union (IoU), while the second is voxelized cosine similarity (CS). CS is a common comparison metric across many domains, although IoU is more common specifically for comparing 3D volumes. The metrics can be defined as:

where $\hat{V}$ and $V$ are voxelized volume representations of the reconstructed and ground-truth meshes, respectively. Norms and summations are over voxels, and each voxel is boolean-valued to indicate whether it lies inside or outside the mesh. Note that since voxels are boolean-valued, the numerators of equations 11 and 12 will be equal. The boolean-valued voxels also mean that the denominator in equation 12 is equivalent to the geometric-mean of the two volume sizes, while the denominator in equation 11 is instead the size of the union between the two volumes. Although both metrics yield values between [0,1], the difference in denominators makes IoU less-forgiving of small deviations between the compared volumes.

Translation-invariant versions of these metrics may also be useful. This removes the influence of potential translational misalignment, so the scores reflect only how well the 3D volumes match in terms of shape. The translation-invariant versions of these metrics take the max score over a sliding window, defined as:

where subscripts $i$, $j$, and $k$ represent translations of $\hat{V}$ in each of the three spatial dimensions, bounded by $\pm n$.

The experiment is performed on three objects: Large Ramp, Long Cuboid, and Elliptic Cone. The primary object of interest to test is the Elliptic Cone because it was reserved exclusively for testing. Results on the Long Cuboid (validation object) and Large Ramp (training object) serve as points of reference to understand how much the mesh-fitting performance may be impacted by hyperparameter over-tuning and Neural Shader overfitting, respectively. The experiment is run five times for each object at different random seeds. The reconstructed meshes for each object shown in Figure 12 are sourced from the median-scoring trial, while Table 3 provides the mean across all trials for each reconstruction metric.

The reconstruction scores from Table 3 trend similarly to what was observed with the $L_{L2}^{*}$ metric that compared rendering accuracy in Table 2. When comparing Tables 2 and 3, it is worth noting that the $L_{L2}^{*}$ score for the Large Ramp by itself was -10.15 dB, which is better than the average amongst all training objects (-8.93 dB). The most noteworthy difference is that the issue of hyperparameter over-tuning appears to be slightly more prominent in the reconstruction results than it is in the rendering accuracy results. This change may be attributable to the hyperparameters introduced by the 3D reconstruction optimization process, which presents additional opportunities for over-tuning. Similar to the conclusion from subsection 5.5.2, the reconstruction results suggest that the issues of overfitting and over-tuning exist but are both relatively minor.

In the method’s current form, objects are assumed to be of a uniform material, and changing the assumed material requires new corresponding weights for the Neural Shader. Support for multiple material types, as well as backpropagation to material properties, would broaden the tool’s utility to other differentiable SAR rendering applications. This could manifest itself as additional feature maps output by the Feature Rasterizer to convey material property context to the Neural Shader.

The current method also only supports one object segment (i.e., one mesh) at a time. Support for multiple segments would allow handling of background clutter, which is typically present in real SAR images, and may make support for multiple materials easier to implement.

This work also only considered monostatic SAR collections. Generalizing the approach to accommodate multistatic SAR modalities (where transmitter and receiver are not co-located) would expand the coverage of SAR collection modes.

Additionally, since this differentiable rendering method is based on rasterization, it is limited to modeling first-order (single-bounce) effects. Depending on object geometry and material, SAR images may have pronounced multi-path effects as well, which cannot be fully captured as-is. In our internal experiments, there were not significant differences between versions of the data simulated with single vs. double-bounce effects, but other settings could differ. A differentiable rendering scheme based on raytracing may be able to capture such multi-path effects, but is outside the scope of this current work.

In real-world SAR systems, contributions from facets that are equidistant from the sensor are coherently summed, causing the effect known as layover. However, in the differentiable renderer’s current form, contributions from facets that get mixed due to layover are combined (non-coherently) via a weighted mean. This is a limitation shared with [7], which utilizes alpha blending. This may partially explain why the peaks are not as sharp on images produced by the differentiable renderer where layover is known to occur, like in Figure 13.

Lastly, the SAR data used to train and evaluate our method was synthetically generated. The cost of collecting large quantities of measured data make it prohibitively expensive to use for training. However, future experiments should at least use measured data for validation and test objects.

We chose the task of 3D object reconstruction for demonstration purposes, but future work should investigate other applications enabled by differentiable rendering. Particularly, we would like to see this applied to the task of generating adversarial examples subject to 3D shape or material constraints. This task may be useful on its own for assisting in vehicle design or augmentation, but also enables the downstream tasks of geometrically-constrained data augmentation and adversarial training for deep neural networks.

An approach based on SAR raw signal simulation would be fully coherent and could capture effects from transmitter and receiver polarizations. It would also allow the user to employ an image formation algorithm of choice. Additionally, it would eliminate the need to train a CGAN for shading, which is cumbersome because of the large number of high-fidelity SAR images needed to train it. However, such an approach would likely be computationally more costly than the one proposed in this paper, which may hinder its utility in optimization problems.

This subsection provides a visual comparison of imagery produced by the differentiable SAR renderer to collected imagery at comparable geometries. Although the differentiable renderer is not intended for use as a general SAR simulator, comparison to additional data sources is helpful for understanding its capabilities and limitations.

Publicly available SAR data with corresponding scene representation ground-truth information (i.e., detailed CAD models) is difficult to come by, and prohibitively expensive to collect. Fortunately, the Moving and Stationary Target Acquisition and Recognition (MSTAR) public dataset [48] includes a synthetic calibration target known as SLICY (Sandia Laboratory Implementation of Cylinders). While detailed CAD models are not directly available for SLICY, information about the size and shape of the components that compose it is known [49, 50], as well as general information about its material (i.e., smooth metal). We manually constructed a CAD model that is nominally matched to the geometry of SLICY, seen in Figure 13.

The SLICY data was collected at X-band, but at a resolution about 4x as coarse as what was used in our main experiments, and image-formation was done in the slant-plane. We resampled the SLICY imagery to project it into the ground-plane and change its pixel-spacing to 0.15m in both range and cross-range.

To make images from our differentiable renderer comparable, we down-sampled the images it produced to match the resampled SLICY data described above. Also, since the differentiable renderer directly produces images with a PEDF-remapping, we (approximately) reverse-processed the remap to estimate what the raw pixel magnitudes would be. Since the PEDF-remap is lossy, the reverse-processing is imperfect; most notably, only roughly 30dB of dynamic range is retained due to clipping that occurs as part of PEDF-remapping.

Lastly, because scaling differs between image sources, we must normalize each image in order to compare them on a common scale. We normalize each image by its peak magnitude, and display the images on a decibel scale with 40dB of dynamic range. The resulting images are displayed in Figure 13.

Despite a number of mismatches impacting the comparison in Figure 13, the images maintain good structural similarity. The most prevalent difference seems to stem from the differing material types, which explains why a greater proportion of the object’s body appears illuminated in the rendered images compared to the collected counterparts. The SLICY object was also explicitly designed to demonstrate certain multi-path effects, which the differentiable renderer does not emulate. The hollow short cylinder is one such example, and it seems to be the source of the top-right scatterer in collected image HB15315.015. The delayed return from the cavity is what causes the scatterer to appear displaced in range.