MarkNerf:Watermarking for Neural Radiance Field

The embedded network, denoted as E, takes the watermark information with the size of 3×W×H and the image k from the Lego dataset as inputs. The watermark image w is then passed through a Conv layer with an output channel of 32, resulting in the feature map A with the size of 32×W×H. Furthermore, the image k is fed through another Conv layer with an output channel of 32 to obtain the feature map B, also with the size of 32×W×H.

Within the embedded network, we utilize a convolutional kernel with a size of 3×3, a step size of 1, and padding of 1. The ReLU activation function is employed, and a BN (batch normalization) layer is applied to normalize the data. The specific architecture of the embedded network is illustrated in Figure 4.

Upon exposure to noise, the neural radiation field may result in image distortion under the secret view [1]. To enhance the robustness of the proposed method, a noise layer is introduced for model processing. This study posits two approaches to incorporate noise into the implicitly represented 3D model: one involves adding noise to the network model parameters themselves, as demonstrated in reference [1], while the other entails introducing noise to the input data to simulate alterations in the network parameters themselves. For the purpose of simulating NeRF’s susceptibility to noise, this study adopts the latter approach. Specifically, Gaussian noise, salt and pepper noise, speckle noise, and Poisson noise are utilized to emulate various types of distortion. Refer to Figure 5 for the visual representation of the noise image.

The neural radiation field is a methodology employed for the generation of 3D scenes, entailing the utilization of a neural network model to establish a mapping relationship between pixel space and scene space. This mapping enables the prediction of radiation transmission within the scene. In this study, NeRF is trained using camera parameters associated with a confidential angle, facilitating the generation of a clandestine view image.

1. 1)

   NeRF network structure: The NeRF model takes as input the 3D coordinate position, denoted as x=(x, y, z), and the direction, represented as d=($\theta$ , $\phi$), of the point in the input space. The output space comprises the color c=(r, g, b) of the corresponding point and the density $\theta$ of the corresponding voxel position.In the specific implementation, the position information of x and d is firstly encoded, followed by the input of x into the MLP network. This results in the output of $\theta$ and a 256-dimensional intermediate feature. The intermediate feature and d are then inputted into a fully connected layer to predict the color. Finally, 2D images are generated through volume rendering, which is a process transforming 3D data into 2D representation.Volume rendering involves obtaining the final pixel value of a 2D image by generating pixel samples along a ray in the viewing direction and performing a weighted sum of the pixel values c and volume density $\theta$ obtained from the 3D points reconstructed by 3D reconstruction. This process is expressed mathematically as shown in formula (1).

   $$\begin{array}[]{l}C(r)=\int_{{t_{n}}}^{{t_{f}}}{T(t)\sigma(r(t))c(r(t),d)dt,}\\ whereT(t)=\exp(-\int_{{t_{n}}}^{{t_{f}}}{\sigma(r(s))}ds)\end{array}$$

   (2)

   Here, a ray, denoted as $\ {r(t)}$, is defined as $\ {r(t)=o+td}$, where (o) represents the photocenter position of the camera and (d) represents the directional vector. The cumulative transmittance of the ray along the path from the near end $\ {t_{n}}$ to the far end boundary $\ {t_{f}}$ is denoted as $\ {T(t)}$ .During the training process, the rendering of the overall scene is achieved by obtaining the image pixels and comparing them with the original perspective using the mean square error, also known as $\ {\cal L}_{2}$ loss or quadratic loss $\ {\cal L}_{2}$. The network is optimized based on this loss function. Additionally, a hierarchical representation rendering approach is adopted to improve rendering efficiency. This involves the simultaneous optimization of both the coarse network $\ {C_{c}}(r)$ and the fine network $\ {C_{f}}(r)$. By doing so, the weight distribution of the coarse network is utilized to better allocate samples in the fine network.

   $$\begin{array}[]{l}L=\sum\nolimits_{r\in R}{\left[{\left\|{{{\hat{C}}_{c}}(r)-C(r)}\right\|_{2}^{2}+\left\|{{{\hat{C}}_{f}}(r)-C(r)}\right\|_{2}^{2}}\right]}\end{array}$$

   (3)

   In formula (2), R represents all the rays in the input view,$\ {C(r)}$ represents the actual pixel value of the input view, and by $\hat{C}(r)$ represents the predicted pixel value.

2. 2)

   Camera parameter: NeRF takes a finite set of discrete images and camera parameters corresponding to the viewing angle to generate a continuous static 3D scene. It is capable of generating a new view image from an infinite number of angles. In this study, leveraging this feature of NeRF, we select the camera parameter (m) for a specific angle as the key. Since there are infinitely many angle options, the key space is infinite, ensuring the algorithm’s security. Camera parameters are categorized into internal parameters and external parameters. The external parameters determine the camera’s position and orientation, and their role is to transform points from the camera coordinate system to the world coordinate system. The camera parameter determines the projection property by mapping 3D coordinates in the camera coordinate system to the 2D image plane. In this study, we select camera parameters as the key to obtain the secret view image, ensuring image uniqueness. Moreover, the camera parameters consist only of two matrices, resulting in small key information.

The extraction network employs an over-parameterized approach to extract watermark information. Similar to the embedded network, the extraction network incorporates the concept of dense connections to enhance the transfer of features.

1. 1)

   overparameterization: Neural networks often encounter overfitting issues due to the abundance of parameters. However, this paper leverages this drawback as an advantage. Specifically, an extraction network with a substantial number of parameters is employed, where the secret view image and the original watermark image serve as inputs, while the output is the watermark image. Notably, when the secret-angle image is provided as an input, the watermark image can be accurately obtained. Conversely, if any other angle image is provided, the watermark image cannot be retrieved, thereby ensuring the algorithm’s security within this study.

2. 2)

   Extract network structure: The network takes in an image s of size 3×W×H, which is then extracted and processed through a convolutional layer with an output channel of 32 to obtain the feature map E. Subsequently, the feature map F is obtained through an additional convolution on E. By concatenating E, F, and G and performing another convolution, the feature map G is obtained, resulting in a clear watermark image w’. The basic procedure is depicted in formulas (8) to (11).

   $$\begin{array}[]{l}\left\{{\begin{array}[]{*{20}{c}}{E=Con{v_{3->32}}(s)}\\ {F=Con{v_{32->32}}(E)}\\ {G=Con{v_{32+32->32}}(Cat(E,F))}\\ {w^{\prime}=Con{v_{32+32+32->D}}(Cat(E,F,G))}\end{array}}\right.\end{array}$$

   (4)

   The convolutional kernel settings of the extraction network remain consistent with those of the embedded network, and a 3×W×H watermark image is outputted after the final convolutional layer. Once the extraction network is trained, the model parameters are saved. Importantly, the watermark image cannot be extracted from input images generated by other NeRF perspectives. For a comprehensive illustration of the extraction network’s architecture, refer to Figure 6.

   

The loss function primarily comprises two components: carrier image content $\ {\cal L}e$ and watermark image content $\ {\cal L}d$. $\ {\cal L}e$ represents the content loss of the carrier image, which is calculated as the Mean Square Error (MSE) between the carrier image and the image containing the watermark, as specified in Equation (12).

Here, $\alpha$ denotes the weight parameter, k represents the carrier image, and k’ represents the image containing the watermark. The watermark image content loss, denoted as $\ {\cal L}d$, is defined by formula (13).

In the equation, $\beta$, $\gamma$, and $\mu$ represent the weight parameters, w denotes the original watermark image, and w’ denotes the extracted watermark image.$\ {\cal L}_{{\rm{ssim}}}$ refers to the loss evaluated by Structural SIMilarity (SSIM), while $\ {\cal L}_{{\rm{msssim}}}$ corresponds to the loss evaluated by Multi-Scale Structural SIMilarity (MS-SSIM [26]). The computation formulas for SSIM and MS-SSIM are presented in formulas (14) and (15), respectively.

In the aforementioned equation, $\ {\mu_{x}}$ and $\ {\sigma_{x}}$ denote the mean and variance of image x, while $\ {\mu_{y}}$ and $\ {\sigma_{y}}$ represent the mean and variance of image y, respectively. Additionally, $\ {\sigma_{xy}}$ denotes the covariance of images X and Y. The variable M symbolizes the different scales, and $\ {\beta_{m}}$ and $\ {\gamma_{m}}$ represent the relative importance between images. For this study, $\ {\beta_{m}}$ and $\ {\gamma_{m}}$ are both set to 1. Constants C1 and C2 are defined, with a value of k1 = 0.01 and k2 = 0.03, respectively. Notably, L refers to the dynamic range of pixel values, with a specified value of 255.

For this experiment, the NeRF model is trained using the NeRFSynthetic dataset, which includes images of Lego, Chair, Ship, and other objects, while the ImageNet dataset is utilized for selecting random images as watermark images for testing purposes. The NeRFSynthetic dataset encompasses images captured from various angles of the model, accompanied by the respective camera parameters. Notably, the ImageNet dataset is a considerable dataset comprising 14,197,122 images, of which 100 images are specifically chosen as watermark images for this study. The experiment was conducted on a computer running Windows 11 with a capacity of 64GB. The embedded network and extraction network were optimized using the Adam optimizer. The input image size was uniformly adjusted to 256×256 pixels, with a learning rate of 0.0001 and a total of 50,000 epochs. The parameters of the loss function were set as follows: $\ \alpha$ = 0.3, $\ \beta$ = 0.3, $\ \gamma$ = 0.5, and $\ \mu$ = 0.5.

Imperceptibility, also referred to as perceptual transparency, implies that there should be no perceivable difference between the embedded watermark image and the original image. In general, Mean Squared Error (MSE) is employed to evaluate the quality of image alterations, Peak Signal-to-Noise Ratio (PSNR) is used to assess the compression quality of the image, and Structural SIMilarity (SSIM) measures the similarity between the embedded watermark image and the original image. Additionally, LPIPS [27] was employed to evaluate the perception loss and image quality during the evaluation process. PSNR is calculated based on MSE. A higher PSNR and SSIM are indicative of better quality, while a lower LPIPS is desirable. The calculation formulas for MSE and PSNR are presented in equations (16) and (17), respectively.

In the context, X and Y represent two images with dimensions of W×H, respectively. The variable sc indicates the scaling factor, which is typically assigned a value of 2.

1. 1)

   Image quality: The experimental results of this algorithm are presented in the table. The PSNR, SSIM, and LPIPS values reported in the table for both the embedded networks and the extraction networks represent their respective mean experimental results. As evident from the table, the embedded network and extraction network exhibit high PSNR and SSIM values, while demonstrating low LPIPS values. These findings indicate that the algorithm proposed in this paper generates embedded watermark images that possess high imperceptibility, as reflected by the extracted watermark images.

2. 2)

   Image effect:In this experiment, an image was embedded in each of the Lego, Chair, Ship, Drums, Hotdog, and Materials datasets. Subsequently, these images, along with others, underwent training using the NeRF algorithm. Following the training, watermark information was then extracted from the rendered new visual images. The visual outcomes of the embedded watermark image and the extracted watermark image are depicted in Figure 7 and Figure 8, respectively.

   

   

Robustness refers to the watermark’s ability to withstand attacks. In this study, the process of neural radiation field being attacked was simulated by introducing noise to the images in the training set. The robustness of the watermark is typically assessed using metrics such as Bit Error Rate (BER) and Normalized Correlation (NC).

1. 1)

   Bit error rate:The bit error rate (BER) is defined as the reciprocal of the peak signal-to-noise ratio (PSNR), with its formula presented in equation (18).

   $$\begin{array}[]{l}BER=\frac{1}{{PSNR}}\end{array}$$

   (11)

   The bit error rate (BER) quantifies the extent to which the original image is altered during the watermarking process. In this experiment, noise was introduced to the image from the Lego dataset, while the lion image from the ImageNet dataset was utilized as the watermark image to obtain the experimental results. The BER values associated with the extracted watermark under various noise attacks are presented in Table 2.

   noise type	Gaussian noise	Pepper Noise	Speckle noise	Poisson noise
   BER	0.02964	0.03081	0.03185	0.02836

2. 2)

   Normalized correlation coefficient: To quantitatively assess the similarity between the extracted watermark image and the original watermark image, the normalization coefficient is employed as the evaluation criterion. Its definition is presented in equation (19).

   $$\begin{array}[]{l}NC(w,w^{\prime})=\frac{{\sum\limits_{x=1}^{M}{\sum\limits_{y=1}^{N}{w(x,y)w^{\prime}(x,y)}}}}{{\sqrt{\sum\limits_{x=1}^{M}{\sum\limits_{y=1}^{N}{w{{(x,y)}^{2}}}}}\sqrt{\sum\limits_{x=1}^{M}{\sum\limits_{y=1}^{N}{w^{\prime}{{(x,y)}^{2}}}}}}}\end{array}$$

   (12)

   Here, w(x,y) refers to the original watermark image, w’(x,y) represents the extracted watermark image, M and N denote the image resolution. The normalized correlation coefficient (NC) signifies the similarity between the original image and the extracted watermark image, ranging from 0 to 1. A higher value closer to 1 indicates better robustness.In this experiment, varied noise types were employed to attack the watermark-embedded image, followed by the extraction of the watermark image and calculation of the corresponding NC. The normalized correlation coefficient of the extracted watermark image following the attack is depicted in Figure 9.

   

   Upon analyzing the findings presented in the figure, it is evident that the proposed watermarking scheme in this study exhibits commendable anti-interference capability.

To evaluate the efficacy of the hidden perspective, this study investigates the impact of images generated from different viewpoints on the watermark extraction performance. For this experiment, a specific image from the Lego dataset was selected as the host image for watermark embedding, while an elephant image from the image dataset was chosen as the watermark image. Images rendered using the NeRF technique under various perspectives were inputted into the extractor for evaluation. The experimental outcomes are depicted in Figure 10.

The experimental results indicate that as the rotation angle increases, the extracted watermark image progressively becomes more blurred, eventually reaching a point where the watermark image extraction is no longer feasible. To quantitatively analyze the impact of the rotation angle on watermark extraction, the values of PSNR, SSIM, and LPIPS are also provided, representing the comparison between the extracted watermark and the original image. These results are presented in the table. The angle variation specified in the table corresponds to the rotation angle around the central z-axis.

Based on the experimental data, it is observed that smaller rotation angles yield higher PSNR and SSIM values for the extracted watermark, demonstrating superior similarity to the original watermark image. Additionally, the lower LPIPS values further support the notion of the proposed algorithm’s efficacy in safeguarding model copyright.