ProgrammableGrass: A Shape-Changing Artificial Grass Display Adapted for Dynamic and Interactive Display Features

In addition to grass materials, there are many display methods that use the unique colors of natural materials. Display methods using liquids have been proposed, focusing on water-wet ground or asphalt [12, 13], bubbles and colors created by electrolysis [14, 15], condensation [16], liquid-metal droplets [17], and tubes filled with water [18]. Since carpet shading can be easily changed by simply rubbing one’s finger on a carpet, display methods using carpets have also been proposed [19, 20]. In addition, Yamamoto et al. presented a carpet method to switch images according to the positions of a viewer or a light source [21]. Photo-Chromeleon is a method that changes multi-color textures using photochromic dyes, and the color of the materials can be switched with the light of a certain wavelength [22]. Organic display methods focusing on moss [23], mimosa [24], and foodstuffs [25] have also been published. In the field of art, Daniel Rozin has created multiple artworks focusing on interactive display devices using natural materials such as wood and homespun baskets [26, 27]. In addition, the people of Inakadate village in Japan create rice paddy art using differences between the colors of rice paddies [28].

The advantage of these display methods using natural materials is that they are compatible with our daily living environments. In this paper, we focus on display methods of grass materials and propose a novel green space-friendly display technique that shows dynamic and interactive animations using grass color control.

Pin-array display techniques, in which multiple pins are controlled synchronously, have been developed primarily in the research field of shape-reproducing displays [29, 30]. However, pin-array displays are also used for other purposes. For example, Nakagaki et al. proposed a hands-on animated craft platform using a pin-array display to move crafts such as dolls [31]. Suzuki et al. presented a dynamic and instant three-dimensional (3D) printing method that uses a pin-array display [32]. Haptic devices with a pin-array display have been developed, focusing on palm size [33, 34] and fingertip size [35]. Furthermore, pin-array displays are used as virtual reality (VR) tools [36, 37, 38, 39]. Nakagaki et al. represented a platform that can convert a pin-array display into other interfaces, such as a haptic display and a 3D modeling tool. Engert et al. proposed a platform for shape-changing spatial displays using a pin-array display [40].

These studies benefit from a pin-array display’s ability to control a shape on a point-by-point basis. In this paper, we present a method that uses a pin-array display system, including DC motors, to move the grass length pixel by pixel, allowing for quick control of the grass color.

As with other non-luminescent display methods, grass-based display techniques have been developed in research, commerce, and art. Examples of grass display methods are pressing or mowing the grass. Nvidia generated an image of an electronic component on a grass-like green field using plants pressed against it [41]. Scheible et al. proposed a large-scale grass art method that used a drone to get a bird’s-eye view of the grass while mowing it [42]. Other techniques involve using grass shading, depending on the orientation of the grass tips. Sugiura et al. presented a grass printer device that uses multiple rods to apply pressure to grass and change the direction of the grass tips. The change in direction of the grass tips enables the production of a binary image on the grass [5]. Using a similar technique, NEW GROUND TECHNOLOGY performs a grass print service that uses air pressure to change the direction of the grass tips to show a large-scale image [6]. This method uses a spatial dither technique to represent the shading in the input image by adjusting the placement of light and dark grass shades. In this method, a large number of pixel shades are required on a grass surface. Another grass display method uses grass growth rate differences to generate a rich image. Ackroyd & Harvey created artworks to show photography using grass [7]. Canvas containing grass seeds are exposed to the light of a negative image of photography from a projector. As the grass grows, the photography is generated with differences in grass growth rates. These methods show static images on the grass, and a shape-changing grass system is needed to switch images quickly and play animations. We previously developed a grass display that can dynamically change the grass color by adjusting the grass length using two colors of grass [8, 9]. However, the current grass display cannot swiftly control the grass color linearly at 8-bit levels, and its resolution cannot be adjustable.

In this paper, we propose a novel approach that quickly controls the grass color at 8-bit levels with a high response speed. These technologies enable an artificial grass display to play dynamic animations using 8-bit grayscale images such as an MP4 file, and gesture-based animations from a user. We also design a hardware module for the grass display that can be connected to other modules for easy adjustment of the resolution. Figure 2 shows the position of our work relative to previous grass studies. Table I shows the comparison between our previous work and the current work, focusing on levels of grass color control, frame rate, number of pixels, and resolution scalability.

Plant color evaluation methods are important to develop grass color control. One of the algorithms used to evaluate plant color is CIEDE2000, a color difference formula published by the International Commission on Illumination (CIE, Commission Internationale de l’Éclairage) [11]. The CIEDE2000 is suitable for plant color evaluations because the formula reflects the perceptual model of human eyes. For example, leaf color evaluation systems with the CIEDE2000 have been represented to estimate nitrogen levels [43, 44], while an apple color evaluation method was proposed to measure the discoloration of apple pulp using the CIEDE2000 [45]. A dyed wood color evaluation method also uses the CIEDE2000 [46].

We previously proposed a CIEDE2000-based grass display setting system to map several levels to the grass lengths for controlling the green-yellow grass color [9, 10]. This method sets levels according to whether the grass colors can be distinguished by the human eye. However, in the method, only several levels can be set, and the number of levels varies depending on differences in the environment in which the grass display is installed and the grass color itself. In other words, the current method does not allow the grass display to be calibrated to show the 8-bit images.

On the other hand, in conventional displays such as LCDs, gamma correction is a method of linearly controlling color at 8-bit levels. Gamma correction aims to linearize the relationship between the input 8-bit levels and luminance, taking into account the non-linear characteristics between the display’s luminance and voltage. This color calibration technique, originally developed for Cathode Ray Tube (CRT) displays [47], has been adapted and explored for use with other types of displays as well. This includes research on methods for LCDs [48, 49, 50, 51] and Organic Light-Emitting Diode (OLED) displays [52]. These studies have been dedicated to developing calibration methods specifically tailored to the unique characteristics of each type of display. Moreover, the widespread adoption of standardized color management systems, such as International Color Consortium (ICC) profiles [53], has significantly facilitated the consistent management of gamma correction across various displays.

However, these calibration schemes are primarily designed based on the luminance characteristics of self-emitting displays. Our proposed grass display is one of reflective displays, including electronic papers. Reflective displays are significantly influenced by external environmental conditions in color control [54]. Furthermore, the measurement methods for characteristics of grass color in relation to the grass length are not well established. Consequently, for controlling grass color at 8-bit levels, a calibration approach based on an external environment and a characteristic between grass colors and lengths is important.

We present a novel CIEDE2000-based stable calibration system that allows for linear control of grass color at 8-bit levels using the characteristic of grass color variation with grass lengths. The calibration system also takes into account differences between the colors of multiple grass pixels. Furthermore, we develop a calibration tool using a tablet computer to easily facilitate the calibration of ProgrammableGrass.

The grass color of ProgrammableGrass is determined by the mixture of green and yellow grass, utilizing the principle of spatial additive mixing as shown in Figure 3(a). Spatial additive mixing is a phenomenon in which humans perceive a texture’s color as the average color when the texture is too fine for the human eyes to distinguish. This phenomenon often occurs when humans see grass surfaces, including lively green grass and dying yellow grass.

Using this phenomenon, we previously developed the pixel structure called a grass pixel [8, 9] as shown in Figure 3(b). The pixel structure comprises a green grass pin and a yellow grass multi-slit component. The green grass is attached to the top surface of a pin, and the yellow grass is planted on the top surface of a multi-slit component. By passing the green grass through the gaps of the yellow grass and moving the green grass pins vertically, the pixel structure appears like lively grass growing out among dying grass, thereby changing the perceived grass color. In this study, we adopted the grass pixel as the pixel system for ProgrammableGrass.

We designed a module that is the smallest unit of ProgrammableGrass to facilitate the scalability of the resolution. The module was named a grass module. Figure 4 shows the overview of the grass module hardware concept design. The resolution of the grass module is $2\times 8$ pixels, and the grass module includes green grass pins, yellow grass multi-slit components, and actuators with printed circuit boards (PCBs). In addition, the grass module has a PCB to connect with another grass module. Multiple grass modules can be chained together via the connection PCB to increase the width of the resolution of ProgrammableGrass. For example, when two grass modules are chained, the resolution is $4\times 8$ pixels and ProgrammableGrass can be used as a small artificial grass display. When four grass modules are chained, the resolution is $8\times 8$ pixels and the ProgrammableGrass can show graphical animations such as text and icons. In this paper, we built four grass modules and assembled a ProgrammableGrass with a resolution of $8\times 8$ pixels.

Figure 5 shows the overall hardware system of ProgrammableGrass. The grass module has 16 grass pins, and a DC motor actuates each grass pin via a lead screw (lead: 6 [mm/rotation]). The DC motor is controlled by a full-bridge DC motor driver (TB6643KQ). A magnetic rotary encoder tracks the rotary position of the DC motor. The rotary encoder’s resolution for the DC motor and the lead screw combination is 10/73 [mm/count]. A limit switch is used to check the origin position of the grass pin. The grass module is operated by a microcontroller (Teensy 4.1). The microcontroller sends pulse width modulation (PWM) values to the motor drivers using 8-bit output shift registers (TC74HC595AP). In addition, the microcontroller receives the rotary positions of the DC motors using 8-bit input shift registers (TC74HC165AP). The states of limit switches of the grass pins are sent to the microcontroller through serial peripheral interface (SPI) I/O (Input and Output) expanders (MCP23S08). These shift registers and SPI devices are included in the grass module. Since the shift registers can be cascaded with other shift registers and the I/O expanders can share SPI signal lines, the grass modules can be connected to other grass modules, and several grass modules can then be controlled by the microcontroller. Moreover, a Bluetooth module (ESP32-WROVER-E) is connected to the microcontroller via SPI signal lines, inputting an 8-bit level and outputting the grass length for each pixel wirelessly. For example, this module allowed ProgrammableGrass to be operated wirelessly with a tablet computer during the calibration in Section V and the demonstration in Section IX. The grass module uses 3.3V supplies to operate the shift registers and I/O expanders, and 12V supplies to run the DC motors. The voltages are shared among several grass modules.

Figure 8 illustrates the overall software design of ProgrammableGrass, which is developed in a C++ environment. The microcontroller controls the grass pixel using a PD controller developed with MATLAB and Simulink. The green grass length is adjusted based on the 8-bit levels. First, the 8-bit level is converted into the grass length using a correspondence table for the grass length and the 8-bit level. The correspondence table is obtained from a grass color calibration system that measures the color of the grass pixel and determines the relationship between the grass length and the 8-bit level. The details of the grass color calibration system are described in Section IV. Second, the grass length value is converted into a count value based on the rotary encoder of the DC motor. Since the resolution of the rotary encoder is 10/73 [mm/count], the gain value is set to 7.3 [count/mm]. Then the converted count is also cast to an integer. In the PD control, the converted count value is a desired setpoint (SP), and the output count value from the rotary encoder is a process variable (PV). In setting the PD control parameters in a heuristic manner, the P parameter is first set so that the PV oscillates near the SP, and the D parameter is set to suppress this oscillation. The PD controller then sends a PWM value to the DC motor driver using the SP and the PV. Therefore, the grass length is adjusted based on the 8-bit levels through the PD control. This process is repeated for each grass pixel by the microcontroller.

We conducted experiments on the response speed of ProgrammableGrass. We developed a grass pixel for the experiments as shown in Figure 9. This grass pixel was built based on the hardware and software structures for the grass module. The P and D parameters were set at 6.0 and 10.0, respectively. The size of the grass pixel was $300\times 110\times 110$ [mm]. In the experiments, we sent linear SPs to the grass pixel from 0 to 146 [count] for moving the grass length from 0 to a maximum of 20 [mm] in a fixed time interval. In addition, the PVs from the rotary encoder were measured. When the PVs follow the linear SPs, the grass pixel can control the grass length with the time interval. The time interval was set to $1/n$ [sec] corresponding to a natural number $n$ frames per second (fps). Then, the maximum response speed at which the PVs can follow the linear SPs was evaluated as the $n$ was increased.

Figure 10 shows the results of the response speed experiments. The blue and red lines are trajectories of the linear SPs and the PVs, respectively. The PVs were found to follow the linear SPs between 1 and 1/10 [sec]. Therefore, ProgrammableGrass could control the grass length at a maximum of 10 [fps]. The response speed of ProgrammableGrass was 10 times faster than the grass pixel response speed of 1 [sec] in the grass pixel method proposed by Tanaka et al. [9]. Furthermore, the response speed of 1/10 [sec] satisfies the speed of a limited animation, which is one of the animation methods to limit the number of frames per second to six or fewer [55]. Therefore, ProgrammableGrass was found to be able to control the grass length enough to keep up with the animation speed. However, the grass pixel could not follow the linear SPs of 1/11 [sec] and shorter.

In the grass color calibration system, the CIELAB color space is adopted to quantify the grass color because the CIELAB values of the grass colors can be compared using the color difference formula of the CIEDE2000. The CIEDE2000 is a color difference formula based on human color perception and is often used to measure multiple material colors, such as that of vegetables. In addition, each parameter of a CIELAB value is $L^{*}$ (Lightness), $a^{*}$ (Complementary colors between red and green), and $b^{*}$ (Complementary colors between yellow and blue). The CIELAB value of the grass color is called a grass color value in this study.

Figure 11 shows the measurement of the grass color through image processing. A digital camera is used to capture the grass color. We prepare a color correction board including two types of gray cards, with 18% and 50% reflectance, respectively. The digital camera needs suitable exposure to capture the grass color stably using the color correction board. Before capturing the grass color, the camera’s exposure is adjusted so that the intensity of the reflected light from the gray card with 18% reflectance is in the middle of the light intensity range.

The digital camera captures the grass pixel as a raw image to avoid a unique camera maker’s image processing engine. The raw image is developed as an original image in the linear ProPhotoRGB color space. ProPhotoRGB is a device-independent RGB color space with a wide color gamut and is also used as a working color space in Adobe Lightroom [56]. Since the relationship between light intensity and brightness is linear in the real world, a linear RGB color space of ProPhotoRGB is used to calculate the grass color. When the raw image is developed, the white balance is adjusted using the gray card with 50% reflectance of the color correction board. After the raw image is developed, the original image is cropped to focus on the surface of the grass pixel. Specifically, the three slits of the grass pixel fit into the cropped image.

The grass color is quantified based on the cropped image. Since the grass color is perceived as the average color based on the spatial additive mixing, the average RGB value is calculated with the cropped image. The average RGB value is then converted to a CIELAB value via CIE 1931 color space. Since the white point of the linear ProPhotoRGB color space is 5000 [K], the white point of the CIELAB color space is also set to 5000 [K]. Therefore, the grass color value is calculated as a CIELAB value through image processing.

In this subsection, we focus on how to obtain the correspondence table between the grass length and the 8-bit level using the measured grass color values. We describe the process of creating a correspondence table for a single grass pixel as shown in Figure 12.

First, the grass colors are measured at grass length intervals of about 1 [mm] between the minimum and maximum of the grass length, as shown in Figure 12(a). Second, the color difference of the grass color value at each grass length is calculated based on the grass color value of 0 [mm]. In this paper, OGCD (Origin Grass Color Difference) is defined as the color difference, quantified using the CIEDE2000 color difference formula in the CIELAB color space, between the grass color value at 0 [mm] and another measured grass color value. Therefore, the characteristic between the grass color and the grass length can be expressed by the OGCDs. Moreover, a nonlinear model is calculated by a sixth-order curve fitting using the calculated OGCDs between the minimum and maximum of the grass length, as shown in Figure 12(b). The nonlinear model is called an OGCD characteristic.

Finally, assuming a linear relationship between the OGCD and the 8-bit level where the maximum value of OGCD corresponds to 255 [level], a correspondence table between the grass length and the 8-bit level is obtained based on the OGCD characteristic model as shown in Figure 12(c). Therefore, the grass pixel can be calibrated when the obtained correspondence table is implemented in its software as described in Subsection III-C. Thus, ProgrammableGrass can control the grass color linearly at the 8-bit levels.

The grass pixels have different OGCD characteristics from pixel to pixel due to slight differences in green grass movement and the shape of the grass multi-slit component. Therefore, it is necessary to obtain correspondence tables that consider the OGCD characteristics of multiple grass pixels. This approach enables the control of the colors of multiple grass pixels at 8-bit levels, with minimal variation from pixel to pixel. Using the calibration of four grass pixels as an example, Figure 13 shows the output of the correspondence table for each grass pixel.

As outlined in Subsection IV-B, the OGCD characteristic for each grass pixel is obtained. Then, among the multiple OGCD characteristics, the grass pixel with the smallest OGCD range is determined. The OGCD of this grass pixel at 20 [mm] grass length is called the reference OGCD. Then, the correspondence table for each grass pixel is calculated so that the relationship between the OGCD and the 8-bit level is linear and the value of OGCD is the reference OGCD when the 8-bit level is 255.

For example, as illustrated in Figure 13, the reference OGCD is the OGCD at a grass length of 20 [mm] of the 2nd grass pixel, which has the smallest range of OGCD values. In the 0th grass pixel, the grass length $a$, which corresponds to this reference OGCD, is determined based on the OGCD characteristic. Then, the correspondence table is calculated where the grass length $a$ corresponds to an 8-bit level value of 255. Similarly, in the 1st and 3rd grass pixels, the correspondence tables are calculated using the grass lengths $b$ and $c$, respectively. These lengths correspond to the reference OGCD in each respective pixel, based on their unique OGCD characteristics.

Thus, by implementing these individual correspondence tables into each grass pixel, the calibration system accounts for the unique OGCD characteristics, providing a consistent method for controlling the color of multiple grass pixels at 8-bit levels.

In this phase, raw images are taken while moving the grass length from 0 to 20 [mm]. Before capturing, the exposure and white balance of the tablet’s camera are adjusted using the color correction board as shown in Figure 14(a1). To move the grass length at regular intervals, the tablet and ProgrammableGrass are connected through Bluetooth. The tablet is fixed at a certain position based on where a user looks at ProgrammableGrass. As shown in Figure 14(a2), the preview and the shoot button of the tablet’s camera are displayed. When the user pushes the shoot button, a raw image is captured each time the grass length is incrementally adjusted by approximately 1 mm, spanning the range from 0 to the maximum of 20 [mm]. At this time, the tablet computer sends the grass lengths as the SPs of the PD control instead of the 8-bit levels.

After the raw images are captured, a crop area is set for each grass pixel to measure the grass colors. The user slides or pinches the tablet screen to determine the crop area as shown in Figure 14(b1). Then, when the user pushes the check button, the crop area is decided as shown in Figure 14(b2). Since the tablet position is fixed, the crop area is applied to all raw images.

Once the crop areas are determined, the CIELAB values of the grass colors for each grass length are obtained, and the correspondence table for each grass pixel can be calculated according to Section IV. Thus, ProgrammableGrass can be calibrated with the resulting correspondence tables.

Figure 15 shows the experimental environment. In the experiments, the grass pixel described in Subsection III-D was used as ProgrammableGrass. To measure the grass color stably, we built a color measurement environment focusing on International Standards Organization (ISO) 3664:2009 [57], which is a standard of color measurements in the fields of graphic arts and photography where color management is a strict discipline. As shown in Figure 15, the measurement environment was a dark room to avoid external lights. Two LED fluorescent lamps for color measurements were used as light sources (ECORICA LED, ECL-LD2EGN-L3A, color temperature: 5000 [K], average of color rendering index (Ra): 97). The light sources were placed at the height of 0.4 [m] from the top surface of the grass pixel so that the illuminance of the top surface was about 2000 [lx]. The color measurement environment with the light sources generally satisfies ISO 3664:2009. Silk gray cards with 18% and 50% reflectance were used as gray cards of the color correction board. An iPad Pro was used to measure the grass colors of the grass pixel and to use the calibration tool of Section V.

In the experiments, since the slits of the grass multi-slit component caused the grass pixel appearance differently depending on a viewer’s position, the grass color was evaluated from several positions. Figure 16 shows the overview of the viewpoint positions. Since we assumed that ProgrammableGrass was put down, the distance between the grass pixel and the viewpoint was 2.0 [m], and the height of the viewpoint was 1.7 [m] from the top surface of the grass pixel for the average adult height. The horizontal angles between the grass pixel and the viewpoint were $0^{\circ},30^{\circ},60^{\circ}$, and $90^{\circ}$ around the center of the grass pixel. When the horizontal angles were $0^{\circ}$ and $90^{\circ}$, it was perpendicular and parallel between the slits and the viewpoint, respectively. The iPad Pro captured the grass color at these viewpoints.

Herein, the grass colors of the calibrated grass pixel were measured when the 8-bit levels were input at regular intervals. The grass pixel was calibrated at each viewpoint using the calibration tool in the experiments. In addition, the blue area of Figure 18 shows the OGCD characteristic used to calibrate the grass pixel at each viewpoint. These OGCD characteristics of the blue curves illustrate the nonlinear characteristics between the grass color and the grass length for the grass pixel.

The red area of Figure 18 shows the experimental results of the single calibrated grass pixel. In the experimental results, the viewpoints from which the grass pixel was calibrated and measured are referred to as calibration viewpoints and measurement viewpoints, respectively. The rows and columns are classified by the calibration viewpoints and the measurement viewpoints, respectively. For the experimental results, the blue dots show the difference values of the OGCD from the average OGCD for each 8-bit level of the ten trials together. The red line is the linear model calculated from the single regression analysis based on the difference values of the blue dots. In addition, the $R^{2}$ is shown in each experimental result. The blue area in Figure 18 also shows the OGCD characteristics for comparison with the experimental results. The OGCD characteristics represent the nonlinear characteristics between the grass length and color of the grass pixel. In addition, the OGCD characteristics were also evaluated with a single regression analysis to show the linear model as the red line and the $R^{2}$.

As a result, we revealed that the grass color of the grass pixel could be controlled linearly based on the 8-bit levels using the grass color calibration system. Additionally, in all experimental results, the $R^{2}$ values were greater than 0.9, indicating that the grass pixel could consistently control the grass color across all viewpoints. In the results at the $0^{\circ}$ calibration viewpoint, the blue dots were plotted linearly at each measurement viewpoint. In particular, the experimental graph of the $0^{\circ}$ measurement viewpoint shows that the strong nonlinear OGCD characteristic at the $0^{\circ}$ measurement viewpoint was improved. In addition, the $R^{2}$ of the OGCD characteristic improved from 0.939 to 0.996. In the result at even the $30^{\circ}$ and $60^{\circ}$ measurement viewpoint, the $R^{2}$ values of the OGCD characteristic improved from 0.904 to 0.995 and from 0.953 to 0.993, respectively. At the $90^{\circ}$ measurement viewpoints, the $R^{2}$ of the OGCD characteristics did not improve. However, the plot results show that the grass color changed linearly. Figure 19 shows a set of images of the grass pixel, calibrated at the $0^{\circ}$ viewpoint and captured from four different viewpoints.

Similar to the results of the $0^{\circ}$ calibration viewpoints, the calibration at the $30^{\circ}$ improved the $R^{2}$ of the OGCD characteristic at $0^{\circ}$, $30^{\circ}$, and $60^{\circ}$ measurement viewpoints except for $90^{\circ}$ measurement viewpoint. Moreover, the $R^{2}$ at all measurement viewpoints improved at the $60^{\circ}$ and $90^{\circ}$ calibration viewpoints.

Furthermore, we also revealed that the effectiveness of the grass color calibration system depended on the calibration viewpoints. At the $0^{\circ}$ and $30^{\circ}$ measurement viewpoint, the OGCD characteristic was nonlinear with the grass length. Then, the OGCD characteristic began to change linearly from the $60^{\circ}$ measurement viewpoint. This was caused by the slit appearance of the grass pixel. At the $60^{\circ}$ and $90^{\circ}$ measurement viewpoints, the green grass was seen through the slits of the grass pixel earlier than at the $0^{\circ}$ and $30^{\circ}$ measurement viewpoints. Therefore, since the grass pixel was calibrated with the OGCD characteristic, the effectiveness of the grass color calibration system depended on the calibration viewpoints.

For example, when the grass pixel calibrated at the $0^{\circ}$ viewpoint was measured from the $60^{\circ}$ viewpoint, the plotted graph was more convex above the OGCD characteristic at the $60^{\circ}$ measurement viewpoint. This was due to the strong effect of linearizing the nonlinear OGCD characteristic at the $0^{\circ}$ measurement viewpoint. In contrast, when the grass pixel calibrated at the $60^{\circ}$ viewpoint was measured at the $0^{\circ}$ viewpoint, the grass color changed linearly; however, the grass pixel was not improved as much as the grass pixel calibrated at the $0^{\circ}$ viewpoint. This was because the effect of the grass color calibration system of the $60^{\circ}$ viewpoint was weak in controlling the grass color observed from the $0^{\circ}$ viewpoint.

The experimental results show that the range of the OGCD of the $90^{\circ}$ measurement viewpoint was smaller than that of the other measurement viewpoints. This was because the green grass was seen even though the grass length was 0 [mm] at the $90^{\circ}$ measurement viewpoint. This phenomenon did not occur from other measurement viewpoints. Therefore, the amount of the change in the OGCD of the $90^{\circ}$ measurement viewpoint was smaller than that of other measurement viewpoints.

We obtained the average value of the $R^{2}$ from the $0^{\circ}$ to $90^{\circ}$ measurement viewpoints, in order of the $0^{\circ},30^{\circ},60^{\circ}$, and $90^{\circ}$ calibration viewpoints: 0.9908, 0.9855, 0.9893, and 0.9785, respectively. The average $R^{2}$ value was highest at the $0^{\circ}$ calibration viewpoint. Therefore, we concluded that the $0^{\circ}$ viewpoint is the most suitable calibration viewpoint when ProgrammableGrass is observed from angles ranging from $0^{\circ}$ to $90^{\circ}$.

In the previous section, we conducted experiments on the calibration effects of the grass pixel under the standard lighting conditions of ISO with high luminance, high color rendering, and a color temperature of 5000K. In this section, we delve into the calibration effects within actual indoor environments by undertaking experiments in several indoor settings.

The study was carried out in three distinct locales within the University of Tsukuba campus: a classroom, a meeting room, and a dimly lit space. These experiments employed three viewpoints: $0^{\circ},30^{\circ}$ and $60^{\circ}$ as discussed in Section VI. Here, we installed the grass pixel and performed calibration at $0^{\circ}$ in each indoor environment. Subsequently, we input levels ranging from 0 to 255 at fixed intervals into the grass pixel and measured the OGCD with the viewpoints $0^{\circ},30^{\circ}$ and $60^{\circ}$. This procedure was aimed at ascertaining whether the OGCD changes follow a linear pattern as the environment changes. Details of the illuminance and the color temperature at the location of the grass pixel in the experimental environments are summarized in Table II.

The experimental outcomes are depicted in Figure 20. These illustrations not only incorporate the three indoor environments but also the results from the previous section’s ISO standard environment experiments. As a result, we revealed that calibrating the grass pixel in three different indoor settings, similar to the ISO environment, allows for a linear control of the grass color in response to the input from the 8-bit levels. These color controls are consistent across the three viewpoints. Additionally, despite the changes in luminance and color temperature across the four environments presented in these graphs, the range of OGCD was found to be approximately 30.

The uniformity in the OGCD range across different environments can be attributed to the calibration process, where exposure and white balance were adjusted using 18% and 50% gray cards, respectively. This adjustment ensures that the changes in the grass color’s $L^{*},a^{*}$, and $b^{*}$ components in relation to the 8-bit levels are similar. To facilitate discussion on these results, Figure 21 showcases the changes in $L^{*},a^{*}$, and $b^{*}$ observed from the viewpoint $0^{\circ}$ in each environment. The graph plots the differences from the 0 level values of $L^{*},a^{*}$, and $b^{*}$, color-coded in gray, green, and yellow, respectively. The plots reveal that, across all environments, the range of differences in $a^{*}$ and $b^{*}$ is around 15, whereas for $L^{*}$, it’s about 35. In these experiments, even in the dimly lit space with only 113 [lx], the change in $L^{*}$ was similar to that in the other three environments. However, we suppose that in environments with even lower illumination, despite adjusting the exposure of the digital camera, the color of the grass tends to be measured as darker, almost black. This, in turn, would lead to a smaller range of the OGCD.

The experimental environment and procedure are similar to the experiments in Section VI. We measured the grass colors in the same color measurement environment as used in Section VI. A grass module was used as ProgrammableGrass to be measured. The iPad Pro of the previous section was also used to measure the grass colors and to calibrate the grass pixels with the calibration tool. The iPad Pro was located a distance of 2.0 [m] from the grass module and at a height of 1.7 [m] above the surface of the grass module.

In these experiments, we evaluated the effect of calibrating the eight pixels of the grass module based on their respective OGCD characteristics. In Section VI, we revealed that it is suitable to calibrate ProgrammableGrass from a $0^{\circ}$ viewpoint where the camera and the slits are perpendicular. Thus, the eight grass pixels were also calibrated from the $0^{\circ}$ viewpoint using the tablet calibration tool. The calibration generated multiple correspondence tables for the eight grass pixels. After calibration, as in the experiments in Section VI, 8 level intervals were input to the grass pixels and the OGCD of each grass pixel was measured each time from the $0^{\circ}$ viewpoint. We also measured the changes in OGCD for each grass pixel when applying a single correspondence table simultaneously to all eight pixels. In this procedure, every grass pixel employed the correspondence table derived from the grass pixel that demonstrated the smallest OGCD range during the calibration of multiple grass pixels. This ensured that the range of grass length in the single correspondence table was from 0 to 20 [mm].

Figure 22(a) and (b) show the results with the single correspondence table and multiple correspondence tables, respectively. The colors of the dots on the graphs are differentiated for each grass pixel. When the color of the dots is the same in graphs (a) and (b), it indicates that these dots correspond to the same grass pixel. In Figure 22(b), the blue line represents the reference OGCD, which was described in Subsection IV-C. Moreover, a single regression analysis was performed on Figure 22(a) and (b) to determine how similar the OGCD changes were for the eight grass pixels. Then, Figure 22 shows the prediction model of the single regression analysis with a red line, and the coefficient of determination $R^{2}$ is also displayed.

As a result, by creating correspondence tables for the eight grass pixels, it became easier to control the colors of the eight grass pixels. In the plot of (b), the OGCD variation among the grass pixels is smaller than in the plot of (a), and the $R^{2}$ value is improved from 0.919 to 0.982.

For the results in (a), the OGCD for black and cyan dots, which have small OGCD ranges, changes linearly, while other color dots have upward convex trajectories. This is because each grass pixel used the correspondence table for the grass pixel with small OGCD ranges. The largest OGCD difference between the grass pixels was 9.8 at level value 167.

In contrast, in results (b), each grass pixel was calibrated based on its characteristics. Thus the grass colors of all pixels could be controlled linearly. In these experiments, the reference OGCD was set at 24.5. At level 255, the mean OGCD value was 24.3, with a standard deviation of 1.41.

Thus, we revealed that the calibration based on multiple OGCD characteristics was effective in reducing the OGCD variation among the grass pixels.

Here, we demonstrated several animation examples of ProgrammableGrass as shown in Figure 23. An $8\times 8$ pixel artificial grass display was built using four grass modules in the animation examples. In the examples of Figure 23(a), (b), and (c), the artificial grass display played the animations at 10 [fps] using several $8\times 8$ MP4 files with 8-bit grayscale levels. In addition, we calibrated the artificial grass display using the calibration tool in the demonstration environment

Based on the findings in the previous subsection, we made several applications of ProgrammableGrass as shown in Figure 24. To emphasize the practicality of these applications, we also included conceptual images in each application, illustrating the potential of these applications when ProgrammableGrass could be integrated into society as if it were conventional grass.

In the design of the grass pixel, the subtle differences in the appearance of the grass pixels are due to manually inserting the green grass among the yellow grass. If this process could be mechanized, the appearance of the grass pixels could be aligned for a more diverse perspective. In addition, the slit structure was adopted to insert the green grass into the gaps of the yellow grass easily. However, the appearance of the grass pixel varies depending on the viewing angle. We suppose that a yellow grass component with holes in a matrix-like grid will reduce differences in appearance depending on the viewing angles.

The enclosure of the grass module was developed to expand the resolution’s width using multiple grass modules. If the enclosure is designed to increase the resolution’s width and height, the users can adjust the resolution of ProgrammableGrass freely. In addition, ProgrammableGrass was assembled using up to four grass modules. As the number of grass modules increases, signal issues such as delay and noise may arise. Therefore, when more grass modules are used, it is necessary to design electronic circuits considering the signal issues according to the usage environment. For example, this solution includes ferrite beads as noise filters, and employing metal shielding.

The gesture-based animation described in Section IX shows a slight delay in the display of ProgrammableGrass relative to the gesture input. This delay is due to the Bluetooth communication speed between ProgrammableGrass and the tablet computer. It can be resolved by using a wired connection or a more powerful wireless communication method, such as Wi-Fi.

Grass fields are usually used as playing fields and elements that improve green landscapes. We believe that ProgrammableGrass will become familiar with social environments, and the relationship between ProgrammableGrass and users will become closer when ProgrammableGrass is sturdy enough to be stepped on by the users and small enough to be easily installed. Standing on the current artificial grass display is possible due to the aluminum frame. However, the 3D-printed yellow grass blades might break due to the Polylactic acid (PLA) materials. Therefore, the durability of the yellow grass can be further enhanced using resilient artificial grass materials like polyethylene and thermoplastic polyurethane (TPU). It is also needed to design how to control the grass length when people are on the artificial grass display.

Furthermore, to expand the artificial grass display to the size of a sports field and a park, it is important to improve the resolution and display size of a pin display, which is a fundamental technology of ProgrammableGrass. Additionally, if ProgrammableGrass were to be considered as part of the infrastructure for a sustainable society in the future, energy efficiency would also become necessary. The extensive use of actuators in a pin display leads to high power consumption and increases costs. Therefore, by focusing on minimizing development costs and improving energy efficiency, we can not only enhance the resolution and display size of ProgrammableGrass but also contribute significantly to the realization of a sustainable society. For example, MagneShape [58] is an example of technology where a pin display can be operated without electricity and at a lower cost by moving a magnetic sheet against pins embedded with magnets.

The green grass lengths can correspond to the 8-bit levels using the grass color calibration system. It is also possible to display 16-bit images on ProgrammableGrass by creating a correspondence table that focuses on 16-bit levels instead of 8-bit levels. However, when the difference between the grass lengths of adjacent 8-bit or 16-bit levels is less than 10/73 [mm], those grass lengths are converted to the same counts using a gain value of 7.3. This problem arises due to the resolution of the grass module’s magnetic rotary encoder; utilizing a high-resolution rotary encoder would solve this issue.

In this paper, we conducted experiments on the 8-bit calibration of a single grass pixel under various lighting conditions including standard illumination defined by ISO, the classroom, the meeting room, and the dimly lit space. However, the calibration experiments conducted using the tablet PC in this study are subject to limitations, as they require redoing the experiments in the actual field every time the placement of the grass display is changed. To achieve a more straightforward and flexible 8-bit calibration, we are focusing on a method that completes the calibration in a virtual space. This approach allows for calibration in diverse environments by simply changing the High Dynamic Range Image (HDRI) for the lighting environment. Indeed, we have previously proposed a simple calibration method for grass pixels using a virtual space [10]. In future work, we are planning to develop a system that can realize an 8-bit grass color calibration in a virtual space.

In addition to this, it is important to test the effectiveness of the calibration system in actual outdoor environments. Unlike indoor environments, outdoor settings are characterized by very strong illumination, as well as constantly changing illumination and white balance. These changes are due to the movement of the sun and shifts in weather conditions. The current 8-bit calibration system is based on the assumption of constant illumination and white balance. Therefore, addressing fluctuating illumination and white balance is essential for realizing an 8-bit grass color calibration in outdoor environments in the future.

In Subsection IX-B, we introduced the interactions with ProgrammableGrass via the tablet PC. Along with these functionalities, if the resolution and display size of ProgrammableGrass are improved, it would enable the creation of interactive art such as calligraphy art on the grass. Moreover, if ProgrammableGrass is installed as part of the real-world infrastructure, urban landscape designers would be able to flexibly determine the color of the grass to match the surrounding buildings, the ambiance of the city, and the seasons.

In real-life situations, people engage with grass, such as having picnics in a park with family and friends, and this interaction with grass facilitates smooth communication among people. To enhance the relationship between users and grass, it is important to have functionalities that allow interactions such as drawing illustrations or controlling animations directly by touching the grass, without a tablet PC.

Modern artificial grass materials are available in various colors, such as red and blue. Such artificial grass materials can be used to develop ProgrammableGrass with multiple grass colors according to the environment and situation. Furthermore, ProgrammableGrass controls the grass color based on spatial additive mixing. Since spatial additive mixing is a phenomenon based on the perceptual model of the human eyes, user studies are important to control the grass color focusing on psychophysical aspects. In addition, it is also important to confirm whether ProgrammableGrass is friendly to green landscapes through user studies. For example, user studies can be conducted using the Likert scale [59], a method well-suited for measuring user emotions, opinions, and evaluations, to determine whether ProgrammableGrass can be harmonious with green landscapes.