Trash to Treasure:
Using text-to-image models to inform the design of physical artefacts

Of the 30 participants, 27 produced at least two prompt and image pairs. Of those 27, 24 produced all three images (the remaining three produced just one image). 23/30 participants self-reported that their sculptures were informed by the images they saw, and 28/30 participants reported that they would use text-to-image models again for a creative task. Figures 1,  2,  3, and  4 show examples of participants’ three prompt and images pairs and the created sculptures that have strong visual links to the final design highlighted.

Figure  1 shows an example of strong image influence on sculpture design for someone who did not have a sculpture idea going into the process. The rightmost image is a photo of the sculpture they built, which is striking in its visual similarity to the third generated image in particular. Some congruences, including color of materials which were not changeable, are highlighted in colored boxes on the image.

Figure  2 shows the journey of someone who knew from the onset they wanted to make a sculpture of a crab. The three images are outputs from the prompts: “crab, ocean, spider, seaweed, plush toy of a crab” →“crab, ocean, spider, seaweed, plush toy of a crab, colour red” →“crab, ocean, spider, seaweed, plush toy of a crab, colour red, insect, lobster.” Despite already knowing what they wanted to make, the participant said the images gave reminders of a crab’s features so they could create it in real life. The participant tweaked their second prompt to include “colour red” because their physical materials were also red. Common entities like crabs are easy to generate thanks to their high visual determinacy, but some users with more abstract ideas than the example discussed above reported frustration visualizing what they had in mind, even if they knew what they wanted to see in the images.

Of the participants who reported that the images were not informative to their sculpture (n = 7), three explained that this was because images did not sufficiently relate to the materials given and they could not build what they saw. Some of these participants expressed frustration in formulating a text prompt that would produce what they wanted to see in an image. Only 36.2% of images contained elements that the participants expected to see, indicating many visual elements were unexpected. 40% of participants (12/30) directly referenced their chosen materials, like “box” or “sponge.” ²²2We do not find statistically significant differences in the distribution of average cosine distances of user prompts and whether or not the prompt contained mentions of physical materials (t = 1.01, p = 0.32), material qualities (t = -1.25, p = 0.22), or colors (t = 0.12 , p = 0.91). The frustration some users had translating between materials and expected images suggests a need to better support users in translating between physical materials and prompt wording when ideating for physical artefacts.

In post-interviews, the reasons participants reported that images were informative were diverse. Some found that the lateral concepts introduced by the images gave them new conceptual ideas, like the addition of a river in Figure  1 perhaps indicating they were in an early design stage where exploration was particularly useful. Others honed existing ideas by getting implementation inspiration from the images, like in Figure  2 perhaps indicating a later design stage focused on execution. These findings confirm those in Epstein, Schroeder, and Newman (2022), which showed that AI-generated images are helpful in visualizing ideas for two main reasons: they give lateral insight as well as implementation ideas.

Two participants who did not find the images helpful built their sculpture immediately after their first image, and explained that the strength of their pre-visualization ideas meant that they believed no images would further inform the sculptures they would make. This might suggest that seeing images is less helpful for users with an already concrete vision, but we do not find strong evidence to support that claim. Of the individuals with a design idea before starting the activity, 3/8 found seeing the images helpful and 5 did not. Of the individuals with no design idea before, 15/23 found seeing the images helpful and 8 did not, a 44% difference. This provides suggestive evidence (t = -1.365, p = 0.183) that participants who did not have an idea may find the generated images more helpful.

Participants’ wording choices for prompts ranged in their degree of conceptual exploration. We observed and named some patterns we saw in users’ prompting choices. The most minimal amount of conceptual exploration that we observed was a pattern we call the “refiner” style. In these instances, the participant started with a prompt and made minor edits to it. It was the most common pattern we saw (n = 15). For example: Prompt 1: ‘geometric ominous dystopian creature metallic’. Prompt 2: ‘geometric ominous dystopian creature neon heart beating’. Prompt 3: ‘geometric ominous creature neon heart beating rainbow’. The participant described here kept changing the final terms to try to see the image they wanted.

The next pattern observed was the “rephraser” prompting style (n = 5). Here, conceptual subject matter remained the same, but changes to the wording or order of the prompt changed substantially. Example: Prompt 1: ‘idyllic cyber vista with a bottle robot’. Prompt 2: ‘dasani water bottle robot in front of the Windows XP wallpaper’. Prompt 3: ‘robot that looks like a water bottle in front of a grassy hill’. Each prompt contains a reference to a robot, but the way they refer to the same background of a grassy hill changes.

A few participants did even more conceptual exploration. The “explorer” prompting style (n = 2) we describe consisted of three conceptually unrelated prompts. Example: Prompt 1: ‘my printed map required too much plastic’. Prompt 2: ‘I’m exhausted but I’m still having fun’. Prompt 3: ‘my hovercraft is full of eels’. The small size of this category shows that most people had some idea of what they were exploring.

These described styles form an exploration gradient where “explorers” have the most semantic distance traveled, “rephrasers” have a main idea but still explore around it, and “refiners” focus on exploitation of a single string of words.

We observed that a remaining 7 participants did not fit neatly in to these qualitatively defined prompting categories, often using a mix of styles we have described. For example, the user who built the sculpture in Figure 1 prompted as follows: Prompt 1: “Hamburg is the most beautiful city in the world”, Prompt 2: “sun and shiny water, colourful buildings made of trash and ropes”, Prompt 3: “green red and white buildings in front of a shiny river with a white fence on the side”. The first prompt appears to be exploratory, whereas the second two describe a more concrete scene in two similar but distinct ways, reminiscent of behavior we see in those using a “rephraser” style. The final three participants to account for only gave a single prompt, so their prompting style does not form a style that can be attributed over time.

Because not all participants fit neatly into the prompting styles we qualitatively described, we use a computational measure of semantic distance to objectively characterize all prompting journeys in the rest of our analysis. We calculate the average conceptual distance a participant traveled through their prompting journey by taking the cosine distance between the text embeddings of first and second prompts, then the second and third prompts, then averaging the two distances. For participants with two prompts, we treat the single distance between the two prompts as the ”average” distance. For participants with a single prompt, where no distance between prompts was traveled, we defined the distance as 0.

Figure 5 shows a histogram of average cosine distances across the prompting journeys of our 27 participants who gave at least 2 prompts, along with the average and standard deviation of the subgroup we qualitatively grouped into “refiner,” “rephraser,” “inconsistent,” and “explorer.” The participants we describe as “refiners” have the lowest average cosine distance of 0.178, while the participants we described as “explorers” had the highest average cosine distance of 0.793. The participants we describe as “rephraser” and “inconsistent” fall in the middle range.

We visualize three participant journeys, one from each of the main styles we have described, through semantic space in Figure 7. We use TSNE to reduce dimensionality of the shared embedding space to two dimensions, and draw arrows from points representing a participant’s first to second to third prompts. The path between green dots in Figure 7 (representing a single user’s prompting journey) shows considerable conceptual exploration using an “explorer” prompting style. The participant journey represented by blue dots shows a less exploratory “rephraser” style, and the user represented by purple dots shows the “refiner” prompting style. The prompts and corresponding images generated for these three participants can be seen in Figure 6.

We next investigate the relationship between the design stage a user indicated, based on whether they described in our post-interview having started the task with a sculpture idea or not, and the average semantic distance they traveled. Because we gave participants the design task of creating a sculpture, participants skipped what Hwang (2022) calls the “Q&A” stage of defining the creative task. By constraining the category of materials and type of creative output (sculpture), we further define the design problem’s “artefact type” and “media type,” per the Mothersill and Bove (2017) conceptualization of the design process. Those who came into the visualization stage of the exercise with no idea what to make started in the “wandering stage,” wherein creatives explore possible strategies and incubate ideas. Some who started visualization with a sculpture idea skipped this stage and were already in the “hands-on stage” of developing solutions, or were already moving to the “camera-ready stage,” which focuses on selecting and implementing ideas, to use Hwang’s taxonomy. Using these stages of the creative process, we compare the average semantic distance traveled for the group of participants who had sculpture ideas before visualizing with generative AI to those who did not.

We found that the amount of conceptual exploration a participant did was lower for participants who said they had a sculpture idea at the visualization stage than those who did not (t = -2.94, p = 0.006). We also found that participants who stopped early, generating fewer than 3 images, employed less conceptual exploration in their images than those who did not (t = 4.31, p <0.001). All participants who stopped after one (n = 3) or two images (n = 4) had an idea of what to make before starting their visualization.