Generate samples using Midjourney AI
From the beginning, I don’t have a good image model for drawing plants. Midjourney supports running its ai model using a tool that is very friendly to beginners. I generate images by just inputting a set of prompts, including the phases of certain plants, certain ecology settings, certain backgrounds, etc. I can get a set of 4 images as a sample and generate another set of its variants if the images are not satisfactory.
I used this tool to get some visualisations and inspirations for our initial concept of forming the plants with Midjourney’s powerful model that trained with enormous data. Most importantly, a very developed model is essential to the outcome of the generation because the capability of the model to provide a precise understanding of what has been inputted determines how well the generated images come out as intended.
I used Midjourney at the beginning because it provides a relatively consistent style and tone in the same sets of generated results from the same sets of prompts. Midjourney provided me with a set of nice concept images in the early stages. However, due to the limited license. We cannot directly use Midjourney’s images in our finished presentation.
Draw concept images using Stable Diffusion Web UI
Later, I set up Stable Diffusion and started to look around how to run it locally to create concept images.
To gain visual references of concept art illustrations and designs, on the other hand, I appropriated elements for generating the appearance of plants that were drawn from a variety of image websites like Pinterest.
In conjunction with some sources on plant studies, their development and ecological evolution, I have drawn up a series of prompts for additional input beyond just the ChatGPT phrases to achieve the ideal output of the text-to-image process. Since many ideas of ‘bio morphosis’ of the plant are non-specific and sometimes can be fantasy in an environmentalism narrative, the process of making it is rather driven by imagination, even human concept artists would take a hard time for representations of the plants and their environments’ imagery because the subject we are creating doesn’t exist in the real word. Thus I modified every source of the prompt by specifying the settings and contextual information to make the prompt more absorbable to image AI tools.
At the very beginning, sample images generated by a default SD model were rather unsatisfactory, so I tried to adjust the prompts to make the images’ quality. If we just input a few prompts into txt2img to create a plant, it does not work.
As the composition of a plant itself is rather complex, a plant has stalks, leaves, a root system, buds etc. There is no quick way to generate all details of these parts well at the same time. Therefore, I used long strings of prompts to generate single plants in every picture. The number of concept illustrations generated was huge, and only a few quality materials can be selected to take into the following process.
After exploring, I found that if I want to make the generated picture more like a conceptual design, I need to add more specific styling descriptions and mention ‘by certain illustrator/artist’ at some point. Under the prompt words column, the weight of the pre-positioned prompt phases used in the generation is greater than that of the post-position prompt words. Place a prompt before other prompts can let it make a greater impact on the result. This also applies to negative prompts.
I put one of Jiaojiao’s sketches into the Stable Diffusion’s img2img to convert it to new sketches. Different prompts have been input in different steps during the process.
When creating the cactus illustrations, I used Photoshop to remove the background from multiple images, took one/several piece(s) from different plants and patchworked them into a new paint. The samples of plants that were stitched by different parts/tissues of plants were then sent to img2img and redrawn to generate more variants.
In addition, I generated a series of images that depict wasteland and post-apocalyptic landscapes, which would be used as background for the mutating cactus. Then, I removed the background of the cactus concept images, put the cactus in the middle of each scene, and redrawn them. In the last round of redrawing the image, I turned down the CFG scale and refined the prompt so the basic composition and tone of the image would stay consistent.
Used tools:
https://docs.midjourney.com/docs/midjourney-discord
https://github.com/deforum-art/deforum-for-automatic1111-webui
https://huggingface.co/lambdalabs/image-mixer
https://huggingface.co/spaces/fffiloni/CLIP-Interrogator-2
