Prompt Engineering

Introduction

Creating an image that matches a certain expectation is not as trivial as one might expect. The solution space of the AI model is insanely high. To limit this a well engineered prompt can be used to steer the AI in the right direction. This practice is called Prompt Engineering.

Prompt guides

• https://stable-diffusion-art.com/how-to-come-up-with-good-prompts-for-ai-image-generation/#Some_good_keywords_for_you
• https://anakin.ai/blog/stable-diffusion-prompt-guide/
• https://cheatsheet.md/stable-diffusion/stable-diffusion-prompts-guide.en
• https://cheatsheet.md/stable-diffusion/stable-diffusion-webui-styles
• https://stable-diffusion-art.com/
• https://supagruen.github.io/StableDiffusion-CheatSheet/

Resource references

• https://civitai.com/
• https://www.mage.space/explore
• https://tensor.art/
• https://majinai.art/
• https://openmodeldb.info/

Prompt perfecters

Using ChatGPT to optimize the prompt is efficient, because it is not always clear how the AI will respond to a certain prompt. There are templates available to help kickstart the prompt perfectioning via ChatGPT.

• https://www.feedough.com/stable-diffusion-prompt-generator/
• https://huggingface.co/spaces/Gustavosta/MagicPrompt-Stable-Diffusion
• https://sd-prompt-generator.netlify.app/

SD-Web-UI Forge workflow

To get as close to the imagined concept and as efficiently as possible, the following workflow is proposed:

1. Database selection:
   1. Checkpoint selection: Each model is trained for specific output. Use the right one! (Multiple can be combined)
   2. Add additional LoRa or LyCORIS files to improve specific features (download at: https://civitai.com/models) 
2. Prompt creation:
   1. Use prompt perfector to get to a good descriptive prompt
   2. Use CLIP Interrogator to retrieve a prompt from an example image that can be used to get detailed prompts for specific features
   3. Use a fixed seed, in order to same results for the same prompt.
3. Choose settings:
   1. Choose proper sampling method. Each method has it's own benefits and downsides.
   2. Use relative low resolution, but at the desired aspect ratio. Exact resolution is depending on the used database models (e.g. 512x512, 768x768, 1024x1024 px).
4. Image generation:
   1. Iterate the prompt until the concept is mainly displayed.
5. Finalize the details:
   1. Move to the img2img tab to use inpainting to improve certain aspects
6. Upscale image:
   1. Move to Extras tab and set higher resolution and iterations. Fine-tune the prompt until all details are correct.
7. Save image and save prompt & settings.