[docs] Refresh effective and efficient doc #12134

Original file line number	Diff line number	Diff line change
Expand Up		@@ -7,7 +7,7 @@
		- local: quicktour
		title: Quicktour
		- local: stable_diffusion
	title: Effective and efficient diffusion
	title: Basic performance

		- title: DiffusionPipeline
		isExpanded: false
Expand Down

285 changes: 75 additions & 210 deletions docs/source/en/stable_diffusion.md

Show comments View file Open in desktop

Original file line number	Diff line number	Diff line change
Expand Up		@@ -10,252 +10,117 @@ an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express o
		specific language governing permissions and limitations under the License.
		-->

	# Effective and efficient diffusion

		[[open-in-colab]]

	Getting the [`DiffusionPipeline`] to generate images in a certain style or include what you want can be tricky. Often times, you have to run the [`DiffusionPipeline`] several times before you end up with an image you're happy with. But generating something out of nothing is a computationally intensive process, especially if you're running inference over and over again.

	This is why it's important to get the most computational (speed) and memory (GPU vRAM) efficiency from the pipeline to reduce the time between inference cycles so you can iterate faster.

	This tutorial walks you through how to generate faster and better with the [`DiffusionPipeline`].

	Begin by loading the [`stable-diffusion-v1-5/stable-diffusion-v1-5`](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5) model:
	# Basic performance

	```python
	from diffusers import DiffusionPipeline

	model_id = "stable-diffusion-v1-5/stable-diffusion-v1-5"
	pipeline = DiffusionPipeline.from_pretrained(model_id, use_safetensors=True)
	```
	Diffusion is a random process that is computationally demanding. You may need to run the [`DiffusionPipeline`] several times before getting a desired output. That's why it's important to carefully balance generation speed and memory usage in order to iterate faster,

	The example prompt you'll use is a portrait of an old warrior chief, but feel free to use your own prompt:
	This guide recommends some basic performance tips for using the [`DiffusionPipeline`]. Refer to the Inference Optimization section docs such as [Accelerate inference](./optimization/fp16) or [Reduce memory usage](./optimization/memory) for more detailed performance guides.

	```python
	prompt = "portrait photo of a old warrior chief"
	```

	## Speed

	<Tip>

	💡 If you don't have access to a GPU, you can use one for free from a GPU provider like [Colab](https://colab.research.google.com/)!

	</Tip>

	One of the simplest ways to speed up inference is to place the pipeline on a GPU the same way you would with any PyTorch module:

	```python
	pipeline = pipeline.to("cuda")
	```
	## Memory usage

	To make sure you can use the same image and improve on it, use a [`Generator`](https://pytorch.org/docs/stable/generated/torch.Generator.html) and set a seed for [reproducibility](./using-diffusers/reusing_seeds):
	Reducing the amount of memory used indirectly speeds up generation and can help a model fit on device.

	```python
	```py
		import torch
	from diffusers import DiffusionPipeline

	generator = torch.Generator("cuda").manual_seed(0)
	```

	Now you can generate an image:
	pipeline = DiffusionPipeline.from_pretrained(
	"stabilityai/stable-diffusion-xl-base-1.0",
	torch_dtype=torch.bfloat16
	).to("cuda")
	pipeline.enable_model_cpu_offload()

	```python
	image = pipeline(prompt, generator=generator).images[0]
	image
	prompt = """
	cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
	highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
	"""
	pipeline(prompt).images[0]
	print(f"Max memory reserved: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB")
		```

	<div class="flex justify-center">
	<img src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/stable_diffusion_101/sd_101_1.png">
	</div>
	## Inference speed

	This process took ~30 seconds on a T4 GPU (it might be faster if your allocated GPU is better than a T4). By default, the [`DiffusionPipeline`] runs inference with full `float32` precision for 50 inference steps. You can speed this up by switching to a lower precision like `float16` or running fewer inference steps.
	Denoising is the most computationally demanding process during diffusion. Methods that optimizes this process accelerates inference speed. Try the following methods for a speed up.

	Let's start by loading the model in `float16` and generate an image:
	- Add `.to("cuda")` to place the pipeline on a GPU. Placing a model on an accelerator, like a GPU, increases speed because it performs computations in parallel.
	- Set `torch_dtype=torch.bfloat16` to execute the pipeline in half-precision. Reducing the data type precision increases speed because it takes less time to perform computations in a lower precision.

	```python
	```py
		import torch
	import time
	from diffusers import DiffusionPipeline, DPMSolverMultistepScheduler

	pipeline = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16, use_safetensors=True)
	pipeline = pipeline.to("cuda")
	generator = torch.Generator("cuda").manual_seed(0)
	image = pipeline(prompt, generator=generator).images[0]
	image
	pipeline = DiffusionPipeline.from_pretrained(
	"stabilityai/stable-diffusion-xl-base-1.0",
	torch_dtype=torch.bfloat16
	).to("cuda")
		```

	<div class="flex justify-center">
	<img src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/stable_diffusion_101/sd_101_2.png">
	</div>

	This time, it only took ~11 seconds to generate the image, which is almost 3x faster than before!

	<Tip>

	💡 We strongly suggest always running your pipelines in `float16`, and so far, we've rarely seen any degradation in output quality.

	</Tip>

	Another option is to reduce the number of inference steps. Choosing a more efficient scheduler could help decrease the number of steps without sacrificing output quality. You can find which schedulers are compatible with the current model in the [`DiffusionPipeline`] by calling the `compatibles` method:

	```python
	pipeline.scheduler.compatibles
	[
	diffusers.schedulers.scheduling_lms_discrete.LMSDiscreteScheduler,
	diffusers.schedulers.scheduling_unipc_multistep.UniPCMultistepScheduler,
	diffusers.schedulers.scheduling_k_dpm_2_discrete.KDPM2DiscreteScheduler,
	diffusers.schedulers.scheduling_deis_multistep.DEISMultistepScheduler,
	diffusers.schedulers.scheduling_euler_discrete.EulerDiscreteScheduler,
	diffusers.schedulers.scheduling_dpmsolver_multistep.DPMSolverMultistepScheduler,
	diffusers.schedulers.scheduling_ddpm.DDPMScheduler,
	diffusers.schedulers.scheduling_dpmsolver_singlestep.DPMSolverSinglestepScheduler,
	diffusers.schedulers.scheduling_k_dpm_2_ancestral_discrete.KDPM2AncestralDiscreteScheduler,
	diffusers.utils.dummy_torch_and_torchsde_objects.DPMSolverSDEScheduler,
	diffusers.schedulers.scheduling_heun_discrete.HeunDiscreteScheduler,
	diffusers.schedulers.scheduling_pndm.PNDMScheduler,
	diffusers.schedulers.scheduling_euler_ancestral_discrete.EulerAncestralDiscreteScheduler,
	diffusers.schedulers.scheduling_ddim.DDIMScheduler,
	]
	```

	The Stable Diffusion model uses the [`PNDMScheduler`] by default which usually requires ~50 inference steps, but more performant schedulers like [`DPMSolverMultistepScheduler`], require only ~20 or 25 inference steps. Use the [`~ConfigMixin.from_config`] method to load a new scheduler:

	```python
	from diffusers import DPMSolverMultistepScheduler
	- Use a faster scheduler, such as [`DPMSolverMultistepScheduler`], which only requires ~20-25 steps.
	- Set `num_inference_steps` to a lower value. Reducing the number of inference steps reduces the overall number of computations. However, this can result in lower generation quality.

	```py
		pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
	```

	Now set the `num_inference_steps` to 20:

	```python
	generator = torch.Generator("cuda").manual_seed(0)
	image = pipeline(prompt, generator=generator, num_inference_steps=20).images[0]
	image
	```

	<div class="flex justify-center">
	<img src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/stable_diffusion_101/sd_101_3.png">
	</div>

	Great, you've managed to cut the inference time to just 4 seconds! ⚡️

	## Memory

	The other key to improving pipeline performance is consuming less memory, which indirectly implies more speed, since you're often trying to maximize the number of images generated per second. The easiest way to see how many images you can generate at once is to try out different batch sizes until you get an `OutOfMemoryError` (OOM).

	Create a function that'll generate a batch of images from a list of prompts and `Generators`. Make sure to assign each `Generator` a seed so you can reuse it if it produces a good result.
	prompt = """
	cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
	highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
	"""

	```python
	def get_inputs(batch_size=1):
	generator = [torch.Generator("cuda").manual_seed(i) for i in range(batch_size)]
	prompts = batch_size * [prompt]
	num_inference_steps = 20
	start_time = time.perf_counter()
	image = pipeline(prompt).images[0]
	end_time = time.perf_counter()

	return {"prompt": prompts, "generator": generator, "num_inference_steps": num_inference_steps}
	print(f"Image generation took {end_time - start_time:.3f} seconds")
		```

	Start with `batch_size=4` and see how much memory you've consumed:
	## Generation quality

	```python
	from diffusers.utils import make_image_grid
	Many modern diffusion models deliver high-quality images out-of-the-box. However, you can still improve generation quality by trying the following.

	images = pipeline(**get_inputs(batch_size=4)).images
	make_image_grid(images, 2, 2)
	```

	Unless you have a GPU with more vRAM, the code above probably returned an `OOM` error! Most of the memory is taken up by the cross-attention layers. Instead of running this operation in a batch, you can run it sequentially to save a significant amount of memory. All you have to do is configure the pipeline to use the [`~DiffusionPipeline.enable_attention_slicing`] function:

	```python
	pipeline.enable_attention_slicing()
	```

	Now try increasing the `batch_size` to 8!

	```python
	images = pipeline(**get_inputs(batch_size=8)).images
	make_image_grid(images, rows=2, cols=4)
	```

	<div class="flex justify-center">
	<img src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/stable_diffusion_101/sd_101_5.png">
	</div>

	Whereas before you couldn't even generate a batch of 4 images, now you can generate a batch of 8 images at ~3.5 seconds per image! This is probably the fastest you can go on a T4 GPU without sacrificing quality.

	## Quality

	In the last two sections, you learned how to optimize the speed of your pipeline by using `fp16`, reducing the number of inference steps by using a more performant scheduler, and enabling attention slicing to reduce memory consumption. Now you're going to focus on how to improve the quality of generated images.

	### Better checkpoints

	The most obvious step is to use better checkpoints. The Stable Diffusion model is a good starting point, and since its official launch, several improved versions have also been released. However, using a newer version doesn't automatically mean you'll get better results. You'll still have to experiment with different checkpoints yourself, and do a little research (such as using [negative prompts](https://minimaxir.com/2022/11/stable-diffusion-negative-prompt/)) to get the best results.

	As the field grows, there are more and more high-quality checkpoints finetuned to produce certain styles. Try exploring the [Hub](https://huggingface.co/models?library=diffusers&sort=downloads) and [Diffusers Gallery](https://huggingface.co/spaces/huggingface-projects/diffusers-gallery) to find one you're interested in!
	- Try a more detailed and descriptive prompt. Include details such as the image medium, subject, style, and aesthetic. A negative prompt may also help by guiding a model away from undesirable features by using words like low quality or blurry.

	### Better pipeline components
	```py
	import torch
	from diffusers import DiffusionPipeline

	You can also try replacing the current pipeline components with a newer version. Let's try loading the latest [autoencoder](https://huggingface.co/stabilityai/stable-diffusion-2-1/tree/main/vae) from Stability AI into the pipeline, and generate some images:
	pipeline = DiffusionPipeline.from_pretrained(
	"stabilityai/stable-diffusion-xl-base-1.0",
	torch_dtype=torch.bfloat16
	).to("cuda")

	```python
	from diffusers import AutoencoderKL
	prompt = """
	cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
	highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
	"""
	negative_prompt = "low quality, blurry, ugly, poor details"
	pipeline(prompt, negative_prompt=negative_prompt).images[0]
	```

	vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-mse", torch_dtype=torch.float16).to("cuda")
	pipeline.vae = vae
	images = pipeline(**get_inputs(batch_size=8)).images
	make_image_grid(images, rows=2, cols=4)
	```

	<div class="flex justify-center">
	<img src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/stable_diffusion_101/sd_101_6.png">
	</div>

	### Better prompt engineering

	The text prompt you use to generate an image is super important, so much so that it is called prompt engineering. Some considerations to keep during prompt engineering are:

	- How is the image or similar images of the one I want to generate stored on the internet?
	- What additional detail can I give that steers the model towards the style I want?

	With this in mind, let's improve the prompt to include color and higher quality details:

	```python
	prompt += ", tribal panther make up, blue on red, side profile, looking away, serious eyes"
	prompt += " 50mm portrait photography, hard rim lighting photography--beta --ar 2:3 --beta --upbeta"
	```

	Generate a batch of images with the new prompt:
	For more details about creating better prompts, take a look at the [Prompt techniques](./using-diffusers/weighted_prompts) doc.

	```python
	images = pipeline(**get_inputs(batch_size=8)).images
	make_image_grid(images, rows=2, cols=4)
	```
	- Try a different scheduler, like [`HeunDiscreteScheduler`] or [`LMSDiscreteScheduler`], that gives up generation speed for quality.

	<div class="flex justify-center">
	<img src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/stable_diffusion_101/sd_101_7.png">
	</div>
	```py
	import torch
	from diffusers import DiffusionPipeline, HeunDiscreteScheduler

	Pretty impressive! Let's tweak the second image - corresponding to the `Generator` with a seed of `1` - a bit more by adding some text about the age of the subject:
	pipeline = DiffusionPipeline.from_pretrained(
	"stabilityai/stable-diffusion-xl-base-1.0",
	torch_dtype=torch.bfloat16
	).to("cuda")
	pipeline.scheduler = HeunDiscreteScheduler.from_config(pipeline.scheduler.config)

	```python
	prompts = [
	"portrait photo of the oldest warrior chief, tribal panther make up, blue on red, side profile, looking away, serious eyes 50mm portrait photography, hard rim lighting photography--beta --ar 2:3 --beta --upbeta",
	"portrait photo of an old warrior chief, tribal panther make up, blue on red, side profile, looking away, serious eyes 50mm portrait photography, hard rim lighting photography--beta --ar 2:3 --beta --upbeta",
	"portrait photo of a warrior chief, tribal panther make up, blue on red, side profile, looking away, serious eyes 50mm portrait photography, hard rim lighting photography--beta --ar 2:3 --beta --upbeta",
	"portrait photo of a young warrior chief, tribal panther make up, blue on red, side profile, looking away, serious eyes 50mm portrait photography, hard rim lighting photography--beta --ar 2:3 --beta --upbeta",
	]

	generator = [torch.Generator("cuda").manual_seed(1) for _ in range(len(prompts))]
	images = pipeline(prompt=prompts, generator=generator, num_inference_steps=25).images
	make_image_grid(images, 2, 2)
	```

	<div class="flex justify-center">
	<img src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/stable_diffusion_101/sd_101_8.png">
	</div>
	prompt = """
	cinematic film still of a cat sipping a margarita in a pool in Palm Springs, California
	highly detailed, high budget hollywood movie, cinemascope, moody, epic, gorgeous, film grain
	"""
	negative_prompt = "low quality, blurry, ugly, poor details"
	pipeline(prompt, negative_prompt=negative_prompt).images[0]
	```

		## Next steps

	In this tutorial, you learned how to optimize a [`DiffusionPipeline`] for computational and memory efficiency as well as improving the quality of generated outputs. If you're interested in making your pipeline even faster, take a look at the following resources:

	- Learn how [PyTorch 2.0](./optimization/fp16) and [`torch.compile`](https://pytorch.org/docs/stable/generated/torch.compile.html) can yield 5 - 300% faster inference speed. On an A100 GPU, inference can be up to 50% faster!
	- If you can't use PyTorch 2, we recommend you install [xFormers](./optimization/xformers). Its memory-efficient attention mechanism works great with PyTorch 1.13.1 for faster speed and reduced memory consumption.
	- Other optimization techniques, such as model offloading, are covered in [this guide](./optimization/fp16).
	Diffusers offers more advanced and powerful optimizations such as [group-offloading](./optimization/memory#group-offloading) and [regional compilation](./optimization/fp16#regional-compilation). To learn more about how to maximize performance, take a look at the Inference Optimization section.

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[docs] Refresh effective and efficient doc #12134

Uh oh!

[docs] Refresh effective and efficient doc #12134

Filter by extension

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing