I'm very interested in this PR; I wish I had time to test DeepCache in Comfy UI and compare the results with your PR.

My implementation doesn’t seem to work without CFG, which is really odd since DeepCache is supposed to be CFG-agnostic.

My guess is that it's sharing the same cache between uncond and conditioned pass and it's probably not supposed to.

My implementation doesn’t seem to work without CFG, which is really odd since DeepCache is supposed to be CFG-agnostic.

My guess is that it's sharing the same cache between uncond and conditioned pass and it's probably not supposed to.

I tried to create a separate cache for conditional and unconditional passes, but it broke things even more. In any case, I think we should fix things with CFG first before addressing the CFG-free issue, don't think those are related

What is CFG?

According to my understanding, it's when we pass the --cfg-scale parameter. Why do they refer to it as something that's missing in this project?

Or is it a deepcache configuration?

What is CFG?

CFG means Classifier-Free guidance. It's basically a way to change how much of an effect the prompt has for conditional generation by linearly extrapolating from the conditioned prediction away from the prediction without text conditioning (or with a negative prompt). So it needs 2 forward passes at each step: 1 with the positive prompt, and 1 with empty/negative prompt.

What is CFG?

According to my understanding, it's when we pass the --cfg-scale parameter. Why do they refer to it as something that's missing in this project?

Or is it a deepcache configuration?

It's just the --cfg-scale. When you try to run inference with a CFG of 1, the results are significantly worse, almost garbage. It seems that the more steps it takes, the further off the output gets, as if some error is accumulating at each step.

I did try separating the cache between the conditional and unconditional passes, but that didn’t help and in fact, it broke the case where we run with CFG > 1. From my understanding, DeepCache operates at a higher level and shouldn't be affected by this conditional/unconditional distinction stuff. Something is seriously wrong here, but I can't quite put my finger on it

EDIT: I may have wrongly assumed that you're familiar with the concept of CFG, but @stduhpf already explained it well. Basically, during inference, you're doing:

final_prediction = prediction_unconditional + w * (prediction_conditional - prediction_unconditional)

When w = 1, you're effectively running only the conditional pass. That’s useful because it means you can double your inference speed, and distilled models support this approach. However, you do trade off some prompt fidelity when doing so.

I recently read a paper that concluded CFG might actually be useless. It only appears to work because we end up using twice the compute