ComfyUI implementation for ELLA.

• [2024年4月30日] Add a new node ELLA Text Encode to automatically concat ella and clip condition.
• [2024年4月24日] Upgraded ELLA Apply method. Better compatibility with the comfyui ecosystem. Refer to the method mentioned in ComfyUI_ELLA PR #25
  • DEPRECATED: Apply ELLA without simgas is deprecated and it will be removed in a future version.
• [2024年4月22日] Fix unstable quality of image while multi-batch. Add CLIP concat (support lora trigger words now).
• [2024年4月19日] Documenting nodes.
• [2024年4月19日] Initial repo.

• SIGMAS from node BasicScheduler or TIMESTEPS by node Set ELLA Timesteps must be the same as the KSampler settings. Because Timestep-Aware Semantic Connector (TSC), which dynamically adapts semantics features over sampling time steps, has been introduced.
• If you need concat clip CONDITIONING to make LoRA trigger words effective, ELLA output CONDITIONING always needs to be linked to the conditioning_to of Conditioning (Concat) node.

The examples directory has workflow examples. You can directly load these images as workflow into ComfyUI for use.

workflow_example

All legacy workflows was compatible. But it is deprecated and will be removed in a future version.

workflow_example_legacy

🎉 It works with controlnet!

workflow_controlnet

🎉 It works with lora trigger words by concat CLIP CONDITIONING!

⚠️ NOTE again that ELLA CONDITIONING always needs to be linked to the conditioning_to of Conditioning (Concat) node.

workflow_lora

With ELLA Text Encode node, can simplify the workflow.

With the upgrade(2024年4月24日), some interesting workflow can be implemented, such as using ELLA only in positive. As shown below:

workflow_lora_positive_ella_only

However, there is no guarantee that positive-only will bring better results.

Workflow with AYS.

workflow_ella_ays

AYS got with more visual details and better text-alignment, ref to paper.

And EMMA is working in progress.

Download or git clone this repository inside ComfyUI/custom_nodes/ directory. ComfyUI-ELLA requires the latest version of ComfyUI. If something doesn't work be sure to upgrade.

cd ComfyUI/custom_nodes
git clone https://github.com/TencentQQGYLab/ComfyUI-ELLA

Next install dependencies.

cd ComfyUI-ELLA
pip install -r requirements.txt

These models must be placed in the corresponding directories under models.

Remember you can also use any custom location setting an ella & ella_encoder entry in the extra_model_paths.yaml file.

• ComfyUI/models/ella, create it if not present.
  • Place ELLA Models here
• ComfyUI/models/ella_encoder, create it if not present.
  • Place FLAN-T5 XL Text Encoder here, it should be a folder of transfomers structure with config.json

In summary, you should have the following model directory structure:

ComfyUI/models/ella/
└── ella-sd1.5-tsc-t5xl.safetensors
ComfyUI/models/ella_encoder/
└── models--google--flan-t5-xl--text_encoder
 ├── config.json
 ├── model.safetensors
 ├── special_tokens_map.json
 ├── spiece.model
 ├── tokenizer_config.json
 └── tokenizer.json

Nodes reference

• XXX not implemented for 'Half'. See issue #12
• AYS + Ella getting dark image generations. See issue #39
  • Check if add_noise of SamplerCustom node is enabled.
  • Lower the cfg of SamplerCustom node.

• Support prompt weighting

• ComfyUI: https://github.com/comfyanonymous/ComfyUI
• Diffusers (borrowed timestep modules): https://github.com/huggingface/diffusers

@misc{hu2024ella,
 title={ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment}, 
 author={Xiwei Hu and Rui Wang and Yixiao Fang and Bin Fu and Pei Cheng and Gang Yu},
 year={2024},
 eprint={2403.05135},
 archivePrefix={arXiv},
 primaryClass={cs.CV}
}