使用中文微调的控制Stable Diffusion的空间信息的轻量级网络
ControlLoRA是使用LoRA技术可以简单调试stable diffusion来控制其空间信息的工程。一般使用的是简单微小的网络(~7M 参数个数, ~25M 存储)。更多的信息可以从ControlLoRA获得。
这个工程可以看作是ControlLoRA的一个fork。并依据ControlLoRA的方法给出两个中文领域的模型。
你可以使用在线的huggingface space,上传你的图片和中文提示文本看输出结果。由于是在cpu上进行部署使用,我推荐你下载这些工程到本地并使用你的gpu进行运行。(由于"is_available"的设定,将会动态根据是否有gpu切换设备)
| 名称 | HuggingFace 模型链接 | HuggingFace 空间链接 |
|---|---|---|
| ControlNet By Canny Chinese 🔪 | https://huggingface.co/svjack/canny-control-lora-zh | https://huggingface.co/spaces/svjack/ControlNet-Canny-Chinese |
| ControlNet By Pose Chinese 🏃 | https://huggingface.co/svjack/pose-control-lora-zh | https://huggingface.co/spaces/svjack/ControlNet-Pose-Chinese |
pip install -r requirements.txt
在安装后,可以cd进入ControlNet-Canny-Chinese 和 ControlNet-Pose-Chinese 分别运行
python app.py
打开你的浏览器,进入 http://localhost:7860 在浏览器进行实验。
| Name | Prompt | Original Image | Backbone Image | Transformed Image |
|---|---|---|---|---|
| ControlNet By Canny Chinese 🔪 | 表情俏皮的小丑 | Girl in a jacket | Girl in a jacket | Girl in a jacket |
| ControlNet By Canny Chinese 🔪 | 满布流星的夜晚 | Girl in a jacket | Girl in a jacket | Girl in a jacket |
| ControlNet By Canny Chinese 🔪 | 猫咪吸血鬼 | Girl in a jacket | Girl in a jacket | Girl in a jacket |
| ControlNet By Pose Chinese 🏃 | 麦田守望者 | Girl in a jacket | Girl in a jacket | Girl in a jacket |
| ControlNet By Pose Chinese 🏃 | 身穿军服的军官 | Girl in a jacket | Girl in a jacket | Girl in a jacket |
LoRA: Low-Rank Adaptation of Large Language Models
LoRA通过学习秩分解矩阵对并冻结原来的权重减少了训练参数的个数。这极大地减少了大模型对下游任务微调和任务切换的存储限制,开放部署时的推断潜力。LoRA也超过了其它很多调节模型(如:adapter, prefix-tuning, 和 fine-tuning)
在Stable Diffusion领域,我也提供了3个使用Lora进行微调的Stable Diffusion模型。
CC3M数据集由svjack/img2dataset-pq2hf-transform-toolkit进行下载和转换。
| 名称 | HuggingFace模型链接 | 语言 | 微调数据集 |
|---|---|---|---|
| svjack/pokemon-sd-lora-zh | https://huggingface.co/svjack/pokemon-sd-lora-zh | Chinese | svjack/pokemon-blip-captions-en-zh |
| svjack/concept-caption-3m-sd-lora-en | https://huggingface.co/svjack/concept-caption-3m-sd-lora-en | English | Conceptual Captions (CC3M) |
| svjack/concept-caption-3m-sd-lora-zh | https://huggingface.co/svjack/concept-caption-3m-sd-lora-zh | Chinese | Conceptual Captions (CC3M) |
你可以通过模型卡片发现如何使用这些模型。
svjack - svjackbt@gmail.com - ehangzhou@outlook.com
Project Link:https://github.com/svjack/ControlLoRA-Chinese