diff --git a/ICEdit-MoE-LoRA.safetensors b/ICEdit-MoE-LoRA.safetensors new file mode 100644 index 0000000..dda2217 --- /dev/null +++ b/ICEdit-MoE-LoRA.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fa03f92c4f1ffb5c3107236c314ef1c2872e6485544130bf144331cb233aba58 +size 134 diff --git a/README.md b/README.md index cd8363b..921029d 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,179 @@ --- -license: Apache License 2.0 - -#model-type: -##如 gpt、phi、llama、chatglm、baichuan 等 -#- gpt - -#domain: -##如 nlp、cv、audio、multi-modal -#- nlp - -#language: -##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa -#- cn - -#metrics: -##如 CIDEr、Blue、ROUGE 等 -#- CIDEr - -#tags: -##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他 -#- pretrained - -#tools: -##如 vllm、fastchat、llamacpp、AdaSeq 等 -#- vllm +license: apache-2.0 +datasets: +- osunlp/MagicBrush +- TIGER-Lab/OmniEdit-Filtered-1.2M +language: +- en +base_model: +- black-forest-labs/FLUX.1-Fill-dev +pipeline_tag: image-to-image +library_name: diffusers +tags: +- art --- -### 当前模型的贡献者未提供更加详细的模型介绍。模型文件和权重，可浏览“模型文件”页面获取。 -#### 您可以通过如下git clone命令，或者ModelScope SDK来下载模型 +
+ +

In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer

+ +
+ Zechuan Zhang  + Ji Xie  + Yu Lu  + Zongxin Yang  + Yi Yang✉  +
+
+ ReLER, CCAI, Zhejiang University; Harvard University +
+
+ ✉Corresponding Author +
+
+ Arxiv  + Huggingface 🤗  + Github  + Huggingface Demo 🤗  + Project Page +
+ + +
+ +

We present In-Context Edit, a novel approach that achieves state-of-the-art instruction-based editing using just 0.5% of the training data and 1% of the parameters required by prior SOTA methods. The first row illustrates a series of multi-turn edits, executed with high precision, while the second and third rows highlight diverse, visually impressive single-turn editing results from our method.

+
+ +:open_book: For more visual results, go checkout our project page + +This repository will contain the official implementation of _ICEdit_. + + +
+ +# ⚠️ Tips + +### If you encounter such a failure case, please **try again with a different seed**! + +- Our base model, FLUX, does not inherently support a wide range of styles, so a large portion of our dataset involves style transfer. As a result, the model **may sometimes inexplicably change your artistic style**. + +- Our training dataset is **mostly targeted at realistic images**. For non-realistic images, such as **anime** or **blurry pictures**, the success rate of the editing **drop and could potentially affect the final image quality**. + +- While the success rates for adding objects, modifying color attributes, applying style transfer, and changing backgrounds are high, the success rate for object removal is relatively lower due to the low quality of the OmniEdit removal dataset. + +The current model is the one used in the experiments in the paper, trained with only 4 A800 GPUs (total `batch_size` = 2 x 2 x 4 = 16). In the future, we will enhance the dataset, and do scale-up, finally release a more powerful model. + +# To Do List + +- [x] Inference Code +- [ ] Inference-time Scaling with VLM +- [x] Pretrained Weights +- [ ] More Inference Demos +- [x] Gradio demo +- [ ] Comfy UI demo +- [ ] Training Code + +# 🎆 News +- **[2025/4/30]** 🔥 We release the [Huggingface Demo](https://huggingface.co/spaces/RiverZ/ICEdit) 🤗! Have a try! +- **[2025/4/30]** 🔥 We release the inference code and [pretrained weights](https://huggingface.co/sanaka87/ICEdit-MoE-LoRA/tree/main) on Huggingface 🤗! +- **[2025/4/30]** 🔥 We release the [paper](https://arxiv.org/abs/2504.20690) on arXiv! +- **[2025/4/29]** We release the [project page](https://river-zhang.github.io/ICEdit-gh-pages/) and demo video! Codes will be made available in next week~ Happy Labor Day! + + +# 💼 Installation + +## Conda environment setup -SDK下载 ```bash -#安装ModelScope -pip install modelscope -``` -```python -#SDK模型下载 -from modelscope import snapshot_download -model_dir = snapshot_download('AI-ModelScope/ICEdit-MoE-LoRA') -``` -Git下载 -``` -#Git模型下载 -git clone https://www.modelscope.cn/AI-ModelScope/ICEdit-MoE-LoRA.git +conda create -n icedit python=3.10 +conda activate icedit +pip install -r requirements.txt +pip install -U huggingface_hub ``` -

如果您是本模型的贡献者，我们邀请您根据模型贡献文档，及时完善模型卡片内容。

\ No newline at end of file +## Download pretrained weights + +If you can connect to Huggingface, you don't need to download the weights. Otherwise, you need to download the weights to local. + +- [Flux.1-fill-dev](https://huggingface.co/black-forest-labs/flux.1-fill-dev). +- [ICEdit-MoE-LoRA](https://huggingface.co/sanaka87/ICEdit-MoE-LoRA). + +## Inference in bash (w/o VLM Inference-time Scaling) + +Now you can have a try! + +> Our model can **only edit images with a width of 512 pixels** (there is no restriction on the height). If you pass in an image with a width other than 512 pixels, the model will automatically resize it to 512 pixels. + +> If you found the model failed to generate the expected results, please try to change the `--seed` parameter. Inference-time Scaling with VLM can help much to improve the results. + +```bash +python scripts/inference.py --image assets/girl.png \ + --instruction "Make her hair dark green and her clothes checked." \ + --seed 42 \ +``` + +Editing a 512×768 image requires 35 GB of GPU memory. If you need to run on a system with 24 GB of GPU memory (for example, an NVIDIA RTX3090), you can add the `--enable-model-cpu-offload` parameter. + +```bash +python scripts/inference.py --image assets/girl.png \ + --instruction "Make her hair dark green and her clothes checked." \ + --enable-model-cpu-offload +``` + +If you have downloaded the pretrained weights locally, please pass the parameters during inference, as in: + +```bash +python scripts/inference.py --image assets/girl.png \ + --instruction "Make her hair dark green and her clothes checked." \ + --flux-path /path/to/flux.1-fill-dev \ + --lora-path /path/to/ICEdit-MoE-LoRA +``` + +## Inference in Gradio Demo + +We provide a gradio demo for you to edit images in a more user-friendly way. You can run the following command to start the demo. + +```bash +python scripts/gradio_demo.py --port 7860 +``` + +Like the inference script, if you want to run the demo on a system with 24 GB of GPU memory, you can add the `--enable-model-cpu-offload` parameter. And if you have downloaded the pretrained weights locally, please pass the parameters during inference, as in: + +```bash +python scripts/gradio_demo.py --port 7860 \ + --flux-path /path/to/flux.1-fill-dev (optional) \ + --lora-path /path/to/ICEdit-MoE-LoRA (optional) \ + --enable-model-cpu-offload (optional) \ +``` + +Then you can open the link in your browser to edit images. + +### 🎨 Enjoy your editing! + + + +# Comparison with Commercial Models + +
+
+ +

Compared with commercial models such as Gemini and GPT-4o, our methods are comparable to and even superior to these commercial models in terms of character ID preservation and instruction following. We are more open-source than them, with lower costs, faster speed (it takes about 9 seconds to process one image), and powerful performance.

+
+ + +
+ + +# Bibtex +If this work is helpful for your research, please consider citing the following BibTeX entry. + +``` +@misc{zhang2025ICEdit, + title={In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformer}, + author={Zechuan Zhang and Ji Xie and Yu Lu and Zongxin Yang and Yi Yang}, + year={2025}, + eprint={2504.20690}, + archivePrefix={arXiv}, + primaryClass={cs.CV}, + url={https://arxiv.org/abs/2504.20690}, +} +``` \ No newline at end of file diff --git a/configuration.json b/configuration.json new file mode 100644 index 0000000..7f3861d --- /dev/null +++ b/configuration.json @@ -0,0 +1 @@ +{"framework": "pytorch", "task": "image-to-image", "allow_remote": true} \ No newline at end of file diff --git a/docs/images/gpt4o_comparison.png b/docs/images/gpt4o_comparison.png new file mode 100644 index 0000000..23f8a6b Binary files /dev/null and b/docs/images/gpt4o_comparison.png differ diff --git a/docs/images/gradio.png b/docs/images/gradio.png new file mode 100644 index 0000000..304dc3b Binary files /dev/null and b/docs/images/gradio.png differ diff --git a/docs/images/teaser.png b/docs/images/teaser.png new file mode 100644 index 0000000..ae276be Binary files /dev/null and b/docs/images/teaser.png differ