From 8e4fc7f0e90e1218fabaa5f1a7aeed578526cfa7 Mon Sep 17 00:00:00 2001 From: lxlxlxlxlxlx Date: Mon, 16 Jun 2025 08:13:46 +0000 Subject: [PATCH] Update README.md --- README.md | 202 ++++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 165 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 810ab93..38a921d 100644 --- a/README.md +++ b/README.md @@ -1,47 +1,175 @@ ---- -license: Apache License 2.0 + +

+ +

-#model-type: -##如 gpt、phi、llama、chatglm、baichuan 等 -#- gpt +

+ project page + arxiv + demo + model + model + model +

-#domain: -##如 nlp、cv、audio、multi-modal -#- nlp +

+

+ News | + Quick Start | + Usage Tips | + Online Demos | + Citation | + License +

+

-#language: -##语言代码列表 https://help.aliyun.com/document_detail/215387.html?spm=a2c4g.11186623.0.0.9f8d7467kni6Aa -#- cn +**OmniGen2** is a powerful and efficient unified multimodal model. Its architecture is composed of two key components: a 3B Vision-Language Model (VLM) and a 4B diffusion model. In this design, the frozen 3B VLM ([Qwen-VL-2.5](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct)) is responsible for interpreting both visual signals and user instructions, while the 4B diffusion model leverages this understanding to perform high-quality image generation. -#metrics: -##如 CIDEr、Blue、ROUGE 等 -#- CIDEr +This dual-component architecture enables strong performance across four primary capabilities: -#tags: -##各种自定义，包括 pretrained、fine-tuned、instruction-tuned、RL-tuned 等训练方法和其他 -#- pretrained +- **Visual Understanding**: Inherits the robust ability to interpret and analyze image content from its Qwen-VL-2.5 foundation. +- **Text-to-Image Generation**: Creates high-fidelity and aesthetically pleasing images from textual prompts. +- **Instruction-guided Image Editing**: Executes complex, instruction-based image modifications with high precision, achieving state-of-the-art performance among open-source models. +- **In-context Generation**: A versatile capability to process and flexibly combine diverse inputs—including humans, reference objects, and scenes—to produce novel and coherent visual outputs. -#tools: -##如 vllm、fastchat、llamacpp、AdaSeq 等 -#- vllm ---- -### 当前模型的贡献者未提供更加详细的模型介绍。模型文件和权重，可浏览“模型文件”页面获取。 -#### 您可以通过如下git clone命令，或者ModelScope SDK来下载模型 +As an open-source project, OmniGen2 provides a powerful yet resource-efficient foundation for researchers and developers exploring the frontiers of controllable and personalized generative AI. + +**We will release the training code, dataset, and data construction pipeline soon. Stay tuned!** + +

+ +
+ Demonstration of OmniGen2's overall capabilities. +

+ +

+ +
+ Demonstration of OmniGen2's image editing capabilities. +

+ +

+ +
+ Demonstration of OmniGen2's in-context generation capabilities. +

+ +## 🔥 News +- **2025-06-16**: [Gradio](https://github.com/VectorSpaceLab/OmniGen2?tab=readme-ov-file#-gradio-demo) and [Jupyter](https://github.com/VectorSpaceLab/OmniGen2/blob/main/example.ipynb) demo is available. +- **2025-06-16**: We release **OmniGen2**, a multimodal generation model, model weights can be accessed in [huggingface](https://huggingface.co/OmniGen2/OmniGen2). + +## 📌 TODO +- [ ] Technical report. +- [ ] In-context generation benchmark: **OmniContext**. +- [ ] Support CPU offload and improve inference efficiency. +- [ ] Training data and scripts. +- [ ] Data construction pipeline. +- [ ] ComfyUI Demo (**commuity support will be greatly appreciated!**). + +## 🚀 Quick Start + +### 🛠️ Environment Setup + +#### ✅ Recommended Setup -SDK下载 ```bash -#安装ModelScope -pip install modelscope -``` -```python -#SDK模型下载 -from modelscope import snapshot_download -model_dir = snapshot_download('OmniGen2/OmniGen2') -``` -Git下载 -``` -#Git模型下载 -git clone https://www.modelscope.cn/OmniGen2/OmniGen2.git +# 1. Clone the repo +git clone git@github.com:VectorSpaceLab/OmniGen2.git +cd OmniGen2 + +# 2. (Optional) Create a clean Python environment +conda create -n omnigen2 python=3.11 +conda activate omnigen2 + +# 3. Install dependencies +# 3.1 Install PyTorch (choose correct CUDA version) +pip install torch==2.6.0 torchvision --extra-index-url https://download.pytorch.org/whl/cu124 + +# 3.2 Install other required packages +pip install -r requirements.txt +pip install flash-attn --no-build-isolation ``` -

如果您是本模型的贡献者，我们邀请您根据模型贡献文档，及时完善模型卡片内容。

\ No newline at end of file +#### 🌏 For users in Mainland China + +```bash +# Install PyTorch from a domestic mirror +pip install torch==2.6.0 torchvision --index-url https://mirror.sjtu.edu.cn/pytorch-wheels/cu124 + +# Install other dependencies from Tsinghua mirror +pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple +pip install flash-attn --no-build-isolation -i https://pypi.tuna.tsinghua.edu.cn/simple +``` + +--- + +### 🧪 Run Examples + +```bash +# Visual Understanding +bash example_understanding.sh + +# Text-to-image generation +bash example_t2i.sh + +# Instruction-guided image editing +bash example_edit.sh + +# Subject-driven image editing +bash example_subject_driven_edit.sh +``` + +--- + +### 🌐 Gradio Demo + +* **Online Demo**: +We are temporarily providing 8 GPUs to support the online demos. If you notice a long queue for a particular link, please try other links: + + [Demo1](https://be5916033313307354.gradio.live), [Demo2](https://281efc44b736406f42.gradio.live), [Demo3](https://a27912fbaef54294f8.gradio.live), [Demo4](https://bbf305e391bc769d22.gradio.live) + + [Chat-Demo1](https://a79e0445bb498554e8.gradio.live), [Chat-Demo2](https://7f922fdca66e47c427.gradio.live), [Chat-Demo3](https://6568f4b2a8353be3ae.gradio.live), [Chat-Demo4](https://f38c30ed99f0f6caab.gradio.live) + + + +* **Run Locally**: + ```bash + pip install gradio + python app.py + # Optional: Share demo with public link (You need to be able to access huggingface) + python app.py --share + ``` + +## 💡 Usage Tips +To achieve optimal results with OmniGen2, you can adjust the following key hyperparameters based on your specific use case. +- `num_inference_step`: The number of sampling steps per generation. Higher values generally improve quality but increase generation time. + - Recommended Range: 28 to 50 +- `text_guidance_scale`: Controls how strictly the output adheres to the text prompt (Classifier-Free Guidance). + - **For Text-to-Image**: Use a higher value (e.g., 6-7) for simple or less detailed prompts. Use a lower value (e.g., 4) for complex and highly detailed prompts. + - **For Editing/Composition**: A moderate value around 4-5 is recommended. +- `image_guidance_scale`: This controls how much the final image should resemble the input reference image. + - **The Trade-off**: A higher value (~2.0) makes the output more faithful to the reference image's structure and style, but it might ignore parts of your text prompt. A lower value (~1.5) gives the text prompt more influence. + - **Tip**: Start with 1.5 and increase it if you need more consistency with the reference image. For image editing task, we recommend to set it between 1.3 and 2.0; for in-context generateion task, a higher image_guidance_scale will maintian more details in input images, and we recommend to set it between 2.5 and 3.0. +- `max_pixels`: Automatically resizes images when their total pixel count (width × height) exceeds this limit, while maintaining its aspect ratio. This helps manage performance and memory usage. +- `max_input_image_side_length`: Maximum side length for input images. +- `negative_prompt`: Tell the model what you don't want to see in the image. + - **Example**: blurry, low quality, text, watermark + - **Tip**: For the best results, try experimenting with different negative prompts. If you're not sure, just leave it blank. + + + +## ❤️ Citing Us +If you find this repository or our work useful, please consider giving a star :star: and citation :t-rex:, which would be greatly appreciated (OmniGen2 report will be available as soon as possible): + +```bibtex +@article{xiao2024omnigen, + title={Omnigen: Unified image generation}, + author={Xiao, Shitao and Wang, Yueze and Zhou, Junjie and Yuan, Huaying and Xing, Xingrun and Yan, Ruiran and Wang, Shuting and Huang, Tiejun and Liu, Zheng}, + journal={arXiv preprint arXiv:2409.11340}, + year={2024} +} +``` + +## License +This work is licensed under Apache 2.0 license.