InstructMoLE: Enhancing Diffusion Transformers with Precision

In a notable advancement for generative AI, researchers have introduced InstructMoLE, a framework that fine-tunes diffusion transformers using an Instruction-Guided Mixture of Low-Rank Experts. This approach addresses task interference and enhances compositional control in multi-conditional tasks.

Why This Matters

Fine-tuning generative models like diffusion transformers is essential for precise and varied outputs, particularly in complex tasks requiring multiple conditions. Traditional methods, such as LoRA, often struggle with task interference, leading to inaccuracies. InstructMoLE tackles these challenges by introducing a global routing signal to streamline the process.

The innovation lies in Instruction-Guided Routing (IGR), a global signal that applies a coherent expert council uniformly across all input tokens. This maintains global semantics and structural integrity, improving upon previous token-level routing methods.

Key Details

• Global Routing Signal: InstructMoLE reduces artifacts like spatial fragmentation and semantic drift, common in complex image generation tasks.
• Output-Space Orthogonality Loss: This feature promotes expert functional diversity and prevents representational collapse, ensuring a broad range of capabilities.
• Performance: Experiments show that InstructMoLE significantly outperforms existing LoRA adapters and other MoLE variants, setting new benchmarks in multi-conditional generation tasks.

The Team Behind the Innovation

This breakthrough is credited to Jinqi Xiao, Qing Yan, Liming Jiang, Zichuan Liu, Hao Kang, Shen Sang, Tiancheng Zhi, Jing Liu, Cheng Yang, Xin Lu, and Bo Yuan. Their work presents a robust framework for instruction-driven fine-tuning, expanding the potential of generative models.

Key Implications

• Enhanced Task Control: InstructMoLE minimizes task interference, improving model accuracy and reliability.
• Global Routing: Aligns model processing with user instructions, preserving semantic integrity.
• Expert Diversity: Output-space orthogonality loss ensures diverse expert functions, preventing collapse.
• Superior Performance: Outperforms existing methods like LoRA, setting new standards for multi-conditional tasks.

Recommended Category

Research