In-context concept learning using diffusion models

This thesis addresses the challenge of learning contextual visual concepts from limited image sets. We leverage pre-trained text-to-image diffusion models and introduce a novel optimization framework targeting token embeddings and linear projection parameters in the model's cross-attention layers. Our approach is guided by three primary objectives: perceiving the concept within its context, ensuring focused attention on the concept, and accurately capturing the concept's details. Additionally, we develop a controllable image-editing workflow that allows for precise control over the strength and location of the transferred concepts in new images. Comprehensive qualitative and quantitative experiments demonstrate the effectiveness of our method in learning and transferring contextual visual concepts, significantly outperforming existing techniques.