FashionCLIP is a specialized CLIP-like model fine-tuned for the fashion domain, offering enhanced zero-shot performance on fashion-related tasks like retrieval and classification. It targets researchers and practitioners in the fashion industry seeking more accurate and domain-specific multimodal understanding.

How It Works

FashionCLIP builds upon the CLIP architecture, fine-tuning it on a large dataset of over 700K fashion image-text pairs from the Farfetch dataset. This fine-tuning process adapts the model to better capture domain-specific fashion concepts, leading to improved generalization and performance in zero-shot scenarios compared to general-purpose CLIP models. The latest version, FashionCLIP 2.0, leverages the laion/CLIP-ViT-B-32-laion2B-s34B-b79K checkpoint, further boosting performance due to its larger training data.

Quick Start & Requirements

• Install via pip: $ pip install fashion-clip
• For local development: $ pip install -e . from the project root.
• Requires Python and standard ML libraries. Specific version requirements are not detailed but implied by Hugging Face Transformers usage.
• Colab notebook available for more functionalities.
• Hugging Face model available at patrickjohncyh/fashion-clip.

Highlighted Details

• FashionCLIP 2.0 achieves a weighted macro F1 score of 0.83 on FMNIST, 0.71 on KAGL, and 0.58 on DEEP datasets, outperforming OpenAI CLIP and Laion CLIP.
• The project is associated with a publication in Nature Scientific Reports.
• Provides an API for feature extraction, classification, and retrieval, with FCLIPDataset and FashionCLIP classes for data handling and model interaction.
• Supports local and S3 image sources and private Hugging Face repositories.

Maintenance & Community

• The project is associated with Patrick John Chia and Giuseppe Attanasio.
• Model weights are available on Hugging Face.
• A related project, RustEmbed, integrates FashionCLIP via gRPC.

Licensing & Compatibility

• The README does not explicitly state a license. The code is released as open-source. Compatibility for commercial use or closed-source linking is not specified.

Limitations & Caveats

• The official Farfetch dataset, used for training, is pending release, meaning pre-processed data and model weights might not be fully available or reproducible without it.