flickify/text_encoder_2

--- tags: - vision widget: - src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/cat-dog-music.png candidate_labels: playing music, playing sports example_title: Cat & Dog --- # Model Card: CLIP Disclaimer: The model card is taken and modified from the official CLIP repository, it can be found [here](https://github.com/openai/CLIP/blob/main/model-card.md). ## Model Details The CLIP model was developed by researchers at OpenAI to learn about what contributes to robustness in computer vision tasks. The model was also developed to test the ability of models to generalize to arbitrary image classification tasks in a zero-shot manner. It was not developed for general model deployment - to deploy models like CLIP, researchers will first need to carefully study their capabilities in relation to the specific context they’re being deployed within. ### Model Date January 2021 ### Model Type The base model uses a ViT-L/14 Transformer architecture as an image encoder and uses a masked self-attention Transformer as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. The original implementation had two variants: one using a ResNet image encoder and the other using a Vision Transformer. This repository has the variant with the Vision Transformer. ### Documents - [Blog Post](https://openai.com/blog/clip/) - [CLIP Paper](https://arxiv.org/abs/2103.00020) ### Use with Transformers ```python from PIL import Image import requests from transformers import CLIPProcessor, CLIPModel model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14") processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14") url = "http://images.cocodataset.org/val2017/000000039769.jpg" image = Image.open(requests.get(url, stream=True).raw) inputs = processor(text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True) outputs = model(**inputs) logits_per_image = outputs.logits_per_image # this is the image-text similarity score probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ``` ## Model Use ### Intended Use The model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models - the CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis. #### Primary intended uses The primary intended users of these models are AI researchers. We primarily imagine the model will be used by researchers to better understand robustness, generalization, and other capabilities, biases, and constraints of computer vision models. ### Out-of-Scope Use Cases **Any** deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful. Certain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use. Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases. ## Data The model was trained on publicly available image-caption data. This was done through a combination of crawling a handful of websites and using commonly-used pre-existing image datasets such as [YFCC100M](http://projects.dfki.uni-kl.de/yfcc100m/). A large portion of the data comes from our crawling of the internet. This means that the data is more representative of people and societies most connected to the internet which tend to skew towards more developed nations, and younger, male users. ### Data Mission Statement Our goal with building this dataset was to test out robustness and generalizability in computer vision tasks. As a result, the focus was on gathering large quantities of data from different publicly-available internet data sources. The data was gathered in a mostly non-interventionist manner. However, we only crawled websites that had policies against excessively violent and adult images and allowed us to filter out such content. We do not intend for this dataset to be used as the basis for any commercial or deployed model and will not be releasing the dataset. ## Performance and Limitations ### Performance We have evaluated the performance of CLIP on a wide range of benchmarks across a variety of computer vision datasets such as OCR to texture recognition to fine-grained classification. The paper describes model performance on the following datasets: - Food101 - CIFAR10 - CIFAR100 - Birdsnap - SUN397 - Stanford Cars - FGVC Aircraft - VOC2007 - DTD - Oxford-IIIT Pet dataset - Caltech101 - Flowers102 - MNIST - SVHN - IIIT5K - Hateful Memes - SST-2 - UCF101 - Kinetics700 - Country211 - CLEVR Counting - KITTI Distance - STL-10 - RareAct - Flickr30 - MSCOCO - ImageNet - ImageNet-A - ImageNet-R - ImageNet Sketch - ObjectNet (ImageNet Overlap) - Youtube-BB - ImageNet-Vid ## Limitations CLIP and our analysis of it have a number of limitations. CLIP currently struggles with respect to certain tasks such as fine grained classification and counting objects. CLIP also poses issues with regards to fairness and bias which we discuss in the paper and briefly in the next section. Additionally, our approach to testing CLIP also has an important limitation- in many cases we have used linear probes to evaluate the performance of CLIP and there is evidence suggesting that linear probes can underestimate model performance. ### Bias and Fairness We find that the performance of CLIP - and the specific biases it exhibits - can depend significantly on class design and the choices one makes for categories to include and exclude. We tested the risk of certain kinds of denigration with CLIP by classifying images of people from [Fairface](https://arxiv.org/abs/1908.04913) into crime-related and non-human animal categories. We found significant disparities with respect to race and gender. Additionally, we found that these disparities could shift based on how the classes were constructed. (Details captured in the Broader Impacts Section in the paper). We also tested the performance of CLIP on gender, race and age classification using the Fairface dataset (We default to using race categories as they are constructed in the Fairface dataset.) in order to assess quality of performance across different demographics. We found accuracy >96% across all races for gender classification with ‘Middle Eastern’ having the highest accuracy (98.4%) and ‘White’ having the lowest (96.5%). Additionally, CLIP averaged ~93% for racial classification and ~63% for age classification. Our use of evaluations to test for gender, race and age classification as well as denigration harms is simply to evaluate performance of the model across people and surface potential risks and not to demonstrate an endorsement/enthusiasm for such tasks. ## Feedback ### Where to send questions or comments about the model Please use [this Google Form](https://forms.gle/Uv7afRH5dvY34ZEs9)--- tags: - 视觉 widget: - src: https://huggingface.co/datasets/mishig/sample_images/resolve/main/cat-dog-music.png candidate_labels: 演奏音乐，进行体育运动 example_title: 猫咪与狗狗 --- # 模型卡片：CLIP 免责声明：本模型卡片改编自官方CLIP仓库，原始版本可查阅[此处](https://github.com/openai/CLIP/blob/main/model-card.md)。 ## 模型详情 CLIP模型由OpenAI的研究人员开发，旨在探究提升计算机视觉任务鲁棒性的关键因素，同时测试模型以**零样本（zero-shot）**方式泛化至任意图像分类任务的能力。本模型并非为通用模型部署而开发——若要部署类似CLIP的模型，研究人员需首先结合其具体部署场景，对模型的各项能力开展严谨研究。 ### 模型发布日期 2021年1月 ### 模型类型 基础模型采用ViT-L/14 Transformer（Transformer）架构作为图像编码器，并使用带掩码的自注意力Transformer（Transformer）作为文本编码器。上述编码器通过对比损失函数训练，以最大化（图像，文本）配对的相似度。 原始实现包含两种变体：一种使用ResNet作为图像编码器，另一种使用视觉Transformer（Vision Transformer）。本仓库仅包含视觉Transformer变体。 ### 相关文档 - [博客文章](https://openai.com/blog/clip/) - [CLIP研究论文](https://arxiv.org/abs/2103.00020) ### 结合Transformers库使用 python from PIL import Image import requests from transformers import CLIPProcessor, CLIPModel model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14") processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14") url = "http://images.cocodataset.org/val2017/000000039769.jpg" image = Image.open(requests.get(url, stream=True).raw) inputs = processor(text=["一张猫咪的照片", "一张狗狗的照片"], images=image, return_tensors="pt", padding=True) outputs = model(**inputs) logits_per_image = outputs.logits_per_image # 该值为图像-文本相似度得分 probs = logits_per_image.softmax(dim=1) # 可通过softmax运算得到标签概率 ## 模型用途 ### 预期用途 本模型旨在作为面向研究社区的研究成果。我们期望该模型能够帮助研究人员更好地理解和探索零样本、任意图像分类任务。同时，我们也希望该模型可用于跨学科研究，分析此类模型的潜在影响——CLIP研究论文中已讨论了潜在的下游应用场景，可为此类分析提供参考范例。 #### 核心预期用途 本模型的核心目标用户为AI研究人员。 我们预期研究人员将使用本模型，进一步理解计算机视觉模型的鲁棒性、泛化能力及其他特性、偏差与约束条件。 ### 超出适用范围的使用场景 **任何**已部署的模型使用场景（无论商用与否）目前均不属于适用范围。非部署的使用场景，如受限环境下的图像搜索，同样不被推荐，除非已针对特定固定类别分类体系对模型开展了全面的领域内测试。这是因为我们的安全评估结果显示，亟需开展任务针对性测试，尤其考虑到CLIP在不同类别分类体系下的性能差异较大。这意味着，目前在任何未经过测试、无约束的场景中部署该模型，都可能带来潜在危害。 属于监控与人脸识别领域的特定使用场景，无论模型性能如何，均始终不属于适用范围。这是因为目前缺乏确保公平使用的测试规范与校验机制，此类AI任务的应用仍为时过早。 由于本模型未针对英语以外的其他语言进行专门训练或评估，其使用应仅限英语场景。 ## 训练数据 本模型基于公开可用的图像-字幕数据进行训练。训练数据通过两种方式收集：一是爬取少量网站的公开数据，二是使用现有常用图像数据集，例如[YFCC100M](http://projects.dfki.uni-kl.de/yfcc100m/)。其中绝大部分数据来自我们的互联网爬取工作，这意味着该数据更能代表与互联网连接度较高的人群与社会群体，这类群体往往偏向于发达国家、年轻男性用户。 ### 数据构建宗旨 我们构建该数据集的初衷是测试计算机视觉任务中的鲁棒性与泛化能力。因此，我们的重点是从各类公开互联网数据源中收集大规模数据，且采集过程基本保持非干预性。不过，我们仅爬取那些明确禁止发布过度暴力与成人内容，并允许我们过滤此类内容的网站。我们不希望本数据集被用作任何商用或部署模型的基础，且不会公开该数据集。 ## 性能与局限性 ### 性能评估 我们已在涵盖多种计算机视觉数据集的大量基准测试中评估了CLIP的性能，测试范围涵盖光学字符识别、纹理识别至细粒度分类等任务。研究论文中详细描述了模型在以下数据集上的表现： - Food101 - CIFAR10 - CIFAR100 - Birdsnap - SUN397 - 斯坦福汽车数据集（Stanford Cars） - FGVC Aircraft - VOC2007 - DTD - Oxford-IIIT 宠物数据集（Oxford-IIIT Pet dataset） - Caltech101 - Flowers102 - MNIST - SVHN - IIIT5K - 仇恨表情包数据集（Hateful Memes） - SST-2 - UCF101 - Kinetics700 - Country211 - CLEVR计数任务（CLEVR Counting） - KITTI 距离估计（KITTI Distance） - STL-10 - RareAct - Flickr30 - MSCOCO - ImageNet - ImageNet-A - ImageNet-R - ImageNet Sketch - ObjectNet（ImageNet重叠数据集） - Youtube-BB - ImageNet-Vid ### 局限性 CLIP模型及我们对其的分析存在若干局限性。目前，CLIP在细粒度分类、物体计数等特定任务上表现欠佳。此外，CLIP还存在公平性与偏差相关的问题，我们已在研究论文中对此展开讨论，并将在下一节简要说明。另外，我们对CLIP的测试方法也存在重要局限：在多数场景下，我们使用线性探针（linear probe）评估模型性能，而已有研究表明线性探针可能会低估模型的实际性能。 ### 偏差与公平性 我们发现，CLIP的性能及其所表现出的特定偏差，在很大程度上取决于类别设计以及我们对包含与排除类别的选择。我们通过将[Fairface](https://arxiv.org/abs/1908.04913)数据集里的人物图像分类为与犯罪相关类别及非人类动物类别，测试了CLIP存在的某些贬低性风险。结果显示，在种族与性别维度上存在显著差异。此外，我们发现这些差异会随类别构建方式的不同而发生变化（详细内容可见论文中的「更广泛影响」章节）。 我们还使用Fairface数据集对CLIP在性别、种族与年龄分类任务上的性能进行了测试（我们默认采用Fairface数据集中定义的种族类别），以评估模型在不同人口群体中的表现。结果显示，在性别分类任务中，所有种族的准确率均超过96%，其中「中东」种族准确率最高（98.4%），「白人」种族准确率最低（96.5%）。此外，CLIP在种族分类任务中的平均准确率约为93%，在年龄分类任务中的平均准确率约为63%。我们开展性别、种族、年龄分类及贬低性风险测试，仅为评估模型在人群中的表现并揭示潜在风险，而非为了表明对这类任务的支持或热衷。 ## 反馈与咨询 ### 模型相关问题或意见反馈渠道 请使用[此谷歌表单](https://forms.gle/Uv7afRH5dvY34ZEs9)提交反馈。