Visual Reasoning On Winoground

Metrics

Group Score

Image Score

Text Score

Results

Performance results of various models on this benchmark

Model Name	Group Score	Image Score	Text Score	Paper Title	Repository
ViLBERT base	4.75	7.25	23.75	Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
METER (finetuned, Flickr30k)	14.75	20.75	43.5	Equivariant Similarity for Vision-Language Foundation Models
BLIP (ITM)	13.3	15.8	35.8	Revisiting the Role of Language Priors in Vision-Language Models
BLIP2 (SGVL)	23.3	28.5	42.8	Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs	-
Gemini + CoCoT	27.75	32.5	40	CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs
GPT-4V (CoT, pick b/w two options)	58.75	68.75	75.25	The Role of Chain-of-Thought in Complex Vision-Language Reasoning Task	-
OpenFlamingo + CoCoT	41.5	55.25	58.25	CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs
COCA ViT-L14 (f.t on COCO)	8.25	11.50	28.25	What You See is What You Read? Improving Text-Image Alignment Evaluation
OFA large (ITM)	7.25	10.25	30.75	Simple Token-Level Confidence Improves Caption Correctness	-
VSE++ (COCO, VGG)	3.50	5.50	18.75	Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality
METER	12.00	15.75	39.25	Equivariant Similarity for Vision-Language Foundation Models
OpenFlamingo	33.25	41.25	39	CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs
LDM-CLIP (SelfEval)	-	7.25	22.75	SelfEval: Leveraging the discriminative nature of generative models for evaluation	-
CLIP (ViT-L/14)	-	8.0	30.25	SelfEval: Leveraging the discriminative nature of generative models for evaluation	-
BLIP 129M (CapFilt/L)	12.2	15.2	34.7	Measuring Progress in Fine-grained Vision-and-Language Understanding
KeyComp* (GPT-4)	18.2	28.7	43.5	Prompting Large Vision-Language Models for Compositional Reasoning
LLaVA-7B (GPTScore)	10.50	17.00	25.50	An Examination of the Compositionality of Large Generative Vision-Language Models
TIFA	11.30	12.50	19.00	What You See is What You Read? Improving Text-Image Alignment Evaluation
Diffusion Classifier (zero-shot)	-	-	34.00	Your Diffusion Model is Secretly a Zero-Shot Classifier
KeyComp* (GPT-3.5)	17.4	27.8	42.7	Prompting Large Vision-Language Models for Compositional Reasoning

0 of 113 row(s) selected.