Tutorial Introduction

This tutorial demo can classify a given biological image by family, genus, species, etc. It is the best student paper of CVPR2024.BioCLIP: A Vision Foundation Model for the Tree of Life"The Gradio version Demo of ".

BioCLIP Research Background

Compared with general tasks, the label space of biological computer vision is richer. Not only is the number of classification annotations huge, but the annotations are also interconnected in the hierarchical classification system. This undoubtedly brings huge challenges to training basic models with high species coverage and strong generalization capabilities.

With the experience accumulated from hundreds of years of biological research, researchers believe that if the basic model can successfully encode the structure of the annotation space, then even if a specific species has not been seen, the model may be able to identify its corresponding genus or family and give a corresponding representation, and this hierarchical representation will help achieve few-shot or even zero-shot learning of new taxa. Based on this, the researchers chose CLIP, a multimodal model architecture developed by OpenAI, and used CLIP's multimodal comparative learning objective to continuously pre-train on TREEOFLIFE-10M.

Effect Preview

Run steps

1. After cloning the tutorial and starting it, directly copy the API address and paste it into any URL (real-name authentication must have been completed, and there is no need to open the workspace for this step)

2. Enter the Gradio interface and upload the image to be identified.

This demo provides two modes: "open-ended" and "zero-shot".

• The "open-ended" mode provides seven classification levels: kingdom, phylum, class, order, family, genus, and species. Users can upload images and select the level to be classified to perform classification tasks. The more detailed the level of classification, the more difficult the classification will be.
• The "zero-shot" mode allows users to provide the categories to be classified. After uploading the picture, the model can give the categories to which the picture belongs.

Open-Ended

Select the level you want to classify and click the "Submit" button to generate the classification results.

Zero-Shot

Enter several possible categories to be classified and click the "submit" button to generate the classification results.

Discussion and Exchange

🖌️ If you see a high-quality project, please leave a message in the background to recommend it! In addition, we have also established a tutorial exchange group. Welcome friends to scan the QR code and remark [Tutorial Exchange] to join the group to discuss various technical issues and share application effects↓