[paper] [project page]
Official implementation of the paper "Deep ViT Features as Dense Visual Descriptors".
We demonstrate the effectiveness of deep features extracted from a self-supervised, pre-trained ViT model (DINO-ViT) as dense patch descriptors via real-world vision tasks: (a-b) co-segmentation & part co-segmentation: given a set of input images (e.g., 4 input images), we automatically co-segment semantically common foreground objects (e.g., animals), and then further partition them into common parts; (c-d) point correspondence: given a pair of input images, we automatically extract a sparse set of corresponding points. We tackle these tasks by applying only lightweight, simple methodologies such as clustering or binning, to deep ViT features.
Our code is developed in pytorch on and requires the following modules: tqdm, faiss, timm, matplotlib, pydensecrf, opencv, scikit-learn. We use python=3.9 but our code should be runnable on any version above 3.6. We recomment running our code with any CUDA supported GPU for faster performance. We recommend setting the running environment via Anaconda by running the following commands:
Otherwise, run the following commands in your conda environment:
We provide a wrapper class for a ViT model to extract dense visual descriptors in extractor.py. You can extract descriptors to .pt files using the following command:
You can specify the pretrained model using the -model flag with the following options:
- dino_vits8, dino_vits16, dino_vitb8, dino_vitb16 from the DINO repo.
- vit_small_patch8_224, vit_small_patch16_224, vit_base_patch8_224, vit_base_patch16_224 from the timm repo.
You can specify the stride of patch extracting layer to increase resolution using the -stride flag.
We provide a notebook for running on a single example in part_cosegmentation.ipynb.
To run on several image sets, arrange each set in a directory, inside a data root directory:
The following command will produce results in the specified <save_root_name>:
Note: The default configuration in part_cosegmentation.ipynb is suited for running on small sets (e.g. < 10). Increase amount of num_crop_augmentations for more stable results (and increased runtime). The default configuration in part_cosegmentation.py is suited for larger sets (e.g. >> 10).
We provide a notebook for running on a single example in cosegmentation.ipynb.
To run on several image sets, arrange each set in a directory, inside a data root directory:
The following command will produce results in the specified <save_root_name>:
We provide a notebook for running on a single example in correpondences.ipynb.
To run on several image pairs, arrange each image pair in a directory, inside a data root directory:
The following command will produce results in the specified <save_root_name>:
We provide code for computing the PCA of several images in a single directory:
We provide code for interactively visualizing the similarity of a chosen descriptor in the source image to all target descriptors in a terget image.
If you found this repository useful please consider starring ⭐ and citing :
