CLIP Image Search

Traditional image search needs labels or fine-tuning per task. CLIP maps images and text into a shared embedding space—can we build a semantic search engine over an entire validation split using only pre-trained weights, and how well does zero-shot classification work on fine-grained classes?

Tiny-ImageNet validation set: 10,000 images across 200 classes (3×224×224 after preprocessing). Class names loaded from words.txt (WordNet synset descriptions). Images resized to 224×224 and tensorized. No training data used—CLIP ViT-B/32 runs in eval mode throughout.

Extract L2-normalized image embeddings for all 10,000 validation images in batches of 64. Build text embeddings for each class with the prompt template “a photo of a {class_name}” (using the first word of the WordNet description). At search time, rank images by cosine similarity (dot product of normalized vectors) to a text or image query embedding. Zero-shot classification assigns each image to the class with highest image–text similarity.

No fine-tuning. One-time embedding extraction (~60 seconds for 10k images at ~2.6 it/s). Embeddings saved to tiny_imagenet_val_embeddings.pt for reuse. Top-K=5 for all retrieval queries.

• Loaded openai/clip-vit-base-patch32 via HuggingFace CLIPModel and CLIPProcessor
• Implemented batched extract_image_embeddings() with L2 normalization
• Built load_wnid_to_words() to map class IDs to human-readable labels
• Ran zero-shot classification via full image–text similarity matrix
• Built search_text_to_image() for natural-language queries (dog, sports car, pasta, hiking, container ship)
• Built search_image_to_image() using query images downloaded from Unsplash URLs

CLIP turns language and vision into shared geometry—search becomes nearest-neighbor in embedding space. Zero-shot accuracy is low on 200 fine-grained Tiny-ImageNet classes, but retrieval quality for broad natural-language queries is immediately useful. This reinforced how self-supervised multimodal pre-training enables flexible applications without task-specific training.