Fine-tuning CLIP models for peripheral blood cell images retrieval based on morphological descriptions

This master's thesis explores the fine-tuning of Contrastive Language-Image Pre-training (CLIP) models for retrieving peripheral blood cell images based on morphological descriptions, aiming to assist hematologists in diagnostic processes. The study addresses the limitations of manual peripheral blood smear analysis, which is time-consuming and prone to variability, by leveraging state-of-the-art AI techniques. The methodology involved preparing a dataset of lymphocyte images with textual descriptions of 12 morphological features. Pre-trained CLIP models (ViT-B/32, ViT-L/14, ViT-B/16) were fine-tuned using Cosine Similarity, Contrastive Loss, and Multiple Negatives Ranking Loss (MNRL). Performance was evaluated using a novel descriptor-specific recall metric. Initial zero-shot performance of the pre-trained CLIP ViT-B/32 was limited, with a Recall@10 of 0.080. Fine-tuning yielded substantial improvements. The MNRL function proved most effective, and the CLIP ViT-B/32 architecture offered a strong balance of performance and efficiency, achieving a Recall@5 increasing from 0.055 (baseline) to 0.356 and a Recall@10 of 0.660. Performance varied across descriptors, with features like 'Cytoplasmic Hairiness' being more reliably retrieved. A web application was developed to demonstrate these capabilities. The study concludes that fine-tuned CLIP models hold significant potential for specialized medical image retrieval. Future work includes exploring advanced data augmentation, refining captioning strategies, and clinical validation.

Grantor dc:publisher

Universitat Oberta de Catalunya (UOC)

Year dc:date.issued

2025

Author dc:creator

• Piazza Amat, Ivana

• vision language models
• blood cell morphology
• medical image retrieval

Statement dc:rights

• CC BY-NC-ND

Licence dc:rights.uri

• https://creativecommons.org/licenses/by-nc-nd/4.0/

Language dc:language.iso

eng

Handle dc:identifier.uri

https://hdl.handle.net/10609/153609

OAI identifier oai:identifier

oai:openaccess.uoc.edu:10609/153609