CNNFORMER interpolation: a demosaicing framework for single-chip color–NIR1 imaging in fluorescence-guided cancer surgery

Single-chip multispectral imaging sensors that integrate vertically stacked photodiodes with pixelated spectral filters offer significant promise for advanced, real-time visualization during image-guided cancer surgery. However, the inherent design trade-offs of these sensors lead to a substantial reduction in spatial resolution, which can limit their clinical utility. Conventional demosaicing algorithms, such as bilinear interpolation and convolutional neural network (CNN)-based approaches, have been developed to mitigate these limitations by improving the effective resolution and reducing common image artifacts. While these methods have achieved meaningful improvements, they still exhibit deficiencies in accuracy and often introduce smaller, yet clinically relevant, reconstruction artifacts, particularly in regions with high spatial frequency content. To address these persistent challenges, this work presents a CNN–Transformer-based demosaicing algorithm specifically optimized for reconstructing high-resolution color and near-infrared (NIR) images acquired by a hexachromatic multispectral imaging sensor. Transformer architectures, originally developed for natural language processing to capture long-range dependencies within sequential data, have recently shown remarkable success when adapted to vision tasks due to their ability to model complex, global relationships across an image. By integrating convolutional layers for local feature extraction with Transformer modules for capturing long-range context, the proposed hybrid model was developed and trained on extensive color-image datasets and rigorously evaluated on both color and NIR imagery. Quantitative evaluation demonstrates that the CNN–Transformer demosaicing method achieves an average mean squared error (MSE) reduction of approximately 85% for color images and 76% for NIR images, along with improvements in structural dissimilarity (DSSIM) of roughly 72% and 79%, respectively, when compared to state-of-the-art CNN-based demosaicing algorithms on preclinical datasets. In clinical datasets, the model similarly achieves significant reductions in MSE and DSSIM, particularly excelling at reconstructing fine image details critical for accurate intraoperative visualization. These results underscore the capability of Transformer-based models to surpass the limitations of traditional CNN-only approaches by capturing more comprehensive spatial information. Leveraging recent advances in GPU computing, the proposed CNN–Transformer demosaicing framework offers a practical, real-time solution for enhancing spatial resolution and image fidelity in multispectral imaging applications, ultimately advancing the effectiveness of image-guided cancer surgery.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Yang, Jiankun

Contributors dc:contributor

• Gruev, Viktor

• Machine Learning
• In-vivo Imaging
• Computer Vision

Statement dc:rights

• Copyright 2025 Jiankun Yang

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/129754

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F129754