Towards a foundation model for multi-modal and hyperspectral geospatial data

Geospatial imagery data, such as that collected by different satellite-based sensing systems at different times, holds immense potential for enabling a wide range of high-impact applications. Such potential comes from the rich and contextualized information provided by geospatial imagery across multiple dimensions, channels, and sensing modalities. To unlock the insights from geospatial data, recent work has adapted existing self-supervised learning (SSL) approaches; however, they fall short of tailored training objects and model architectures, leading to inflexibility and computational inefficiencies especially when facing an increasing number of channels and modalities. In light of existing limitations, we introduce a novel framework consisting of three key components: i) a Multi-Modal Masked Autoencoder (MM-MAE) that fuses features from different modalities; ii) a Masked-Channel Reconstruction objective that exploits interchannel relationships in hyperspectral data; and iii) a Spatial-Spectral Vision Transformer (S2ViT), incorporating novel Low-Rank Spatial-Spectral Attention Blocks, which flexibly assigns attention to different dimensions. Experimental results demonstrate that our proposed method surpasses current state-of-the-art multi-modal geospatial foundation models, achieving superior performance with less computation and fewer parameters. The flexibility and extensibility of our framework make it a promising solution for future geospatial data analysis tasks that involve a wide range of modalities and dimensions. Consequently, our pretrained model can be effectively applied to various downstream tasks, such as land-cover classification, land functionality management, and marine debris detection, eventually supporting informed decision-making for sustainable development and environmental conservation.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2024

Author dc:creator

• Si, Haozhe

Contributors dc:contributor

• Zhao, Han

• Computer Vision
• Machine Learning

Statement dc:rights

• Copyright 2024 Haozhe Si

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/125729

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F125729