MACON: memory-augmented continual learning for open-world classification

Emerging concepts and rapidly changing environments necessitate AI models that can quickly adapt to new scenarios without losing their inherited capabilities. Large foundation models like CLIP offer a strong zero-shot learning baseline under an open-vocabulary classification scenario. However, their massive training data makes re-training or fine-tuning impractical without sacrificing zero-shot performance. We introduce Memory-Augmented CONtinual learning (MACON), a novel framework for addressing open-world continual learning challenges. The core idea is to augment foundation models, such as CLIP, with memory to provide context and flexibility for better decision-making and prevention of catastrophic forgetting. We propose various memory retrieval methods tailored to different continual learning scenarios. Our results demonstrate that MACON exhibits fast adaptation capabilities, minimal forgetting issues, and robust generalization abilities, making it suitable for a wide range of open-world applications.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Computer Science

Grantor

University of Illinois at Urbana-Champaign

Year dc:date

2023

Author dc:creator

• Lyu, Weijie

Contributors dc:contributor

• Hoiem, Derek

• Continual Learning
• Open-world Recognition
• Open-vocabulary Classification
• Memory-augmented Neural Network

Statement dc:rights

• Copyright 2023 Weijie Lyu

Language dc:language

en, eng

Handle dc:identifier

https://hdl.handle.net/2142/120293

OAI record oai:identifier

https://www.ideals.illinois.edu/oai-pmh?verb=GetRecord&metadataPrefix=etdms&identifier=oai%3Awww.ideals.illinois.edu%3A2142%2F120293