Non-equilibrium physics: from spin glasses to machine and neural learning

Disordered many-body systems exhibit a wide range of emergent phenomena across different scales. These complex behaviors can be utilized for various information processing tasks such as error correction, learning, and optimization. Despite the empirical success of utilizing these systems for intelligent tasks, the underlying principles that govern their emergent intelligent behaviors remain largely unknown. In this thesis, we aim to characterize such emergent intelligence in disordered systems through statistical physics. We chart a roadmap for our efforts in this thesis based on two axes: learning mechanisms (long-term memory vs. working memory) and learning dynamics (artificial vs. natural). We begin our exploration from the long-term memory and artificial dynamics continent of this atlas, where we examine the structure-function relationships in feedforward neural networks, the prototypical example of neural learning. Using replica theory, information theory, and optimal transport, we study the computational consequences of imposing connectivity constraints on the network, such as distribution constraints, sign constraints, and disentangling constraints. We evaluate the performances based on metrics such as capacity, generalization, and generative ability. Next, we explore the working memory and artificial dynamics corner of the atlas and investigate the non-equilibrium driven dynamics of recurrent neural networks under external inputs. Then, we move to the working memory and natural dynamics island and study the ability of driven spin-glasses to perform discriminative tasks such as novelty detection and classification. Finally, we conclude our exploration at the long-term memory and natural dynamics kingdom and investigate the generative modeling ability in many-body localized systems. Throughout our journey, we uncover relationships between learning mechanisms and physical dynamics that could serve as guiding principles for designing intelligent systems. We hope that our investigation into the emergent intelligence of seemingly disparate learning systems can expand our current understanding of intelligence beyond neural systems and uncover a wider range of computational substrates suitable for AI applications.

Name thesis:degree_name

Doctoral

Department dc:contributor.department

Massachusetts Institute of Technology. Department of Physics

Grantor dc:publisher

Massachusetts Institute of Technology

Year dc:date.issued

2023

Author dc:creator

• Zhong, Weishun

Advisors dc:contributor.advisor

• Sompolinsky, Haim
• Kardar, Mehran

Statement dc:rights

• In Copyright - Educational Use Permitted
• Copyright retained by author(s)

Licence dc:rights.uri

• https://rightsstatements.org/page/InC-EDU/1.0/

Handle dc:identifier.uri

https://hdl.handle.net/1721.1/152568

OAI identifier oai:identifier

oai:dspace.mit.edu:1721.1/152568