PFHE: partially homomorphic encryption on CNN inference

Fully Homomorphic Encryption (FHE) enables secure computation on encrypted data, but its high computational overhead poses significant challenges for practical deep learning inference. In many application scenarios, high-resolution images may only contain a small portion of sensitive information. We noticed that none of the previous works consider this, so in this work, we propose a new framework that accelerates encrypted Convolutional Neural Network (CNN) inference by only encrypting the privacy-sensitive regions of input while processing the remaining parts in plaintext. Our method significantly reduces computational cost without compromising data confidentiality. We developed a new data layout for ciphertexts to utilize the sparse nature of the data. Besides that, we first leverage the capability of FHE to scheme switch between different schemes, such as Cheon-Kim-Kim-Song (CKKS) and Fast Homomorphic Encryption over the Torus (FHEW), achieving the implementation of nonlinear activation like ReLU in CNN with high precision, without the consumption of a significant amount of multiplications. We evaluate our framework under the ImageNet dataset. For the first six convolutional layers, our method achieves at least 4.12× speedup in latency and 28.72× less memory usage compared to the traditional Channel-wise convolution method under various settings. We then present a case study for a hybrid solution, combining our partially FHE encrypted convolution method with Channel-wise convolution, which also shows a theoretical latency reduction of 10.30× and 33.30× less memory consumption.

Name thesis:degree_name

M.S.

Level thesis:degree_level

Thesis

Discipline thesis:degree_discipline

Electrical & Computer Engr

Grantor

University of Illinois Urbana-Champaign

Year dc:date

2025

Author dc:creator

• Dai, Bill

Contributors dc:contributor

• Chen, Deming

• Homomorphic encryption
• Machine learning
• Cybersecurity
• Privacy-preserved machine learning

Statement dc:rights

• Copyright 2025 Bill Dai

Language dc:language

en

Handle dc:identifier

https://hdl.handle.net/2142/132598

OAI identifier oai:identifier

oai:www.ideals.illinois.edu:2142/132598