Computer Science Institute seminar with a paper entitled “Facilitating training in deep learning using Marchenko-Pastur decomposition”.

Dear All,

We cordially invite you to the Seminar of the Institute of Computer Science, which will be held on January 9, 2025 at 12:00 in room 110INF. The paper entitled. “Facilitate Training in Deep Learning using Marchenko-Pastur Distribution” will be presented by Mariia Kiyashko from Penn State University.

Synopsis:
In this talk we present several aspects of Random Matrix Theory (RMT) and its applications to Deep Neural Networks (DNNs). We begin with a short overview of RMT, focusing on the Marchenko-Pastur (MP) spectral approach. Next, we present recent results (both analytical and numerical) on enhancing DNN training efficiency through MP-based pruning techniques ([1]). Furthermore, we explore how combining this pruning method with L2 regularization can significantly reduce randomness in weight layers, speeding up the training process. The talk concludes with the discussion of the novel idea of extending the MP-approach to the input-output Jacobian matrix of DNNs, with a particular focus on identifying fixed points. We support our analytical results by numerical examples for various DNN architectures: fully connected networks, CNNs, and ViTs. This is a joint work with my PhD advisor Prof. L. Berlyand (PSU), PSU PhD students Y. Shmalo, L. Zelong, and with I. Afanasiev and V. Slavin (Kharkiv, Ukraine).

[1] Berlyand, Leonid, et al. “Enhancing accuracy in deep learning using random matrix theory.” Journal of Machine Learning. (2024).