Seminarium Instytutu Informatyki z referatem pt. „Facilitate Training in Deep Learning using Marchenko-Pastur Distribution”

Szanowni Państwo,

Serdecznie zapraszamy na Seminarium Instytutu Informatyki, które odbędzie się 9 stycznia 2025 r. o godzinie 12:00 w sali 110INF.
Referat pt. „Facilitate Training in Deep Learning using Marchenko-Pastur Distribution” wygłosi Mariia Kiyashko z Penn State University.

Streszczenie:
In this talk we present several aspects of Random Matrix Theory (RMT) and its applications to Deep Neural Networks (DNNs). We begin with a short overview of RMT, focusing on the Marchenko-Pastur (MP) spectral approach. Next, we present recent results (both analytical and numerical) on enhancing DNN training efficiency through MP-based pruning techniques ([1]). Furthermore, we explore how combining this pruning method with L2 regularization can significantly reduce randomness in weight layers, speeding up the training process. The talk concludes with the discussion of the novel idea of extending the MP-approach to the input-output Jacobian matrix of DNNs, with a particular focus on identifying fixed points. We support our analytical results by numerical examples for various DNN architectures: fully connected networks, CNNs, and ViTs.
This is a joint work with my PhD advisor Prof. L. Berlyand (PSU), PSU PhD students Y. Shmalo, L. Zelong, and with I. Afanasiev and V. Slavin (Kharkiv, Ukraine).

[1] Berlyand, Leonid, et al. „Enhancing accuracy in deep learning using random matrix theory.” Journal of Machine Learning. (2024).