The UCF Mathematics and Applications Seminar, which takes place every Friday from 10:30am to 11:30am in MSB 318, provides a venue for researchers to present their current work, foster new collaborations, and showcase both foundational mathematics and its applications to graduate and undergraduate students.
Dr. Ouayl Chadli will be speaking at this seminar about smoothing technics for the heavy ball method with applications in Machine Learning
Abstract: First-order optimization is at the heart of most training algorithms in Machine Learning. Thanks to backpropagation, neural networks have achieved a number of impressive successes over the past few years, including natural language processing, image processing, and reinforcement learning. However, training complex state-of-the-art architectures remains difficult due to the highly non-smooth and non-convex nature of their loss functions. While stochastic gradient descent and its variants (Adam, RMSProp, or Nesterov’s accelerated gradient descent) showed surprisingly good performance in many practical scenarios, but these algorithms remain fragile for particularly non-smooth or non-convex objectives. One of the main reasons is that the computation of the (sub)gradients and convergence analysis (in batch or mini-batch settings) essentially rely on the sum-rule in smooth or convex settings, i.e. ∂(f1 + f2) = ∂f1 + ∂f2. Unfortunately, this sum rule does not hold in general in the non-convex setting using the standard Clarke’s subdifferential. In many deep-learning studies, the failure of the sum rule is ignored. They use it in practice but circumvent the theoretical problem by modeling their method through simple smooth or convex dynamics. This practice can create additional spurious stationary points that are not Clarke’s critical points. To overcome this difficulty, we introduce an inertial second-order dynamical system, which is obtained by approximating the nonsmooth function by a class of smooth functions. By studying the asymptotic behavior of the algorithm generated by this dynamical system, we prove that each trajectory converges to the Clarke’s critical points under some appropriate conditions on the smoothing parameters.
Read More