250063 VO Mathematics of Deep Learning (2024S)

The lecture attempts to teach multiple facets of deep learning theory. Concretely, we will study the following:

1. What is Deep Learning: A gentle introduction into the language, notation, and main concepts of deep learning on a high level.
2. Feed-forward neural networks: The main building block of deep learning is that of a neural network. We will introduce this in a formal way.
3. Universal Approximation: We will study multiple results that study the (absence of) limitations of deep neural networks to represent general functions.
4. Connection to Splines: There are close relationships between neural networks and classical approximation methods. We will discuss these.
5. ReLU neural networks: There is a special subclass of neural networks that is the most frequently used. We study this special case in more detail.
6. Affine pieces of ReLU neural networks: A standard tool to understand what deep neural networks can and cannot do is to count the number of affine regions that they can generate. We will find upper and lower bounds for this.
7. Deep ReLU neural networks: We study the effect of depth, specifically for deep ReLU neural networks and find reasons for the fact that in practice deeper architectures reign supreme.
8. Curse of Dimensionality: We study the phenomenon of high dimensional approximation, which neural networks seemingly do well, despite the fact that this is typically very hard.
9. Interpolation: Deep neural networks can interpolate data under certain assumptions. We will formalize those.
10. Training of deep neural networks: We will understand how to train neural networks, and under which conditions this training can work
11. Loss landscape analysis: The optimization problem can be understood by studying the topography of the so-called loss landscape.
12. Neural network spaces: The set of all neural networks with a fixed architecture will be studied and conclusions for optimization will be drawn.
13. Generalization: We will describe general statistical learning theory in the context of deep neural networks. This will show under which conditions one can generalize from training sets to test sets.
14. Generalization in the overparameterized regime: The most common regime, i.e., overparameterized is excluded from the results before. We will describe why in practice things still work
15. Adversarial examples and Robustness: Finally, we study under which conditions deep neural networks are robust to changes in their inputs.