Deep Learning from Scratch - Week 7
Course starts soon..
We will start now with a quiz based on the first week material
You have 6 minutes to answer the quiz.
The quiz link:
It will be copied in Mattermost and in the Zoom chat.
Short Overview of different Networks
Feed Forward Neural Networks
The network we saw during the course:
input layer, hidden layer(s), output layer
Introduced in 1958 as the perceptron 
What is an encoder doing?
An autoencoder is a neural network that learns to copy its input to its output. 
It has an internal (hidden) layer that describes a code used to represent the input, and it is constituted by two main parts: an encoder that maps the input into the code, and a decoder that maps the code to a reconstruction of the original input. 
Often when people write autoencoders, the hope is that the middle layer h will take on useful properties in some compressed format. 
An example of when we do not need it [2,3]
An example of when we do need it [2,3]
 Autoencoder on Wikipedia
 Article about autoencoders and denoising example (Towardsdatascience.com)
 Building Autoencoders in Keras
 Bourlard, Hervé, and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics 59.4-5 (1988): 291-294.
 Kingma, Diederik P., and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
 Vincent, Pascal, et al. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning. ACM, 2008.
Convolutional Neural Networks
What is a convolution?
A Convolution, animated [1,2]
A Convolutional Neural Network explained 
Compare image classifiers
Not always as expected..
Deep Visualization Toolbox
 A guide to convolution arithmetic for deep learning, Dumoulin et al. (2018)
 Github repository for the animations related to the convolution arithmetic 
 3blue1brown: a great resource for math explanation and visualizations.
 Slides from Samek's presentation at ICIP 2018
 Toolbox for Deep Visualization
 Github Repository of the Toolbox
 CNN Explainer Demo: play with a CNN in your browser
 CNN Visualizer Demo: Flat 2D Visualization
Recurrent Neural Networks and Long Short Term Memory
A leap into language processing
What makes Recurrent Networks so special? 
they [Neural Networks] accept a fixed-sized vector as input (e.g. an image) and produce a fixed-sized vector as output (e.g. probabilities of different classes). [..] The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both. 
What does Long Short Term Memory means?
[LSTM] are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997)  [..] LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn! 
Natural Language Processing Application 
Compare text classifiers
 The unreasonable Effectiveness of RNN - Andrej Karpathy
 Long Short Term Memory, Hochreiter et al. (1997)
 Understanding LSTMs - Colah's Blog
 Memorization RNN/LSTM
 Slide from Samek's presentation at ICIP 2018
 Animation RNN
 Detailed Explanation, LSTM
 RNN representation
EXERCISE (15-20 mins)We go through the programming assignment that were planned for this week.
Register your project
For the next week