Quiz

We will start now with a quiz based on the first week material

You have 6 minutes to answer the quiz.

The quiz link:
Quiz Link
It will be copied in Mattermost and in the Zoom chat.

Waterkant

03-12 June

Hackathon

04-05 June

Schedule

The Challenges

Short Overview of different Networks

Feed Forward Neural Networks

The network we saw during the course:
input layer, hidden layer(s), output layer

Introduced in 1958 as the perceptron [1]

Biologically inspired

Bibliography

[1] Rosenblatt, Frank. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review 65.6 (1958): 386.

Autoencoders

What is an encoder doing?

An autoencoder is a neural network that learns to copy its input to its output. [1]

It has an internal (hidden) layer that describes a code used to represent the input, and it is constituted by two main parts: an encoder that maps the input into the code, and a decoder that maps the code to a reconstruction of the original input. [1]

Often when people write autoencoders, the hope is that the middle layer h will take on useful properties in some compressed format. [2]

An example of when we do not need it [2,3]

An example of when we do need it [2,3]

Bibliography

Articles:
[1] Autoencoder on Wikipedia
[2] Article about autoencoders and denoising example (Towardsdatascience.com)
[3] Building Autoencoders in Keras
Papers:
[4] Bourlard, Hervé, and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics 59.4-5 (1988): 291-294.
[5] Kingma, Diederik P., and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
[6] Vincent, Pascal, et al. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning. ACM, 2008.

Convolutional Neural Networks

What is a convolution?

A Convolution, animated [1,2]

A Convolutional Neural Network explained [3]

Compare image classifiers

Slide from Samek's presentation at ICIP 2018 [4]

Not always as expected..

Slide from Samek's presentation at ICIP 2018 [4]

Deep Visualization Toolbox

Image from the website[5]. Here also the source code of the Toolbox [6]

Bonus: CNN Explainer Demo [7]

Bonus: CNN Visualizer Demo [8]

CNN Bibliography

[1] A guide to convolution arithmetic for deep learning, Dumoulin et al. (2018)
[2] Github repository for the animations related to the convolution arithmetic [1]
[3] 3blue1brown: a great resource for math explanation and visualizations.
[4] Slides from Samek's presentation at ICIP 2018
[5] Toolbox for Deep Visualization
[6] Github Repository of the Toolbox
[7] CNN Explainer Demo: play with a CNN in your browser
[8] CNN Visualizer Demo: Flat 2D Visualization

Recurrent Neural Networks and Long Short Term Memory

A leap into language processing

What makes Recurrent Networks so special? [1]

they [Neural Networks] accept a fixed-sized vector as input (e.g. an image) and produce a fixed-sized vector as output (e.g. probabilities of different classes). [..] The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both. [1]

What does Long Short Term Memory means?

[LSTM] are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997) [2] [..] LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn! [3]

Natural Language Processing Application [4]

Compare text classifiers

Slide from Samek's presentation at ICIP 2018 [5]

RNN/LSTM Bibliography

[1] The unreasonable Effectiveness of RNN - Andrej Karpathy
[2] Long Short Term Memory, Hochreiter et al. (1997)
[3] Understanding LSTMs - Colah's Blog
[4] Memorization RNN/LSTM
[5] Slide from Samek's presentation at ICIP 2018
Additional Resources:
[6] Animation RNN
[7] Detailed Explanation, LSTM
[8] RNN representation

EXERCISE (15-20 mins)