Deep Learning from Scratch - Week 7

Course starts soon..


We will start now with a quiz based on the first week material

You have 6 minutes to answer the quiz.

The quiz link:
Quiz Link
It will be copied in Mattermost and in the Zoom chat.


03-12 June


04-05 June


The Challenges

Short Overview of different Networks





Feed Forward Neural Networks

The network we saw during the course:
input layer, hidden layer(s), output layer

Introduced in 1958 as the perceptron [1]

Biologically inspired


[1] Rosenblatt, Frank. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological review 65.6 (1958): 386.


What is an encoder doing?

An autoencoder is a neural network that learns to copy its input to its output. [1]

It has an internal (hidden) layer that describes a code used to represent the input, and it is constituted by two main parts: an encoder that maps the input into the code, and a decoder that maps the code to a reconstruction of the original input. [1]

Often when people write autoencoders, the hope is that the middle layer h will take on useful properties in some compressed format. [2]

An example of when we do not need it [2,3]

An example of when we do need it [2,3]


[1] Autoencoder on Wikipedia
[2] Article about autoencoders and denoising example (
[3] Building Autoencoders in Keras
[4] Bourlard, Hervé, and Yves Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics 59.4-5 (1988): 291-294.
[5] Kingma, Diederik P., and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
[6] Vincent, Pascal, et al. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning. ACM, 2008.

Convolutional Neural Networks

What is a convolution?

A Convolution, animated [1,2]

A Convolutional Neural Network explained [3]

Compare image classifiers

Slide from Samek's presentation at ICIP 2018 [4]

Not always as expected..

Slide from Samek's presentation at ICIP 2018 [4]

Deep Visualization Toolbox

Image from the website[5]. Here also the source code of the Toolbox [6]

Bonus: CNN Explainer Demo [7]

Bonus: CNN Visualizer Demo [8]

CNN Bibliography

[1] A guide to convolution arithmetic for deep learning, Dumoulin et al. (2018)
[2] Github repository for the animations related to the convolution arithmetic [1]
[3] 3blue1brown: a great resource for math explanation and visualizations.
[4] Slides from Samek's presentation at ICIP 2018
[5] Toolbox for Deep Visualization
[6] Github Repository of the Toolbox
[7] CNN Explainer Demo: play with a CNN in your browser
[8] CNN Visualizer Demo: Flat 2D Visualization

Recurrent Neural Networks and Long Short Term Memory

A leap into language processing

What makes Recurrent Networks so special? [1]

they [Neural Networks] accept a fixed-sized vector as input (e.g. an image) and produce a fixed-sized vector as output (e.g. probabilities of different classes). [..] The core reason that recurrent nets are more exciting is that they allow us to operate over sequences of vectors: Sequences in the input, the output, or in the most general case both. [1]

What does Long Short Term Memory means?

[LSTM] are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter & Schmidhuber (1997) [2] [..] LSTMs are explicitly designed to avoid the long-term dependency problem. Remembering information for long periods of time is practically their default behavior, not something they struggle to learn! [3]

Natural Language Processing Application [4]

Compare text classifiers

Slide from Samek's presentation at ICIP 2018 [5]

RNN/LSTM Bibliography

[1] The unreasonable Effectiveness of RNN - Andrej Karpathy
[2] Long Short Term Memory, Hochreiter et al. (1997)
[3] Understanding LSTMs - Colah's Blog
[4] Memorization RNN/LSTM
[5] Slide from Samek's presentation at ICIP 2018
Additional Resources:
[6] Animation RNN
[7] Detailed Explanation, LSTM
[8] RNN representation

EXERCISE (15-20 mins)

We go through the programming assignment that were planned for this week.
Here M. Peixeiro actually wrote an article that explains exactly the solution of the assignment for this week.

Register your project

Fill out this Form

For the next week

  • Finish the third week of the course! Second course is also done!
  • Do the Programming Assignment on Tensorflow