Deep Learning from Scratch - Week 4

Course starts soon..


We will start now with a quiz based on the first week material

You have 6 minutes to answer the quiz.

The quiz link:
Quiz Link
It will be copied in Mattermost and in the Zoom chat.

Questions from last week

Leaky ReLU and negative numbers on the activation

Cost function for unsupervised learning

Leaky ReLU

difference between activation functions in hidden layers and output layer

from CS231:

Cost function for unsupervised learning

much wider topic: as an idea, the function it is used for is usually different

a typical example is to "learn" the characteristics of the input, for example dividing into different clusters or learning a lower dimensional representation. In this case the cost function can be defined as the difference betweeen the "learned" representation and the input and is then minimized through the training.

For more information, check out this paper from Adam Coates and Andrew Ng.


Project Template Folder

Please use the project template

Quiz (15 mins)

  • What are the characteristics of an activation function?
  • Why we need to have random initialization and we cannot leave all weights equal to zeros?
  • We initialize the weights of the neurons randomly. Is this really the best we can do? Do you have an idea about how we could do it?
  • Are all neurons of a layer always connected to all the neurons of the next one?
  • What is the relation between the cost function, gradient descent, the activation function and the weights of the network?


Exercise (15-20 mins)

We go through the programming assignment that were planned for this week.
Q: Is having more hidden units better in terms of performances? Is it worse? Does it depends on a specific parameter? Do you have an intuition why?
Q2: Does having more hidden layers require more iterations to train?

Open Source Project of the Week

Machine Learning Toolkit for Unity

ML-Agents Community Challenge Winners

Hide / Escape - Avoidance of Pursuing Enemies

A cool video on state-of-the-art on hide/escape with AI!

For the next week

  • Finish the fourth week of the course
  • Do the Programming Assignments: Build your Neural Network Step by Step and Deep Neural Network Application
  • Prepare an idea for a project, or be ready to discuss projects ideas and start forming teams