Getting started with the data for class project

I added a page to the menu (under project description) to help people get started with loading and playing with the data for the class projects using Fuel, and a quick example of training a neural network on this data using Blocks. If you have any questions, feel free to leave a comment on the page.

Lecture 8, Feb. 1st, 2016: Convolutional Neural Networks I

In this lecture, we will discuss the architecture responsible for virtually all the successes of deep learning in computer vision: the convolutional neural network.

Please study the following material in preparation for the class:

• Hugo Larochelle’s video lectures 9.1 to 9.9.
• Chapter 11 of the Deep Learning textbook.

Other relevant material:

• Slides used in class (based on Hugo’s slides).
• Hinton’s coursera lecture 5, videos 1 to 4.
• Christopher Olah’s blog posts on:
  • Conv Nets: A Modular Perspective
  • Understanding Convolutions
  • Groups & Group Convolutions (Warning: advanced material)

Lecture 7, Jan. 28th, 2016: Neural Networks, Odds and Ends

In this lecture, we will conclude our discussion of the standard multi-layer perceptron.

Please study the following material in preparation for the class:

• Hugo Larochelle’s video lectures 2.8 to 2.11.
• Chapter 6 of the Deep Learning textbook  (end of Chapter 6: 6.4, 6.5, 6.6 – also review 6.2.3 and 6.2.4)

Other relevant material:

• Christopher Olah’s blog post on Neural Networks, Manifolds, and Topology  (highly recommended!)
• Sigmoid versus Tanh and weight initialization: Understanding the difficulty of training deep feedforward neural networks.  Xavier Glorot and Yoshua Bengio
• Rectified linear vs sigmoid, tanh: Deep Sparse Rectifier Neural Networks. Xavier Glorot, Antoine Bordes and Yoshua Bengio

Assignment solution and Theano intro

Please have a look at the following regarding the first assignment:

• The notebook with the layer-wise NumPy implementation
• Vincent’s solution with the gradient derivations
• Theano-based solution
• Theano slides and the tutorials

Lecture 6, Jan. 25th, 2016: Training Neural Networks

In this lecture we continue with neural networks and the back-propagation algorithm.

Please study the following material in preparation for the class:

• Hugo Larochelle’s video lectures 2.1 to 2.7.
• Chapter 6 of the Deep Learning textbook

Other relevant material:

• Hinton’s coursera lecture 3, videos 1 to 5.

Git, Git Pages

The notebook from class with a few Git references and commands as well as a quickstart to setting up Jekyll.

The stash Git command that was mentioned in class is indeed useful to look at, allowing you to temporarily store some changes without actually committing them.

Lecture 5, Jan. 21, 2016

 

Lecture material

Please study the following material in preparation for the discussion (first 2 are same as last lecture):

• Hugo Larochelle’s video lectures 2.1 to 2.7.
• Chapter 6 of the Deep Learning textbook  (all sections)
• Hinton’s coursera lecture 2, videos 1 to 5.

Question-Answer session on Quora

Yoshua Bengio will answer questions about deep learning Tuesday at 6pm (January 19th) on Quora:

https://www.quora.com/profile/Yoshua-Bengio/session/37

Those interested can post questions or upvote the questions they would most like to see answered.

Lecture 4, Jan. 18, 2016

Lecture 4 will start with Bart’s tutorial on github and introduction to Theano.

Then we will have our first discussion session (i.e. flipped class). Please study the following material in preparation for the discussion:

• Hugo Larochelle’s video lectures 2.1 to 2.7.
• Chapters 6 of the Deep Learning textbook  (all sections)

Please post questions for the discussion (we will cover all comments to Lectures 1-4).

Also, we can also cover background material (by request):

• Chapters 1-5 of a book draft on Deep Learning,  by Yoshua Bengio, Ian Goodfellow and Aaron Courville.

 

 

Notebook NumPy/Python intro

During the introduction in class I forgot to mention Jupyter notebooks. (Commonly known as IPython notebooks, which is the old name of this project.) They can be a great way of organising your Python code for e.g. assignments, allowing you to mix Python code, graphs and text, and repeatedly changing and executing Python code while maintaining a global state.

If you installed Python using Anaconda you already have Jupyter, and you can simply start it by typing jupyter notebook on the command line. If you installed Python in another way, you might need to install it first using pip install jupyter.

I uploaded the notebook (PDF) used in class that contained the introduction to NumPy and Python.