Deep Learning, Malek-Ashtar University of Technology (Fall 2017)


last update:


Course Description:

This course introduces you to deep learning: the state-of-the-art approach to building artificial intelligence algorithms. Deep learning is the machine learning technique behind the most exciting capabilities in diverse areas like natural language processing, image recognition, speech recognition, robotics, etc. In this course, we cover the basic components of deep learning, what it means, how it works, and develop code necessary to build various algorithms such as deep convolutional networks, auto-encoders, generative adversarial networks, and recurrent neural networks.

Deep learning offers enormous potential for creative applications and in this course we interrogate what's possible. Through practical applications and guided homework assignments, you'll be expected to create datasets, develop and train neural networks, explore your own media collections using existing state-of-the-art deep nets, synthesize new content from generative algorithms, and understand deep learning's potentials.


Useful References

o   Deng, Li, and Dong Yu. "Deep learning: methods and applications.", 2014

o   Yu, Dong, and Li Deng. “Automatic speech recognition: A deep learning approach”, 2015.

o   Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. “Deep learning” , 2016.

o   Bishop, Christopher M. Pattern recognition and machine learning, 2006.

Grading Policy


Grade Breakdown




The assignments submission must be electronically through Yadamooz. Do not email us your assignments.



Presentation of a new method for deep learning based on an approved paper.

Final exam



Final Project



Teaching Assistant

Ebrahim Khademi (contact at Yadamooz)




Course Materials

Lecture 1

Course Introduction

·         Recent success with Deep Learning

·         Historical context

·         Course plan

Suggested Readings:

1.     [Linear Algebra Review]

2.     [Probability Review]

3.      [Optimization (SGD) Review]

[python tutorial]

Lecture 2

A Review of Artificial Neural Networks 1

·         Perceptron

·         Stochastic Gradient Descent

·         Backpropagation

Suggested Readings:

1.     [Classification and Linear Classifiers]

2.     [Backpropagation]


[slides] [video]

Lecture 3

A Review of Artificial Neural Networks 2

·         Activation Functions (Sigmoid, Tanh, ReLU, Leaky ReLU, ELU, Maxout)

·         Neural Network architectures

Suggested Readings:

1.    [Neural Nets notes 1]


[slides] [video]

Lecture 4

A Review of Artificial Neural Networks 3

·         Data Preprocessing

·         Weight Initialization

·         Batch Normalization

Suggested Readings:

1.    [Neural Nets notes 2]

2.    [Batch normalization]


[slides] [video]

Lecture 5

A Review of Artificial Neural Networks 4

·         Model Ensembles

·         Regularization

·         Dropout

·         Regularization: A common pattern

Suggested Readings:

1.    [Neural Nets notes 2]

2.    [Neural Nets notes 3]

3.    [Dropout]

4.    [Regularization]


[slides] [video]

Lecture 6

A Review of Artificial Neural Networks 5

·         Hyperparameter Optimization

·         Loss functions

·         Parameter updates

·         Transfer Learning

Suggested Readings:

1.    [Neural Nets notes 2]

2.    [Neural Nets notes 3]

3.    [linear classify]

4.    [intro-to-cross-entropy-loss]

5.    [transfer-learning]


[slides] [video]

Lecture 7

A quick review of Probabilistic Graphical Models

·         Probabilistic Graphical Models

·         Bayesian Networks

·         Dynamic Bayesian Networks

·         Markov Random Fields


Suggested Readings:

1.    [Pattern Recognition and Machine Learning]

2.    [Probabilistic graphical models: principles and techniques]

3.    [Dynamic Bayesian networks]

4.    [PGM coursera]

5.    [PGM AUT: CL1, CL2, CL3, CL4]



Lecture 8

Deep Belief Network

·         Sampling

·         Restricted Boltzmann Machine

·         Deep Belief Network

·         Applications DBN


Suggested Readings:

1.    [Reducing the Dimensionality of Data]

2.    [A practical guide to training RBM]

3.    [DeeBNet]

4.    [FEPCD]


[slides] [video]

Lecture 9


·         Sparse Autoencoders

·         Denoising autoencoders

·         Contractive Autoencoders

·         Applications of Autoencoders

Suggested Readings:

1.    [Sparse autoencoder]

2.    [Deep learning]

3.    [From PCA to Autoencoders]


[slides] [video]

Lecture 10

Convolutional Neural Networks 1

·         Convolutional

·         Pooling


Suggested Readings:

1.    [ConvNet notes]

2.    [Wikipedia]


[slides] [video]

Lecture 11

Convolutional Neural Networks 2

·         AlexNet

·         ZF Net

·         VGG Net

·         GoogLeNet

·         ResNet


Suggested Readings:

1.    [AlexNet]

2.    [ZF Net]

3.    [VGG Net]

4.    [GoogLeNet]

5.    [ResNet]


[slides] [video]

Lecture 12

Convolutional Neural Networks 3

·         R-CNN

·         Fast R-CNN

·         GAN


Suggested Readings:

1.    [R-CNN]

2.    [Fast R-CNN]

3.    [GAN]


[slides1] [slides2]  [video]


Lecture 13

Computational Graphs

·         Computational Graph Language

·         Derivative using Computational Graph

·         Factoring Paths

·         Forward- and Reverse-Mode Differentiation

Suggested Readings:

1.    [Calculus on Computational Graphs: Backpropagation]

2.    [Computation Graphs]


[slides] [video]

Lecture 14

Natural Language Processing

·         Word Vectors

·         SVD Based Methods

·         Iteration Based Methods – Word2vec

Suggested Readings:

1.    [NLP Notes 1]

2.    [NLP Notes 2]


[slides] [video]

Lecture 15

Recurrent Neural Networks

·         RNN computation

·         Character-Level Language Models

·         Backpropagation through time

·         RNN Examples

Suggested Readings:

1.    [Neural Nets Lecture 10]

2.    [The Unreasonable Effectiveness of RNNs]

3.    [Computational Graph, RNN, CNN]


[slides] [video]

Lecture 16


·         Gradient Vanishing

·         Long Short Term Memory

·         Variants on LSTM


Suggested Readings:

1.    [Neural Nets Lecture 10]

2.    [wikipedia]

3.    [Understanding LSTM Networks]

4.    [Neural Networks for Machine Learning]

5.    [RNN Tutorial]


[slides] [video]

Lecture 17

Neural Text Generation

·         Machine Translation

·         Bidirectional LSTM

·         Attention Mechanism

·         Google’s Multilingual NMT


Suggested Readings:

1.    [Neural machine translation]

2.    [Attention-based NMT]

3.    [Google's multilingual NMT]

4.    [Google's NMT ]


[slides] [video]

Lecture 18

Automatic Speech Recognition (ASR)

·         Major Speech Processing Tasks

·         Automatic Speech Recognition

·         Early DNN based ASR systems

·         End-to-End DNN based ASR sys.

·         End-to-End Speech Synthesis


Suggested Readings:

1.    [ASR and DL]

2.    [End-to-end models for Speech]

3.    [end-to-end ASR]



Lecture 19

Deep Reinforcement Learning

·         Reinforcement Learning

·         Mathematical formulation of the RL problem

·         Deep Q-learning

·         Deep Q-learning examples


Suggested Readings:

1.    [CNN, Lecture 14]

2.    [DRL, Nature]

3.    [DRL, Vehicle Intelligence Lab]


[slides] [video]





Assignment #1

Activation Functions

[Assignment #1] [code]

Assignment #2

Simple Neural Network

[Assignment #2]

Suggested Readings:

·         [Tensorflow and Deep Learning - without a PhD]

Assignment #3


[Assignment #3]

Assignment #4

Word Embedding

[Assignment #4] [Data + Tensorflow]