University of Pavia

Ph.D. School of Electrical and Electronics Engineering and Computer Science

Deep Learning and TensorFlow

A short course: June 30 – July 24, 2020

Due to the recent sanitary emergency, the course will be held entirely online.

Registered participants will receive links to virtual rooms before the course starts.

Instructor: Marco Piastra, PhD, marco.piastra@unipv.it

Assistants:

• Mirto Musci, PhD, mirto.musci@unipv.it
• Andrea Pedrini, PhD, andrea.pedrini@unipv.it

Deep Learning techniques are at the heart of the recent revolution in Artificial Intelligence that has opened many new perspectives for engineering applications. On a closer look, nonetheless, such techniques appear to be more evolutionary - in terms of substantial improvements over existing methods - than revolutionary. This short course is intended as a first and reasonably self-contained introduction to the topic, with the purpose of analyzing Deep Learning in a broader theoretical context and exploring the reasons of its unquestionable success, in several specific application cases. The short course includes also an introductory, hands-on tutorial on TensorFlow, one of the best known open-source deep learning frameworks which is supported and developed by Google. The tutorial aims to give the fundamental notions about this framework by working on specific examples.

The book “Deep Learning”, by I. Goodfellow, Y. Bengio and A. Courville, MIT Press, 2017, will be assumed as reference for these episodes. At present, an online, html version of the book is freely accessible online at http://www.deeplearningbook.org/.

For Keras and Python programming for Deep Learning in general: “Deep Learning with Python” by Francois Chollet, Manning, 2017

For TensorFlow, a good introductory book is “Hands-On Machine Learning with Scikit-Learn & TensorFlow” by Aurélien Géron, O’Reilly, 2017

Online registration

The registration is free of charge and reserved to the students and staff of University of Pavia.

Prerequisites (for the hands-on tutorial)

Previous knowledge of the topics below will be assumed as prerequisites:

• Python programming: basic operations, lists, tuples, dictionaries, conditions, loops, functions, classes and instances, modules, exception handling (all code examples will be in Python 3)
• Numpy basics: array creation, basic operations and linear algebra (matmul), indexing, slicing, iteration, reshaping
• Python notebooks (i.e. Google Colab): minimal survival skills

Detailed Syllabus

1 - Lecture (3 hours), Tuesday June 30, 2:15 pm

Machine learning: Artificial Neural Networks

• An introductory example: linear regression
• Supervised machine learning: representation, evaluation, optimization
• Feed-forward networks as universal representation
• Evaluation function and numerical optimization
• Gradient descent, stochastic gradient descent, mini-batch gradient descent
• Feed-forward network as a flow graph
• Automatic differentiation: computing the gradient
• Tensorial representation, implicit operations (transposing, broadcasting)

2 - Lecture (3 hours)), Friday July 3, 2:15 pm

Deep Learning for Neural Networks

• Deep network representation: fundamentals, potential advantages and major issues
• Layerwise representation as a flow graph (with layerwise gradient)
• From regression to classification: Softmax layer
• First order vs. second order optimization, potential advantages and major issues
• Quasi second-order methods: momentum, Nesterov, AdaDelta, AdaGrad, Adam
• The choice of non-linear functions: ReLU
• Empirical tricks: batch normalization, dropout 

3 - Lecture (3 hours), Tuesday July 7, 2:15 pm

Deep Convolutional Neural Networks

• The ImageNet challenge
• Convolutional filters: representation and gradients
• Max pooling, normalizations: representation and gradients
• Advanced architectures: Inception and ResNet
• Transfer learning
• Visualizing representations in deep layers, merging
• Active learning, adversarial generation
• Beyond single-object classification: object location, segmentation (hints)

4 - Lecture (4 hours), Friday July 10, 2:15 pm

TensorFlow Fundamentals

• Sessions: computing flow graphs
• Tensors: constant, variables, placeholders and ops
• Tensor transformations: slicing, broadcasting, reshaping
• Optimization: loss functions, gradients, optimizers

5 - Lab (4 hours), ), Tuesday July 14, 2:15 pm

A first example with TensorFlow

• Graph and sessions
• Randomization
• Linear regression: stochastic gradient descent in detail and with an optimizer
• Running a session and visualizing results

6 - Lab (4 hours), Friday July 17, 2:15 pm

TensorFlow for DCNN (I)

• Logistic regression for MNIST
• DCNN for MNIST
• A bag of tricks: batch normalization, dropout, improved optimizers

7 - Lab (4 hours), , Tuesday July 21, 2:15 pm

TensorFlow eager mode, Keras

• Graph mode and eager mode: differences
• A complete example: regression revisited
• From eager mode to graphs
• The Keras front end

8 - Lab (4 hours), Friday July 24, 2:15 pm

TensorFlow for DCNN (II)

• Dataset preparation: setup, tensor representation, mini batches
• Transfer learning: pre-trained models
• More tricks: data augmentation
• Fine tuning and network surgery: telling cats from dogs

Organizers

• Prof. Virginio Cantoni
• Prof. Francesco Leporati

Ph.D. Coordinator

• Prof. Paolo Di Barba 

All presentations will be made available at:https://vision.unipv.it/AI/AIRG.html

Source code for the examples will be made available at: https://bitbucket.org/unipv_cvmlab/deep-learning-and-tensorflow-a-short-course-2020