Introduction
From the 2nd to the 13th July 2018 I am teaching deep learning at the African Institute for Mathematical Sciences South Africa as part of the winter data science workshop held by Dr. Bubacarr Bah. The goal is to teach participants the foundations of neural networks and to convey the general ideas and key points. For this workshop I use Google Colab for all programming.
Recommended tutorials to help with Python programming: Python is commonly used when implementing neural networks. In particular, the Numpy package is used a lot. I would recommend getting up to speed with Numpy – there is a tutorial here to explain the basics and another tutorial here. The primary focus of this course is on neural networks so I would recommend getting familiar with Python. There are Python tutorials here and here. There are plenty of great online courses especially from Udacity and Coursera, for example this one (you can apply for financial aid).
Feedback from July participants:
- How would you score the course experience overall: 4.6/5
- How would you score the lecturer: 4.9/5
- Did the lecturer create an environment that allowed you to ask questions: 4.9/5
- Were there enough examples provided for each of the topics covered: 4.6/5
Comments from July participants:
- “Excellent teaching. Appreciate the way you are connecting every details of the algorithms with real examples.”
- “I enjoyed the thorough step by step building of the network.”
- “I did enjoy the programming since it was my first time, especially using Python.”
- “The foundation had been laid.”
Course Material:
Day 1 (4 hours):
- I kick off the session by presenting some interesting and fun deep learning advances over the recent years.
- We then move on with a numpy refresher to remind everyone about some of the important operations which are commonly used with neural networks.
- I then present an intro to neural networks by discussing forward passes, activation functions, sum of square error and gradient descent. My solutions to the notebook are available here.
Day 2 (6 hours):
- We move on with a tutorial on how to implement a gradient descent step in numpy. My solutions are available here.
- Next, we implement gradient descent over all of the training data instead of just one sample as was done above. My solution is here.
- We then learn how to implement back propagation and then code it for a single record in the training data. My solution is available here.
- Finally, we code back propagation for all of the training data, and my solution is here.
- I introduce the participants to using Keras for regression problems using the Boston house pricing dataset. The notebook for the regression problem is available here. Following this, I introduce them to classification in Keras using MNIST.
- The participants and presented with a notebook on how to implement a CNN in Keras.
Day 3 (4 hours):
- Deep learning methods often require large datasets. I introduce the participants to data augmentation, the code is available here.
- We discuss the importance of network architecture design.
- Next, we engage in a discussion on network architecture design where I challenge the participants with different datasets and ask them to propose a suitable network architecture, hyper-parameters and loss function.
- In the previous examples we coded in Keras. There are alternative frameworks available. I introduce the participants to TFLearn which is a Tensorflow wrapper. The coding exercises are available here, and the solutions are available here.
- We discuss transfer learning in Keras.
Day 4 (2 hours):
- The exercises for transfer learning are available here, and my solution is available here.
- So far we have covered regression, feature based and image classification. I introduce the participants to a different problem domain, that of natural language processing. We using Google BigQuery to create a dataset and then use a bag of words approach to classify the text into several categories. The code is available here and the solution is here.
Day 5 (2 hours):
- Today we begin with a tutorial on text sentiment analysis using Keras. The exercise is available here and the solution is here.
- Next, we discuss how we can use 1D convolutional neural networks for text classification. The exercises are available here and the solution is available here.
- Training neural networks takes time. We can save the weights to a file so that they can be reused. In this tutorial we learn how to save and load weights, and to create checkpoints so that the best weights across multiple epochs can be obtained instead of just the final weights after the epochs have completed. The notebook is available here.
Day 6 (2 hours):
- I will go over the concept of the embedding layer again and go over the solution for the text classification problem.
- I introduce the participants to Long Short Term Memory (LSTM). We discuss the theory behind these networks and then we engage in a practical. The notes are available here, and the code is available here.
Network Optimisation Challenge:
Here are three programming challenges. In each case you do not need to worry about the data pre-processing. Your primary task is to create the most optimal network architecture. Use Keras to build the model, compile and fit the training data. Once you are satisfied with your solution, share your notebook with me. The instructions are in the notebooks. Good luck!
Challenge 1: Regression
Challenge 2: Multi-label regression
Challenge3: Multi-class classification
Want additional machine learning tutorials?
Subscribe below, and I will email you my latest tutorials and material related to machine learning: