Applied Machine Learning

Lecturer: Hossein Hajiabolhassan Data Science Center Shahid Beheshti University
Teaching Assistants:
Zahra Taheri	Ali Hojatnia	Yavar T. Yeganeh
Fatemeh Amanian	Mahdis Hosseini	Sohrab Faridi

---

Index:

• Course Overview

• TextBooks

• Slides and Papers

  1. Lecture 1: Toolkit Lab (Part 1)
  2. Lecture 2: Introduction
  3. Lecture 3: Empirical Risk Minimization
  4. Lecture 4: PAC Learning
  5. Lecture 5: The Bias-Complexity Tradeoff
  6. Lecture 6: Learning via Uniform Convergence
  7. Lecture 7: The VC-Dimension
  8. Lecture 8: Toolkit Lab (Part 2)
  9. Lecture 9: Linear Predictors
  10. Lecture 10: Decision Trees
  11. Lecture 11: Nearest Neighbor
  12. Lecture 12: Ensemble Methods
  13. Lecture 13: Model Selection and Validation
  14. Lecture 14: Neural Networks
  15. Lecture 15: Convex Learning Problems
  16. Lecture 16: Regularization and Stability
  17. Lecture 17: Support Vector Machines
  18. Lecture 18: Multiclass Classification
  • Additional NoteBooks and Slides

• Class Time and Location

  • Recitation and Assignments

• Projects

  • Practical Guide
  • Fascinating Guide to Use Python Libraries (Machine Learning)
  • Google Colab
  • Latex
  • Useful NoteBooks

• Grading

  • Two Written Exams

• Prerequisites

  • Linear Algebra
  • Probability and Statistics
  • Discrete Mathematics

• Topics

• Account

• Academic Honor Code

• Questions

• Miscellaneous

  • Tutorials
  • Data Handling
  • Projects

---

Course Overview:

Machine learning is an area of artificial intelligence that provides systems the ability to 
automatically learn. Machine learning allows machines to handle new situations via analysis, 
self-training, observation and experience. The wonderful success of machine learning has made 
it the default method of choice for artificial intelligence experts. In this course, we review 
the fundamentals and algorithms of machine learning. 

TextBooks:

Main TextBooks:

• Understanding Machine Learning: From Theory to Algorithms, by Shai Shalev-Shwartz and Shai Ben-David
• An Introduction to Statistical Learning: with Applications in R by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani
• Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow (2nd Edition) by Aurelien Geron

Additional TextBooks:

• Pattern Recognition and Machine Learning by Christopher Bishop
  
• Hands-On Machine Learning with R by Bradley Boehmke and Brandon Greenwell

Slides and Papers:

Recommended Slides & Papers:

1. Toolkit Lab (Part 1: Anaconda, Jupyter Lab, Markdown, Git, GitHub, and Google Colab)

Required Reading:

Anaconda, Jupyter Lab, Markdown, Git, GitHub, and Google Colab:

• Blog: Managing Environments
• Blog: Kernels for Different Environments
• Slide: Practical Data Science: Jupyter NoteBook Lab by Zico Kolter
• Awesome JupyterLab by Hai Nguyen Mau
• Blog: Learn Markdown Online
• Slide: An Introduction to Git by Politecnico di Torino
• Blog: Google Colab Free GPU Tutorial by Fuat

Teaching Assitant Class:
Python continues to take leading positions in solving data science tasks and challenges. Here are three of the most important of libraries.
Numpy is the fundamental package for scientific computing with Python.	Pandas is an easy-to-use data structures and data analysis tools	Matplotlib is a Python 2D plotting library
Resources:
Scipy Lecture Notes	Data Science iPython Notebooks

• Homeworks: Python Libraries for Data Science
  • Exercises: Practice Numpy in LabEx
    
  • Exercises: Practice Pandas in LabEx
    
  • Exercises: Practice Matplotlib in LabEx
    

Suggested Reading:

• Tools in Data Science
• 28 Jupyter Notebook Tips, Tricks, and Shortcuts by Josh Devlin
• Cheat Sheet: Markdown Syntax
• Git Cheat Sheet
• R Tutorial for Beginners: Learning R Programming

Additional Resources:

• PDF: Conda Cheat Sheet
• Blog: Conda Commands (Create Virtual Environments for Python with Conda) by LipingY
  
• Blog: Colab Tricks by Rohit Midha
• Paper: Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence by Sebastian Raschka, Joshua Patterson, and Corey Nolet
• The following table was adopted from Applied Machine Learning and Deep Learning created by Cuixian Chen

Python Overview [Word] Python Tutorial [PDF] [Code]	Numpy [PDF] [Code] User Guide [Link] Quickstart [Link] Reference [Link] Practice Numpy in LabEx [Link] Cheatsheet [Link]	Matplotlib [PDF][Code] Example [Link] Tutorials [Link] Reference [Link] Practice Matplotlib in LabEx [Link] Cheatsheet [Link]
Pandas [Code] 10 Min to Pandas [Link] Cookbook [Link] Tutorials [Link] Reference [Link] Practice Pandas in LabEx [Link] Cheatsheet [Link]	Seaborn: Stat data Visulization [Link] Example [Link] Tutorials [Link] Reference [Link] Cheatsheet [Link]	Scikit Learn [Link] Scikit Image [Link] Scikit Tutorial #1 [Code] Scikit Tutorial #2 [Code] Cheatsheet [Link]

2. Introduction

   Required Reading:
   

   • Introduction
     Chapter 1 of Understanding Machine Learning: From Theory to Algorithms

3. Empirical Risk Minimization

   Required Reading:
   

   • A Formal Model – The Statistical Learning Framework & Empirical Risk Minimization
     Chapter 2 of Understanding Machine Learning: From Theory to Algorithms
     • Exercises: 2.1, 2.2, and 2.3
   • Slide: Machine Learning by Roland Kwitt
     
   • Slide: Lecture 1 by Shai Shalev-Shwartz
   • Blog: Some Key Machine Learning Definitions by Joydeep Bhattacharjee

4. PAC Learning

   Required Reading:
   

   • Chapter 3 of Understanding Machine Learning: From Theory to Algorithms
     • Exercises: 3.2, 3.3, 3.4, 3.5, 3.6, 3.7
   • Slide: Machine Learning by Roland Kwitt
   • Slide: Lecture 2 by Shai Shalev-Shwartz

5. Learning via Uniform Convergence

   Required Reading:
   

   • Chapter 4 of Understanding Machine Learning: From Theory to Algorithms
     

   • Slide: Machine Learning by Roland Kwitt

6. The Bias-Complexity Tradeoff

   Required Reading:
   

   • Chapter 5 of Understanding Machine Learning: From Theory to Algorithms
     • Exercise: 5.2
   • Slide: Machine Learning by Roland Kwitt
     
   • Slide: Lecture 3 by Shai Shalev-Shwartz
     
   • Paper: The Bias-Variance Dilemma by Raul Rojas

   Suggested Reading:
   

   • Paper: A Unified Bias-Variance Decomposition by Pedro Domingos

   Additional Reading:
   

   • NoteBook: Exploring the Bias-Variance Tradeoff by Kevin Markham
   • Blog: Bias-Variance Decomposition by Sebastian Raschka
   • Slide: Bias-Variance Theory by Thomas G. Dietterich

7. The VC-Dimension

   Required Reading:
   

   • Chapter 6 of Understanding Machine Learning: From Theory to Algorithms
     
     • Exercises: 6.2, 6.4, 6.6, 6.9, 6.10, and 6.11
       
   • Slide: Machine Learning by Roland Kwitt
     

8. Toolkit Lab (Part 2)

   Required Reading:
   

   • Machine Learning Mastery With Python by Jason Brownlee
     
     • Python Codes
       
   • Data Exploration:
     • NoteBook: Titanic 1 – Data Exploration by John Stamford
     • NoteBook: Kaggle Titanic Supervised Learning Tutorial
     • NoteBook: An Example Machine Learning Notebook by Randal S. Olson
     • Homework: Take the 7-Day Machine Learning Challenge of Kaggle: Machine learning is the hottest field in data science, and this track will get you started quickly.

9. Linear Predictors

   Required Reading:
   

   • Chapter 9 of Understanding Machine Learning: From Theory to Algorithms
     • Exercises: 9.1, 9.3, 9.4, and 9.6
   • Slide: Machine Learning by Roland Kwitt
     
   • Slide: Tutorial 3: Consistent linear predictors and Linear regression by Nir Ailon
     
   • NoteBook: Perceptron in Scikit by Chris Albon
     
   • Blog: Why Linear Regression is not Suitable for Classification by Hong Jing
   • Slide: Logistic Regression by Jeff Howbert

   Additional Reading:
   

   • Blog: MAE, MSE, RMSE, Coefficient of Determination, Adjusted R Squared — Which Metric is Better? by Akshita Chugh
   • Blog: Key Difference between R-squared and Adjusted R-squared for Regression Analysis by Aniruddha Bhandari
   • NoteBook: Linear Regression by Kevin Markham
     
   • Paper: Matrix Differentiation by Randal J. Barnes
   • Lecture: Logistic Regression by Cosma Shalizi
   • Lecture: Multiclass Classification by Yossi Keshet
   • NoteBook: Logistic Regression-Analysis by Nitin Borwankar
     
     • DataSets
       
   • NoteBook: Logistic Regression by Kevin Markham
     
   • Infographic and Code: Simple Linear Regression (100 Days Of ML Code) by Avik Jain
     
   • Infographic and Code: Multiple Linear Regression (100 Days Of ML Code) by Avik Jain
     
   • Infographic and Code: Logistic Regression (100 Days Of ML Code) by Avik Jain
     

   R (Programming Language):
   

   • Book: Machine Learning Mastery With R by Jason Brownlee
     
   • Blog: Linear Regression by UC Business Analytics R Programming Guide
     
   • Blog: Linear Regression with lm() by Nathaniel D. Phillips
     
   • Blog: Logistic Regression by UC Business Analytics R Programming Guide
     

10. Decision Trees

    Required Reading:
    

    • Chapter 18 of Understanding Machine Learning: From Theory to Algorithms
      • Exercise: 18.2
    • Slide: Decision Trees by Nicholas Ruozzi
    • Slide: Representation of Boolean Functions by Troels Bjerre Sørensen
      
    • Slide: Overfitting in Decision Trees by Reid Johnson
      
    • NoteBook: Decision Trees
      

    Additional Reading:
    

    • Paper: Do We Need Hundreds of Classifiers to Solve Real World Classification Problems? by Manuel Fernandez-Delgado, Eva Cernadas, Senen Barro, and Dinani Amorim
      
    • Blog: Introduction to Random Forest and its Hyper-parameters
    • Blog: Random Forest Classifier Example by Chris Albon. This tutorial is based on Yhat’s 2013 tutorial on Random Forests in Python. - NoteBook
      
    • NoteBook: Titanic Competition with Random Forest by Chris Albon
      
    • Infographic and Code: Decision Trees (100 Days Of ML Code) by Avik Jain
      

    R (Programming Language):
    

    • Book: Machine Learning Mastery With R by Jason Brownlee
      
    • Blog: Decision Tree Classifier Implementation in R by Rahul Saxena
      
    • Blog: Regression Trees by UC Business Analytics R Programming Guide
      

11. Nearest Neighbor

    Required Reading:
    

    • Chapter 19 (Section 1) of Understanding Machine Learning: From Theory to Algorithms
      
    • Slide: Nearest Neighbor Classification by Vivek Srikumar
      
    • NoteBook: k-Nearest Neighbors
      

    Additional Reading:
    

    • Blog: When to perform a Feature Scaling? by By Raghav Vashisht
    • Blog: Voronoi Tessellations
    • Blog: Mahalanobis Distance by Chris McCormick
    • NoteBook: Training a Machine Learning Model with Scikit-Learn by Kevin Markham
      
      • Video
        
    • NoteBook: Comparing Machine Learning Models in Scikit-Learn by Kevin Markham
      
      • Video
        
    • Infographic: K-Nearest Neighbours (100 Days Of ML Code) by Avik Jain
      

    R (Programming Language):
    

    • Book: Machine Learning Mastery With R by Jason Brownlee
      
    • Blog: Knn Classifier Implementation in R with Caret Package by Rahul Saxena
      

12. Ensemble Methods

    Required Reading:
    

    • Chapter 10 of Understanding Machine Learning: From Theory to Algorithms and Chapter 8 of An Introduction to Statistical Learning: with Applications in R
      
      • Exercises: 10.1 and 10.4 from Understanding Machine Learning: From Theory to Algorithms
    • Slide: Ensemble Learning through Diversity Management: Theory, Algorithms, and Applications by Huanhuan Chen and Xin Yao
    • Slide: Ensemble Learning, Model selection, Statistical validation by João Mendes Moreira and José Luís Borges
    • Slide: Bagging and Random Forests by David Rosenberg
      
    • Slide: Machine Learning by Roland Kwitt
      
    • Slide: Introduction to Machine Learning (Boosting) by Shai Shalev-Shwartz
      
    • Paper: Ensemble Methods in Machine Learnin by Thomas G. Dietterich
      
    • NoteBook: AdaBoost
    • Question: Adaboost with a Weak Versus a Strong Learner

    Additional Reading:
    

    • Blog: Ensemble Methods by Rai Kapil
    • Blog: Ensemble Methods (Part 3): Meta-learning, Stacking and Mixture of Experts by Marta Enesco and Keshav Dhandhania
    • Blog: Consensus Clustering by Lance Fernando
    • Blog: Boosting, Bagging, and Stacking — Ensemble Methods with sklearn and mlens by Robert R.F. DeFilippi
      • NoteBook
    • NoteBook: Introduction to Python Ensembles by Sebastian Flennerhag
    • Library (ML-Ensemble): Graph handles for deep computational graphs and ready-made ensemble classes for ensemble networks by Sebastian Flennerhag
      
    • NoteBook: Ensemble Methods by Vadim Smolyakov
      
    • Paper: On Agnostic Boosting and Parity Learning by A. T. Kalai, Y. Mansour, and E. Verbin
      
    • Paper: Faster Face Detection Using Convolutional Neural Networks & the Viola-Jones Algorithm by Karina Enriquez

    R (Programming Language):
    

    • Book: Machine Learning Mastery With R by Jason Brownlee
      
    • Blog: Random Forests by UC Business Analytics R Programming Guide
      

13. Model Selection and Validation

    Required Reading:
    

    • Chapter 11 of Understanding Machine Learning: From Theory to Algorithms
      - Exercises: 11.1 and 11.2 from Understanding Machine Learning: From Theory to Algorithms
    • Blog: What is the Difference Between a Parameter and a Hyperparameter? by Jason Brownlee
    • Blog: A “short” introduction to model selection by David Schönleber
    • Blog: K-Fold and Other Cross-Validation Techniques by Renu Khandelwal
    • Tutorial: Learning Curves for Machine Learning in Python by Alex Olteanu

    Suggested Reading:
    

    • NoteBook: Split the Dataset Using Stratified K-Folds Cross-Validator
    • Blog: Hyperparameter Tuning the Random Forest in Python by Will Koehrsen
      • Jupyter NoteBook
        
    • Blog: Hyperparameter Optimization: Explanation of Automatized Algorithms by Dawid Kopczyk
      
      • Code (Python):
    • Blog: How to Use Random Seeds Effectively by Jai Bansal

    Additional Reading:
    

    • Blog: Nested Cross Validation Explained by Weina Jin
    • NoteBook: Cross Validation by Ritchie Ng
      
    • NoteBook: Cross Validation With Parameter Tuning Using Grid Search by Chris Albon
      
    • Blog: Random Test/Train Split is not Always Enough by Win-Vector
      
    • Slide: Cross-Validation: What, How and Which? by Pradeep Reddy Raamana
      
    • Paper: Algorithms for Hyper-Parameter Optimization (NIPS 2011) by J. Bergstra, R. Bardenet,Y. Bengio, and B. Kégl
      
    • Library: Yellowbrick (Machine Learning Visualization)
      • Learning Curve
      • Validation Curve
      • Double Descent by Arjun Ahuja
      • Deep Double Descent by Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever

    R (Programming Language):
    

    • Book: Machine Learning Mastery With R by Jason Brownlee
      
    • Blog: Resampling Methods by UC Business Analytics R Programming Guide
      
    • Blog: Linear Model Selection by UC Business Analytics R Programming Guide
      

14. Neural Networks

    Required Reading:
    

    • Chapter 20 of Understanding Machine Learning: From Theory to Algorithms
      
    • Slide: Neural Networks by Shai Shalev-Shwartz
      
    • Blog: 7 Types of Neural Network Activation Functions: How to Choose?
      
    • Blog: Activation Functions
    • Blog: Back-Propagation, an Introduction by Sanjeev Arora and Tengyu Ma
      

    Additional Reading:
    

    • Blog: The Gradient by Khanacademy
      
    • Blog: Activation Functions by Dhaval Dholakia
      
    • Paper: Why Does Deep & Cheap Learning Work So Well? by Henry W. Lin, Max Tegmark, and David Rolnick
      
    • Slide: Basics of Neural Networks by Connelly Barnes
      

    R (Programming Language):
    

    • Blog: Classification Artificial Neural Network by UC Business Analytics R Programming Guide
      

15. Convex Learning Problems

    Required Reading:
    

    • Chapter 12 of Understanding Machine Learning: From Theory to Algorithms
      
    • Slide: Machine Learning by Roland Kwitt
      

    Additional Reading:
    

    • Blog: Escaping from Saddle Points by Rong Ge
      

16. Regularization and Stability

    Required Reading:
    

    • Chapter 13 of Understanding Machine Learning: From Theory to Algorithms
      
    • Slide: Machine Learning by Roland Kwitt
      
    • Blog: L1 and L2 Regularization by Renu Khandelwal
      
    • Blog: L1 Norm Regularization and Sparsity Explained for Dummies by Shi Yan
      

    Additional Resources:
    

    • NoteBook: Regularization by Ethen
      

    R (Programming Language):
    

    • Book: Machine Learning Mastery With R by Jason Brownlee
      
    • Blog: Regularized Regression by UC Business Analytics R Programming Guide
      

17. Support Vector Machines

    Required Reading:
    

    • Chapter 15 of Understanding Machine Learning: From Theory to Algorithms
      
    • Slide: Support Vector Machines and Kernel Methods by Shai Shalev-Shwartz
      
    • Blog: Understanding the Mathematics behind Support Vector Machines by Nikita Sharma

    Additional Reading:
    

    • Infographic: Support Vector Machines (100 Days Of ML Code) by Avik Jain
      
      • Markdown (NoteBook)

    R (Programming Language):
    

    • Book: Machine Learning Mastery With R by Jason Brownlee
      
    • Blog: Support Vector Machine Classifier Implementation in R with Caret Package by Rahul Saxena
      
    • Blog: Support Vector Machine by UC Business Analytics R Programming Guide
      

18. Multiclass Classification

    Required Reading:
    

    • Chapter 17 of [Understanding Machine Learning: From Theory to Algorithms]
    • Slide: Machine Learning Basics Lecture 7: Multiclass Classification by Yingyu Liang

• Additional NoteBooks and Slides:

  • Course: Fondations of Machine Learning by David S. Rosenberg
    
  • Python Machine Learning Book Code Repository
    
  • Dive into Machine Learning
    
  • Python code for "An Introduction to Statistical Learning with Applications in R" by Jordi Warmenhoven
    
  • iPython-NoteBooks by John Wittenauer
    
  • Scikit-Learn Tutorial by Jake Vanderplas
    
  • Data Science Roadmap by Javier Estraviz
    

Class Time and Location:

Saturday, Monday, Wednesday 10:30-12:00 PM

Recitation and Assignments:

Monday and Wednesday 17:00-18:30 PM Refer to the following link to check the assignments.
Also, I recommend to study link of recitation and assignments of machine learning in 2020.

Projects:

Projects are programming assignments that cover the topic of this course. Any project is written by
Jupyter Notebook. Projects will require the use of Python 3.7, as well as
additional Python libraries as follows.

• Python 3.7: An interactive, object-oriented, extensible programming language.
  
• NumPy: A Python package for scientific computing.
  
• Pandas: A Python package for high-performance, easy-to-use data structures and data analysis tools.
  
• Scikit-Learn: A Python package for machine learning.
  
• Matplotlib: A Python package for 2D plotting.
  
• SciPy: A Python package for mathematics, science, and engineering.
  
• IPython: An architecture for interactive computing with Python.
  

Practical Guide:

• Slide: Practical Advice for Building Machine Learning Applications by Vivek Srikumar
  
• Blog: Comparison of Machine Learning Models by Kevin Markham
  

Fascinating Guide to Use Python Libraries (Machine Learning):

• Technical Notes On Using Data Science & Artificial Intelligence: To Fight For Something That Matters by Chris Albon

Google Colab:

Google Colab is a free cloud service and it supports free GPU!

• How to Use Google Colab by Souvik Mandal
• Primer for Learning Google Colab
• Deep Learning Development with Google Colab, TensorFlow, Keras & PyTorch

Latex:

The students can include mathematical notation within markdown cells using LaTeX in their Jupyter Notebooks.

• A Brief Introduction to LaTeX PDF
  
• Math in LaTeX PDF
  
• Sample Document PDF
  

Useful NoteBooks:

• Preparing and Cleaning Data for Machine Learning by Josh Devlin
• Getting Started with Kaggle: House Prices Competition by Adam Massachi
• Scikit-learn Tutorial: Machine Learning in Python by Satyabrata Pal

Grading:

• Homework – 30%
  — Will consist of mathematical problems and/or programming assignments.
• Midterm – 20%
• Endterm – 50%

Three Written Exams:

Midterm Examination: Saturday 1400/09/27, 10:30-12:00
Final Examination: Thursday 1400/11/07, 14:00-16:00

Prerequisites:

General mathematical sophistication; and a solid understanding of Algorithms, Linear Algebra, and Probability Theory, at the advanced undergraduate or beginning graduate level, or equivalent.

Linear Algebra:

• Video: Professor Gilbert Strang's Video Lectures on linear algebra.

Probability and Statistics:

• Learn Probability and Statistics Through Interactive Visualizations: Seeing Theory was created by Daniel Kunin while an undergraduate at Brown University. The goal of this website is to make statistics more accessible through interactive visualizations (designed using Mike Bostock’s JavaScript library D3.js).
• Statistics and Probability: This website provides training and tools to help you solve statistics problems quickly, easily, and accurately - without having to ask anyone for help.
• Jupyter NoteBooks: Introduction to Statistics by Bargava
• Video: Professor John Tsitsiklis's Video Lectures on Applied Probability.
• Video: Professor Krishna Jagannathan's Video Lectures on Probability Theory.

Discrete Mathematics:

Course (Videos, Lectures, Assignments): MIT OpenCourseWare (Discrete Mathematics)

Topics:

Have a look at some reports of Kaggle or Stanford students (CS224N, CS224D) to get some general inspiration.

Account:

It is necessary to have a GitHub account to share your projects. It offers plans for both private repositories and free accounts. Github is like the hammer in your toolbox, therefore, you need to have it!

Academic Honor Code:

Honesty and integrity are vital elements of the academic works. All your submitted assignments must be entirely your own (or your own group's).

We will follow the standard of Department of Mathematical Sciences approach:

• You can get help, but you MUST acknowledge the help on the work you hand in
• Failure to acknowledge your sources is a violation of the Honor Code
• You can talk to others about the algorithm(s) to be used to solve a homework problem; as long as you then mention their name(s) on the work you submit
• You should not use code of others or be looking at code of others when you write your own: You can talk to people but have to write your own solution/code

Questions?

I will be having office hours for this course on Monday (09:30 AM--12:00 AM). If this is not convenient, email me at hhaji@sbu.ac.ir or talk to me after class.