10-715 Fall 2021: Advanced Introduction to Machine Learning
This course is designed for Ph.D. students whose primary field of study is machine learning, or who intend to make machine learning methodological research a main focus of their thesis. It will give students a thorough grounding in the algorithms, mathematics, theories, and insights needed to do in-depth research and applications in machine learning. The topics of this course will in part parallel those covered in the general graduate machine learning course (10-701), but with a greater emphasis on depth in theory and algorithms. The course will also include additional recent topics such as fairness in machine learning.
IMPORTANT NOTE: Students entering the class are expected to have a pre-existing strong working knowledge of linear algebra (e.g., mathematical representation of subspaces, singular value decomposition), probability (e.g., multivariate Gaussians, Bayes' rule, conditional expectation), and calculus (e.g., derivative of a vector with respect to another vector), and programming (we'll mostly be supporting Python). This class is best for you if you have machine learning at the
CORE of your studies/research, and want to understand the fundamentals. This class will be
HEAVY and will move
FAST. If machine learning is an auxiliary component of your studies/research or if you do not have the required background, you may consider the general graduate Machine Learning course (10-701) or the Masters-level Machine Learning course (10-601).
Click here for an ML course comparison. Note that machine learning itself is NOT a prerequisite for this course. Also note that by departmental policy, this course is open only to graduate students (5th year masters are allowed).
Waitlist: If you are in the waitlist, meet the aforementioned requisites, please send an email to the instructor and cc Diane Stidle (stidle@andrew.cmu.edu) outlining how you meet them (e.g., some courses you took on these topics etc.) if you haven't already done so. We will keep clearing the waitlist if some students drop the course (which often happens in the first two weeks of classes when students are still shopping around and deciding the classes that best fit their needs).
Time and location: The class is scheduled for Monday and Wednesday 10.10am to 11.30am in the Singleton room in the Roberts Engineering Hall. We will reserve Fridays for recitations, which will be held 10.10am to 11.30am in the same location. Note that the recitation will not take place every Friday: please see the schedule below.
Units: 12
Instructor: Nihar B. Shah
Textbook: [SB] Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz and Shai Ben-David (
available online)
Last year's videos: The course content is similar to the offering last year. You can find last year's videos
here.
Course staff and contact details:
Nihar Shah: nihars at cs.cmu.edu Office hours by appointment. To set up a meeting, please send Nihar an email with your availability, as well as the topics you would like to discuss (e.g., specific lectures or paper dissection choice).
Filipe de Avila Belbute Peres
: filiped at andrew.cmu.edu Office hours details to be posted on Diderot
Qiqi Xu
: qiqixu at andrew.cmu.edu Office hours details to be posted on Diderot
Saurabh Garg
: sgarg2 at andrew.cmu.edu Office hours details to be posted on Diderot
Senyu Tong
: senyut at andrew.cmu.edu Office hours details to be posted on Diderot
Syllabus and tentative schedule (subject to change): Note that most lectures will be taught on the "board'', and hence there are no "slides''.
| Date | Topic | References/Remarks |
| PART I: DEPTH |
| Aug 30 | Logistics and introduction to introduction to ML | SB chapter 2 |
| Sep 1 | Perceptrons: Hope, hopelessness, and hope again | SB chapter 9 |
| Sep 6 | No class (labor day) |
| Sep 8 | Optimization for ML | Notes |
| Sep 10 | Recitation: Optimization |
| Sep 13 | Support vector machines | SB chapter 15 |
| Sep 15 | Kernel methods 1 | SB chapter 16 |
| Sep 17 | Recitation: Tail bounds |
| Sep 20 | Kernel methods 2 | SB chapter 16 |
| Sep 22 | Learning theory 1 | SB Chapters 2 - 5 |
| Sep 24 | Recitation: Linear regression, Logistic regression |
| Sept 27 | Recitation: MLE and MAP [Note: This is a recitation on a Monday since Nihar is giving a tutorial at the same time] |
| Sep 29 | Learning theory 3 | SB Chapters 2 - 6 |
| Oct 1 | Learning theory 2 [Note: This is a regular lecture on Friday to make up for Monday] | SB Chapters 2 - 6 |
| Oct 4 | Learning theory 4 | SB Chapters 6 - 7 |
| Oct 6 | Midterm | All material in previous lectures |
| Oct 8 | Recitation: Rademacher Complexity |
| PART II: BREADTH |
| Oct 11 | Neural networks 1: Introduction. Also, midterm discussion. | SB Chapter 20 |
| Oct 13 | Neural networks 2: Representation power |
| Oct 18 | Neural networks 3: Training, automatic differentiation, CNNs, ResNet etc. |
| Oct 20 | Theory paper dissection |
| Oct 25 | Model complexity, cross-validation bias-variance tradeoff, interpolation regime, and Neural networks 4 (neural architecture search) | |
| Oct 27 | Decision trees, random forests, bagging, bootstrapping | SB Chapter 18 |
| Nov 1 | Unsupervised learning: Clustering | SB Chapter 22 |
| Nov 3 | Dimensionality reduction | SB Chapter 23 |
| Nov 8 | Boosting | SB Chapter 10 |
| Nov 10 | Online learning | SB Chapter 21 |
| Nov 15 | Semi-supervised learning, Active learning, Multi-armed bandits | Transductive SVM, Active learning, Multi-armed bandits, Ranking via MABs |
| Nov 17 | Reinforcement learning | Survey |
| Nov 22 | Graphical models, Causality | Graphical models |
| Nov 24 | No class (Thanksgiving break) |
| Nov 29 | Fairness, interpretability, explanability | Hiring example, Paper 1, Paper 2 |
| Dec 1 | Applied paper dissection |
Accommodations for Students with Disabilities:
If you have a disability and have an accommodations letter from the Disability Resources office, I encourage you to discuss your accommodations and needs with me as early in the semester as possible. I will work with you to ensure that accommodations are provided as appropriate. If you suspect that you may have a disability and would benefit from accommodations but are not yet registered with the Office of Disability Resources, I encourage you to contact them at access@andrew.cmu.edu.
Finally, to all students: Please take care! The COVID pandemic can lead to stress and uncertainty and can take its toll on our mental health. Make sure to move regularly, eat well, and reach out to your support system or
me if you need to. We can all benefit from support in times of stress, and this semester is no exception.