Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

rasbt/deeplearning-models

Repository files navigation

Deep Learning Models

A collection of various deep learning architectures, models, and tips for TensorFlow and PyTorch in Jupyter Notebooks.

Traditional Machine Learning

Title Dataset Description Notebooks
Perceptron 2D toy data TBD PyTorch TensorFlow
Logistic Regression 2D toy data TBD PyTorch TensorFlow
Softmax Regression (Multinomial Logistic Regression) MNIST TBD PyTorch TensorFlow
Softmax Regression with MLxtend's plot_decision_regions on Iris Iris TBD PyTorch

Multilayer Perceptrons

Title Dataset Description Notebooks
Multilayer Perceptron MNIST TBD PyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Dropout MNIST TBD PyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Batch Normalization MNIST TBD PyTorch Lightning PyTorch TensorFlow
Multilayer Perceptron with Backpropagation from Scratch MNIST TBD PyTorch TensorFlow

Convolutional Neural Networks

Basic

Title Dataset Description Notebooks
Convolutional Neural Network TBD TBD PyTorch Lightning PyTorch TensorFlow
CNN with He Initialization TBD TBD PyTorch Lightning PyTorch

Concepts

Title Dataset Description Notebooks
Replacing Fully-Connected by Equivalent Convolutional Layers TBD TBD PyTorch

AlexNet

Title Dataset Description Notebooks
AlexNet Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch
AlexNet with Grouped Convolutions Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch

DenseNet

Title Description Daset Notebooks
DenseNet-121 Digit Classifier Trained on MNIST TBD TBD PyTorch Lightning PyTorch
DenseNet-121 Image Classifier Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch

Fully Convolutional

Title Dataset Description Notebooks
"All Convolutionl Net" -- A Fully Convolutional Neural Network TBD TBD PyTorch Lightning PyTorch

LeNet

Title Dataset Description Notebooks
LeNet-5 on MNIST TBD TBD PyTorch Lightning PyTorch
LeNet-5 on CIFAR-10 TBD TBD PyTorch Lightning PyTorch
LeNet-5 on QuickDraw TBD TBD PyTorch Lightning PyTorch

MobileNet

Title Dataset Description Notebooks
MobileNet-v2 on Cifar-10 TBD TBD PyTorch Lightning PyTorch
MobileNet-v3 small on Cifar-10 TBD TBD PyTorch Lightning PyTorch
MobileNet-v3 large on Cifar-10 TBD TBD PyTorch Lightning PyTorch
MobileNet-v3 large on MNIST via Embetter TBD TBD PyTorch

Network in Network

Title Dataset Description Notebooks
Network in Network Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch

VGG

Title Dataset Description Notebooks
Convolutional Neural Network VGG-16 Trained on CIFAR-10 TBD TBD PyTorch Lightning PyTorch TensorFlow
VGG-16 Smile Classifier CelebA TBD PyTorch Lightning PyTorch
VGG-16 Dogs vs Cats Classifier TBD TBD PyTorch
Convolutional Neural Network VGG-19 TBD TBD PyTorch Lightning PyTorch

ResNet

Title Dataset Description Notebooks
ResNet and Residual Blocks MNIST TBD PyTorch
ResNet-18 Digit Classifier MNIST TBD PyTorch
ResNet-18 Gender Classifier CelebA TBD PyTorch
ResNet-34 Digit Classifier MNIST TBD PyTorch
ResNet-34 Object Classifier QuickDraw TBD PyTorch
ResNet-34 Gender Classifier CelebA TBD PyTorch
ResNet-50 Digit Classifier MNIST TBD PyTorch
ResNet-50 Gender Classifier CelebA TBD PyTorch
ResNet-101 Gender Classifier CelebA TBD PyTorch
ResNet-101 CIFAR-10 TBD PyTorch
ResNet-152 Gender Classifier CelebA TBD PyTorch

Transformers

Title Dataset Description Notebooks
Multilabel DistilBERT Jigsaw Toxic Comment Challenge DistilBERT classifier fine-tuning PyTorch
DistilBERT as feature extractor IMDB movie review DistilBERT classifier with sklearn random forest and logistic regression PyTorch
DistilBERT as feature extractor using embetter IMDB movie review DistilBERT classifier with sklearn random forest and logistic regression using the scikit-learn embetter library PyTorch
Fine-tune DistilBERT I IMDB movie review Fine-tune only the last 2 layers of DistilBERT classifier PyTorch Lightning
Fine-tune DistilBERT II IMDB movie review Fine-tune the whole DistilBERT classifier PyTorch PyTorch Lightning

Ordinal Regression and Deep Learning

Please note that the following notebooks below provide reference implementations to use the respective methods. They are not performance benchmarks.

Title Dataset Description Notebooks
Baseline multilayer perceptron Cement A baseline multilayer perceptron for classification trained with the standard cross entropy loss PyTorch PyTorch Lightning
CORAL multilayer perceptron Cement Implementation of Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation 2020 PyTorch PyTorch Lightning
CORN multilayer perceptron Cement Implementation of Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities 2022 PyTorch PyTorch Lightning
Binary extension multilayer perceptron Cement Implementation of Ordinal Regression with Multiple Output CNN for Age Estimation 2016 PyTorch PyTorch Lightning
Reformulated squared-error multilayer perceptron Cement Implementation of A simple squared-error reformulation for ordinal classification 2016 PyTorch PyTorch Lightning
Class distance weighted cross-entropy loss Cement Implementation of Class Distance Weighted Cross-Entropy Loss for Ulcerative Colitis Severity Estimation 2022 PyTorch PyTorch Lightning

Normalization Layers

Title Dataset Description Notebooks
BatchNorm before and after Activation for Network-in-Network CIFAR-10 Classifier TBD TBD PyTorch
Filter Response Normalization for Network-in-Network CIFAR-10 Classifier TBD TBD PyTorch

Metric Learning

Title Dataset Description Notebooks
Siamese Network with Multilayer Perceptrons TBD TBD TensorFlow

Autoencoders

Fully-connected Autoencoders

Title Dataset Description Notebooks
Autoencoder (MNIST) TBD TBD PyTorch TensorFlow
Autoencoder (MNIST) + Scikit-Learn Random Forest Classifier TBD TBD PyTorch TensorFlow

Convolutional Autoencoders

Title Dataset Description Notebooks
Convolutional Autoencoder with Deconvolutions / Transposed Convolutions TBD TBD PyTorch TensorFlow
Convolutional Autoencoder with Deconvolutions and Continuous Jaccard Distance TBD TBD PyTorch
Convolutional Autoencoder with Deconvolutions (without pooling operations) TBD TBD PyTorch
Convolutional Autoencoder with Nearest-neighbor Interpolation TBD TBD PyTorch TensorFlow
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on CelebA TBD TBD PyTorch
Convolutional Autoencoder with Nearest-neighbor Interpolation -- Trained on Quickdraw TBD TBD PyTorch

Variational Autoencoders

Title Dataset Description Notebooks
Variational Autoencoder TBD TBD PyTorch
Convolutional Variational Autoencoder TBD TBD PyTorch

Conditional Variational Autoencoders

Title Dataset Description Notebooks
Conditional Variational Autoencoder (with labels in reconstruction loss) TBD TBD PyTorch
Conditional Variational Autoencoder (without labels in reconstruction loss) TBD TBD PyTorch
Convolutional Conditional Variational Autoencoder (with labels in reconstruction loss) TBD TBD PyTorch
Convolutional Conditional Variational Autoencoder (without labels in reconstruction loss) TBD TBD PyTorch

Generative Adversarial Networks (GANs)

Title Dataset Description Notebooks
Fully Connected GAN on MNIST TBD TBD PyTorch TensorFlow
Fully Connected Wasserstein GAN on MNIST TBD TBD PyTorch
Convolutional GAN on MNIST TBD TBD PyTorch TensorFlow
Convolutional GAN on MNIST with Label Smoothing TBD TBD PyTorch TensorFlow
Convolutional Wasserstein GAN on MNIST TBD TBD PyTorch
Deep Convolutional GAN (DCGAN) on Cats and Dogs Images TBD TBD PyTorch
Deep Convolutional GAN (DCGAN) on CelebA Face Images TBD TBD PyTorch

Graph Neural Networks (GNNs)

Title Dataset Description Notebooks
Most Basic Graph Neural Network with Gaussian Filter on MNIST TBD TBD PyTorch
Basic Graph Neural Network with Edge Prediction on MNIST TBD TBD PyTorch
Basic Graph Neural Network with Spectral Graph Convolution on MNIST TBD TBD PyTorch

Recurrent Neural Networks (RNNs)

Many-to-one: Sentiment Analysis / Classification

Title Dataset Description Notebooks
A simple single-layer RNN (IMDB) TBD TBD PyTorch
A simple single-layer RNN with packed sequences to ignore padding characters (IMDB) TBD TBD PyTorch
RNN with LSTM cells (IMDB) TBD TBD PyTorch
RNN with LSTM cells (IMDB) and pre-trained GloVe word vectors TBD TBD PyTorch
RNN with LSTM cells and Own Dataset in CSV Format (IMDB) TBD TBD PyTorch
RNN with GRU cells (IMDB) TBD TBD PyTorch
Multilayer bi-directional RNN (IMDB) TBD TBD PyTorch
Bidirectional Multi-layer RNN with LSTM with Own Dataset in CSV Format (AG News) TBD TBD PyTorch

Many-to-Many / Sequence-to-Sequence

Title Dataset Description Notebooks
A simple character RNN to generate new text (Charles Dickens) TBD TBD PyTorch

Model Evaluation

K-Fold Cross-Validation

Title Dataset Description Notebooks
Baseline CNN MNIST A simple baseline with traditional train/validation/test splits PyTorch PyTorch Lightning
K-fold with pl_cross MNIST A 5-fold cross-validation run using the pl_cross library PyTorch Lightning

Data Augmentation

Title Dataset Description Notebooks
AutoAugment & TrivialAugment for Image Data CIFAR-10 Trains a ResNet-18 using AutoAugment and TrivialAugment PyTorch Lightning

Tips and Tricks

Title Dataset Description Notebooks
Cyclical Learning Rate TBD TBD PyTorch
Annealing with Increasing the Batch Size (w. CIFAR-10 & AlexNet) TBD TBD PyTorch
Gradient Clipping (w. MLP on MNIST) TBD TBD PyTorch

Transfer Learning

Title Dataset Description Notebooks
Transfer Learning Example (VGG16 pre-trained on ImageNet for Cifar-10) TBD TBD PyTorch

Visualization and Interpretation

Title Dataset Description Notebooks
Vanilla Loss Gradient (wrt Inputs) Visualization (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) TBD TBD PyTorch
Guided Backpropagation (Based on a VGG16 Convolutional Neural Network for Kaggle's Cats and Dogs Images) TBD TBD PyTorch

PyTorch Workflows and Mechanics

PyTorch Lightning Examples

Title Dataset Description Notebooks
MLP in Lightning with TensorBoard -- continue training the last model TBD TBD PyTorch
MLP in Lightning with TensorBoard -- checkpointing best model TBD TBD PyTorch

Custom Datasets

Title Dataset Description Notebooks
Custom Data Loader Example for PNG Files TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- CSV files converted to HDF5 TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Face Images from CelebA TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from Quickdraw TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Drawings from the Street View House Number (SVHN) Dataset TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Asian Face Dataset (AFAD) TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Dating Historical Color Images TBD TBD PyTorch
Using PyTorch Dataset Loading Utilities for Custom Datasets -- Fashion MNIST TBD TBD PyTorch

Training and Preprocessing

Title Dataset Description Notebooks
PyTorch DataLoader State and Nested Iterations Toy Explains DataLoader behavior when in nested functions PyTorch
Generating Validation Set Splits TBD TBD PyTorch
Dataloading with Pinned Memory TBD TBD PyTorch
Standardizing Images TBD TBD PyTorch
Image Transformation Examples TBD TBD PyTorch
Char-RNN with Own Text File TBD TBD PyTorch
Sentiment Classification RNN with Own CSV File TBD TBD PyTorch

Improving Memory Efficiency

Title Dataset Description Notebooks
Gradient Checkpointing Demo (Network-in-Network trained on CIFAR-10) TBD TBD PyTorch

Parallel Computing

Title Description Notebooks
Using Multiple GPUs with DataParallel -- VGG-16 Gender Classifier on CelebA TBD PyTorch
Distribute a Model Across Multiple GPUs with Pipeline Parallelism (VGG-16 Example) TBD PyTorch

Other

Title Dataset Description Notebooks
PyTorch with and without Deterministic Behavior -- Runtime Benchmark TBD TBD PyTorch
Sequential API and hooks TBD TBD PyTorch
Weight Sharing Within a Layer TBD TBD PyTorch
Plotting Live Training Performance in Jupyter Notebooks with just Matplotlib TBD TBD PyTorch

Autograd

Title Dataset Description Notebooks
Getting Gradients of an Intermediate Variable in PyTorch TBD TBD PyTorch

TensorFlow Workflows and Mechanics

Custom Datasets

Title Description Notebooks
Chunking an Image Dataset for Minibatch Training using NumPy NPZ Archives TBD TensorFlow
Storing an Image Dataset for Minibatch Training using HDF5 TBD TensorFlow
Using Input Pipelines to Read Data from TFRecords Files TBD TensorFlow
Using Queue Runners to Feed Images Directly from Disk TBD TensorFlow
Using TensorFlow's Dataset API TBD TensorFlow

Training and Preprocessing

Title Dataset Description Notebooks
Saving and Loading Trained Models -- from TensorFlow Checkpoint Files and NumPy NPZ Archives TBD TBD TensorFlow

Related Libraries

Title Description Notebooks
TorchMetrics How do we use it, and what's the difference between .update() and .forward()? PyTorch

About

A collection of various deep learning architectures, models, and tips

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 10

Languages

AltStyle によって変換されたページ (->オリジナル) /