Neural networks and deep learning

Acknowledgements

Neural Networks and Deep Learning

What this book is about

On the exercises and problems

Using neural nets to recognize handwritten digits

Sigmoid neurons

The architecture of neural networks

A simple network to classify handwritten digits

Learning with gradient descent

Implementing our network to classify digits

Toward deep learning

How the backpropagation algorithm works

Warm up: a fast matrix-based approach to computing the output from a neural network

The two assumptions we need about the cost function

The Hadamard product, $s \odot t$

The four fundamental equations behind backpropagation

Proof of the four fundamental equations (optional)

The backpropagation algorithm

The code for backpropagation

In what sense is backpropagation a fast algorithm?

Backpropagation: the big picture

Improving the way neural networks learn

The cross-entropy cost function

Overfitting and regularization

Weight initialization

Handwriting recognition revisited: the code

How to choose a neural network's hyper-parameters?

Other techniques

A visual proof that neural nets can compute any function

Universality with one input and one output

Many input variables

Extension beyond sigmoid neurons

Fixing up the step functions

Why are deep neural networks hard to train?

The vanishing gradient problem

What's causing the vanishing gradient problem? Unstable gradients in deep neural nets

Unstable gradients in more complex networks

Other obstacles to deep learning

Introducing convolutional networks

Convolutional neural networks in practice

The code for our convolutional networks

Recent progress in image recognition

Other approaches to deep neural nets

On the future of neural networks

Appendix: Is there a simple algorithm for intelligence?

Acknowledgements

Frequently Asked Questions

If you benefit from the book, please make a small donation. I suggest 5,ドル but you can choose the amount.

Alternately, you can make a donation by sending me Bitcoin, at address 1Kd6tXH5SDAmiFb49J9hknG5pqj7KStSAx

Sponsors

Deep Learning Workstations, Servers, and Laptops

Thanks to all the supporters who made the book possible, with especial thanks to Pavel Dudrenov. Thanks also to all the contributors to the Bugfinder Hall of Fame.

Resources

Michael Nielsen on Twitter

Code repository

Michael Nielsen's project announcement mailing list

Deep Learning, book by Ian Goodfellow, Yoshua Bengio, and Aaron Courville

cognitivemedium.com

By Michael Nielsen / Dec 2019

The book grew out of a set of notes I prepared for an online study group on neural networks and deep learning. Many thanks to all the participants in that study group: Paul Bloore, Chris Dawson, Andrew Doherty, Ilya Grigorik, Alex Kosorukoff, Chris Olah, and Rob Spekkens. I learned a lot from all of you. I am particularly grateful to Rob, for providing so many insightful questions and ideas, and to Chris, who has continued to share his rapidly expanding knowledge of neural networks. Thanks also to Yoshua Bengio, who read and provided feedback on a chapter.

In academic work, please cite this book as: Michael A. Nielsen, "Neural Networks and Deep Learning", Determination Press, 2015

This work is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License. This means you're free to copy, share, and build on this book, but not to sell it. If you're interested in commercial use, please contact me. Last update: Thu Dec 26 15:26:33 2019

Creative Commons Licence