In this post, we take a look at a timeline of recent innovations in deep neural networks that have led to its popularity and widespread adoption. The list is based on the the review paper Deep Learning published in Nature (2015) by Yann Le Cun, Yoshua Bengio, and Geoffrey Hinton. This is not a comprehensive list, but it will give an insight into when some of the now familiar concepts in deep learning evolved.
- Paper on backpropagation by Rumelhart, Hinton, and Williams, introduces this key concept that is now the norm in deep neural networks
- Convolutional nets are used for handwriting classification by LeCun et al. at AT&T labs
- Hochreiter and Schmidhuber publish the paper that introduces LSTM recurrent networks that are able to learn long-range dependencies in sequences
- The use of GPUs is shown to improve computational speed of deep neural networks compared to CPUs
- Paper on Rectified Linear Units (ReLu) as an alternative to the sigmoid activation function for fast training of neural networks by Glorot, Bordes and Begio at the University of Montreal
Deep convolutional nets are used to classify images in the ImageNet database, reducing error rates by almost half, in a landmark paper published by Krizhevsky, Sutskever and Hinton at the University of Toronto. This was made possible by the efficient use of GPUs, ReLu, and dropout
The above papers have caused a shift towards deep learning in the fields of computer vision and speech recognition and have led to its widespread adoption in those fields and general popularity