A Recent History of Deep Learning

1 minute read

In this post, we take a look at a timeline of recent innovations in deep neural networks that have led to its popularity and widespread adoption. The list is based on the the review paper Deep Learning published in Nature (2015) by Yann Le Cun, Yoshua Bengio, and Geoffrey Hinton. This is not a comprehensive list, but it will give an insight into when some of the now familiar concepts in deep learning evolved.


  • Paper on backpropagation by Rumelhart, Hinton, and Williams, introduces this key concept that is now the norm in deep neural networks


  • Convolutional nets are used for handwriting classification by LeCun et al. at AT&T labs


  • Hochreiter and Schmidhuber publish the paper that introduces LSTM recurrent networks that are able to learn long-range dependencies in sequences


  • The use of GPUs is shown to improve computational speed of deep neural networks compared to CPUs


  • Paper on Rectified Linear Units (ReLu) as an alternative to the sigmoid activation function for fast training of neural networks by Glorot, Bordes and Begio at the University of Montreal


  • Deep convolutional nets are used to classify images in the ImageNet database, reducing error rates by almost half, in a landmark paper published by Krizhevsky, Sutskever and Hinton at the University of Toronto. This was made possible by the efficient use of GPUs, ReLu, and dropout

  • Deep Belief Networks (DBNs) are shown to exceed existing benchmarks for speech recognition on a small and large vocabulary

  • The above papers have caused a shift towards deep learning in the fields of computer vision and speech recognition and have led to its widespread adoption in those fields and general popularity


  • Dropout as a form of regularization for neural networks to prevent it from overfitting published by the Hinton lab

  • Neural Turing machines allow neural networks to learn to read from and write to memory, published by Graves et. al.


Leave a Comment