# A Brief Timeline Deep learning actually has a long and storied history. It's had different monikers like **connectionism** and **cybernetics** and gone through springs and winters until its most resurgence. Deep learning can trace its roots back all the way to the early 1940s when connectionism and artificial neurons piqued the interest of researchers and the public. The following decade saw a rise in enthusiasm and the field is established. The '70s saw the enthusiasm fade as the promised artificial intelligence and automation failed to live up to the hype. The late '80s and '90s saw a resurgence in the field, but new and improved models cannot be fully realized due to the computational costs. The digitization of the 2000s and continual improvement in processing power usher us to the modern times where large datasets and massive computing capabilities result in one of the most productive decades in deep learning research and applications. Following is a sampling of some of the notable events in the DL timeline: ## 1940's * **1943:** Warren McCulloch and Walter Pitts introduce the idea of mathematically modeling biological neurons, the artificial neuron in [**"**A logical calculus of the ideas immanent in nervous activity"](https://link.springer.com/article/10.1007/BF02478259) * **1949:** Claude E Shannon connects [switching circuits](https://ieeexplore.ieee.org/abstract/document/6771698?casa_token=sZTF_NCoEwsAAAAA:wpyvZsJMFJs70dQURHKYEG_pNxp-sYWBjKP7FZhyNEhDeEzTSysJlEKJJ48o5TEjzyzvwxCElQ) and artificial neurons ## 1950's * **1957:** Frank Rosenblatt invents the perceptron, the building block of modern neural networks. It is a linear classifier, unlike the modern feed-forward neural networks with non-linear activations. It was of course not implemented via software but as an actual [machine!](https://blogs.umass.edu/brain-wars/1957-the-birth-of-cognitive-science/the-perceptron-a-perceiving-and-recognizing-automaton/) ## 1960's * **1969:** Marvin Minsky and Seymour Papert published their book, [Perceptron: an introduction to computational geometry](https://direct.mit.edu/books/book/3132/PerceptronsAn-Introduction-to-Computational), highlighting the limitation of perceptrons. The inapplicability of primitive binary perceptrons to real-world tasks halts the enthusiasm for artificial neurons and ushers in the AI Winter that halts progress in the field for a decade. ## 1970's * **1970:** Seppo Linnainmaa publishes his work on automatic differentiation on differentiable compute graphs, a direct pre-cursor to our modern backpropagation-based systems. ## 1980's * **1982:** [Recurrent neural networks](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC346238/) (RNNs) are introduced to tackle series data and to enable memory to artificial neural networks. * **1986:** [Backpropagation is used on networks of "neurons-like units"](https://www.nature.com/articles/323533a0). The hidden units of neurons are automatically updated via backprop and automatically learn representations of the input data. * **1988: Neocognitron**, by Kuniihiko Fukushima uses hierarchical neural networks to perform visual pattern recognition tasks. ## 1990's * **1997:** [**Long Short Term Memory** (LSTMs) RNN](https://direct.mit.edu/neco/article-abstract/9/8/1735/6109/Long-Short-Term-Memory?redirectedFrom=fulltext)s are introduced, to solve the inefficiencies in RNNs for long time ranges. This results in deep learning-based models being practical for time-series problems. * **1998:** The MNIST database, comprising of a set of images handwritten digits, is released. * **1998:** Yann LeCun proposed, **LeNet-5,** a multi-layer deep neural network composed of convolutional neural networks (CNNs) and multi-layer-perceptrons trained via backpropagation. ## 2000's * **2006:** Non-linear dimensionality reduction is performed using multi-layer neural networks called [**Autoencoders**](https://www.science.org/doi/10.1126/science.1127647) using gradient descent and backpropagation. * **2006: Deep Belief Network** training is greatly simplified, and is used to predict the MNIST dataset. * **2009:** The [**ImageNet**](https://www.image-net.org/) project is released. One of the largest hand-annotated image databases, the database and the associated Imagenet Large Scale Visual Recognition Challenge (ILSVRC) become a proving ground for computer vision models. ## 2010's * **2012:** [**AlexNet,** by Alex Krizhevsky et al](https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf), wins the ILSVRC challenge, beating all traditional computer vision and ML-based approaches. With 60 million parameters, 650,000 neurons, and trained on multiple GPUs, AlexNet was one of the first cases of large deep learning models becoming computationally feasible thanks to hardware accelerators such as GPUs. * **2015:** [A 152-layer **residual convolutional neural network** (ResNet-152)](https://arxiv.org/abs/1512.03385) exceeds human-level performance on the ILSVRC. ## And beyond... There has been a Cambrian explosion in the research and application of deep learning starting from the early 2010s. Larger datasets, better models, and increased computing power has resulted in a boom in the research and application of deep learning methods. We've seen ubiquitous adoption of DL models in computer vision, machine translation, and natural language processing. DL methods are revolutionizing scientific domains by significantly reducing computation and simulation times. Improvements in learning algorithms have allowed us to train models with billions and trillions of parameters! In the following chapters, we'll take a look at the building blocks of these powerful models and how they are used to make some of the most powerful models. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://szaman.gitbook.io/intro-to-deep-learning/a-brief-timeline.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.