Blog: Historical Development of Neural Network Principles
The key developments in neural network principles are outlined in
In 1943 Warren McCulloch and Walter Pitts proposed a model of
computing element, called McCulloch-Pitts neuron, which performs a
weighted sum of the inputs to the element followed by a threshold
logic operation [McCulloch and Pitts, 19431. Combinations of these
computing elements were used to realize several logical computations.
The main drawback of this model of computation is that the weights
are fixed and hence the model could not learn fiom examples.
In 1949 Donald Hebb proposed a learning scheme for adjusting
a connection weight based on pre- and post-synaptic values of the
variables [Hebb, 19491. Hebb’s law became a fundamental learning
rule in neural networks literature.
In 1954 a learning machine was developed by Marvin Minsky,
in which the connection strengths could be adapted automatically
Historical Development of Neural Network Principles
[Minsky, 19541. But it was in 1958 that Rosenblatt proposed the
perceptron model, which has weights adjustable by the perceptron
learning law [Rosenblatt, 19581. The learning law was shown to
converge for pattern classification problems, which are linearly
separable in the feature space. While a single layer of perceptrons
could handle only linearly separable classes, it was shown that a
multilayer perceptron could be used to perform any pattern
classification task. But there was no systematic learning algorithm
to adjust the weights to realize the classification task. In 1969 Minsky
and Papert demonstrated the limitations of the perceptron model
through several illustrative examples [Minsky and Papert, 19691.
Lack of suitable learning law for a multilayer perceptron network
had put brakes on the development of neural network models for
pattern recognition tasks for nearly 15 years till 1984.\
In 1960s Widrow and his group proposed an Adaline model for a
eomputing element and an LMS learning algorithm to adjust the
weights of an Adaline model Widrow and Hoff, 19601. The
convergence of the LMS algorithm was proved. The algorithm was
successfully used for adaptive signal processing situations.
The resurgence of interest in artificial neural networks is due to
two key developments in early 1980s. The first one is the energy
analysis of feedback neural networks by John Hopfield, published in
1982 and 1984 [Hopfield, 1982; Hopfield, 19841. The analysis has
shown the existence of stable equilibrium states in a feedback
network, provided that the network has symmetric weights, and that
the state update is made asynchronously. Also, in 1986, Rumelhart
et al have shown that it is possible to adjust the weights of a
multilayer feedforward neural network in a systematic way to learn
the implicit mapping in a set of input-output pattern pairs
[Rumelhart et al, 1986al. The learning law is called generalized delta
rule or error backpropagation learning law.
About the same time Ackley, Hinton and Sejnowski proposed the
Boltzmann machine which is a feedback neural network with
stochastic neuron units [Ackley et al, 19851. A stochastic neuron has
an output function yrhich is implemented using a probabilistic update
rule instead of a deterministic update rule as in the Hopfield model.
Moreover, , the Boltzmann machine has several additional neuron
units, called hidden units, which are used to make a given pattern
storage problem representable in a feedback network.
Besides these key developments, there are many other significant
contributions made in this field during the past thirty years. Notable
among them are the concepts of competitive learning, selforganization
and simulated annealing. Self-organization led to the
realization of feature mapping. Simulated annealing has been very
useful in implementing the learning law for the Boltzmann machine.
Several new learning laws were also developed, the prominent among
Basics of Artificial Neural Networks
them being the reinforcement learning or learning with critic. Several
architectures were developed to address specific issues in pattern
recognition. Some of these architectures are: adaptive resonance
theory (ART), neocognitron and counterpropagation networks.
Currently, fuzzy logic concepts are being used to enhance the
capability of the neural networks to deal with real world problems
such as in speech, image processing, natural language processing and
decision making [Lin and Lee, 19961.