## Blog: Historical Development of Neural Network Principles

The key developments in neural network principles are outlined in

this section.

In 1943 Warren McCulloch and Walter Pitts proposed a model of

computing element, called McCulloch-Pitts neuron, which performs a

weighted sum of the inputs to the element followed by a threshold

logic operation [McCulloch and Pitts, 19431. Combinations of these

computing elements were used to realize several logical computations.

The main drawback of this model of computation is that the weights

are fixed and hence the model could not learn fiom examples.

In 1949 Donald Hebb proposed a learning scheme for adjusting

a connection weight based on pre- and post-synaptic values of the

variables [Hebb, 19491. Hebb’s law became a fundamental learning

rule in neural networks literature.

In 1954 a learning machine was developed by Marvin Minsky,

in which the connection strengths could be adapted automatically

Historical Development of Neural Network Principles

[Minsky, 19541. But it was in 1958 that Rosenblatt proposed the

perceptron model, which has weights adjustable by the perceptron

learning law [Rosenblatt, 19581. The learning law was shown to

converge for pattern classification problems, which are linearly

separable in the feature space. While a single layer of perceptrons

could handle only linearly separable classes, it was shown that a

multilayer perceptron could be used to perform any pattern

classification task. But there was no systematic learning algorithm

to adjust the weights to realize the classification task. In 1969 Minsky

and Papert demonstrated the limitations of the perceptron model

through several illustrative examples [Minsky and Papert, 19691.

Lack of suitable learning law for a multilayer perceptron network

had put brakes on the development of neural network models for

pattern recognition tasks for nearly 15 years till 1984.\

In 1960s Widrow and his group proposed an Adaline model for a

eomputing element and an LMS learning algorithm to adjust the

weights of an Adaline model Widrow and Hoff, 19601. The

convergence of the LMS algorithm was proved. The algorithm was

successfully used for adaptive signal processing situations.

The resurgence of interest in artificial neural networks is due to

two key developments in early 1980s. The first one is the energy

analysis of feedback neural networks by John Hopfield, published in

1982 and 1984 [Hopfield, 1982; Hopfield, 19841. The analysis has

shown the existence of stable equilibrium states in a feedback

network, provided that the network has symmetric weights, and that

the state update is made asynchronously. Also, in 1986, Rumelhart

et al have shown that it is possible to adjust the weights of a

multilayer feedforward neural network in a systematic way to learn

the implicit mapping in a set of input-output pattern pairs

[Rumelhart et al, 1986al. The learning law is called generalized delta

rule or error backpropagation learning law.

About the same time Ackley, Hinton and Sejnowski proposed the

Boltzmann machine which is a feedback neural network with

stochastic neuron units [Ackley et al, 19851. A stochastic neuron has

an output function yrhich is implemented using a probabilistic update

rule instead of a deterministic update rule as in the Hopfield model.

Moreover, , the Boltzmann machine has several additional neuron

units, called hidden units, which are used to make a given pattern

storage problem representable in a feedback network.

Besides these key developments, there are many other significant

contributions made in this field during the past thirty years. Notable

among them are the concepts of competitive learning, selforganization

and simulated annealing. Self-organization led to the

realization of feature mapping. Simulated annealing has been very

useful in implementing the learning law for the Boltzmann machine.

Several new learning laws were also developed, the prominent among

Basics of Artificial Neural Networks

them being the reinforcement learning or learning with critic. Several

architectures were developed to address specific issues in pattern

recognition. Some of these architectures are: adaptive resonance

theory (ART), neocognitron and counterpropagation networks.

Currently, fuzzy logic concepts are being used to enhance the

capability of the neural networks to deal with real world problems

such as in speech, image processing, natural language processing and

decision making [Lin and Lee, 19961.

*Source: Artificial Intelligence on Medium*