As mentioned earlier, artificial neural networks are loosely based on biology. Current research into the brain's physiology has unlocked only a limitedunderstanding of how neurons work or even what constitutes intelligence ingeneral. Researchers are working in both the biological and engineering fields to further decipher the key mechanisms for how man learns and reacts to everyday experiences. Improved knowledge in neural processing helps create better, more succinct artificial networks. It also creates a cornucopia of new, and ever evolving, architectures. Kunihiko Fukushima, a senior research scientist in Japan, describes the give and take of building a neural network model; "We try to follow physiological evidence as faithfully as possible. For parts not yet clear, however, we construct a hypothesis and build a model that follows that hypothesis. We then analyze or simulate the behavior of the model and compare it with that of the brain. If we find any discrepancy in the behavior between the model and the brain, we change the initial hypothesis and modify the model. We repeat this procedure until the model behaves in the same way as the brain." This common process has created thousands of network topologies.

Figure 4.0.1 Processing Element.

Neural computing is about machines, not brains. It is the process of trying to build processing systems that draw upon the highly successful designs naturally occuring in biology. This linkage with biology is the reason that there is a common architectural thread throughout today's artificial neural networks. Figure 4.0.1 shows a model of an artificial neuron, or processing element, which embodies a wide variety of network architectures.

This figure is adapted from NeuralWare's simulation model used in NeuralWorks Profession II/Plus. NeuralWare sells a artificial neural network design and development software package. Their processing element model shows that networks designed for prediction can be very similar to networks designed for classification or any other network category. Prediction, classification and other network categories will be discussed later. The point here is that all artificial neural processing elements have common components.

**Component 1. Weighting Factors:** A neuron usually receives many
simultaneous inputs. Each input has its own relative weight which gives the
input the impact that it needs on the processing element's summation function.
These weights perform the same type of function as do the the varying synaptic
strengths of biological neurons. In both cases, some inputs are made more
important than others so that they have a greater effect on the processing
element as they combine to produce a neural response.

Weights are adaptive coefficients within the network that determine the intensity of the input signal as registered by the artificial neuron. They are a measure of an input's connection strength. These strengths can be modified in response to various training sets and according to a network's specific topology or through its learning rules.

**Component 2. Summation Function:** The first step in a processing
element's operation is to compute the weighted sum of all of the inputs.
Mathematically, the inputs and the corresponding weights are vectors which can
be represented as (i1, i2 . . . i*n*) and (w1, w2 . . . w*n*). The
total input signal is the dot, or inner, product of these two vectors. This
simplistic summation function is found by muliplying each component of the i
vector by the corresponding component of the w vector and then adding up all the
products. Input1 = i1 * w1, input2 = i2 * w2, etc., are added as input1 + input2
+ . . . + input*n*. The result is a single number, not a multi-element
vector.

Geometrically, the inner product of two vectors can be considered a measure of their similarity. If the vectors point in the same direction, the inner product is maximum; if the vectors point in opposite direction (180 degrees out of phase), their inner product is minimum.

The summation function can be more complex than just the simple input and weight sum of products. The input and weighting coefficients can be combined in many different ways before passing on to the transfer function. In addition to a simple product summing, the summation function can select the minimum, maximum, majority, product, or several normalizing algorithms. The specific algorithm for combining neural inputs is determined by the chosen network architecture and paradigm.

Some summation functions have an additional process applied to the result before it is passed on to the transfer function. This process is sometimes called the activation function. The purpose of utilizing an activation function is to allow the summation output to vary with respect to time. Activation functions currently are pretty much confined to research. Most of the current network implementations use an "identity" activation function, which is equivalent to not having one. Additionally, such a function is likely to be a component of the network as a whole rather than of each individual processing element component.

**Component 3. Transfer Function:** The result of the summation function,
almost always the weighted sum, is transformed to a working output through an
algorithmic process known as the transfer function. In the transfer function the
summation total can be compared with some threshold to determine the neural
output. If the sum is greater than the threshold value, the processing element
generates a signal. If the sum of the input and weight products is less than the
threshold, no signal (or some inhibitory signal) is generated. Both types of
response are significant.

The threshold, or transfer function, is generally non-linear. Linear
(straight-line) functions are limited because the output is simply proportional
to the input. Linear functions are not very useful. That was the problem in the
earliest network models as noted in Minsky and Papert's book *Perceptrons*.

The transfer function could be something as simple as depending upon whether the result of the summation function is positive or negative. The network could output zero and one, one and minus one, or other numeric combinations. The transfer function would then be a "hard limiter" or step function. See Figure 4.1.1 for sample transfer functions.

Figure 4.1.1 Sample Transfer Functions.

Another type of transfer function, the threshold or ramping function, could mirror the input within a given range and still act as a hard limiter outside that range. It is a linear function that has been clipped to minimum and maximum values, making it non-linear. Yet another option would be a sigmoid or S-shaped curve. That curve approaches a minimum and maximum value at the asymptotes. It is common for this curve to be called a sigmoid when it ranges between 0 and 1, and a hyperbolic tangent when it ranges between -1 and 1. Mathematically, the exciting feature of these curves is that both the function and its derivatives are continuous. This option works fairly well and is often the transfer function of choice. Other transfer functions are dedicated to specific network architectures and will be discussed later.

Prior to applying the transfer function, uniformly distributed random noise may be added. The source and amount of this noise is determined by the learning mode of a given network paradigm. This noise is normally referred to as "temperature" of the artificial neurons. The name, temperature, is derived from the physical phenomenon that as people become too hot or cold their ability to think is affected. Electronically, this process is simulated by adding noise. Indeed, by adding different levels of noise to the summation result, more brain-like transfer functions are realized. To more closely mimic nature's characteristics, some experimenters are using a gaussian noise source. Gaussian noise is similar to uniformly distributed noise except that the distribution of random numbers within the temperature range is along a bell curve. The use of temperature is an ongoing research area and is not being applied to many engineering applications.

NASA just announced a network topology which uses what it calls a temperature coefficient in a new feed-forward, back-propagation learning function. But this temperature coefficient is a global term which is applied to the gain of the transfer function. It should not be confused with the more common term, temperature, which is simple noise being added to individual neurons. In contrast, the global temperature coefficient allows the transfer function to have a learning variable much like the synaptic input weights. This concept is claimed to create a network which has a significantly faster (by several order of magnitudes) learning rate and provides more accurate results than other feedforward, back-propagation networks.

**Component 4. Scaling and Limiting:** After the processing element's
transfer function, the result can pass through additional processes which scale
and limit. This scaling simply multiplies a scale factor times the transfer
value, and then adds an offset. Limiting is the mechanism which insures that the
scaled result does not exceed an upper or lower bound. This limiting is in
addition to the hard limits that the original transfer function may have
performed.

This type of scaling and limiting is mainly used in topologies to test biological neuron models, such as James Anderson's brain-state-in-the-box.

**Component 5. Output Function (Competition): **Each processing element is
allowed one output signal which it may output to hundreds of other neurons. This
is just like the biological neuron, where there are many inputs and only one
output action. Normally, the output is directly equivalent to the transfer
function's result. Some network topologies, however, modify the transfer result
to incorporate competition among neighboring processing elements. Neurons are
allowed to compete with each other, inhibiting processing elements unless they
have great strength. Competition can occur at one or both of two levels. First,
competition determines which artificial neuron will be active, or provides an
output. Second, competitive inputs help determine which processing element will
participate in the learning or adaptation process.

**Component 6. Error Function and Back-Propagated Value:** In most
learning networks the difference between the current output and the desired
output is calculated. This raw error is then transformed by the error function
to match a particular network architecture. The most basic architectures use
this error directly, but some square the error while retaining its sign, some
cube the error, other paradigms modify the raw error to fit their specific
purposes. The artificial neuron's error is then typically propagated into the
learning function of another processing element. This error term is sometimes
called the current error.

The current error is typically propagated backwards to a previous layer. Yet, this back-propagated value can be either the current error, the current error scaled in some manner (often by the derivative of the transfer function), or some other desired output depending on the network type. Normally, this back-propagated value, after being scaled by the learning function, is multiplied against each of the incoming connection weights to modify them before the next learning cycle.

**Component 7. Learning Function:** The purpose of the learning function
is to modify the variable connection weights on the inputs of each processing
element according to some neural based algorithm. This process of changing the
weights of the input connections to achieve some desired result can also be
called the adaption function, as well as the learning mode. There are two types
of learning: supervised and unsupervised. Supervised learning requires a
teacher. The teacher may be a training set of data or an observer who grades the
performance of the network results. Either way, having a teacher is learning by
reinforcement. When there is no external teacher, the system must organize
itself by some internal criteria designed into the network. This is learning by
doing.

With supervised learning, the artificial neural network must be trained before it becomes useful. Training consists of presenting input and output data to the network. This data is often referred to as the training set. That is, for each input set provided to the system, the corresponding desired output set is provided as well. In most applications, actual data must be used. This training phase can consume a lot of time. In prototype systems, with inadequate processing power, learning can take weeks. This training is considered complete when the neural network reaches an user defined performance level. This level signifies that the network has achieved the desired statistical accuracy as it produces the required outputs for a given sequence of inputs. When no further learning is necessary, the weights are typically frozen for the application. Some network types allow continual training, at a much slower rate, while in operation. This helps a network to adapt to gradually changing conditions.

Training sets need to be fairly large to contain all the needed information if the network is to learn the features and relationships that are important. Not only do the sets have to be large but the training sessions must include a wide variety of data. If the network is trained just one example at a time, all the weights set so meticulously for one fact could be drastically altered in learning the next fact. The previous facts could be forgotten in learning something new. As a result, the system has to learn everything together, finding the best weight settings for the total set of facts. For example, in teaching a system to recognize pixel patterns for the ten digits, if there were twenty examples of each digit, all the examples of the digit seven should not be presented at the same time.

How the input and output data is represented, or encoded, is a major component to successfully instructing a network. Artificial networks only deal with numeric input data. Therefore, the raw data must often be converted from the external environment. Additionally, it is usually necessary to scale the data, or normalize it to the network's paradigm. This pre-processing of real-world stimuli, be they cameras or sensors, into machine readable format is already common for standard computers. Many conditioning techniques which directly apply to artificial neural network implementations are readily available. It is then up to the network designer to find the best data format and matching network architecture for a given application.

After a supervised network performs well on the training data, then it is important to see what it can do with data it has not seen before. If a system does not give reasonable outputs for this test set, the training period is not over. Indeed, this testing is critical to insure that the network has not simply memorized a given set of data but has learned the general patterns involved within an application.

This promising field of unsupervised learning is sometimes called self-supervised learning. These networks use no external influences to adjust their weights. Instead, they internally monitor their performance. These networks look for regularities or trends in the input signals, and makes adaptations according to the function of the network. Even without being told whether it's right or wrong, the network still must have some information about how to organize itself. This information is built into the network topology and learning rules.

An unsupervised learning algorithm might emphasize cooperation among clusters of processing elements. In such a scheme, the clusters would work together. If some external input activated any node in the cluster, the cluster's activity as a whole could be increased. Likewise, if external input to nodes in the cluster was decreased, that could have an inhibitory effect on the entire cluster.

Competition between processing elements could also form a basis for learning. Training of competitive clusters could amplify the responses of specific groups to specific stimuli. As such, it would associate those groups with each other and with a specific appropriate response. Normally, when competition for learning is in effect, only the weights belonging to the winning processing element will be updated.

At the present state of the art, unsupervised learning is not well understood and is still the subject of research. This research is currently of interest to the government because military situations often do not have a data set available to train a network until a conflict arises.

Generally, several factors besides time have to be considered when discussing the off-line training task, which is often described as "tiresome." Network complexity, size, paradigm selection, architecture, type of learning rule or rules employed, and desired accuracy must all be considered. These factors play a significant role in determining how long it will take to train a network. Changing any one of these factors may either extend the training time to an unreasonable length or even result in an unacceptable accuracy.

Most learning functions have some provision for a learning rate, or learning constant. Usually this term is positive and between zero and one. If the learning rate is greater than one, it is easy for the learning algorithm to overshoot in correcting the weights, and the network will oscillate. Small values of the learning rate will not correct the current error as quickly, but if small steps are taken in correcting errors, there is a good chance of arriving at the best minimum convergence.

**Hebb's Rule:** The first, and undoubtedly the best known, learning rule
was introduced by Donald Hebb. The description appeared in his book *The
Organization of Behavior* in 1949. His basic rule is: If a neuron receives an
input from another neuron, and if both are highly active (mathematically have
the same sign), the weight between the neurons should be strengthened.

**Hopfield Law:** It is similar to Hebb's rule with the exception that it
specifies the magnitude of the strengthening or weakening. It states, "if the
desired output and the input are both active or both inactive, increment the
connection weight by the learning rate, otherwise decrement the weight by the
learning rate."

**The Delta Rule:** This rule is a further variation of Hebb's Rule. It is
one of the most commonly used. This rule is based on the simple idea of
continuously modifying the strengths of the input connections to reduce the
difference (the delta) between the desired output value and the actual output of
a processing element. This rule changes the synaptic weights in the way that
minimizes the mean squared error of the network. This rule is also referred to
as the Widrow-Hoff Learning Rule and the Least Mean Square (LMS) Learning Rule.

The way that the Delta Rule works is that the delta error in the output layer is transformed by the derivative of the transfer function and is then used in the previous neural layer to adjust input connection weights. In other words, this error is back-propagated into previous layers one layer at a time. The process of back-propagating the network errors continues until the first layer is reached. The network type called Feedforward, Back-propagation derives its name from this method of computing the error term.

When using the delta rule, it is important to ensure that the input data set is well randomized. Well ordered or structured presentation of the training set can lead to a network which can not converge to the desired accuracy. If that happens, then the network is incapable of learning the problem.

**The Gradient Descent Rule**: This rule is similar to the Delta Rule in
that the derivative of the transfer function is still used to modify the delta
error before it is applied to the connection weights. Here, however, an
additional proportional constant tied to the learning rate is appended to the
final modifying factor acting upon the weight. This rule is commonly used, even
though it converges to a point of stability very slowly.

It has been shown that different learning rates for different layers of a network help the learning process converge faster. In these tests, the learning rates for those layers close to the output were set lower than those layers near the input. This is especially important for applications where the input data is not derived from a strong underlying model.

**Kohonen's Learning Law:** This procedure, developed by Teuvo Kohonen,
was inspired by learning in biological systems. In this procedure, the
processing elements compete for the opportunity to learn, or update their
weights. The processing element with the largest output is declared the winner
and has the capability of inhibiting its competitors as well as exciting its
neighbors. Only the winner is permitted an output, and only the winner plus its
neighbors are allowed to adjust their connection weights.

Further, the size of the neighborhood can vary during the training period. The usual paradigm is to start with a larger definition of the neighborhood, and narrow in as the training process proceeds. Because the winning element is defined as the one that has the closest match to the input pattern, Kohonen networks model the distribution of the inputs. This is good for statistical or topological modeling of the data and is sometimes referred to as self-organizing maps or self-organizing topologies.