SlideShare a Scribd company logo

Introduction Of Artificial neural network

•
•
Nagarajan
Nagarajan

The document summarizes different types of artificial neural networks including their structure, learning paradigms, and learning rules. It discusses artificial neural networks (ANN), their advantages, and major learning paradigms - supervised, unsupervised, and reinforcement learning. It also explains different mathematical synaptic modification rules like backpropagation of error, correlative Hebbian, and temporally-asymmetric Hebbian learning rules. Specific learning rules discussed include the delta rule, the pattern associator, and the Hebb rule.

Related slideshows

Artificial Neural Network
Artificial Neural NetworkArtificial Neural Network
Artificial Neural Network
 
Artificial Intelligence: Artificial Neural Networks
Artificial Intelligence: Artificial Neural NetworksArtificial Intelligence: Artificial Neural Networks
Artificial Intelligence: Artificial Neural Networks
 
Neural networks
Neural networksNeural networks
Neural networks
 
1 of 93
Madras universityDepartment Of Computer Science
Seminar OnIntroduction Of ANN, RulesAndAdaptive Resonance Theory
GROUP MEMBERS ARE :P.JayaVelJ.Joseph Amal RajM.Kaja Mohinden
ARTIFICIAL NEURAL NETWORK (ANN)An artificial neural network (ANN), usually called "neural network" (NN), is a mathematical model or computational model that tries to simulate the structure and/or functional aspects of biological neural networks.It consists of an interconnected group of artificial neurons and processes information using a connectionist approach to computation.In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network during the learning phase.
ARTIFICIAL NEURAL NETWORK (ANN)
ARTIFICIAL NEURAL NETWORK (ANN)Why use neural networks?Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A trained neural network can be thought of as an "expert" in the category of information it has been given to analyse. This expert can then be used to provide projections given new situations of interest and answer "what if" questions.Other advantages include: Adaptive learning: An ability to learn how to do tasks based on the data given for training or initial experience.
Self-Organisation: An ANN can create its own organisation or representation of the information it receives during learning time.
Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability.
Fault Tolerance via Redundant Information Coding: Partial destruction of a network leads to the corresponding degradation of performance. However, some network capabilities may be retained even with major network damage. ARTIFICIAL NEURAL NETWORK (ANN)Learning paradigms :There are three major learning paradigmsSupervised learning
Unsupervised learning
Reinforcement learningsupervised learningIn supervised learning, the learning rule is provided with a set of examples (the training set ) of proper network behavior: , where is an input to the network and is the corresponding correct ( target ) output. As the inputs are applied to the network, the network outputs are compared to the targets. The learning rule is then used to adjust the weights and biases of the network in order to move the network outputs closer to the targets. The perceptron learning rule falls in this supervised learning category. In supervised learning, we are given a set of example pairs    and the aim is to find a function    in the allowed class of functions that matches the examples. In other words, we wish to infer the mapping implied by the data; the cost function is related to the mismatch between our mapping and the data and it implicitly contains prior knowledge about the problem domain.
supervised learning
supervised learningFeedforward Learning RulesQuickprop[fahlman88empirical]Equation 1. Error derivative at previous epoch
Equation 2. Error derivative at this epoch The Quickprop algorithm is loosely based on Newton's method. It is quicker than standard backpropagation because it uses an approximation to the error curve, and second order derivative information which allow a quicker evaluation. Training is similar to backprop except for a copy of (eq. 1) the error derivative at a previous epoch. This, and the current error derivative (eq. 2), are used to minimise an approximation to this error curve.
supervised learningThe update rule is given in equation 3:Equation 3. Quickprop update rule This equation uses no learning rate. If the slope of the error curve is less than that of the previous one, then the weight will change in the same direction (positive or negative). However, there needs to be some controls to prevent the weights from growing too large.
Unsupervised learningIn unsupervised learning we are given some data x and the cost function to be minimized, that can be any function of the data x and the network's output, f.The cost function is dependent on the task (what we are trying to model) and our a priori assumptions (the implicit properties of our model, its parameters and the observed variables).
Unsupervised learning As a trivial example, consider the model f(x) = a, where a is a constant and the cost C = E[(x − f(x))2]. Minimizing this cost will give us a value of a that is equal to the mean of the data. The cost function can be much more complicated. Its form depends on the application: for example, in compression it could be related to the mutual information between x and y, whereas in statistical modelling, it could be related to theposterior probability of the model given the data. (Note that in both of those examples those quantities would be maximized rather than minimized).Tasks that fall within the paradigm of unsupervised learning are in general estimation problems; the applications include clustering, the estimation of statistical distributions, compression and filtering.
Unsupervised learningUnsupervised learning, in contrast to supervised learning, does not provide the network with target output values. This isn't strictly true, as often (and for the cases discussed in the this section) the output is identical to the input. Unsupervised learning usually performs a mapping from input to output space, data compression or clustering.
Reinforcement learningIn reinforcement learning, data x are usually not given, but generated by an agent's interactions with the environment. At each point in time t, the agent performs an action yt and the environment generates an observation xt and an instantaneous cost ct, according to some (usually unknown) dynamics.Tasks that fall within the paradigm of reinforcement learning are control problems, games and other sequential decision making tasks.
Reinforcement learningThe aim is to discover a policy for selecting actions that minimizes some measure of a long-term cost; i.e., the expected cumulative cost. The environment's dynamics and the long-term cost for each policy are usually unknown, but can be estimated.ANNs are frequently used in reinforcement learning as part of the overall algorithm. 
Neural Network “Learning Rules”:Successful learning in any neural network is dependent on how the connections between the neurons are allowed to change in response to activity. The manner of change is what the majority of researchers call "a learning rule". However, we will call it a "synaptic modification rule" because although the network learned the sequence, it is not clear that the *connections* between the neurons in the network "learned" anything in particular.
Mathematical synaptic Modification ruleThere are many categories of mathematical synaptic modification rule which are used to describe how synaptic strengths should be changed in a neural network.  Some of these categories include: backpropgration of error, correlative Hebbian, and temporally-asymmetric Hebbian.
Mathematical synaptic modification ruleBackpropogation of error states that connection strengths should change throughout the entire network in order to minimize the difference between the actual activity and the "desired" activity at the "output" layer of the network.
Mathematical synaptic Modification ruleCorrelative Hebbian states that any two interconnected neurons that are active at the same time should strengthen their connections, so that if one of the neurons is activated again in the future the other is more likely to become activated too.
Mathematical synaptic Modification ruleTemporally-asymmetric Hebbian is described in more detail in the example below, but essentially emphasizes the importants of causality: if a neuron realiably fires before another, its connection to the other neuron should be strengthened. Otherwise, it should be weakened. 
Neural Network “Learning Rules”:The Delta RuleThe Pattern AssociatorThe Hebb Rule
The Delta RuleA generalized form of the delta rule, developed by D.E. Rumelhart, G.E. Hinton, and R.J. Williams, is needed for networks with hidden layers. They showed that this method works for the class of semilinear activation functions (non-decreasing and differentiable).Generalizing the ideas of the delta rule, consider a hierarchical network with an input layer, an output layer and a number of hidden layers.
The Delta Rule. We will consider only the case where there is one hidden layer. The network is presented with input signals which produce output signals that act as input to the middle layer. Output signals from the middle layer in turn act as input to the output layer to produce the final output vector. This vector is compared to the desired output vector. Since both the output and the desired output vectors are known, the delta rule can be used to adjust the weights in the output layer.
The Delta RuleCan the delta rule be applied to the middle layer? Both the input signal to each unit of the middle layer and the output signal are known. What is not known is the error generated from the output of the middle layer since we do not know the desired output. To get this error, backpropagate through the middle layer to the units that are responsible for generating that output. The error genrated from the middle layer could be used with the delta rule to adjust the weights.
The Pattern AssociatorA pattern associator learns associations between input patterns and output patterns. One of the most appealing characteristics of such a network is the fact that it can generate what it learns about one pattern to other similar input patterns. Pattern associators have been widely used in distributed memory modeling.
The Pattern AssociatorThe pattern associator is one of the more basic two-layer networks. Its architecture consists of two sets of units, the input units and the output units.Each input unit connects to each output unit via weighted connections.Connections are only allowed from input units to output units.
The Pattern AssociatorThe effect of a unit ui in the input layer on a unit uj in the output layer is determined by the product of the activation ai of ui and the weight of the connection from ui to uj. The activation of a unit uj in the output layer is given by: SUM(wij * ai).
Adaptive Resonance Theory (ART) Discrete Bidirectional Associative Memory Kochen Self Organization MapCounter Propagation Network (CPN) PerceptronVector RepresentationADALINE (Adaptive Linear Neuron or later Adaptive Linear Element) Madaline (Multiple Adaline) Backpropagation, or propagation of error
Adaptive Resonance Theory (ART) Adaptive Resonance Theory (ART) is a theory developed by Stephen Grossberg and Gail Carpenter on aspects of how the brain processes information. It describes a number of neural network models which use supervised and unsupervised learning methods, and address problems such as pattern recognition and prediction.
Discrete Bidirectional Associative Memory
Kochen Self Organization MapThe self-organizing map (SOM) invented by TeuvoKohonen performs a form of unsupervised learning. A set of artificial neurons learn to map points in an input space to coordinates in an output space. The input space can have different dimensions and topology from the output space, and the SOM will attempt to preserve these. 
Kochen Self Organization MapIf an input space is to be processed by a neural network, the first issue of importance is the structure of this space. A neural network with real inputs computes a function f defined from an input space A to an output space B. The region where f is defined can be covered by a Kohonen network in such a way that when, for example,an input vector is selected from the region a1, only one unit in the network fires. Such a tiling in which input space is classified in subregions is also called a chart or map of input space. Kohonen networks learn to create maps of the input space in a self-organizing way.
Kochen Self Organization Map-AdvantagesProbably the best thing about SOMs that they are very easy to understand. It’s very simple, if they are close together and there is grey connecting them, then they are similar. If there is a black ravine between them, then they are different. Unlike Multidimensional Scaling or N-land, people can quickly pick up on how to use them in an effective manner.Another great thing is that they work very well. As I have shown you they classify data well and then are easily evaluate for their own quality so you can actually calculated how good a map is and how strong the similarities between objects are. 
Kochen Self Organization Map
PerceptronThe perceptron is a type of artificial neural network invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt. It can be seen as the simplest kind of feedforward neural network: a linear classifier.The Perceptron is a binary classifier that maps its input x (a real-valued vector) to an output value f(x) (a single binary value) across the matrix.where w is a vector of real-valued weights and   is the dot product (which computes a weighted sum). b is the 'bias', a constant term that does not depend on any input value.
ADALINEDefinitionAdaline is a single layer neural network with multiple nodes where each node accepts multiple inputs and generates one output. Given the following variables:x is the input vectorw is the weight vectorn is the number of inputsθ some constanty is the outputthen we find that the output is  . If we further assume thatxn + 1 = 1wn + 1 = θ then the output reduces to the dot product of x and w   
Madaline Madaline (Multiple Adaline) is a two layer neural network with a set of ADALINEs in parallel as its input layer and a single PE (processing element) in its output layer. For problems with multiple input variables and one output, each input is applied to one Adaline. For similar problems with multiple outputs, madalines in parallel can be used. The madaline network is useful for problems which involve prediction based on multiple inputs, such as weather forecasting (Input variables: barometric pressure, difference in pressure. Output variables: rain, cloudy, sunny).
BackpropagationBackpropagation, or propagation of error, is a common method of teaching artificial neural networks how to perform a given task. It was first described by Arthur E. Bryson and Yu-Chi Ho in 1969,[1][2] but it wasn't until 1986, through the work of David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, that it gained recognition, and it led to a “renaissance” in the field of artificial neural network research.It is a supervised learning method, and is an implementation of the Delta rule. It requires a teacher that knows, or can calculate, the desired output for any given input. It is most useful for feed-forward networks (networks that have no feedback, or simply, that have no connections that loop). The term is an abbreviation for "backwards propagation of errors". Backpropagation requires that the activation function used by theartificial neurons (or "nodes") is differentiable.
BackpropagationBackpropagationCalculation of error dk = f(Dk) -f(Ok)
Network Structure –Back-propagation NetworkOi Output UnitWj,iajHidden UnitsWk,jIkInput Units
Counter propagation network (CPN) (§ 5.3)Basic idea of CPNPurpose: fast and coarse approximation of vector mappingnot to map any given x to its with given precision,input vectors x are divided into clusters/classes.each cluster of x has one output y, which is (hopefully) the average of for all x in that class.Architecture: Simple case: FORWARD ONLY CPN, yzx111yvzwxjj,kkk,iiyzxmpnfrom hidden (class) to outputfrom input to hidden (class)
Learning in two phases: training sample (x, d ) where is the desired precise mappingPhase1: weights coming into hidden nodes are trained by competitive learning to become the representative vector of a cluster of input vectors x: (use only x, the input part of(x, d )) 1. For a chosen x, feedforward to determined the winning 2. 3. Reduce , then repeat steps 1 and 2 until stop condition is metPhase 2: weights going out of hidden nodes are trained by delta rule to be an average output of where x is an input vector that causes to win (use both x andd). 1. For a chosen x, feedforward to determined the winning 2. (optional) 3. 4. Repeat steps 1 – 3 until stop condition is met
Adaptive Resonance Theory
Adaptive Resonance TheoryAdaptive Resonance Theory (ART) was developed by Grossberg (1976)Input vectors which are close to each other according to a specific similarity measure should be mapped to the same clusterART adapts itself by storing input patterns, and tries to match best the input pattern 45
Adaptive Resonance Theory 1 (ART 1)ART 1 is a binary classification model. Various other versions of the model have evolved from ART 1Pointers to these can be found in the bibliographic remarksThe main network comprises the layers F1, F2 and the attentional gain control as the attentional subsystemThe attentional vigilance node forms the orienting subsystem
ART 1: Architecture……Attentional SubsystemOrienting SubsystemF2--++F1-+-GA+++I
ART 1: 2/3 RuleJ…F2Si(yj)vjisi-sGGF1+lliThree kinds of inputs to each F1 neuron decide when the neuron firesExternal input Ii
Top-down feedback through outstar weights vji
Gain control signal sGART 1: 2/3 RuleThe gain control signal sG = 1 if Iis presented and all neurons in F2are inactivesGis nonspecificWhen the input is initially presented to the system, sG= 1As soon as a node Jin F2 fires as a result of competition, sG = 0
Adaptive Resonance Theory (ART) ART1: for binary patterns; ART2: for continuous patterns
Motivations: Previous methods have the following problems:Number of class nodes is pre-determined and fixed. Under- and over- classification may result from training
Some nodes may have empty classes.
no control of the degree of similarity of inputs grouped in one class. Training is non-incremental: with a fixed set of samples,
adding new samples often requires re-train the network with the enlarged training set until a new stable state is reached.Ideas of ART model:suppose the input samples have been appropriately classified into k clusters (say by some fashion of competitive learning).each weight vector is a representative (average) of all samples in that cluster.when a new input vector x arrivesFind the winner j* among all k cluster nodesCompare with x if they are sufficiently similar (x resonates with class j*), then update based on else, find/create a free class node and make x as its first member.
To achieve these, we need:a mechanism for testing and determining (dis)similarity between x and .a control for finding/creating new class nodes.need to have all operations implemented by units of local computation.Only the basic ideas are presentedSimplified from the original ART modelSome of the control mechanisms realized by various specialized neurons are done by logic statements of the algorithm
ART1 Architecture
Working of ART13 phases after each input vector x is appliedRecognition phase: determine the winner cluster for xUsing bottom-up weights bWinner j* with max yj* = bj*ּxx is tentatively classified to cluster j*the winner may be far away from x (e.g., |tj* - x| is unacceptably large)
Working of ART1 (3 phases)Comparison phase: Compute similarity using top-down weights t: vector:If (# of 1’s ins)|/(# of 1’s inx) > ρ, accept the classification, update bj* and tj*else: remove j* from further consideration, look for other potential winner or create a new node with x as its first patter.
Weight update/adaptive phaseInitial weight: (no bias) bottom up: top down:When a resonance occurs withIf k sample patterns are clustered to node jthen = pattern whose 1’s are common to all these k samples
Introduction Of Artificial neural network
Example for input x(1)Node 1 wins
Introduction Of Artificial neural network
NotesClassification as a search processNo two classes have the same b and tOutliers that do not belong to any cluster will be assigned separate nodesDifferent ordering of sample input presentations may result in different classification.Increase of r increases # of classes learned, and decreases the average class size.Classification may shift during search, will reach stability eventually.There are different versions of ART1 with minor variationsART2 is the same in spirit but different in details.
RG1G2ART1 Architecture++--++++
cluster units: competitive, receive input vector x through weights b: to determine winner j. input units: placeholder or external inputs interface units: pass s to x as input vector for classification by compare x and controlled by gain control unit G1Needs to sequence the three phases (by control units G1, G2, and R)
R = 0: resonance occurs, update andR = 1: fails similarity test, inhibits J from further computation
ART clustering algorithmsART1
ART2
ART3
ARTMAP
Fuzzy ARTFuzzy ART Modeling
Fuzzy ARTLayer1 consists of neurons that are connected to the neurons in Layer 2 through weight vectors.Thenumber of neurons in Layer 1 depends on the characteristics of the input data.The Layer 2 represent clusters.
67
Fuzzy ART Architecture
Fuzzy ART FMEAFMEA values are evaluated separately with severity, detection and occurrence valuesThe aim is to apply Fuzzy ART algorithm to FMEA method and by performing FMEA on test problems, most favorable parameter combinations (α , β and ρ) are investigated.
Hand-worked ExampleCluster the vectors 11100, 11000, 00001, 00011Low vigilance: 0.3High vigilance: 0.7
Hand-worked Example:  = 0.3
ART 1: Clustering Application = 0.3
Hand-worked Example:  = 0.7
ART 1: Clustering Application = 0.7
Neurophysiological Evidence for ARTMechanismsThe attentional subsystem of an ART network has been used to model aspects of the inferotemporal cortexOrienting subsystem has been used to model a part of the hippocampal system, which is known to contribute to memory functionsThe feedback prevalent in an ART network can help focus attention in models of visual object recognition
Other ApplicationsAircraft Part Design Classification System.See text for details.
Ehrenstein Pattern Explained by ART !The bright disc disappears when the alignment of the dark lines is disturbed!Generates a circular illusory contour – a circular disc of enhanced brightness
78Other Neurophysiological EvidenceAdam Sillito [University College, London]Cortical feedback in a cat tunes cells in its LGN to respond best to lines of a specific length. Chris Redie [MPI Entwicklungsbiologie, Germany]Found that some visual cells in a cat’s LGN and cortex respond best at line ends— more strongly to line ends than line sides. Sillito et al. [University College, London] Provide neurophysiological data suggesting that the cortico-geniculate feedback closely resembles the matching and resonance of an ART network. Cortical feedback has been found to change the output of specific LGN cells, increasing the gain of the input for feature linked events that are detected by the cortex.
Computational ExperimentAnon-binarydataset of FMEA isused to evaluate theperformance of theFuzzy ART neuralnetwork on differenttest problems79
80Computational ExperimentFor acomprehensiveanalysis of the effectsof parameters on theperformance of FuzzyART in FMEA case, anumber of levels ofparameters areconsidered.
81Computational ExperimentThe Fuzzy ART neural network method is applied to determine the most favorable parameter (α, β and ρ) combinations during application of FMEA on test problems
82ResultsFor any test problem 900 solutions are obtained. The β-ρ interactions for parameter combinations are considered where solutions are obtained. For each test problem, all the combinations are evaluated and frequency distribution of clusters are constituted

More Related Content

Introduction Of Artificial neural network

  • 2. Seminar OnIntroduction Of ANN, RulesAndAdaptive Resonance Theory
  • 3. GROUP MEMBERS ARE :P.JayaVelJ.Joseph Amal RajM.Kaja Mohinden
  • 4. ARTIFICIAL NEURAL NETWORK (ANN)An artificial neural network (ANN), usually called "neural network" (NN), is a mathematical model or computational model that tries to simulate the structure and/or functional aspects of biological neural networks.It consists of an interconnected group of artificial neurons and processes information using a connectionist approach to computation.In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network during the learning phase.
  • 6. ARTIFICIAL NEURAL NETWORK (ANN)Why use neural networks?Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A trained neural network can be thought of as an "expert" in the category of information it has been given to analyse. This expert can then be used to provide projections given new situations of interest and answer "what if" questions.Other advantages include: Adaptive learning: An ability to learn how to do tasks based on the data given for training or initial experience.
  • 7. Self-Organisation: An ANN can create its own organisation or representation of the information it receives during learning time.
  • 8. Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability.
  • 9. Fault Tolerance via Redundant Information Coding: Partial destruction of a network leads to the corresponding degradation of performance. However, some network capabilities may be retained even with major network damage. ARTIFICIAL NEURAL NETWORK (ANN)Learning paradigms :There are three major learning paradigmsSupervised learning
  • 11. Reinforcement learningsupervised learningIn supervised learning, the learning rule is provided with a set of examples (the training set ) of proper network behavior: , where is an input to the network and is the corresponding correct ( target ) output. As the inputs are applied to the network, the network outputs are compared to the targets. The learning rule is then used to adjust the weights and biases of the network in order to move the network outputs closer to the targets. The perceptron learning rule falls in this supervised learning category. In supervised learning, we are given a set of example pairs    and the aim is to find a function    in the allowed class of functions that matches the examples. In other words, we wish to infer the mapping implied by the data; the cost function is related to the mismatch between our mapping and the data and it implicitly contains prior knowledge about the problem domain.
  • 13. supervised learningFeedforward Learning RulesQuickprop[fahlman88empirical]Equation 1. Error derivative at previous epoch
  • 14. Equation 2. Error derivative at this epoch The Quickprop algorithm is loosely based on Newton's method. It is quicker than standard backpropagation because it uses an approximation to the error curve, and second order derivative information which allow a quicker evaluation. Training is similar to backprop except for a copy of (eq. 1) the error derivative at a previous epoch. This, and the current error derivative (eq. 2), are used to minimise an approximation to this error curve.
  • 15. supervised learningThe update rule is given in equation 3:Equation 3. Quickprop update rule This equation uses no learning rate. If the slope of the error curve is less than that of the previous one, then the weight will change in the same direction (positive or negative). However, there needs to be some controls to prevent the weights from growing too large.
  • 16. Unsupervised learningIn unsupervised learning we are given some data x and the cost function to be minimized, that can be any function of the data x and the network's output, f.The cost function is dependent on the task (what we are trying to model) and our a priori assumptions (the implicit properties of our model, its parameters and the observed variables).
  • 17. Unsupervised learning As a trivial example, consider the model f(x) = a, where a is a constant and the cost C = E[(x − f(x))2]. Minimizing this cost will give us a value of a that is equal to the mean of the data. The cost function can be much more complicated. Its form depends on the application: for example, in compression it could be related to the mutual information between x and y, whereas in statistical modelling, it could be related to theposterior probability of the model given the data. (Note that in both of those examples those quantities would be maximized rather than minimized).Tasks that fall within the paradigm of unsupervised learning are in general estimation problems; the applications include clustering, the estimation of statistical distributions, compression and filtering.
  • 18. Unsupervised learningUnsupervised learning, in contrast to supervised learning, does not provide the network with target output values. This isn't strictly true, as often (and for the cases discussed in the this section) the output is identical to the input. Unsupervised learning usually performs a mapping from input to output space, data compression or clustering.
  • 19. Reinforcement learningIn reinforcement learning, data x are usually not given, but generated by an agent's interactions with the environment. At each point in time t, the agent performs an action yt and the environment generates an observation xt and an instantaneous cost ct, according to some (usually unknown) dynamics.Tasks that fall within the paradigm of reinforcement learning are control problems, games and other sequential decision making tasks.
  • 20. Reinforcement learningThe aim is to discover a policy for selecting actions that minimizes some measure of a long-term cost; i.e., the expected cumulative cost. The environment's dynamics and the long-term cost for each policy are usually unknown, but can be estimated.ANNs are frequently used in reinforcement learning as part of the overall algorithm. 
  • 21. Neural Network “Learning Rules”:Successful learning in any neural network is dependent on how the connections between the neurons are allowed to change in response to activity. The manner of change is what the majority of researchers call "a learning rule". However, we will call it a "synaptic modification rule" because although the network learned the sequence, it is not clear that the *connections* between the neurons in the network "learned" anything in particular.
  • 22. Mathematical synaptic Modification ruleThere are many categories of mathematical synaptic modification rule which are used to describe how synaptic strengths should be changed in a neural network.  Some of these categories include: backpropgration of error, correlative Hebbian, and temporally-asymmetric Hebbian.
  • 23. Mathematical synaptic modification ruleBackpropogation of error states that connection strengths should change throughout the entire network in order to minimize the difference between the actual activity and the "desired" activity at the "output" layer of the network.
  • 24. Mathematical synaptic Modification ruleCorrelative Hebbian states that any two interconnected neurons that are active at the same time should strengthen their connections, so that if one of the neurons is activated again in the future the other is more likely to become activated too.
  • 25. Mathematical synaptic Modification ruleTemporally-asymmetric Hebbian is described in more detail in the example below, but essentially emphasizes the importants of causality: if a neuron realiably fires before another, its connection to the other neuron should be strengthened. Otherwise, it should be weakened. 
  • 26. Neural Network “Learning Rules”:The Delta RuleThe Pattern AssociatorThe Hebb Rule
  • 27. The Delta RuleA generalized form of the delta rule, developed by D.E. Rumelhart, G.E. Hinton, and R.J. Williams, is needed for networks with hidden layers. They showed that this method works for the class of semilinear activation functions (non-decreasing and differentiable).Generalizing the ideas of the delta rule, consider a hierarchical network with an input layer, an output layer and a number of hidden layers.
  • 28. The Delta Rule. We will consider only the case where there is one hidden layer. The network is presented with input signals which produce output signals that act as input to the middle layer. Output signals from the middle layer in turn act as input to the output layer to produce the final output vector. This vector is compared to the desired output vector. Since both the output and the desired output vectors are known, the delta rule can be used to adjust the weights in the output layer.
  • 29. The Delta RuleCan the delta rule be applied to the middle layer? Both the input signal to each unit of the middle layer and the output signal are known. What is not known is the error generated from the output of the middle layer since we do not know the desired output. To get this error, backpropagate through the middle layer to the units that are responsible for generating that output. The error genrated from the middle layer could be used with the delta rule to adjust the weights.
  • 30. The Pattern AssociatorA pattern associator learns associations between input patterns and output patterns. One of the most appealing characteristics of such a network is the fact that it can generate what it learns about one pattern to other similar input patterns. Pattern associators have been widely used in distributed memory modeling.
  • 31. The Pattern AssociatorThe pattern associator is one of the more basic two-layer networks. Its architecture consists of two sets of units, the input units and the output units.Each input unit connects to each output unit via weighted connections.Connections are only allowed from input units to output units.
  • 32. The Pattern AssociatorThe effect of a unit ui in the input layer on a unit uj in the output layer is determined by the product of the activation ai of ui and the weight of the connection from ui to uj. The activation of a unit uj in the output layer is given by: SUM(wij * ai).
  • 33. Adaptive Resonance Theory (ART) Discrete Bidirectional Associative Memory Kochen Self Organization MapCounter Propagation Network (CPN) PerceptronVector RepresentationADALINE (Adaptive Linear Neuron or later Adaptive Linear Element) Madaline (Multiple Adaline) Backpropagation, or propagation of error
  • 34. Adaptive Resonance Theory (ART) Adaptive Resonance Theory (ART) is a theory developed by Stephen Grossberg and Gail Carpenter on aspects of how the brain processes information. It describes a number of neural network models which use supervised and unsupervised learning methods, and address problems such as pattern recognition and prediction.
  • 36. Kochen Self Organization MapThe self-organizing map (SOM) invented by TeuvoKohonen performs a form of unsupervised learning. A set of artificial neurons learn to map points in an input space to coordinates in an output space. The input space can have different dimensions and topology from the output space, and the SOM will attempt to preserve these. 
  • 37. Kochen Self Organization MapIf an input space is to be processed by a neural network, the first issue of importance is the structure of this space. A neural network with real inputs computes a function f defined from an input space A to an output space B. The region where f is defined can be covered by a Kohonen network in such a way that when, for example,an input vector is selected from the region a1, only one unit in the network fires. Such a tiling in which input space is classified in subregions is also called a chart or map of input space. Kohonen networks learn to create maps of the input space in a self-organizing way.
  • 38. Kochen Self Organization Map-AdvantagesProbably the best thing about SOMs that they are very easy to understand. It’s very simple, if they are close together and there is grey connecting them, then they are similar. If there is a black ravine between them, then they are different. Unlike Multidimensional Scaling or N-land, people can quickly pick up on how to use them in an effective manner.Another great thing is that they work very well. As I have shown you they classify data well and then are easily evaluate for their own quality so you can actually calculated how good a map is and how strong the similarities between objects are. 
  • 40. PerceptronThe perceptron is a type of artificial neural network invented in 1957 at the Cornell Aeronautical Laboratory by Frank Rosenblatt. It can be seen as the simplest kind of feedforward neural network: a linear classifier.The Perceptron is a binary classifier that maps its input x (a real-valued vector) to an output value f(x) (a single binary value) across the matrix.where w is a vector of real-valued weights and   is the dot product (which computes a weighted sum). b is the 'bias', a constant term that does not depend on any input value.
  • 41. ADALINEDefinitionAdaline is a single layer neural network with multiple nodes where each node accepts multiple inputs and generates one output. Given the following variables:x is the input vectorw is the weight vectorn is the number of inputsθ some constanty is the outputthen we find that the output is  . If we further assume thatxn + 1 = 1wn + 1 = θ then the output reduces to the dot product of x and w   
  • 42. Madaline Madaline (Multiple Adaline) is a two layer neural network with a set of ADALINEs in parallel as its input layer and a single PE (processing element) in its output layer. For problems with multiple input variables and one output, each input is applied to one Adaline. For similar problems with multiple outputs, madalines in parallel can be used. The madaline network is useful for problems which involve prediction based on multiple inputs, such as weather forecasting (Input variables: barometric pressure, difference in pressure. Output variables: rain, cloudy, sunny).
  • 43. BackpropagationBackpropagation, or propagation of error, is a common method of teaching artificial neural networks how to perform a given task. It was first described by Arthur E. Bryson and Yu-Chi Ho in 1969,[1][2] but it wasn't until 1986, through the work of David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, that it gained recognition, and it led to a “renaissance” in the field of artificial neural network research.It is a supervised learning method, and is an implementation of the Delta rule. It requires a teacher that knows, or can calculate, the desired output for any given input. It is most useful for feed-forward networks (networks that have no feedback, or simply, that have no connections that loop). The term is an abbreviation for "backwards propagation of errors". Backpropagation requires that the activation function used by theartificial neurons (or "nodes") is differentiable.
  • 45. Network Structure –Back-propagation NetworkOi Output UnitWj,iajHidden UnitsWk,jIkInput Units
  • 46. Counter propagation network (CPN) (§ 5.3)Basic idea of CPNPurpose: fast and coarse approximation of vector mappingnot to map any given x to its with given precision,input vectors x are divided into clusters/classes.each cluster of x has one output y, which is (hopefully) the average of for all x in that class.Architecture: Simple case: FORWARD ONLY CPN, yzx111yvzwxjj,kkk,iiyzxmpnfrom hidden (class) to outputfrom input to hidden (class)
  • 47. Learning in two phases: training sample (x, d ) where is the desired precise mappingPhase1: weights coming into hidden nodes are trained by competitive learning to become the representative vector of a cluster of input vectors x: (use only x, the input part of(x, d )) 1. For a chosen x, feedforward to determined the winning 2. 3. Reduce , then repeat steps 1 and 2 until stop condition is metPhase 2: weights going out of hidden nodes are trained by delta rule to be an average output of where x is an input vector that causes to win (use both x andd). 1. For a chosen x, feedforward to determined the winning 2. (optional) 3. 4. Repeat steps 1 – 3 until stop condition is met
  • 49. Adaptive Resonance TheoryAdaptive Resonance Theory (ART) was developed by Grossberg (1976)Input vectors which are close to each other according to a specific similarity measure should be mapped to the same clusterART adapts itself by storing input patterns, and tries to match best the input pattern 45
  • 50. Adaptive Resonance Theory 1 (ART 1)ART 1 is a binary classification model. Various other versions of the model have evolved from ART 1Pointers to these can be found in the bibliographic remarksThe main network comprises the layers F1, F2 and the attentional gain control as the attentional subsystemThe attentional vigilance node forms the orienting subsystem
  • 51. ART 1: Architecture……Attentional SubsystemOrienting SubsystemF2--++F1-+-GA+++I
  • 52. ART 1: 2/3 RuleJ…F2Si(yj)vjisi-sGGF1+lliThree kinds of inputs to each F1 neuron decide when the neuron firesExternal input Ii
  • 53. Top-down feedback through outstar weights vji
  • 54. Gain control signal sGART 1: 2/3 RuleThe gain control signal sG = 1 if Iis presented and all neurons in F2are inactivesGis nonspecificWhen the input is initially presented to the system, sG= 1As soon as a node Jin F2 fires as a result of competition, sG = 0
  • 55. Adaptive Resonance Theory (ART) ART1: for binary patterns; ART2: for continuous patterns
  • 56. Motivations: Previous methods have the following problems:Number of class nodes is pre-determined and fixed. Under- and over- classification may result from training
  • 57. Some nodes may have empty classes.
  • 58. no control of the degree of similarity of inputs grouped in one class. Training is non-incremental: with a fixed set of samples,
  • 59. adding new samples often requires re-train the network with the enlarged training set until a new stable state is reached.Ideas of ART model:suppose the input samples have been appropriately classified into k clusters (say by some fashion of competitive learning).each weight vector is a representative (average) of all samples in that cluster.when a new input vector x arrivesFind the winner j* among all k cluster nodesCompare with x if they are sufficiently similar (x resonates with class j*), then update based on else, find/create a free class node and make x as its first member.
  • 60. To achieve these, we need:a mechanism for testing and determining (dis)similarity between x and .a control for finding/creating new class nodes.need to have all operations implemented by units of local computation.Only the basic ideas are presentedSimplified from the original ART modelSome of the control mechanisms realized by various specialized neurons are done by logic statements of the algorithm
  • 62. Working of ART13 phases after each input vector x is appliedRecognition phase: determine the winner cluster for xUsing bottom-up weights bWinner j* with max yj* = bj*ּxx is tentatively classified to cluster j*the winner may be far away from x (e.g., |tj* - x| is unacceptably large)
  • 63. Working of ART1 (3 phases)Comparison phase: Compute similarity using top-down weights t: vector:If (# of 1’s ins)|/(# of 1’s inx) > ρ, accept the classification, update bj* and tj*else: remove j* from further consideration, look for other potential winner or create a new node with x as its first patter.
  • 64. Weight update/adaptive phaseInitial weight: (no bias) bottom up: top down:When a resonance occurs withIf k sample patterns are clustered to node jthen = pattern whose 1’s are common to all these k samples
  • 66. Example for input x(1)Node 1 wins
  • 68. NotesClassification as a search processNo two classes have the same b and tOutliers that do not belong to any cluster will be assigned separate nodesDifferent ordering of sample input presentations may result in different classification.Increase of r increases # of classes learned, and decreases the average class size.Classification may shift during search, will reach stability eventually.There are different versions of ART1 with minor variationsART2 is the same in spirit but different in details.
  • 70. cluster units: competitive, receive input vector x through weights b: to determine winner j. input units: placeholder or external inputs interface units: pass s to x as input vector for classification by compare x and controlled by gain control unit G1Needs to sequence the three phases (by control units G1, G2, and R)
  • 71. R = 0: resonance occurs, update andR = 1: fails similarity test, inhibits J from further computation
  • 73. ART2
  • 74. ART3
  • 77. Fuzzy ARTLayer1 consists of neurons that are connected to the neurons in Layer 2 through weight vectors.Thenumber of neurons in Layer 1 depends on the characteristics of the input data.The Layer 2 represent clusters.
  • 78. 67
  • 80. Fuzzy ART FMEAFMEA values are evaluated separately with severity, detection and occurrence valuesThe aim is to apply Fuzzy ART algorithm to FMEA method and by performing FMEA on test problems, most favorable parameter combinations (α , β and ρ) are investigated.
  • 81. Hand-worked ExampleCluster the vectors 11100, 11000, 00001, 00011Low vigilance: 0.3High vigilance: 0.7
  • 83. ART 1: Clustering Application = 0.3
  • 85. ART 1: Clustering Application = 0.7
  • 86. Neurophysiological Evidence for ARTMechanismsThe attentional subsystem of an ART network has been used to model aspects of the inferotemporal cortexOrienting subsystem has been used to model a part of the hippocampal system, which is known to contribute to memory functionsThe feedback prevalent in an ART network can help focus attention in models of visual object recognition
  • 87. Other ApplicationsAircraft Part Design Classification System.See text for details.
  • 88. Ehrenstein Pattern Explained by ART !The bright disc disappears when the alignment of the dark lines is disturbed!Generates a circular illusory contour – a circular disc of enhanced brightness
  • 89. 78Other Neurophysiological EvidenceAdam Sillito [University College, London]Cortical feedback in a cat tunes cells in its LGN to respond best to lines of a specific length. Chris Redie [MPI Entwicklungsbiologie, Germany]Found that some visual cells in a cat’s LGN and cortex respond best at line ends— more strongly to line ends than line sides. Sillito et al. [University College, London] Provide neurophysiological data suggesting that the cortico-geniculate feedback closely resembles the matching and resonance of an ART network. Cortical feedback has been found to change the output of specific LGN cells, increasing the gain of the input for feature linked events that are detected by the cortex.
  • 90. Computational ExperimentAnon-binarydataset of FMEA isused to evaluate theperformance of theFuzzy ART neuralnetwork on differenttest problems79
  • 91. 80Computational ExperimentFor acomprehensiveanalysis of the effectsof parameters on theperformance of FuzzyART in FMEA case, anumber of levels ofparameters areconsidered.
  • 92. 81Computational ExperimentThe Fuzzy ART neural network method is applied to determine the most favorable parameter (α, β and ρ) combinations during application of FMEA on test problems
  • 93. 82ResultsFor any test problem 900 solutions are obtained. The β-ρ interactions for parameter combinations are considered where solutions are obtained. For each test problem, all the combinations are evaluated and frequency distribution of clusters are constituted
  • 94. 83
  • 95. ResultsFor example, for testproblem 1, four groupswhich consist the 70% ofcombinations are selected,cluster numbers thatcontains minimum 80% ofthe all combinations aredetermined according tothe results of paretoanalysis. These are groups2-3 and 484
  • 96. ResultsParameter combinations, β-ρ interactions and the number of α parameters in any combination of β and ρ, is shown at the side. Favorable solutions are marked as bold and italic85
  • 97. ResultsNumber of cluster increases with the increase in ρ.Number of cluster increases with the increase in β.Clustering of the data in most problems depends on the interaction between the β and ρ parameters. α parameter has no effect on solution in small scaled problems, but in large scale problems, effect of α turns to an irregular state Also with the increase in problem scale, the change in number of clusters is defined.
  • 98. ResultsIn FMEA test problems, which determine most favorable parameter combinations, β-ρ interactions providing appropriate cluster numbers are noted on the summary table that evaluates each test problem separately. The values involve favorable β-ρ combinations are marked with the blue area. This is a suitable solution area for FMEA problem. 87
  • 99. 88ART 1: Clustering ApplicationClustering pixel based alphabet images
  • 100. 89Conclusion and DiscussionFuzzy ART neural network is applied to FMEA Appropriate parameter intervals are investigated for giving successful results of Fuzzy ART in FMEA problems. The investigations show us, if input number is smaller than or equal to 30, FMEA problem is defined as small scale, otherwise it is large scale. We suggest that cluster numbers should be determined between 2 and 6 at small scale problems for practical studies.Cluster numbers of large scale problems should be maximum 12for practical studies.
  • 101. 90Conclusion and DiscussionDeterminations about α:In small scale problems, alfa increases cluster number only if β is greater than or equal to 0.8. In other conditions, it is observed that α values have no effect on solution. In large scale problems, appropriate interval cannot be determined because the effect of α becomes irregularDeterminations about β:For both small and large scale problems, number of cluster increases with the increase in β. Determinations about ρ:For both small and large scale problems, number of cluster increases with the increase in ρ.
  • 102. 91Conclusion and DiscussionFor small and large scale problems in FMEA, Fuzzy ART algorithm is fast, effective and easy to implement. Parameter combinations are acquired where the best solution is obtained for non-binary problems.
  • 104. References :Carpenter, G.A. and Grossberg, S. (1987a), "A massively parallel architecture for a self-organizing neural pattern recognition machine", Computer Vision, Graphics, and Image Processing 37, 54-115.Carpenter, G.A. and Grossberg, S. (1987b), "ART 2: Stable self-organization of pattern recognition codes for analog input patterns", Applied Optics 26, 4919-4930.Carpenter, G.A. and Grossberg, S. (1990), "ART 3: Hierarchical search using chemical transmitters in self-organizing pattern recognition architectures", Neural Networks 3, 129-152.Carpenter, G.A., Grossberg, S. and Reynolds, J.H. (1991a), "ARTMAP: Supervised real-time learning and classification of non-stationary data by a self-organizing neural network", Neural Networks 4, 565-588.