Sunday, November 7, 2010

Single layer perceptron as linear classifier

Perceptron is the simplest type of feed forward neural network. It was designed by Frank Rosenblatt as dichotomic classifier of two classes which are linearly separable. This means that the type of problems the network can solve must be linearly separable. Basic perceptron consists of 3 layers:
  • sensor layer
  • associative layer
  • output neuron
There are a number of inputs (xn) in sensor layer, weights (wn) and an output. Sometimes w0 is called bias and x0 = +1/-1 (In this case is x0=-1).

For every input on the perceptron (including bias), there is a corresponding weight. To calculate the output of the perceptron, every input is multiplied by its corresponding weight. Then weighted sum is computed of all inputs and fed it through a limiter function that evaluates the final output of the perceptron.
The output of neuron is formed by activation of the output neuron, which is function of input:

The activation function F can be linear so that we have a linear network, or nonlinear. In this example I decided to use threshold (signum) function:


Output of network in this case is either +1 or -1 depending on the input. If the total input (weighted sum of all inputs) is positive, then the pattern belongs to class +1, otherwise to class -1. Because of this behavior, we can use perceptron for classification tasks.

Lets consider we have a perceptron with 2 inputs and we want to separate input patterns into 2 classes. In this case the separation between the classes is straight line, given by equation:


When we set x0=-1 and mark w0=θ then we can rewrite equation (3) into form:


Here I will describe learning method for perceptron. Learning method of perceptron is iterative procedure that adjust the weights. A learning sample is presented to the network. For each weight the new value is computed by adding a correction to the old value. The threshold is updated in the same way:


where y is output of perceptron, d is desired output and γ is the learning parameter.

More about program and source code, you can find on

1 comment:

  1. I learnt this when I was having the course Theory of Computation. Our teacher gave us the task to code out a program where there will 16 inputs by default which are the binary forms of 0-15. Binary of 0-7 is in class A and 8-15 is in class B. User will provide the 4 weights (as each member will have 4 digit as 4 inputs, such as the binary of 1 has the inputs 0,0,0,1) and the threshold value. Now the program will prepare 4 weights those will be able to distinguish the members of both class as

    (i1*w1)+(i2*w2)+(i3*w3)+(i4*w4) < T means Class A

    otherwise class B

    I liked you briefing about the algorithm.

    I have posted the algorithm and the code of the program in this post on my blog. I hope you will like that too.