Pattern Recognition Applet: Relevance Vector Machines

The relevance vector machine algorithm is a pattern classification algorithm. Basically, the algorithm tries to separate or classify two or more different classes with a line of discrimination. In this case, the different colored points represent different classes, and the algorithm tries to separate the collections of points with a line. A classification algorithm is deemed successful if there are no points of different color on the same side of the line. The RVM algorithm classifies in two main steps.

RVM Training: The computation of the relevance vectors involve the algorithm picking a few points from each class to that will help characterize the line of discrimination. These points lay out a path for the line to go between. The selected relative vectors are designated considering all the points from the input.
Decision Regions: This calculation entails the final drawing of the line of discrimination. The relevance vectors or points determined in the previous step are given more weight in the decision of the path of the line.

The two steps are run in succession. Then the error is noted. Again error occurs if a point is misclassifyied or it is on the wrong side of the line of discrimination.

Here is an example run through of the relevance vector machine algorithm.

First, the user must enter points in the Input Display box. Obviously, more than one class or color of points must be entered for the algorithm to be able to classify anything. To draw points manually click on the Input Display box with more than one class of points. Different classes of points can be selected from the Classes menu. The applet has built-in patterns that can be rendered anytime. These patterns can be found in the Patterns menu.

For this example, points were manually drawn with the Draw Gaussian capability listed under the Patterns menu. Then, select Relevance Vector Machine from the Algorithms menu. The data should then be initialized from the Go menu.
To go through each step click Next from the Go for each step. Step 1, after initialization, displays the input points in the Output Display box.
Step 2 computes the relevance vectors. This is the stage that determines which points are crucial in determining the path of the line of discrimination that will be drawn in Step 3.
The last step completes the algorithm by drawing the line of discrimination. Also, the classification error is given. If any of the points are misclassifyied, or they fall on the wrong side of the line, it will be noted how many and what percentage.

The RVM algorithm is similar to the SVM algorithm. They both derive a line of discrimination by using only the support or relevance vectors. The differences lie in the way those vectors are created.

The support vectors in SVM training are created considering only points near the eventual boundary of the classes. Then all the other points are ignored.
The relevance vectors in RVM training are created considering all points from the input.

The differences in RVM/SVM training causes differences in the following:

Training Speed - SVM training generally produces more support vectors faster. It is less computationally involved than RVM training because RVM training takes into account all the points from the input.
Final Classification Speed - RVM final classification is faster than SVM final classification because it involves many fewer points. SVM final classification has to work with all the points that are near the boundary.

Much of the work in the RVM algorithm is done in the training whereas much of the work in the SVM algorithm lies after the training.

Click here to go back to the main tutorial page.