py4sci

Table Of Contents

Previous topic

Nearest neighbour classifier

Next topic

Random Forests

This Page

Support vector machines

This example covers Support Vector Machines (SVMs) and two ensemble methods: boosting (via gradient boosting in the gbm package) and random forests in the randomForest package.

Support Vector Machines

The default kernel for SVM is radial. In this example, we will use a linear kernel, following up later with a radial kernel. Read the help for svm to find out what kinds of kernels one can use, as well as the parameters of the kernels.

    library(e1071)
    iris = read.table("http://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data",
        sep = ",", header = FALSE)
    names(iris) = c("sepal.length", "sepal.width", "petal.length", "petal.width",
        "iris.type")
    model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel = "linear")
    model

    ##
    ## Call:
    ## svm(formula = iris.type ~ sepal.length + sepal.width, data = iris,
    ##     kernel = "linear")
    ##
    ##
    ## Parameters:
    ##    SVM-Type:  C-classification
    ##  SVM-Kernel:  linear
    ##        cost:  1
    ##       gamma:  0.5
    ##
    ## Number of Support Vectors:  77

In[55]:

def svm_iris_linear():
    grid = np.mgrid[3.5:8:500j,2:5:400j]
    gridT = grid.reshape((2,-1)).T
    %R -i gridT colnames(gridT) = c('sepal.length', 'sepal.width')
    %R model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel='linear')
    %R gridT = data.frame(gridT); labels = predict(model, gridT)
    %R -o labels -d iris

    plt.imshow(labels.reshape((500,400)).T, interpolation='nearest', origin='lower', alpha=0.4, extent=[3.5,8,2,5], cmap=pylab.cm.RdYlGn)
    plt.scatter(iris['sepal.length'], iris['sepal.width'], c=[col[t] for t in iris['iris.type']])
    a = plt.gca()
    a.set_xlim((3.5,8))
    a.set_ylim((2,5))

svm_iris_linear()
_images/svms_fig_00.png
<matplotlib.figure.Figure at 0x86cc290>

We can see which vectors are support vectors using a plot: the + points are support vectors.

    model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel = "linear")
    plot(iris$sepal.length, iris$sepal.width, col = as.integer(iris[, 5]), pch = c("o",
        "+")[1:150 %in% model$index + 1], cex = 2, xlab = "Sepal length", ylab = "Sepal width")

_images/svms_fig_01.png

There is a plot method for SVMs that makes it possible to see the decision boundaries.

    plot(model, iris, sepal.width ~ sepal.length, slice = list(sepal.width = 1,
        sepal.length = 2))

_images/svms_fig_02.png

Nonlinear kernel


    model = svm(iris.type ~ sepal.length + sepal.width, data = iris)
    model

    ##
    ## Call:
    ## svm(formula = iris.type ~ sepal.length + sepal.width, data = iris)
    ##
    ##
    ## Parameters:
    ##    SVM-Type:  C-classification
    ##  SVM-Kernel:  radial
    ##        cost:  1
    ##       gamma:  0.5
    ##
    ## Number of Support Vectors:  86

    plot(model, iris, sepal.width ~ sepal.length, slice = list(sepal.width = 1,
        sepal.length = 2))

_images/svms_fig_03.png

In[59]:

def svm_iris_kernel(gamma=0.5):
    grid = np.mgrid[3.5:8:500j,2:5:400j]
    gridT = grid.reshape((2,-1)).T
    %R -i gridT,gamma colnames(gridT) = c('sepal.length', 'sepal.width')
    %R model = svm(iris.type ~ sepal.length + sepal.width, data = iris, gamma=gamma)
    %R gridT = data.frame(gridT); labels = predict(model, gridT)
    %R -o labels -d iris

    plt.imshow(labels.reshape((500,400)).T, interpolation='nearest', origin='lower', alpha=0.4, extent=[3.5,8,2,5], cmap=pylab.cm.RdYlGn)
    plt.scatter(iris['sepal.length'], iris['sepal.width'], c=[col[t] for t in iris['iris.type']])
    a = plt.gca()
    a.set_xlim((3.5,8))
    a.set_ylim((2,5))

svm_iris_kernel()
_images/svms_fig_04.png
<matplotlib.figure.Figure at 0x86cfbb0>

Nonlinear kernels also have support vectors.

    model = svm(iris.type ~ sepal.length + sepal.width, data = iris)
    plot(iris$sepal.length, iris$sepal.width, col = as.integer(iris[, 5]), pch = c("o",
        "+")[1:150 %in% model$index + 1], cex = 2, xlab = "Sepal length", ylab = "Sepal width")

_images/svms_fig_05.png

The parameter \gamma controls the bandwidth of the kernel.

In[61]:

svm_iris_kernel(gamma=3)
_images/svms_fig_06.png
<matplotlib.figure.Figure at 0xace76d0>

In[62]:

def svm_iris_polynomial(gamma=0.5, degree=4):
    grid = np.mgrid[3.5:8:500j,2:5:400j]
    gridT = grid.reshape((2,-1)).T
    %R -i gridT,gamma,degree colnames(gridT) = c('sepal.length', 'sepal.width')
    %R model = svm(iris.type ~ sepal.length + sepal.width, data = iris, gamma=gamma, degree=degree, kernel='polynomial')
    %R gridT = data.frame(gridT); labels = predict(model, gridT)
    %R -o labels -d iris

    plt.imshow(labels.reshape((500,400)).T, interpolation='nearest', origin='lower', alpha=0.4, extent=[3.5,8,2,5], cmap=pylab.cm.RdYlGn)
    plt.scatter(iris['sepal.length'], iris['sepal.width'], c=[col[t] for t in iris['iris.type']])
    a = plt.gca()
    a.set_xlim((3.5,8))
    a.set_ylim((2,5))

svm_iris_polynomial(degree=2)
_images/svms_fig_07.png
<matplotlib.figure.Figure at 0xace4e90>
    model = svm(iris.type ~ sepal.length + sepal.width, data = iris, kernel = "polynomial",
        degree = 2, gamma = 0.5)
    plot(model, iris, sepal.width ~ sepal.length, slice = list(sepal.width = 1,
        sepal.length = 2))

_images/svms_fig_08.png

In[64]:

svm_iris_polynomial(degree=3)
_images/svms_fig_09.png
<matplotlib.figure.Figure at 0xad0db50>

In[65]:

svm_iris_polynomial(degree=4)
_images/svms_fig_10.png
<matplotlib.figure.Figure at 0xad10af0>