LDA (Linear Discriminant Analysis)

1. Class conditional prob. is assumed MVN

2. Compute the posterior prob. P(G=k |X=x)  (by Bayes rule)

3. Assign x to the class k with the highest posterior prob. (it can be also done by using discriminant function)

4. Fisher's approach : Find the linear combination of X s.t the ratio of between variance and within variance is maximized

5. LDA can be used the dimension reduction like PCA. PCA finds the direction(coordinates) that maximizes variance, LDA finds the direction that maximizes separateness.

sample code

lda1 <- lda(Species~., data=iris)

> lda1

Call:

lda(Species ~ ., data = iris)

Prior probabilities of groups:

    setosa versicolor  virginica

 0.3333333  0.3333333  0.3333333

Group means:

           Sepal.Length Sepal.Width Petal.Length Petal.Width

setosa            5.006       3.428        1.462       0.246

versicolor        5.936       2.770        4.260       1.326

virginica         6.588       2.974        5.552       2.026

Coefficients of linear discriminants:

                    LD1         LD2

Sepal.Length  0.8293776  0.02410215

Sepal.Width   1.5344731  2.16452123

Petal.Length -2.2012117 -0.93192121

Petal.Width  -2.8104603  2.83918785

Proportion of trace:

   LD1    LD2

0.9912 0.0088

#Here, LD1 separates the class 99.12% and LD2 0.0088

lda1$svd^2/sum(lda1$svd^2)

[1] 0.991212605 0.008787395

#Let's check the plot of LD1, LD2

plot(lda1)

#change the plot

#change colors

plot(lda1,col = as.integer(iris$Species))

##too messy?

plot(lda1, panel = function(x, y, ...) points(x, y, ...),

 col = as.integer(iris$Species), pch=20)

plot(lda1, panel = function(x, y, ...) points(x, y, ...),

 col = as.integer(iris$Species), pch=as.integer(iris$Species),

main="LDA plot for iris data")

legend(7,-5,levels(iris$Species),pch=1:3,col=1:3)

lda1.pred<-predict(lda1,iris)

#missclassification rate?

mean(ifelse(iris$Species!=lda1.pred$class,1,0))

#confusion matrix

table(iris$Species,lda1.pred$class)