1. Direct Methods

Direct methods will build classifiers that directly minimize the misclassification costs. In other words, direct methods will take into consideration the cost while building models.

• Ling et al. (2004) proposed a method which uses a criterion of minimal total cost on training data, instead of minimal entropy, to build decision trees. However, I did not find any library of this algorithm. If you do, please let me know!

• Chen et al. (2004) proposed a way to incorporate class weights into Random Forest, thus making it cost-sensitive. Specifically, class weights are used to weight the Gini criterion for finding splits during the tree growing process; class weights are also taken into consideration in the terminal nodes of each tree, i.e. the class prediction of each terminal node is determined by “weighted majority vote”. Free softwares can be downloaded here.

• CART, penalized-SVM and C5.0 are some common approaches that can incorportate costs into the training process.

2. Meta-learning

Different from direct methods, meta-learning methods do NOT handle costs in the learning process. Meta-learning methods build cost-sensitive classifiers by pre-processing the training data or post-processing the output, i.e. sampling or thresholding. Technically speaking, meta-learning can convert any cost-insensitive algorithm into a cost-sensitive one.

1.Baseline

First, let us define training parameters and train a random forest model to serve as the baseline.

## CV
cvCtrl <- trainControl(method = "cv",
                       summaryFunction = custom_cost,
                       classProbs = TRUE)

## Baseline
set.seed(789)
rfFit <- train(score ~., data = training,
                 method = 'rf',
                 trControl = cvCtrl,
                 ntree = 1000,
                 tuneLength = 10,
                 metric = "cost")

baseProb <- predict(rfFit, newdata = test, type = "prob")[,1]
basePred <- predict(rfFit, newdata = test)
baseCM <- confusionMatrix(basePred, test$score)
baseCM$table

##           Reference
## Prediction good bad
##       good  140  53
##       bad     0   7

2. Sampling

For the sampling method, I will use a hybrid sampling method – SMOTE. Different from oversampling, which oversamples the minority class with replacement, SMOTE generates synthetic examples from the minority class.

## Sampling
set.seed(111)
smoteTrain <- SMOTE(score ~ ., data=training)
table(smoteTrain$score) # class distribution after sampling

## 
## good  bad 
##  840  630

set.seed(789)
smoteFit <- train(score ~., data = smoteTrain,
                  method = 'rf',
                  trControl = cvCtrl,
                  ntree = 1000,
                  tuneLength = 10,
                  metric = "cost")

smotePred <- predict(smoteFit, newdata = test)
smoteCM <- confusionMatrix(smotePred, test$score)
smoteCM$table

##           Reference
## Prediction good bad
##       good  122  39
##       bad    18  21

3. Thresholding

For the thresholding method, I will use the baseline model to predict the evaluation set and find the best threshold that is closest to the perfect model (specificity = sensitivity =1). Then I will relabel the predicted class from the baseline model using the new threshold.

## Thresholding
evalResults <- data.frame(score = evaluation$score)
evalResults$RF <- predict(rfFit,
                          newdata = evaluation,
                          type = "prob")[,1]
# best threshold
pred <- prediction(evalResults$RF, evalResults$score)
cost.perf <- performance(pred, "cost", cost.fp = 7, cost.fn = 3)
rfThresh <- pred@cutoffs[[1]][which.min(cost.perf@y.values[[1]])]
# predict the test set using the new threshold
threPred <- factor(ifelse(baseProb > rfThresh,
                          "good", "bad"),
                   levels = levels(evalResults$score))
threCM <- confusionMatrix(threPred, test$score)
threCM$table

##           Reference
## Prediction good bad
##       good   91  12
##       bad    49  48

4. Direct methods

For direct methods, I will use a C5.0 model. Since the predict function for C5.0 does not produce probabilities when employing costs, we cannot use ROC as the metric, so I will use Kappa instead. Besides, C5.0 function can only have 100 boosting iterations at most.

## C5.0
cvCtrlNoProb <- cvCtrl
cvCtrlNoProb$summaryFunction <- defaultSummary
cvCtrlNoProb$classProbs <- FALSE

set.seed(789)
c5Fit <- train(score ~., data = training,
               method = 'C5.0',
               trControl = cvCtrlNoProb,
               tuneGrid = data.frame(.model = "tree",
                                     .trials = c(1:100),
                                     .winnow = FALSE),
               metric = "Kappa",
               cost = cost_mat)

c5Pred <- predict(c5Fit, newdata = test)
c5CM <- confusionMatrix(c5Pred, test$score)
c5CM$table

##           Reference
## Prediction good bad
##       good  131  35
##       bad     9  25

5. Comparison

Here, we only tested various approaches on 1/10 of the whole dataset and there are many random procedures involved. To compare these methods and get a consistent result, I would repeat all the above steps times and take the average of the test set cost.

source("code/costsensitive.R")
res = test_cost_repeat(10)
apply(res, 2, mean)

##     Baseline        SMOTE thresholding         C5.0 
##        388.5        297.5        264.7        306.9

6. Next steps

In terms of the method comparison, the cross-validation might be a better way compared to repeated sampling. My next step would be to use an external cross-validation to validate all the instances.