evaluation {CMA} | R Documentation |
The performance of classifiers can be evaluted by six different measures and two different schemes that are described more precisely below.
For S4 method information, s. evaluation-methods
.
evaluation(clresult, cltrain = NULL, cost = NULL, y = NULL, measure = c("misclassification", "sensitivity", "specifity", "average probability", "brier score", "auc", "0.632", "0.632+"), scheme = c("iterationwise", "observationwise", "classwise"))
clresult |
A list of objects of class cloutput or clvarseloutput
|
cltrain |
An object of class cloutput in which the
whole dataset was used as learning set. Only
used if method = "0.632" or method = "0.632+"
in order to obtain an estimation for the resubsitution
error rate. |
cost |
An optional cost matrix used if measure = "misclassification" .
If it is not specified (default), the cost is the usual indicator
loss. Otherwise, entry i,j of cost quantifies the
loss when the true class is class i-1 and the predicted
class is j-1 , provided the conventional coding 0,...,K-1
in the case of K classes is used. Usually, the matrix contains
only non-negative entries with zeros on the diagonal, but this is
not obligatory. Make sure that the dimension of the matrix matches
the number of classes. |
y |
A vector containing the true class labels. Only needed if scheme = "classwise" . |
measure |
Peformance measure to be used:
|
scheme |
|
An object of class evaloutput
.
Martin Slawski martin.slawski@campus.lmu.de
Anne-Laure Boulesteix http://www.slcmsr.net/boulesteix
Efron, B. and Tibshirani, R. (1997). Improvements on cross-validation: The .632+ bootstrap method.
Journal of the American Statistical Association, 92, 548-560.
evaloutput
, classification
, compare
### simple linear discriminant analysis example using bootstrap datasets: ### datasets: data(golub) golubY <- golub[,1] ### extract gene expression from first 10 genes golubX <- as.matrix(golub[,2:11]) ### generate 25 bootstrap datasets set.seed(333) bootds <- GenerateLearningsets(y = golubY, method = "bootstrap", ntrain = 30, niter = 10, strat = TRUE) ### run classification() ldalist <- classification(X=golubX, y=golubY, learningsets = bootds, classifier=ldaCMA) ### Evaluation: eval_iter <- evaluation(ldalist, scheme = "iter") eval_obs <- evaluation(ldalist, scheme = "obs") show(eval_iter) show(eval_obs) summary(eval_iter) summary(eval_obs) ### auc with boxplot eval_auc <- evaluation(ldalist, scheme = "iter", measure = "auc") boxplot(eval_auc) ### which observations have often been misclassified ? obsinfo(eval_obs, threshold = 0.75)