Example-5 (Sample weights)

Without weights

In [1]:
from pycm import ConfusionMatrix
y_test = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
y_pred = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]
In [2]:
cm1=ConfusionMatrix(y_test, y_pred)
cm1
Out[2]:
pycm.ConfusionMatrix(classes: [0, 1, 2])
In [3]:
print(cm1)
Predict          0    1    2    
Actual
0                3    0    0    

1                0    1    2    

2                2    1    3    





Overall Statistics : 

95% CI                                                           (0.30439,0.86228)
AUNP                                                             0.66667
AUNU                                                             0.69444
Bennett S                                                        0.375
CBA                                                              0.47778
Chi-Squared                                                      6.6
Chi-Squared DF                                                   4
Conditional Entropy                                              0.95915
Cramer V                                                         0.5244
Cross Entropy                                                    1.59352
Gwet AC1                                                         0.38931
Hamming Loss                                                     0.41667
Joint Entropy                                                    2.45915
KL Divergence                                                    0.09352
Kappa                                                            0.35484
Kappa 95% CI                                                     (-0.07708,0.78675)
Kappa No Prevalence                                              0.16667
Kappa Standard Error                                             0.22036
Kappa Unbiased                                                   0.34426
Lambda A                                                         0.16667
Lambda B                                                         0.42857
Mutual Information                                               0.52421
NIR                                                              0.5
Overall ACC                                                      0.58333
Overall CEN                                                      0.46381
Overall J                                                        (1.225,0.40833)
Overall MCC                                                      0.36667
Overall MCEN                                                     0.51894
Overall RACC                                                     0.35417
Overall RACCU                                                    0.36458
P-Value                                                          0.38721
PPV Macro                                                        0.56667
PPV Micro                                                        0.58333
Phi-Squared                                                      0.55
RCI                                                              0.34947
RR                                                               4.0
Reference Entropy                                                1.5
Response Entropy                                                 1.48336
SOA1(Landis & Koch)                                              Fair
SOA2(Fleiss)                                                     Poor
SOA3(Altman)                                                     Fair
SOA4(Cicchetti)                                                  Poor
Scott PI                                                         0.34426
Standard Error                                                   0.14232
TPR Macro                                                        0.61111
TPR Micro                                                        0.58333
Zero-one Loss                                                    5

Class Statistics :

Classes                                                          0                       1                       2                       
ACC(Accuracy)                                                    0.83333                 0.75                    0.58333                 
AUC(Area under the roc curve)                                    0.88889                 0.61111                 0.58333                 
AUCI(Auc value interpretation)                                   Very Good               Fair                    Poor                    
BM(Informedness or bookmaker informedness)                       0.77778                 0.22222                 0.16667                 
CEN(Confusion entropy)                                           0.25                    0.49658                 0.60442                 
DOR(Diagnostic odds ratio)                                       None                    4.0                     2.0                     
DP(Discriminant power)                                           None                    0.33193                 0.16597                 
DPI(Discriminant power interpretation)                           None                    Poor                    Poor                    
ERR(Error rate)                                                  0.16667                 0.25                    0.41667                 
F0.5(F0.5 score)                                                 0.65217                 0.45455                 0.57692                 
F1(F1 score - harmonic mean of precision and sensitivity)        0.75                    0.4                     0.54545                 
F2(F2 score)                                                     0.88235                 0.35714                 0.51724                 
FDR(False discovery rate)                                        0.4                     0.5                     0.4                     
FN(False negative/miss/type 2 error)                             0                       2                       3                       
FNR(Miss rate or false negative rate)                            0.0                     0.66667                 0.5                     
FOR(False omission rate)                                         0.0                     0.2                     0.42857                 
FP(False positive/type 1 error/false alarm)                      2                       1                       2                       
FPR(Fall-out or false positive rate)                             0.22222                 0.11111                 0.33333                 
G(G-measure geometric mean of precision and sensitivity)         0.7746                  0.40825                 0.54772                 
IS(Information score)                                            1.26303                 1.0                     0.26303                 
J(Jaccard index)                                                 0.6                     0.25                    0.375                   
MCC(Matthews correlation coefficient)                            0.68313                 0.2582                  0.16903                 
MCEN(Modified confusion entropy)                                 0.26439                 0.5                     0.6875                  
MK(Markedness)                                                   0.6                     0.3                     0.17143                 
N(Condition negative)                                            9                       9                       6                       
NLR(Negative likelihood ratio)                                   0.0                     0.75                    0.75                    
NPV(Negative predictive value)                                   1.0                     0.8                     0.57143                 
P(Condition positive or support)                                 3                       3                       6                       
PLR(Positive likelihood ratio)                                   4.5                     3.0                     1.5                     
PLRI(Positive likelihood ratio interpretation)                   Poor                    Poor                    Poor                    
POP(Population)                                                  12                      12                      12                      
PPV(Precision or positive predictive value)                      0.6                     0.5                     0.6                     
PRE(Prevalence)                                                  0.25                    0.25                    0.5                     
RACC(Random accuracy)                                            0.10417                 0.04167                 0.20833                 
RACCU(Random accuracy unbiased)                                  0.11111                 0.0434                  0.21007                 
TN(True negative/correct rejection)                              7                       8                       4                       
TNR(Specificity or true negative rate)                           0.77778                 0.88889                 0.66667                 
TON(Test outcome negative)                                       7                       10                      7                       
TOP(Test outcome positive)                                       5                       2                       5                       
TP(True positive/hit)                                            3                       1                       3                       
TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.33333                 0.5                     
Y(Youden index)                                                  0.77778                 0.22222                 0.16667                 
dInd(Distance index)                                             0.22222                 0.67586                 0.60093                 
sInd(Similarity index)                                           0.84287                 0.52209                 0.57508                 

With random weights

In [4]:
from random import randint
weights = [randint(1,10) for i in range(len(y_test))]
weights[2]*=9
In [5]:
cm2=ConfusionMatrix(y_test, y_pred, sample_weight = weights)
cm2
Out[5]:
pycm.ConfusionMatrix(classes: [0, 1, 2])
In [6]:
print(cm2)
Predict          0     1     2     
Actual
0                18    0     0     

1                0     9     10    

2                9     9     69    





Overall Statistics : 

95% CI                                                           (0.7006,0.84779)
AUNP                                                             0.77955
AUNU                                                             0.80432
Bennett S                                                        0.66129
CBA                                                              0.64448
Chi-Squared                                                      92.88202
Chi-Squared DF                                                   4
Conditional Entropy                                              0.81412
Cramer V                                                         0.61198
Cross Entropy                                                    1.20222
Gwet AC1                                                         0.69957
Hamming Loss                                                     0.22581
Joint Entropy                                                    1.99166
KL Divergence                                                    0.02468
Kappa                                                            0.54762
Kappa 95% CI                                                     (0.40019,0.69506)
Kappa No Prevalence                                              0.54839
Kappa Standard Error                                             0.07522
Kappa Unbiased                                                   0.54546
Lambda A                                                         0.24324
Lambda B                                                         0.4
Mutual Information                                               0.4833
NIR                                                              0.70161
Overall ACC                                                      0.77419
Overall CEN                                                      0.35152
Overall J                                                        (1.69944,0.56648)
Overall MCC                                                      0.55399
Overall MCEN                                                     0.45518
Overall RACC                                                     0.50085
Overall RACCU                                                    0.50322
P-Value                                                          0.04494
PPV Macro                                                        0.68003
PPV Micro                                                        0.77419
Phi-Squared                                                      0.74905
RCI                                                              0.41044
RR                                                               41.33333
Reference Entropy                                                1.17754
Response Entropy                                                 1.29743
SOA1(Landis & Koch)                                              Moderate
SOA2(Fleiss)                                                     Intermediate to Good
SOA3(Altman)                                                     Moderate
SOA4(Cicchetti)                                                  Fair
Scott PI                                                         0.54546
Standard Error                                                   0.03755
TPR Macro                                                        0.7556
TPR Micro                                                        0.77419
Zero-one Loss                                                    28

Class Statistics :

Classes                                                          0                       1                       2                       
ACC(Accuracy)                                                    0.92742                 0.84677                 0.77419                 
AUC(Area under the roc curve)                                    0.95755                 0.69398                 0.76142                 
AUCI(Auc value interpretation)                                   Excellent               Fair                    Good                    
BM(Informedness or bookmaker informedness)                       0.91509                 0.38797                 0.52283                 
CEN(Confusion entropy)                                           0.23219                 0.50312                 0.35007                 
DOR(Diagnostic odds ratio)                                       None                    9.6                     10.35                   
DP(Discriminant power)                                           None                    0.54155                 0.55957                 
DPI(Discriminant power interpretation)                           None                    Poor                    Poor                    
ERR(Error rate)                                                  0.07258                 0.15323                 0.22581                 
F0.5(F0.5 score)                                                 0.71429                 0.49451                 0.85608                 
F1(F1 score - harmonic mean of precision and sensitivity)        0.8                     0.48649                 0.83133                 
F2(F2 score)                                                     0.90909                 0.47872                 0.80796                 
FDR(False discovery rate)                                        0.33333                 0.5                     0.12658                 
FN(False negative/miss/type 2 error)                             0                       10                      18                      
FNR(Miss rate or false negative rate)                            0.0                     0.52632                 0.2069                  
FOR(False omission rate)                                         0.0                     0.09434                 0.4                     
FP(False positive/type 1 error/false alarm)                      9                       9                       10                      
FPR(Fall-out or false positive rate)                             0.08491                 0.08571                 0.27027                 
G(G-measure geometric mean of precision and sensitivity)         0.8165                  0.48666                 0.83229                 
IS(Information score)                                            2.19931                 1.70627                 0.316                   
J(Jaccard index)                                                 0.66667                 0.32143                 0.71134                 
MCC(Matthews correlation coefficient)                            0.78107                 0.39672                 0.49751                 
MCEN(Modified confusion entropy)                                 0.26416                 0.52841                 0.48721                 
MK(Markedness)                                                   0.66667                 0.40566                 0.47342                 
N(Condition negative)                                            106                     105                     37                      
NLR(Negative likelihood ratio)                                   0.0                     0.57566                 0.28352                 
NPV(Negative predictive value)                                   1.0                     0.90566                 0.6                     
P(Condition positive or support)                                 18                      19                      87                      
PLR(Positive likelihood ratio)                                   11.77778                5.52632                 2.93448                 
PLRI(Positive likelihood ratio interpretation)                   Good                    Fair                    Poor                    
POP(Population)                                                  124                     124                     124                     
PPV(Precision or positive predictive value)                      0.66667                 0.5                     0.87342                 
PRE(Prevalence)                                                  0.14516                 0.15323                 0.70161                 
RACC(Random accuracy)                                            0.03161                 0.02224                 0.447                   
RACCU(Random accuracy unbiased)                                  0.03292                 0.02226                 0.44804                 
TN(True negative/correct rejection)                              97                      96                      27                      
TNR(Specificity or true negative rate)                           0.91509                 0.91429                 0.72973                 
TON(Test outcome negative)                                       97                      106                     45                      
TOP(Test outcome positive)                                       27                      18                      79                      
TP(True positive/hit)                                            18                      9                       69                      
TPR(Sensitivity, recall, hit rate, or true positive rate)        1.0                     0.47368                 0.7931                  
Y(Youden index)                                                  0.91509                 0.38797                 0.52283                 
dInd(Distance index)                                             0.08491                 0.53325                 0.34037                 
sInd(Similarity index)                                           0.93996                 0.62294                 0.75932