06 Classifiction / Supervised ML

In decision trees, the error rate refers to the proportion of incorrect predictions made by the model.

Error rate

 Information gain

 Gini impurity (Gini index)

 Variance reduction



high Entropie -> high uncertainty

Calculate entropy before and after the split.

Information Gain = Initial Entropie der Zielvariable - Bedingungsentropie der Zielvariable gegeben das Merkmal

—> Da wo IG am höchsten ist, wird getrennt

Overfitting in decision trees occurs when the model is too complex and is able to fit the noise in the training data, leading to poor performance on unseen data. Common causes include having too many branches or too deep of a tree, using too many features or variables.

"Pruning" means removing nodes from a tree

after training has finished

Redundant nodes are removed, sometimes tree is remodeled

• - Remove a non-leaf node from the tree

  - Evaluate the performance without the pruned node

Stopping criteria are sometimes referred to as

"pre-pruning"

• Stoppen, wenn die absolute Anzahl an Proben niedrig ist (unter einem Schwellenwert)

• Stoppen, wenn die Entropie bereits relativ niedrig ist (unter einem Schwellenwert)

• Stoppen, wenn der Informationsgewinn niedrig ist

• Stoppen, wenn die Entscheidung zufällig sein könnte (Chi-Quadrat-Test)

• Schwellenwerte sind Hyperparameter

max. 7

• type of supervised learning algorithm

• can be used for classification and regression tasks.

• SVMs arbeiten indem sie den Hyperplan finden, der die verschiedenen Klassen im Merkmalsraum am besten trennt, während gleichzeitig den Abstand zwischen den nächstgelegenen Datenpunkten der verschiedenen Klassen maximiert. Die Punkte, die dem Hyperplan am nächsten sind, werden Support Vectors genannt und haben eine Schlüsselfunktion bei der Bestimmung der Position des Hyperplans. SVMs werden häufig für hochdimensionalen Daten verwendet und können reguliert werden, um Überanpassung zu vermeiden.

Soft margin

Hyper plane that splits “as cleanly as possible/desirable”

• While maximising margin

Linear separation works more likely in higher

dimensional space

Idea of SVMs – projection of data into higher

dimensional space

Add addition coordiante

To overcome limitation of not linearly sperable data , the SVM algorithm can be extended by using a kernel trick. A kernel is a mathematical function that maps the input data into a higher dimensional space, where the data may be more easily separated. By using a kernel, the SVM algorithm can find a non-linear decision boundary that can separate the different classes in the data.

Quadratic Kernel

 Radial Basis Kernel

 General Polynomial Kernel (arbitrary degree)

 Linear Kernel (=no kernel)

(when could this be the optimal kernel?)

100

Multi-class SVM is an extension of the standard SVM algorithm that can handle multiple classes by training multiple binary classifiers. There are several strategies to perform multi-class classification with SVMs, including:

• One vs. All

  - Binary classifiers that distinguish between class i and the rest

  For example, in a 3-class problem (A, B, C), three binary classifiers are trained: one to separate class A from classes B and C, one to separate class B from classes A and C, and one to separate class C from classes A and B. The final prediction is based on the classifier that has the highest confidence score.r

• One vs. One

  - Build binary classifier for each pair of classes

  - Class with highest number of votes wins

  For example, in a 3-class problem (A, B, C), three binary classifiers are trained: one to separate class A from class B, one to separate class A from class C, and one to separate class B from class C. The final prediction is based on a voting system, where the class that wins the majority of the binary classifiers is chosen.

High classification accuracy

Linear kernels: Good for sparse, high dimensional data, e.g. text mining

Much research has been directed at SVM,

Implementation available in open-source software

1. Finds the k number of closest training examples

2. Uses majority class or average value of those examples to predict for new data point

3. Distance metric (e.g. Euclidean) used to measure similarity

4. k is a hyperparameter that can be adjusted for performance

1. azy learning algorithms do not require a training phase, thus they are faster to execute.

2. Lazy learning algorithms are memory efficient as they only store the training dataset and not the model.

3. Lazy learning algorithms can handle large datasets with high dimensionality.

Precision

True Positive

False Positive

How much more we could have had

R = TP / (TP+FN)

True positive

False negativ

Generalization is an issue in machine learning because a model that has been trained on a specific dataset may not perform well when applied to new, unseen data. This is because the model has learned patterns and relationships specific to the training dataset, which may not be representative of the population as a whole.

Create new input data from existing instances

- Adding noise

- Distorting data (scaling, sampling, rotating, …)

- Combining individual instances

Be aware of potential systematic bias

Only for training data (possibly for validation data)

Valdidation of model -> It involves dividing the data into k subsets (folds) and training the model k times, each time using a different subset as the validation set and the remaining subsets as the training set. The performance of the model is then averaged across all k iterations.

1. k-CV is useful to obtain a more robust estimate of the model performance.

2. k-CV is useful to use the whole dataset for training and validating.

3. k-CV is useful to avoid overfitting by rotating the validation set.

4. 
   

A bootstrap sample is a random subset of the data

sample

The goal of bootstrapping is to estimate the distribution of a sample statistic by repeatedly drawing samples from the original dataset and calculating the statistic of interest for each sample

Statistical Significance Testing is useful to determine whether an observed difference is real or due to chance.

precision / recall

- micro / macro performance

- baseline and bias

- k-fold cross-validation; leave-one-out

- statistical significance AND relevance

- human validation; consistency in evaluator decisions

- how fine-granular is the evaluation scale?