03_Regression

• regression

• classification

• clustering

• predict continuous values

• supervised

• e.g.

  • house pricing

  • sales

  • person weight

• predict discrete values

• supervised

• e.g.

  • object detection

  • spam detectoin

  • cancer detection

• predict discrete values

• unsupervised

• e.g.

  • genome patterns

  • google news

  • lidar processing

• vehicle parameters often only roughly known

• -> estimatoin by regression techniques

• financial relations -> e.g. car pricing

-> given data and model structure

=> possible to predict outcome of a process or system

• training dataset usually representaiton at sparse points

• contains lots of noise

• allows usage of information in simulation, optimization…

learning:

• model structure -> predictive model <- training data

prediction:

prevoiusly unseen sets of input variables

->

predictive model

->

predictions about output variables

• how can we extract information from data

• and use them to reason and predict

• in beforehand unseen cases?

• (learning)

• nearly all classical ML methods can be reinterpreted in terms of statistics

• focus in machine learning is mainly on prediction

• statistics focusses on relation analysis

• lots of advanced regression techniques build upon a statistical interpretatino of regression

• y = b + SUM wi * Φi(x)

• y := output variables

• b := Bias term

• wi := weight parameters

• Φi := basis functions

y = [1 Φ1(x) … Φk(x)] [b w1 … wk]T

• nonlinear optimization (fit curve through datapoints…)

• gaussian process

  • kernels, non parametric, bayesian interpretation)

• SVM

  • kernels, non parameteric, mostly used for classification

1. Choose model

2. Find Parameters

3. Validate Model

• type of basis functions

• number of basis functions

• specify loss function (-> measure how good model fits the data)

• specify constraints on the parameters

• use explicit solution or iterative algorithms to obtain parameters

• use second dataset which was not used for training

  • to evaluate the performance of your model

• this ensures ability to generalize to unseen data

• linear function

• sinusodia function

• polynomial function

• gaussian basis function

• golbally defined on independent variable domain

• design matrix (model) becomes ill-conditioned for large input domain variables for standard polynomiels

• hyperparameter:

  • polynomial degree

• Φ0(x) = 1

• Φ1(x) = x

• Φ2(x) = x^2

• Φ3(x) = x^3

• …

• Φi(x) = x^i

• locally defined on independent variable domain

• sparse design matrix

• infinitely differentiable (e^x…)

• hyperparameters

  • number of gaussian functions

  • width s of each basis function

  • mean my of each basis function

• polynomial -> gets worse globallly (affect globally…)

• gaussian -> gets worse only locally…

• sigmoid

  • bounded

  • globally defined

• bin functions (box aroud [-1,1]

  • bounded

  • locally defined

  • no continuity

• piercewise linear

  • not bounded

  • globally defined

  • max(0,x)

• measures accuracy of a model based on training dataset

• best model we can obtain is minimum loss model

  • choice of loss function fundamental in regression problem

pros:

• very important in practical applications

• solution can be easily obtained analytically

cons:

• not robuts to outliers

examples:

• basic regression

• energy optimization

• control applications

pros:

• robust to outliers

cons:

• no analytical solutoin

• non-differentiable at the origin

examples:

• financial applications

pros:

• combines strengths and weaknesses of L1 and L2

• robust + differentiable

cons:

• more hyperparameters

• no analytical solution

• L2 is differentiable

• L1 more intuitive

• Huber combines theoretical steengths of both

=> start with L2

• optimization problem

• with model y = w1*x + b

• insert datapoints

• => optimal solutions are obtained where the gradient is zero…

=> calculate gradient w.r.t w and b

=> set to 0

=> find optimal parameter for given y,x pairs

• apply regression during operation

• not enough memory to store all data points / not all data points available at the same time…

• recursive least squares (RLS)

• we have memory matrix

• and weigths

• we iteratively update each step

• initialization

  • possibilities:

  • use first datapoints available

  • use values wich you asusme to be right (e.g. based on pyhsical knowledge / laws)

• some applicatoins show slowly varying conditions in logn term

• but can be considered stationary on short to medium time periods

• -> e.g. aging of products lead to slight parameter changes…

• other: vehicle mass usually constant over significantly time period…

=> RLS algo can deal with this by introducint forgetting factor

-> reduces the weight of old samples…

• solution to regression

• important for large-scale problems

• and non-quadratic loss functions

• gradient descend

• gauss-newton

• levenberg-marquardt

pros:

• very generic

cons:

• knowledge about numeric optimization necessary

• weights can be interpreted as physical quantities

  • temperatire

  • spring constants

  • mass

• valid range known for the weithts

• introduce c1 <= w <= c2…

• => improves robustness

• more difficult to solve

1. quadratic cost funciton?

   • no -> numeric iterative

   • yes -> 2

2. are there parameter constraints?

   • yes -> numeric iterative

   • no -> 3

3. is the dataset very large?

   • yes -> numeric iterative

   • no -> 4

4. is all data available instantanously?

   • yes -> analytic

   • no -> sequential analytic

underfitted <-> well done <-> overfitted

underfitted:

• not enough features

• wrong structure

overfitted:

• too many featuers

• irrelevant features

• overly complex model

• overfitting: failure to generalize properly between datapoints

• cost function decreases with increased model complexity

• noise and irrelevant effects become too important

• large model complexity most common cause

• advisable: look for least complex model with decent performance on training and testing dataset

  • -> more likely to generarlize and less prone to numerical issues and outliers

• difficult to quantify wether model is overfitting

• depends on application and structure of dataset…

• overfitting occurs if:

  • data points sparse

  • model complexity high

• sparsity of data points is difficutlt to graps

• sparsity increases fast with increased input dimensionality

=> helpful if data points “nicely” located in sample space…

• difficult to judge overfitting in high-dimensional domains and autonomous sysetems

• => standard technique: separate data in training and validation data

• validation dataset must reflect future properties of underlying physical relationship

• do not reuse validation datsaets

  • if used again and again for testing model

  • -> somehow incorporated into moddelin proces

  • -> does not give expected results anymore

• split before fitting model is essential

  • -> 2/3 good start for training and 1/3 for vlaidatoin

• !! visualize data as much as posisble

• e.g. have limited data set size

• -> one may not want to remove a substantial part for validatoin

• => smaller validation sets to estimate true prediciton error by splitting data in multiple “folds”

• => variance of estimation error is indicator for model stability…

• => k=4 -> solit dataset into 4 parts, and use each iteration another quater for validation…

• Basic goal: choose model structure based on underlying physical principles and not on characteristics of dataset

• polynomial

  • tend to have larger coefficients for sparse datasets

• gaussian:

  • tend to overfit locallyleading to single, large coefficients

• => circumvent this using regularization…

• penalize high coefficients in the optimization

• weighting of the penalty term gives intuitive hyperparameter to control model complexity

• L2 / Thikonov

• L1

• prevents overfitting well

• analytic solution is available as an extension to the MSE problem

• difficult to apply and tune in high-dimensional feature spaces

• tends to produce sparse solutions

  • -> can therefore be applied for feature selection

• spase solutions mean several coefficients go to zero…

• L1: Lasso Regression

• L2: Ridge Regression

• use high dimensional input space for test model

• apply ridge regression

• regularized solutions perform far better at interpolation…

• note: must evaluate between sample points…

• plot train and test loss for different lambda (regularization weight)

• -> the lower the lambda, the higher the model complexity, the higher, the more underfitting…

• bias and variance…

• => study predictors performance on perviously unseen data

• -> evaluate the bias and variance of the outputs

• => high bias -> model is underfitting, as it is wrong for many data points

• => high variance -> model is overfitting, as it is very wrong for couple of data points…

• consider repeating fitting lots of times

• -> high variance -> very noisy, althoug in general correct fitting…

• -> high bias -> less noisy but more incorrect fitting…

• cannot ensure in general to reach low bias and variance at same time

• -> have to balance according to our objective

• => low bias and variance requires large, high quality datasets

• independent variable (x-axis) assumed to be noise free

• => often not true in engineering applications

• other methods to consider this:

  • total least squares

  • Principal Component Analysis (PCA)