12_AI_Dev

1. problem definition

2. data preparation

3. model

4. hardware

5. training

6. tuning

7. inference

• What kind of problem?

• What are inputs and desired outputs?

• How much data is needed?

• How to get that data?

• Neural Network?

• Which layers?

• Which hardware for training?

• Which hardware for inference?

• Supervised?

• Which loss function?

• Which optimizer?

• Iterative change of model, hyperparamters, etc.

• Deploy the model for your desired problem

• public datasets

  • how well does this fit your problem defintion (inputs, outputs,…)

  • can the data be processed to fit your problem? (e.g. 3d bounding boxes -> 2d bounding boxes)

• create your own dataset

  • prototype needed?

  • data infrastructure (how to save data? how to handle large files?)

  • time consuming and expensive

• data prepartaion makes up 80% of your project -> crucial!!!

• but only sees around 1% of AI research…

• e.g. use statistics to see where planes most often hit by bullets to reinforce these areas

• -> bias: only planes that did survive these bullets came back to be evaluated (-> should reinforce areas where no bullets hit…)

• unbalanced sampling from a population

• e.g. tank detection

• -> images with tanks only at day

• -> images without tanks only at night…

• e.g. have less with people of color

• -> e.g. guy holding tehrmometer

  • white: thermometer

  • black: gun

  • …

  • 
    

    
    

• e.g. detect different vehicles

• -> 2000 motor cycles

• -> 10 cars

• => will not yield good results…

-> repeat samples with under-represented data

-> consider imbalance in loss function

• what to do when objects are overlapping?

• -> inconsistency can be problematic…

• => different ground truths…?

• IoU (intersection over union) = area of overlap / area of union

• => inconsistent labels can be considered as noise…

• -> smaller is worse

• for an x value -> no consistent y values…

• use more data to better middle…

• or clean data to have consistent labels…

Data augmentation!!!

• flip images

• rotate images

• scale images outward or inward

• crop images

• translation of objects in x,y position (verschieben); similar to crop…

• add gaussian noise…

  • defacto changes all pixle values…

• deep photo style transfer

  • fancy…

  • -> generate new image by new style (using generative NN…)

• depending on mehtod aroudn 10 percent

• -> overall feeing augmentation useful…

• sample biase

  • dataset does not actually represent the world

• exclusion bias

  • systematic exclusion of information

• measurement bias

  • data measurement for training differs from inference (e.g. for training differnt camera used…)

• recall bias

  • label similar types of data inconsistent

• observer bias

  • effect of seeing what you expect ot see or want to see in data

• Fully connected

• convolutional neural networks

• graph neural networks

• recurrent neural networks

• yes…

• step functoin

• sigmoid

• relu

• CPU

  • solve general / wide range of tasks

• GPU

  • designed to accelerate rendering of graphic

  • focus on parallelizuation

• TPU (tensor processing unit)

  • Designed to accelerate deep learning tasks

  • TPUs from Google especially for Tensorflow

• FPGA

• AISEC

• mainly serielles abarbeiten von inputs

• GPU stark parallelisiert…

• L1 (MAE)

• L2 (MSE)

• Binary Cross Entropy

-> BCE good for:

binary classification for clasificaiotn…

• computaton time

• local / global minima

• conergence tyme

• learning rate dependency

• number of hyperparameters (e.g. momentum)

• neccesarry to have starting point for optimization

• two important dimensions

  • value range

  • distribution

• best practice: start with default initialization from your ML library

• too small value lead to slow learning

• too large values may lead to divergence

• keep in mind vanishing and exploding gradients

• constant initialization performs poorly -> need for randomness

• xavier initialization with

  • mean of activations is 0

  • variance of activations stay the same across every layer

• keep weights in initiali layers

  • => in image classification: early layers have generic features (low level)

• re-train later layers

  • train the actually important, complex features specific to our task….

1. increase data set / data set augmentaiton

2. reduce model size

3. early stopping

4. regularization

5. dropouts

• sequence of data

  • else, find global minima difficult for optimizer

• initialization

• optimizer (e.g. SGD)

• randoim retularization (e.g. dropout)

• calculation architecture may non-dtereministic

• keep random seed

• -> but not all calculations might be deterministic

• => train NN several times with same settings to account for impact of randomness

• => comparability of NN results often difficult…

• learning rate

• decay rate

• bacth size

• number of epochs

• dropout rate

• regularization

• number of layers

• number of parameters in layer

• lots of them

• -> high dimensional optimization problem

• hyperparameters not independent of each other (-> change one influences effect of other)

• => for some rough explainable (e.g. learning rate and number of epochs)

• => for some relation not explainable (e.g. dropout rate and batch size)

• training may take very long (e.g. days)

• -> exhaustive serarch of hyperparameter combinations not feasible…

  
  

• bayesian optimiozation

• is gaussian process and gradient free

• not feaible for lots of hyperparameers (e.g. 4 okay, 30 not)

• featuer visualizatoin

• -> plot graphs to the weights / kernels….

• activation maps -> which inputs are important?

• precision calibration

• layer and tensor fusion

• kernel auto-tuning

• multi stream execution

• dynamic tensor memroy

  
  

• tensoroboard