Neural Networks

• introduce non-linearity for higher expressiveness/capacity and better learning

• linear layers stacked on top of each other are just linear

• easily viewed as matrices -> matrix calculations

• helps with abstracting calculations (e.g. gradient descent)

1. sigmoid

2. tanH

3. ReLU

4. Leaky ReLU

5. parametric ReLU

6. maxout

7. ELU

• compute nodes: matrix operations

• vertex nodes: variable/operators

• directed edges: flow of input

sum of absolute differences:

• robust, but optimization costly

• optimum = median

sum of squared differences

• prone to outliers

• efficient optimization, optimum = mean

• one-hot encoding

log(p) where p in [0,1] -> negative values that need to be turned around

• start at last layer and use chain rule

easily modularize complex functions

• introduce non-linearity: more expressiveness/capacity and better learning

• linear layers on top of each other are just linear

• higher-level purpose: select key features