Optimization and Backpropagation

• calculates function

• caches values needed for gradient computation

• takes upstream derivative

• return all partial derivatives

wrt W

#neurons * #inputchannels + #biases

• too high: model may not converge to an optimal solution or converge too fast to suboptimum

• too low: model converges very slowly

• compute gradient for individual training pairs/batches -> take average

• prevent overfitting / hopefully better generalization

• make training harder so model learns better features

• lower validation error / increase training error

• add term weighted with a lambda

• enforce sparsity

• focus on few key features

• enforce similar weights

• consider all info