4 - Information Theory

‘information is reduction of uncertainity’

Uncertainty

refers to lack of knowledge/predactability about the outcome of an event

the more uncertain a system —> the less we know about the state

Information

reduces uncertainty

when event occurrs —> knowledge of outcome reduces number of possible states

the more uncertain a system, the more information is gained when the uncertainty is resolved

1. if all possible states are equally like, uncertainty grows with number of states

2. uncertainty should be continuous in probability of states

   (small changes in probability of states should lead to small changes in measures)

3. ‘Branching’ property

   (how uncertainty behaves when decisions split in sequential steps (branching))

—> there is only one measure of uncertainty that satisfies all 3 axioms

measured by using entropy(H)

R = random variable, representing possible outcomes

p(Ri​) = prob of outcome Ri

log2 = measurement of information in bits

higher entropy —> greater uncertainty

lower entropy —> less uncertainty

= measure of uncertainty about its state.

it describes the capacity of that system to carry information

1. Stimulus (S) = input to system (e.g. sensory signal)

2. Channel (e.g neuron) = through which stimulus is processed/transmitted to produce a response

3. Response (R) = ouput of system (e.g. neuronal firing)

= generalization of Pearson’s correlation to nonlinear dependencies

• Quantitative Description of Experimental Observations

  —> quantify and explain observations

• Rigorous Predictions of Unobserved Phenomena

  —> predict untested bahaviors and outcomes

• Unifying Perspective on Diverse Systems

  —> compare and analyze seemingly unrelated biological systems