- traditionally associated with uncertainty
- unknown form of the underlying model
- definition could vary by defining entropy as relative entropy wrt a prior distribution -- Jaynes definition
- used as a distance measure between distributions -- relative entropy