THE IMPACT OF CROSS-ENTROPY AND MUTUAL INFORMATION ON HUMAN LANGUAGE ENGINEERING

Jie Zhao
Institute for Signal and Information Processing
Mississippi State University, Mississippi State, MS 39762
email: zhao@isip.msstate.edu

ABSTRACT

Information theory plays an important role in the field of natural language processing. A language model is a statistical model to predict the next word given the previous word sequences. And it's necessary to measure how accurate this language model is. Cross entropy and mutual information can be used to estimate the model quality.

Entropy is a measure of uncertainty of a random variable given a probability model. In most cases, the true probably is unknown, hence cross entropy --- the entropy calculated with respect to an estimated probability model, is used to estimate the true entropy and estimate how close our estimated probability model is to the actual model. Mutual information is a measure of the amount of information that one random variable contains about another random variable. It is the reduction in the uncertainty of one random variable due to the knowledge of the other.

In this talk, I will introduce:

  • fundamentals of Information theory entropy, cross entropy, relative entropy and mutual information
  • language models
  • applications of infomation theory in language models
Additional items of interest: