Information Indices: Unification and Applications

Numerous information indices have been developed since the unified framework of information theoretic statistics was established by Kullback (1959). Often, the indices are developed using mathematical properties of information functions and in contexts of specific problems rather than as integral parts of a system of analysis. This paper presents a summary of informational aspects of the basic information functions, explication of traditional measures in terms of information, unification of various information-theoretic modeling approaches and information indices. The unification thread is the discrimination information function: information indices are all logarithmic measures of discrepancy between two probability distributions. Examples of information indices for maximum entropy modeling, covariate information, and influence diagnostics are tabulated. The subjects of the indices presented include parametric model fitting, nonparametric entropy estimation, categorical data analysis, the linear and exponential family regression, and time series. The coverage however, is not exhaustive. The tabulation includes sampling theory and Bayesian indices, but the focus is on interpretation as descriptive measures and inferential propertics are noted tangentially. Applications of some information indices are illustrated through modeling data for customer duration and choice of long distance provider.

Information Indices: Unification and Applications Keywords:

Entropy , Kullback-Leibler information , Model fitting.