| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
liam{at}cns.nyu.edu, Center for Neural Science, New York University, New York, NY 10003, U.S.A.
We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This "inconsistency" theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual 
formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if "bias-corrected" estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods.
Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.
This article has been cited by other articles:
|
|
 |
|
  G. A. Jacobs, J. P. Miller, and Z. Aldworth Computational mechanisms of mechanosensory processing in the cricket J. Exp. Biol., June 1, 2008; 211(11): 1819 - 1828. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 M. M. Van Hulle Sequential Fixed-Point ICA Based on Mutual Information Minimization Neural Comput., May 1, 2008; 20(5): 1344 - 1365. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 T. O. Sharpee, K. D. Miller, and M. P. Stryker On the Importance of Static Nonlinearity in Estimating Spatiotemporal Neural Filters With Natural Stimuli J Neurophysiol, May 1, 2008; 99(5): 2496 - 2509. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. A. Montemurro, R. Senatore, and S. Panzeri Tight Data-Robust Bounds to Mutual Information Combining Shuffling and Model Selection Techniques Neural Comput., November 1, 2007; 19(11): 2913 - 2957. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Shlens, M. B. Kennel, H. D. I. Abarbanel, and E. J. Chichilnisky Estimating information rates with confidence intervals in neural spike trains. Neural Comput., July 1, 2007; 19(7): 1683 - 1719. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 A. V. M. Herz, T. Gollisch, C. K. Machens, and D. Jaeger Modeling single-neuron dynamics and computations: a balance of detail and abstraction. Science, October 6, 2006; 314(5796): 80 - 85. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 W. Zhao, E. Serpedin, and E. R. Dougherty Inferring gene regulatory networks from time series data using the minimum description length principle Bioinformatics, September 1, 2006; 22(17): 2129 - 2135. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 B. N. Lundstrom and A. L. Fairhall Decoding stimulus variance from a distributional neural code of interspike intervals. J. Neurosci., August 30, 2006; 26(35): 9030 - 9037. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. Shlens, G. D. Field, J. L. Gauthier, M. I. Grivich, D. Petrusca, A. Sher, A. M. Litke, and E. J. Chichilnisky The structure of multi-neuron firing patterns in primate retina. J. Neurosci., August 9, 2006; 26(32): 8254 - 8266. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. J. Chacron Nonlinear Information Processing in a Model Sensory System J Neurophysiol, May 1, 2006; 95(5): 2933 - 2946. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
S. M. Chase and E. D. Young Spike-Timing Codes Enhance the Representation of Multiple Simultaneous Sound-Localization Cues in the Inferior Colliculus J. Neurosci., April 12, 2006; 26(15): 3889 - 3898. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
A. Rokem, S. Watzl, T. Gollisch, M. Stemmler, A. V. M. Herz, and I. Samengo Spike-Timing Precision Underlies the Coding Efficiency of Auditory Receptor Neurons J Neurophysiol, April 1, 2006; 95(4): 2541 - 2552. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. M. Van Hulle Edgeworth Approximation of Multivariate Differential Entropy Neural Comput., September 1, 2005; 17(9): 1903 - 1910. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. Pola, R. S. Petersen, A. Thiele, M. P. Young, and S. Panzeri Data-Robust Tight Lower Bounds to the Information Carried by Spike Times of a Neuronal Population Neural Comput., September 1, 2005; 17(9): 1962 - 2005. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
M. B. Kennel, J. Shlens, H. D. I. Abarbanel, and E. J. Chichilnisky Estimating Entropy Rates with Bayesian Confidence Intervals Neural Comput., July 1, 2005; 17(7): 1531 - 1576. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
Z. N. Aldworth, J. P. Miller, T. Gedeon, G. I. Cummins, and A. G. Dimitrov Dejittered Spike-Conditioned Stimulus Waveforms Yield Improved Estimates of Neuronal Feature Selectivity and Spike-Timing Precision of Sensory Interneurons J. Neurosci., June 1, 2005; 25(22): 5323 - 5332. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. E. Latham and S. Nirenberg Synergy, Redundancy, and Independence in Population Codes, Revisited J. Neurosci., May 25, 2005; 25(21): 5195 - 5206. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
L. Paninski, S. Shoham, M. R. Fellows, N. G. Hatsopoulos, and J. P. Donoghue Superlinear Population Encoding of Dynamic Hand Trajectory in Primary Motor Cortex J. Neurosci., September 29, 2004; 24(39): 8551 - 8561. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
V. J. Uzzell and E. J. Chichilnisky Precision of Spike Trains in Primate Retinal Ganglion Cells J Neurophysiol, August 1, 2004; 92(2): 780 - 789. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
J. M. Amigo, J. Szczepanski, E. Wajnryb, and M. V. Sanchez-Vives Estimating the Entropy Rate of Spike Trains via Lempel-Ziv Complexity Neural Comput., April 1, 2004; 16(4): 717 - 736. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
T. Sharpee, N. C. Rust, and W. Bialek Analyzing Neural Responses to Natural Signals: Maximally Informative Dimensions Neural Comput., February 1, 2004; 16(2): 223 - 250. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
E. Schneidman, W. Bialek, and M. J. Berry II Synergy, Redundancy, and Independence in Population Codes J. Neurosci., December 17, 2003; 23(37): 11539 - 11553. [Abstract] [Full Text] [PDF]
|
|
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| J COGNITIVE NEUROSCIENCE | NEURAL COMPUTATION | MIT PRESS JOURNALS |
