Ebook: An Information-Theoretic Approach to Neural Computing

Neural networks provide a powerful new technology to model and control nonlinear and complex systems. In this book, the authors present a detailed formulation of neural networks from the information-theoretic viewpoint. They show how this perspective provides new insights into the design theory of neural networks. In particular they show how these methods may be applied to the topics of supervised and unsupervised learning including feature extraction, linear and non-linear independent component analysis, and Boltzmann machines. Readers are assumed to have a basic understanding of neural networks, but all the relevant concepts from information theory are carefully introduced and explained. Consequently, readers from several different scientific disciplines, notably cognitive scientists, engineers, physicists, statisticians, and computer scientists, will find this to be a very valuable introduction to this topic.

---

Neural networks provide a powerful new technology to model and control nonlinear and complex systems. In this book, the authors present a detailed formulation of neural networks from the information-theoretic viewpoint. They show how this perspective provides new insights into the design theory of neural networks. In particular they show how these methods may be applied to the topics of supervised and unsupervised learning including feature extraction, linear and non-linear independent component analysis, and Boltzmann machines. Readers are assumed to have a basic understanding of neural networks, but all the relevant concepts from information theory are carefully introduced and explained. Consequently, readers from several different scientific disciplines, notably cognitive scientists, engineers, physicists, statisticians, and computer scientists, will find this to be a very valuable introduction to this topic.

---

Neural networks provide a powerful new technology to model and control nonlinear and complex systems. In this book, the authors present a detailed formulation of neural networks from the information-theoretic viewpoint. They show how this perspective provides new insights into the design theory of neural networks. In particular they show how these methods may be applied to the topics of supervised and unsupervised learning including feature extraction, linear and non-linear independent component analysis, and Boltzmann machines. Readers are assumed to have a basic understanding of neural networks, but all the relevant concepts from information theory are carefully introduced and explained. Consequently, readers from several different scientific disciplines, notably cognitive scientists, engineers, physicists, statisticians, and computer scientists, will find this to be a very valuable introduction to this topic.
Content:
Front Matter....Pages i-xiii
Introduction....Pages 1-5
Preliminaries of Information Theory and Neural Networks....Pages 7-37
Front Matter....Pages 39-39
Linear Feature Extraction: Infomax Principle....Pages 41-63
Independent Component Analysis: General Formulation and Linear Case....Pages 65-107
Nonlinear Feature Extraction: Boolean Stochastic Networks....Pages 109-133
Nonlinear Feature Extraction: Deterministic Neural Networks....Pages 135-166
Front Matter....Pages 167-167
Supervised Learning and Statistical Estimation....Pages 169-186
Statistical Physics Theory of Supervised Learning and Generalization....Pages 187-217
Composite Networks....Pages 219-224
Information Theory Based Regularizing Methods....Pages 225-241
Back Matter....Pages 243-261

---

Neural networks provide a powerful new technology to model and control nonlinear and complex systems. In this book, the authors present a detailed formulation of neural networks from the information-theoretic viewpoint. They show how this perspective provides new insights into the design theory of neural networks. In particular they show how these methods may be applied to the topics of supervised and unsupervised learning including feature extraction, linear and non-linear independent component analysis, and Boltzmann machines. Readers are assumed to have a basic understanding of neural networks, but all the relevant concepts from information theory are carefully introduced and explained. Consequently, readers from several different scientific disciplines, notably cognitive scientists, engineers, physicists, statisticians, and computer scientists, will find this to be a very valuable introduction to this topic.
Content:
Front Matter....Pages i-xiii
Introduction....Pages 1-5
Preliminaries of Information Theory and Neural Networks....Pages 7-37
Front Matter....Pages 39-39
Linear Feature Extraction: Infomax Principle....Pages 41-63
Independent Component Analysis: General Formulation and Linear Case....Pages 65-107
Nonlinear Feature Extraction: Boolean Stochastic Networks....Pages 109-133
Nonlinear Feature Extraction: Deterministic Neural Networks....Pages 135-166
Front Matter....Pages 167-167
Supervised Learning and Statistical Estimation....Pages 169-186
Statistical Physics Theory of Supervised Learning and Generalization....Pages 187-217
Composite Networks....Pages 219-224
Information Theory Based Regularizing Methods....Pages 225-241
Back Matter....Pages 243-261
....