Ebook: Mathematics of Neural Networks: Models, Algorithms and Applications

This volume of research papers comprises the proceedings of the first International Conference on Mathematics of Neural Networks and Applications (MANNA), which was held at Lady Margaret Hall, Oxford from July 3rd to 7th, 1995 and attended by 116 people. The meeting was strongly supported and, in addition to a stimulating academic programme, it featured a delightful venue, excellent food and accommo­ dation, a full social programme and fine weather - all of which made for a very enjoyable week. This was the first meeting with this title and it was run under the auspices of the Universities of Huddersfield and Brighton, with sponsorship from the US Air Force (European Office of Aerospace Research and Development) and the London Math­ ematical Society. This enabled a very interesting and wide-ranging conference pro­ gramme to be offered. We sincerely thank all these organisations, USAF-EOARD, LMS, and Universities of Huddersfield and Brighton for their invaluable support. The conference organisers were John Mason (Huddersfield) and Steve Ellacott (Brighton), supported by a programme committee consisting of Nigel Allinson (UMIST), Norman Biggs (London School of Economics), Chris Bishop (Aston), David Lowe (Aston), Patrick Parks (Oxford), John Taylor (King's College, Lon­ don) and Kevin Warwick (Reading). The local organiser from Huddersfield was Ros Hawkins, who took responsibility for much of the administration with great efficiency and energy. The Lady Margaret Hall organisation was led by their bursar, Jeanette Griffiths, who ensured that the week was very smoothly run.

---

This book examines the mathematics, probability, statistics, and computational theory underlying neural networks and their applications. In addition to the theoretical work, the book covers a considerable range of neural network topics such as learning and training, neural network classifiers, memory-based networks, self-organizing maps and unsupervised learning, Hopfeld networks, radial basis function networks, and general network modelling and theory. Added to the book's mathematical and neural network topics are applications in chemistry, speech recognition, automatic control, nonlinear programming, medicine, image processing, finance, time series, and dynamics. As a result, the book surveys a wide range of recent research on the theoretical foundations of creating neural network models in a variety of application areas.

---

This book examines the mathematics, probability, statistics, and computational theory underlying neural networks and their applications. In addition to the theoretical work, the book covers a considerable range of neural network topics such as learning and training, neural network classifiers, memory-based networks, self-organizing maps and unsupervised learning, Hopfeld networks, radial basis function networks, and general network modelling and theory. Added to the book's mathematical and neural network topics are applications in chemistry, speech recognition, automatic control, nonlinear programming, medicine, image processing, finance, time series, and dynamics. As a result, the book surveys a wide range of recent research on the theoretical foundations of creating neural network models in a variety of application areas.
Content:
Front Matter....Pages i-xxii
Front Matter....Pages 1-1
N-Tuple Neural Networks....Pages 3-14
Information Geometry of Neural Networks — An Overview —....Pages 15-23
Q-Learning: A Tutorial and Extensions....Pages 24-33
Are There Universal Principles of Brain Computation?....Pages 34-40
On-Line Training of Memory-Driven Attractor Networks....Pages 41-46
Mathematical Problems Arising from Constructing an Artificial Brain....Pages 47-57
Front Matter....Pages 59-59
The Successful Use of Probability Data in Connectionist Models....Pages 61-66
Weighted Mixture of Models For On-Line Learning....Pages 67-72
Local Modifications to Radial Basis Networks....Pages 73-77
A Statistical Analysis of the Modified NLMS Rules....Pages 78-83
Finite Size Effects in On-Line Learning of Multi-Layer Neural Networks....Pages 84-88
Constant Fan-in Digital Neural Networks are VLSI-Optimal....Pages 89-94
The Application of Binary Encoded 2nd Differential Spectrometry in Preprocessing of UV-Vis Absorption Spectral Data....Pages 95-100
A Non-Equidistant Elastic Net Algorithm....Pages 101-106
Unimodal Loading Problems....Pages 107-112
On the Use of Simple Classifiers for the Initialisation of One-Hidden-Layer Neural Nets....Pages 113-117
Modelling Conditional Probability Distributions for Periodic Variables....Pages 118-122
Integro-Differential Equations in Compartmental Model Neurodynamics....Pages 123-128
Nonlinear Models for Neural Networks....Pages 129-133
A Neural Network for the Travelling Salesman Problem with a Well Behaved Energy Function....Pages 134-139
Front Matter....Pages 59-59
Semiparametric Artificial Neural Networks....Pages 140-145
An Event-Space Feedforward Network Using Maximum Entropy Partitioning with Application to Low Level Speech Data....Pages 146-150
Approximating the Bayesian Decision Boundary for Channel Equalisation Using Subset Radial Basis Function Network....Pages 151-155
Applications of Graph Theory to the Design of Neural Networks for Automated Fingerprint Identification....Pages 156-160
Zero Dynamics and Relative Degree of Dynamic Recurrent Neural Networks....Pages 161-165
Irregular Sampling Approach to Neurocontrol: The Band-And Space-Limited Functions Questions....Pages 166-170
Unsupervised Learning of Temporal Constancies by Pyramidal-Type Neurons.....Pages 171-175
Numerical Aspects of Machine Learning in Artificial Neural Networks....Pages 176-180
Learning Algorithms for Ram-Based Neural Networks....Pages 181-185
Analysis of Correlation Matrix Memory and Partial Match-Implications for Cognitive Psychology....Pages 186-191
Regularization and Realizability in Radial Basis Function Networks....Pages 192-197
A Universal Approximator Network for Learning Conditional Probability Densities....Pages 198-203
Convergence of a Class of Neural Networks....Pages 204-208
Applications of the Compartmental Model Neuron to Time Series Analysis....Pages 209-214
Information Theoretic Neural Networks for Contextually Guided Unsupervised Learning....Pages 215-219
Convergence in Noisy Training....Pages 220-224
Non-Linear Learning Dynamics with a Diffusing Messenger....Pages 225-229
A Variational Approach to Associative Memory....Pages 230-234
Transformation of Nonlinear Programming Problems into Separable ones Using Multilayer Neural Networks....Pages 235-239
A Theory of Self-Organising Neural Networks....Pages 240-244
Front Matter....Pages 59-59
Neural Network Supervised Training Based on a Dimension Reducing Method....Pages 245-249
A Training Method for Discrete Multilayer Neural Networks....Pages 250-254
Local Minimal Realisations of Trained Hopfield Networks....Pages 255-258
Data Dependent Hyperparameter Assignment....Pages 259-264
Training Radial Basis Function Networks by Using Separable and Orthogonalized Gaussians....Pages 265-269
Error Bounds for Density Estimation by Mixtures....Pages 270-274
On Smooth Activation Functions....Pages 275-279
Generalisation and Regularisation by Gaussian Filter Convolution of Radial Basis Function Networks....Pages 280-284
Dynamical System Prediction: A Lie Algebraic Approach for a Novel Neural Architecture....Pages 285-289
Stochastic Neurodynamics and the System Size Expansion....Pages 290-294
An Upper Bound on the Bayesian Error Bars for Generalized Linear Regression....Pages 295-299
Capacity Bounds for Structured Neural Network Architectures....Pages 300-305
On-Line Learning in Multilayer Neural Networks....Pages 306-311
Spontaneous Dynamics and Associative Learning in an Assymetric Recurrent Random Neural Network....Pages 312-317
A Statistical Mechanics Analysis of Genetic Algorithms for Search and Learning....Pages 318-322
Volumes of Attraction Basins in Randomly Connected Boolean Networks....Pages 323-327
Evidential Rejection Strategy for Neural Network Classifiers....Pages 328-332
Dynamics Approximation and Change Point Retrieval from a Neural Network Model....Pages 333-338
Query Learning for Maximum Information Gain in a Multi-Layer Neural Network....Pages 339-343
Shift, Rotation and Scale Invariant Signatures for Two-Dimensional Contours, in a Neural Network Architecture....Pages 344-348
Front Matter....Pages 59-59
Function Approximation by Three-Layer Artificial Neural Networks....Pages 349-354
Neural Network Versus Statistical Clustering Techniques: A Pilot Study in a Phoneme Recognition Task....Pages 355-360
Multispectral Image Analysis Using Pulsed Coupled Neural Networks....Pages 361-365
Reasoning Neural Networks....Pages 366-371
Capacity of the Upstart Algorithm....Pages 372-377
Regression with Gaussian Processes....Pages 378-382
Stochastic Forward-Perturbation, Error Surface and Progressive Learning in Neural Networks....Pages 383-388
Dynamical Stability of a High-Dimensional Self-Organizing Map....Pages 389-393
Measurements of Generalisation Based on Information Geometry....Pages 394-398
Towards an Algebraic Theory of Neural Networks: Sequential Composition....Pages 399-403

---

This book examines the mathematics, probability, statistics, and computational theory underlying neural networks and their applications. In addition to the theoretical work, the book covers a considerable range of neural network topics such as learning and training, neural network classifiers, memory-based networks, self-organizing maps and unsupervised learning, Hopfeld networks, radial basis function networks, and general network modelling and theory. Added to the book's mathematical and neural network topics are applications in chemistry, speech recognition, automatic control, nonlinear programming, medicine, image processing, finance, time series, and dynamics. As a result, the book surveys a wide range of recent research on the theoretical foundations of creating neural network models in a variety of application areas.
Content:
Front Matter....Pages i-xxii
Front Matter....Pages 1-1
N-Tuple Neural Networks....Pages 3-14
Information Geometry of Neural Networks — An Overview —....Pages 15-23
Q-Learning: A Tutorial and Extensions....Pages 24-33
Are There Universal Principles of Brain Computation?....Pages 34-40
On-Line Training of Memory-Driven Attractor Networks....Pages 41-46
Mathematical Problems Arising from Constructing an Artificial Brain....Pages 47-57
Front Matter....Pages 59-59
The Successful Use of Probability Data in Connectionist Models....Pages 61-66
Weighted Mixture of Models For On-Line Learning....Pages 67-72
Local Modifications to Radial Basis Networks....Pages 73-77
A Statistical Analysis of the Modified NLMS Rules....Pages 78-83
Finite Size Effects in On-Line Learning of Multi-Layer Neural Networks....Pages 84-88
Constant Fan-in Digital Neural Networks are VLSI-Optimal....Pages 89-94
The Application of Binary Encoded 2nd Differential Spectrometry in Preprocessing of UV-Vis Absorption Spectral Data....Pages 95-100
A Non-Equidistant Elastic Net Algorithm....Pages 101-106
Unimodal Loading Problems....Pages 107-112
On the Use of Simple Classifiers for the Initialisation of One-Hidden-Layer Neural Nets....Pages 113-117
Modelling Conditional Probability Distributions for Periodic Variables....Pages 118-122
Integro-Differential Equations in Compartmental Model Neurodynamics....Pages 123-128
Nonlinear Models for Neural Networks....Pages 129-133
A Neural Network for the Travelling Salesman Problem with a Well Behaved Energy Function....Pages 134-139
Front Matter....Pages 59-59
Semiparametric Artificial Neural Networks....Pages 140-145
An Event-Space Feedforward Network Using Maximum Entropy Partitioning with Application to Low Level Speech Data....Pages 146-150
Approximating the Bayesian Decision Boundary for Channel Equalisation Using Subset Radial Basis Function Network....Pages 151-155
Applications of Graph Theory to the Design of Neural Networks for Automated Fingerprint Identification....Pages 156-160
Zero Dynamics and Relative Degree of Dynamic Recurrent Neural Networks....Pages 161-165
Irregular Sampling Approach to Neurocontrol: The Band-And Space-Limited Functions Questions....Pages 166-170
Unsupervised Learning of Temporal Constancies by Pyramidal-Type Neurons.....Pages 171-175
Numerical Aspects of Machine Learning in Artificial Neural Networks....Pages 176-180
Learning Algorithms for Ram-Based Neural Networks....Pages 181-185
Analysis of Correlation Matrix Memory and Partial Match-Implications for Cognitive Psychology....Pages 186-191
Regularization and Realizability in Radial Basis Function Networks....Pages 192-197
A Universal Approximator Network for Learning Conditional Probability Densities....Pages 198-203
Convergence of a Class of Neural Networks....Pages 204-208
Applications of the Compartmental Model Neuron to Time Series Analysis....Pages 209-214
Information Theoretic Neural Networks for Contextually Guided Unsupervised Learning....Pages 215-219
Convergence in Noisy Training....Pages 220-224
Non-Linear Learning Dynamics with a Diffusing Messenger....Pages 225-229
A Variational Approach to Associative Memory....Pages 230-234
Transformation of Nonlinear Programming Problems into Separable ones Using Multilayer Neural Networks....Pages 235-239
A Theory of Self-Organising Neural Networks....Pages 240-244
Front Matter....Pages 59-59
Neural Network Supervised Training Based on a Dimension Reducing Method....Pages 245-249
A Training Method for Discrete Multilayer Neural Networks....Pages 250-254
Local Minimal Realisations of Trained Hopfield Networks....Pages 255-258
Data Dependent Hyperparameter Assignment....Pages 259-264
Training Radial Basis Function Networks by Using Separable and Orthogonalized Gaussians....Pages 265-269
Error Bounds for Density Estimation by Mixtures....Pages 270-274
On Smooth Activation Functions....Pages 275-279
Generalisation and Regularisation by Gaussian Filter Convolution of Radial Basis Function Networks....Pages 280-284
Dynamical System Prediction: A Lie Algebraic Approach for a Novel Neural Architecture....Pages 285-289
Stochastic Neurodynamics and the System Size Expansion....Pages 290-294
An Upper Bound on the Bayesian Error Bars for Generalized Linear Regression....Pages 295-299
Capacity Bounds for Structured Neural Network Architectures....Pages 300-305
On-Line Learning in Multilayer Neural Networks....Pages 306-311
Spontaneous Dynamics and Associative Learning in an Assymetric Recurrent Random Neural Network....Pages 312-317
A Statistical Mechanics Analysis of Genetic Algorithms for Search and Learning....Pages 318-322
Volumes of Attraction Basins in Randomly Connected Boolean Networks....Pages 323-327
Evidential Rejection Strategy for Neural Network Classifiers....Pages 328-332
Dynamics Approximation and Change Point Retrieval from a Neural Network Model....Pages 333-338
Query Learning for Maximum Information Gain in a Multi-Layer Neural Network....Pages 339-343
Shift, Rotation and Scale Invariant Signatures for Two-Dimensional Contours, in a Neural Network Architecture....Pages 344-348
Front Matter....Pages 59-59
Function Approximation by Three-Layer Artificial Neural Networks....Pages 349-354
Neural Network Versus Statistical Clustering Techniques: A Pilot Study in a Phoneme Recognition Task....Pages 355-360
Multispectral Image Analysis Using Pulsed Coupled Neural Networks....Pages 361-365
Reasoning Neural Networks....Pages 366-371
Capacity of the Upstart Algorithm....Pages 372-377
Regression with Gaussian Processes....Pages 378-382
Stochastic Forward-Perturbation, Error Surface and Progressive Learning in Neural Networks....Pages 383-388
Dynamical Stability of a High-Dimensional Self-Organizing Map....Pages 389-393
Measurements of Generalisation Based on Information Geometry....Pages 394-398
Towards an Algebraic Theory of Neural Networks: Sequential Composition....Pages 399-403
....