Timothy L. H. Watkin

A summary is presented of the statistical mechanical theory of learning a rule with a neural network, a rapidly advancing area which is closely related to other inverse problems frequently encountered by physicists. By emphasizing the relationship between neural networks and strongly interacting physical systems, such as spin glasses, the authors show how learning theory has provided a workshop in which to develop new, exact analytical techniques.

We introduce an inferential approach to unsupervised learning which allows us to define an optimal learning strategy. Applying these ideas to a simple, previously studied model, we show that it is impossible to detect structure in data until a critical number of examples have been presented -- an effect which will be observed in all problems with certain underlying symmetries. Thereafter the advantage of optimal learning over previously studied learning algorithms depends critically upon the distribution of patterns; optimal learning may be exponentially faster. Models with more subtle correlations are harder to analyse, but in a simple limit of one such problem we calculate exactly the efficacy of an algorithm similar to some used in practice, and compare it to that of the optimal prescription. PACS. 87.10 - PACS. 02.50 - PACS. 64.60C Published in J. Phys. A: Math. and Gen. * Laboratoire associ'e au C.N.R.S. (U.R.A. 1306), `a l'E.N.S. et aux Universit'es Paris VI et Paris VII. 1 I...

We introduce optimal learning with a neural network, which we define as minimising the expectation generalisation error. We find that the optimally-trained spherical perceptron may learn a linearly-separable rule as well as any possible network. We sketch an algorithm to generate optimal learning, and simulation results support our conclusions. Optimal learning of a well-known, significant unlearnable problem, the "mismatched weight" problem, gives better asymptotic learning than conventional techniques, and may be simulated enormously more easily. Unlike many other learning schemes, optimal learning extends to more general networks learning more complex rules.

We study how well perceptrons learn to solve problems for which there is no perfect answer (the usual case), taking as examples a rule with a threshold, a rule in which the answer is not a monotonic function of the overlap between question and teacher, and a rule with many teachers (a ``hard'' unlearnable problem). In general there is a tendency for first-order transitions, even using spherical perceptrons, as networks compromise between conflicting requirements. Some existing learning schemes fail completely--occasionally even finding the worst possible solution; others are more successful. High-temperature learning seems more satisfactory than zero-temperature algorithms and avoids ``overlearning'' and ``overfitting,'' but care must be taken to avoid ``trapping'' in spurious free-energy minima. For some rules examples alone are not enough to learn from, and some prior information is required.

We present the complete, equilibrium solution of a neural-network model in which each neuron is connected to a small fraction of the others by symmetric, Hebb-rule synapses. At first replica symmetry is assumed, but the results are then corrected for full symmetry breaking, which leads to a substantial increase in storage capacity.