Optimal Learning with a Neural Network

Abstract

We introduce optimal learning with a neural network, which we define as minimising the expectation generalisation error. We find that the optimally-trained spherical perceptron may learn a linearly-separable rule as well as any possible network. We sketch an algorithm to generate optimal learning, and simulation results support our conclusions. Optimal learning of a well-known, significant unlearnable problem, the "mismatched weight" problem, gives better asymptotic learning than conventional techniques, and may be simulated enormously more easily. Unlike many other learning schemes, optimal learning extends to more general networks learning more complex rules.