These authors have found many of these textbooks to be inadequate as general introductory textbooks for various reasons: in some cases the scope of the book might be too narrow, in other cases the book might lack the level of analytical rigor that allows a critical understanding of the main neural network theories. The book by Hertz, Krogh, and Palmer was arguably the first textbook to avoid these shortcomings.
The textbook Introduction to the Theory of Neural Computation ITNC has been used by one of us as the primary text for a graduate course on neural computation. Several books have been published since the appearance of this text, but none, in our opinion, has surpassed its breadth and quality. It is important to point out that this is not the type of book one would recommend to a beginner looking for a self-contained, nontechnical overview of neural networks.
Rather, this book is invaluable for the student who wishes to begin a rigorous study of neural networks, or the researcher who wishes to have a solid understanding of the properties and limitations of the main classes of neural networks. As such, ITNC is ideal for a graduate or advanced undergraduate introductory course on neural networks. Neural networks are studied primarily for one of two purposes: to elucidate brain function, or to generate new, applied technology. Models that fall under the applied category we use this label loosely are the focus of the text by Hertz, Krogh, and Palmer, and the subject of our review.
These two classes of models differ in their properties and applications. The roots of the differences between these and other models can frequently be traced to the different backgrounds of the scientists who developed them, who typically came either from engineering or from physics. As it is, the text is well suited for a onesemester course focusing on perceptrons and associators. Instructors interested in covering additional topics in more detail would have to supplement this book with other texts or articles.
Given its textbook style, the book could be made more useful by including exercises, and perhaps by including special sections on how one might program some of the models described in the text. Perhaps the authors will consider these possibilities if they decide to prepare a further edition of their book.
An extended review Weigend, discusses more recent developments in supervised feedforward neural networks, some of which are part of the helpful trend towards analyzing the networks as a class of statistical likelihood models. We conclude our review with a brief summary of the contents of each chapter.
Chapter 1 provides a brief review of the main characteristics of neurons in the brain, and how these might be related to artificial neurons. After a brief review of the history of research in neural networks, the chapter concludes with a discussion of some important practical issues regarding neural networks research.
Chapter 2 outlines the problem of associative memory describes the binary, discrete symmetric autoassociator as described by Hopfield, and analyzes its storage capacity. This chapter includes an illuminatingdescription of the similarity between the simple networks of McCulloch and Pitts, and the Ising spin model from statistical mechanics. The chapter summarizes in simple terms how mean field theory can be used to analyze the behavior of a collection of simple, interacting elements, be they atoms in a lattice-like material or binary neurons in a fully connected network.
Chapter 3 describes several modifications of Hopfield's autoassociator, including the extension to continuous units. The chapter also briefly discusses hardware implementations, and application of these networks to the generation of temporal sequences of patterns.
Chapter 4 illustrates several applications of this class of models, focusing on optimization problems. The chapter begins by describing how one might construct a meaningful energy function for a simple problem, and how the handcrafted energy function can be used to derive a network structure to solve the problem. The chapter then describes how the continuous autoassociator can solve the classical traveling salesman problem. After one more example, the chapter closes with a description of applications in image processing.
Chapter 5 moves to the class of supervised, error-based, feedforward networks, beginning with the simple perceptron. The authors do a nice job of summarizing simple concepts of classification, and then describing in an intuitive fashion the workings of the simple perceptron. After a stand-alone section proving perceptron convergence when inputs are linearly separable , the chapter describes extensions to include nonlinear or stochastic units.
Chapter 6 extends the analysis of Chapter 5 to the realm of multilayer perceptrons and back propagation. Palmer gradient descent on the error surface in weight space. Section 6. The chapter concludes with an overview of methods for modifying the network architecture to improve performance.
Chapter 7 focuses on a variety of recurrent or feedback networks, including the Boltzmann Machine an interesting cross between a multilayer perceptron and an autoassociator , recurrent back propagation, and other models for learning time sequences. The closing section describes reinforcement learning models. It is unclear why this class of model appears in this chapter, as the "feedback" here is simply a signal indicating how the network is performing. This section could easily be extended and turned into an independent chapter.
Chapter 8 gives a very nice analysis of feedforward networks that use associative, or Hebbian learning. The chapter provides a useful discussion of how Hebbian networks are related to Principal Component Analysis, a technique for extracting information about the dimensions along which some input data exhibit maximum variance. The closing section discusses superficially the concept of self-organizing feature extraction.
This topic serves as a lead-in to the material in the next chapter. Chapter 9 treats material that could easily take up one or two textbooks, under the heading of unsupervised competitive learning.
In spite of its condensed format, the chapter does a reasonable job of summarizing some of the main computational points of relevant models based on competitive learning, such as adaptive resonance theory and self-organizing feature maps. The chapter includes a theoretical analysis of self-organizing feature maps, and a classical example in which they are applied to the traveling salesman problem.
Chapter 9 concludes with a section on normalized radial basis function networks, which, as an important error-based supervised model, might better have been covered in Chapter 6. Chapter 10 caps the book with a much more formal mathematical treatment of two problems: the recall of stored patterns in the symmetric autoassociator, and the capacity of a simple perceptron.
Machine Learning Fundamentals: From synapses to algorithms. An Introduction. Overview: Machine learning and machine intelligence have rapidly gained importance, both in science and in every day life. As processor speed and memory storage are drastically increasing, now we are often facing the problem that there is too much data to be analyzed by hand, and we need automated tools.
This becomes evident in every day problems, like email spam filtering, and also in modern branches of science, such as astronomy, genomics and bioinformatics. Over half a century later, we now understand more about information processing in the brain, and about learning theory. The advances that have been made in machine learning over the past two decades have resulted in drastic improvements, allowing computer programs to learn from examples, and to adapt.
This course provides a comprehensible picture of the concepts underlying machine learning algorithms. By the end of the course, students will understand the basics of information processing and learning in the nervous system, they will be familiar with a variety of important machine learning methods and computational models, and will be able to apply those to selected problems. Cover and Thomas, "Elements of Information Theory".
See Cover's website. Computing machinery and intelligence.
Mind, 59, John von Neumann. The Computer and the Brain , Yale Univ. Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci. Multilayer feedforward networks are universal approximators. Fukushima: "Neural network model for a mechanism of pattern recognition unaffected by shift in position — Neocognitron —", Trans. LeCun: LeNet J. Maass, E. Hopfield: Neural networks and physical systems with emergent collective computational abilities. PNAS 79, , Neurons with graded response have collective computational properties like those of two-state neurons.
PNAS 81, , Hopfield, C. Brody: Learning rules and network repair in spike-timing based computation networks Proc. USA , Cortes and V. Support vector networks. Oldenbourg Verlag, Munich A. Regression estimation with support vector learning machines. Buhmann, M. Held : Model selection in clustering by uniform convergence bounds. Linsker, Self-organization in a perceptual network.
Computer 21 A. Bell and T. An information-maximisation approach to blind separation and blind deconvolution.