Liquid State Machines: An Annotated Bibliography

Papers Discussing Liquid State Machines Directly

Liquid State Machines: Furthermore, this alternative computation style is supported by theoretical results (see section 4), which suggest that it is in principle as powerful as von Neumann style computational models such as Turing machines, but more adequate for the type of real-time computing on analog input streams that is carried out by the nervous system.[On the Computational Power of Circuits of Spiking Neurons Wolfgang Maass Henry Markram]
LSM in a bucket of water (!): Fernando, C. & Sojakka, S. (2003), Pattern recognition in a bucket, in ‘ Proceedings of the Seventh European Conference on Artificial Life (ECAL 2003)’
LSM for robotics: [Biologically inspired neural networks for the control of embodied agents Razvan V. Florian] Clearly explained review of spiking networks and LSM.
LSM for artificial mouse, Review: The most important aspect of the liquid is to react differently enough on different input sequences; the amount of distance created between those is called the separation property (SP) of the liquid. The SP (see fig. 3) reflects the ability of the liquid to create different trajectories of internal states for each of the input classes. The ability of the readout units to distinguish these trajectories, generalize and relate them to the desired outputs is called the approximation property (AP). This property depends on the adaptability of the chosen readout units, whereas the SP is based directly on the liquid’s complexity. [Liquid State Machines, a review Jilles Vreeken]
LSM with columnar structure.: In the LSM model there is a trade-off between the complexity of the liquid and the complexity of the readout. The optimal point for this tradeoff depends on factors such as the kinds and number of target filters that have to be simultaneously implemented. [P. Joshi. Synthesis of a Liquid State Machine with Hopfield/Brody Transient Synchrony. Master's Thesis, Center for Advanced Computer Studies, University of Louisiana, Lafayette, U.S.A., Nov. 2002.]
A Universal Approximation Theorem for Dynamic Networks: A time-invariant filter with fading memory can be approximated with 1) a dynamic network or 2) a Volterra series. A dynamic network has time-dependent weights; these can be faciliation or depression. [Processing of Time Series by Neural Circuits with Biologically Realistic Synaptic Dynamics, Thomas Natschlaager & Wolfgang Maass, Eduardo D. Sontag Anthony Zador]

LSM Slide deck by Maass
LSM and Radial Basis Functions: Online clustering with spiking neurons using temporal coding. [T. Natschläger and B. Ruf. In L. S. Smith and A. Hamilton, editors, Neuromorphic Systems: Engineering Silicon from Neurobiology], local update rules, radial basis functions
LSM and SOM: We propose a mechanism for unsupervised learning in networks of spiking neurons which is based on the timing of single firing events. Our results show that a topology preserving behaviour quite similar to that of Kohonen's self-organizing map can be achieved using temporal coding. In contrast to previous approaches, which use rate coding, the winner among competing neurons can be determined fast and locally. Hence our model is a further step towards a more realistic description of unsupervised. [Self-Organization of Spiking Neurons Using Action Potential Timing Berthold Ruf, Michael Schmitt]

Echo State Machines: The "echo state" approach looks at RNNs from a new angle. Large RNNs are interpreted as "reservoirs" of complex, excitable dynamics. Output units "tap" from this reservoir. This idea leads to training algorithms where only the network-to-output connection weights have to be trained. This can be done with known, highly efficient linear regression algorithms. See also Adaptive Nonlinear System Identification with Echo State Networks
Herbert Jaeger

Kernel Methods; Support Vector Machines

The solutions sought by kernel-based algorithms such as the support vector machine (SVM) are affine functions in the feature space:

for some weight vector w from the feature space. The kernel can be exploited whenever the weight vector can be expressed as a linear combination of the training points,

implying that we can express f as

[Learning the Kernel Matrix with Semidefinite Programming, Lanckriet, Cristianini Peter Bartlett Laurent El Ghaoui Michael I. Jordan]

Applications: Knowledge-based analysis of microarray gene expression data by using support vector machines [Michael P. S. Brown, William Noble Grundy, David Lin, Nello Cristianini, Charles Walsh Sugnet, Terrence S. Furey, Manuel Ares, Jr., and David Haussler]; KERNEL-BASED DATA FUSION AND ITS APPLICATION TO PROTEIN FUNCTION PREDICTION IN YEAST [G. R. G. LANCKRIET M. DENG N. CRISTIANINI M. I. JORDAN W. S. NOBLE ]
Library of Kernel Matrices: It is natural to envision libraries of kernel matrices in elds such as bioinformatics, computational vision, and information retrieval, in which multiple data sources abound. Such libraries would summarize the statistically-relevant features of primary data, and encapsulate domain specic knowledge. Tools such as the semidenite programming methods that we have presented here can be used to bring these multiple data sources together in novel ways to make predictions and decisions.[Learning the kernel matrix with semidefinite programming. G. R. G. Lanckriet, N. Cristianini, L. El Ghaoui, P. L. Bartlett, and M. I. Jordan. In press: Journal of Machine Learning Research, 2003.]
Spikes and Support Vector Machines: Spikernels: Embedding Spiking Neurons in Inner-Product Spaces Lavi Shpigelman Should connect to Kevin Judd's paper. No ref to Maass.

Does LSM implement adaptive volterra filter?: The message of Maass's work is that the recursive neural network can compute a filter equivalent to a Volterra Filter. If the linear variables in the LSM are set adaptively then we would have an adaptive Volterra filter, the latter defined in e.g [ADAPTIVE VOLTERRA FILTERS FOR NONLINEAR ACOUSTIC ECHO CANCELLATION A. Stenger and R. Rabenstein]
Nonlinear Analysis of Time Series: Several methods exist for adjusting nonlinear parameters. A common technique begins with k basis functions with arbitrarily chosen parameters, then adjusts the parameters by gradient descent to find an optimal model. Another technique makes a grid search over a region of parameter space. However, we observe, in the light of Section 1.2, that parameters need only be specified to some precision. If one is lucky, or careful, the precision required of nonlinear parameters is much less then that required for – parameters, and hence accurate adjustment of nonlinear parameters may not be critical to a model's performance. This does seem to be supported by experience in using radial basis functions, where the literature is full of good models built from even randomly chosen centers. Consequently, we propose the following method to optimize the nonlinear parameters: initially choose a large number of basis functions with various arbitrary values for the nonlinear parameters, and then select the k basis functions that give the best model. Of course, this is what we have already called the restrictedselection problem; now, however, we are using the fact that if enough basis functions were initially chosen, at least some of then would lie near to the optimal values for the nonlinear parameters and be indistinguishable at the precision required of the optimal values.[K. Judd and A. Mees. On selecting models for nonlinear time series. Physica D, 82:426--444, 1995. http://citeseer.nj.nec.com/judd95selecting.html] Perhaps this is the mechanism of LSM?
[see also: Radial-Basis Models for Feedback Systems With Fading Memory (2001) David M. Walker, Nicholas B. Tufillaro, Paul Gross]
May connect SVM and LMS. Also connects SVM to Bayes: In this paper we extend the conformal method of modifying a kernel function to improve the performance of Support Vector Machine classifiers [14, 15]. The kernel function is conformally transformed in a data-dependent way by using the information of Support Vectors obtained in primary training. We further investigate the performances of modified Gaussian Radial Basis Function and Polynomial kernels. Simulation results for two artificial data sets show that the method is very effective, especially for correcting bad kernels. [Conformal Transformation of Kernel Functions: A Data-Dependent Way to Improve Support Vector Machine Classi¢ers SI WU* and SHUN-ICHI AMARI]
A programming mechanism?: Support Vector Machines for Analog Circuit Performance Representation F. De Bernardinisyx M. I. Jordany A.Sangiovanni-Vincentelli

Possible hardware implementations

Nonlinear MEMS filter: Parametrically Excited MEMS-Based Filters Kimberly L. Turner Steven W. Shaw
NONLINEAR BEHAVIOR OF A PARAMETRIC RESONANCE-BASED MASS SENSOR Wenhua Zhang, Rajashree Baskaran and Kimberly L. Turner
Translinear Curcuits. "silicon sliderules": ...multiple-input translinear elements; such elements produce output currents that are proportional to the exponential of a weighted sum of their input voltages. We can implement the weighted voltage summations with either resistive or capacitive voltage dividers. We can obtain the required exponential voltage-to-current transformations from either bipolar transistors or subthreshold MOS transistors. The subthreshold floating-gate MOS transistor naturally implements the exponential-of-a-weighted-sum operation in a single device. [Analysis, Synthesis, and Implementation of Networks of Multiple-Input Translinear Elements, B. Minch] Cornell. Note: subthreshold MOS is 100mv range.
Field Programmable Learning Array

Applications

Just a list of areas...

Music and Speech Synthesis: ...investigate the modeling of musical and speech signals and demonstrate that the model may be used for synthesis of musical and speech signals.[Neural Network Modeling of Speech and Music Signals Axel R¨obel]

?

Nanocomputing with Delays José A. B. Fortes

In this paper, we develop a spatio-temporal memory that blends properties from long and short-term memory and is motivated by reaction diffusion mechanisms. The winning processing element of a self-organizing network creates traveling waves on the output space that gradually attenuate over time and space to diffuse temporal information and create localized spatio-temporal neighborhoods for clustering. The novelty of the model is in the creation of time varying Voronoi tessellations anticipating the learned input signal dynamics even when the cluster centers are fixed. We test the method in a robot navigation task and in vector quantization of speech. This method performs better than conventional static vector quantizers based on the same data set and similar training conditions. [Principles and networks for self-organization in space-time; Jose Principe Neil Euliano Shayan Garani ]

TEMPORAL SELF-ORGANIZATION FOR NEURAL NETWORKS By NEIL R. EULIANO ... Thesis

Another recursive neural network with (fading?) memory: Jose C. Principe, James Kuo, Samel Celebi, "An analysis of the gamma memory in dynamic neural networks," Trans. on Neural Networks, Vol. 5, No. 2, pp. 331-337, Mar. 1994. (pdf)