Independent Research Papers
Links to research papers from independent scientists and engineers about Hierarchical Temporal Memory and about Numenta technology. Descriptions of the papers were written by Numenta employees.
Robust Character Recognition using a Hierarchical Bayesian Network »
Thornton, J. R., Gustafsson, T., Blumenstein, M. & Hine, T.
This paper gives an example of trying a part of the HTM algorithm on a different data set. The MNIST digit data set is a standard benchmark in the machine learning community to verify the performance of recognition algorithms. The authors used this data set for their experiments. The algorithm that they used did not involve the temporal component - it is a hierarchical memory system without the temporal part. Therefore, the generalization properties that we expect from temporal pooling cannot be expected from this implementation. An examination of the data set also shows that the extent of translation and scaling in this data set is very less compared to the pictures data set. Therefore, we would expect a reasonable performance without including the temporal part.
The authors tried this algorithm against other popular algorithms and obtained comparable results. Although the HTM variant does not beat the best algorithm for this purpose, that was not the objective of this paper. This paper would be interesting to anybody who is interested in knowing how to apply HTMs to the MNIST dataset. However, the reader should keep in mind that the temporal part of HTMs was not implemented in this paper.
Thornton, J. R., Gustafsson, T., Blumenstein, M. & Hine, T. (2006). Robust Character Recognition using a Hierarchical Bayesian Network. Proceedings of the 19th Australian Joint Conference on Artificial Intelligence, AI-2006, Hobart. 1259-1264. [copyright © Springer-Verlag].
Memory-Prediction Framework for Pattern Recognition: Performance and Suitability of the Bayesian Model of Visual Cortex »
Saulius J. Garalevicius
In this paper, Saulius extends the original Pictures network by training them on more categories and reports interesting experimental results on the performance of the network under various conditions. He plots the performance of the initial model as a function of number of eye movements and uses that as a benchmark while exploring some modifications to the network and the training strategy. The initial network did not have nodes with overlapping receptive fields. The authors experiment to find out the effect of overlapping on the performance was inconclusive. Although the performance was better for between 1-3 eye movements, the performance for larger number of eye movements fluctuated above and below the original model. The author also does an interesting experiment on memory usage vs recognition accuracy.
The author raises many valid concerns about the original model and many of these have been addressed in the new Pictures example. This paper should be an interesting read for anyone thinking of doing additional experiments based on the Pictures example.
Garalevicius, Saulius. (2006). Memory-Prediction Framework for Pattern Recognition . Department of Computer and Information Sciences, Temple University.
Biomimetic sensory abstraction using hierarchical quilted self-organizing maps »
J. W. Miller and P. H. Lommel
HTMs use temporal and spatial pooling at all levels of the hierarchy to obtain invariant representations. Different algorithms can be used to achieve this temporal and spatial pooling. In our current HTM implementation, we learn a temporal transition matrix and then partition the graph represented by that matrix to obtain temporal pools.
In contrast, this paper uses a Self Organizing Map to achieve temporal pooling (temporal clustering). The SOMs seem to have a closer relation to biology although they could turn out to be theoretically equivalent to temporal transition learning. Moreover, SOMs can also be used to cluster together 'similar' units in space, thus reducing the effective search space for a higher level node. In the current implementation of HTMs, the boundaries of the nodes abruptly stop the information flow. SOMs also provide a convenient way to implement overlapping nodes.
This paper also uses online learning. Our current set of algorithms do not do on-line learning. This is something we would rectify in our next version of algorithms. Disadvantages of using SOMs could include increased processing requirements and poor understanding of the convergence and scaling properties. Incorporating feedback and predicting forward in time could also prove to be tricky.
Understanding and creating efficient algorithms for temporal and spatial pooling and integrating them in a hierarchy is the fundamental process behind creating the HTM technology. This paper demonstrates the use of an alternative set of algorithms that could be used for this purpose.
Disadvantages of using SOMs could include increased processing requirements and poor understanding of the convergence and scaling properties. Incorporating feedback and predicting forward in time could also prove to be tricky.
Miller, J.W., Lommel, P.H. Biomimetic sensory abstraction using hierarchical quilted self-organizing maps. The Charles Stark Draper Laboratory, Inc.
