Peekaboo

Yesterday a week-long scikit-learn coding sprint in Paris ended. And let me just say: a week is pretty long for a sprint. I think most of us were pretty exhausted in the end. But we put together a release candidate for 0.14 that Gael Varoquaux tagged last night. You can install it via:   pip install -U https://github.com/scikit-learn/scikit-learn/archive/0.14a1.zip There are also tarballs on github and binaries on sourceforge . If you want the most current version, you can check out the release branch on github: https://github.com/scikit-learn/scikit-learn/tree/0.14.X The full list of changes can be found in what's new . The purpose of  the release candidate is to give users a chance to give us feedback before the release. So please try it out and report back if you have any issues.

The paper "Single Layer Networks in Unsupervised Feature Learning" by Coates, Lee, Ng, that I talked about in this post , is now available online ( pdf )! Thanks to Olivier Grisel from metasploit for pointing that out.

My colleague Hannes Schulz and me also had a paper in this years NIPS deep learning workshop : “ Investigating Convergence of Restricted Boltzmann Machine Learning “. It is about evaluation of RBM training. One problem one faces when training a RBM is that it is usually not possible to evaluate the actual objective function. This makes it hard to evaluate the training and find the right hyperparameters. This problem is even more severe since contrastive divergence and persistent contrastive divergence learning, which are the most popular learning algorithms for RBMs are know to diverge if the hyperparamters are not tuned well. In our work we train a small RBM for which we can compute the partition function and evaluate the objective function exactly. We trained RBMs with a minimum number of hyper parameters and computed exact learning curves. We confirm the divergence of the algorithms in some cases and we also confirm that the reconstruction error is not a good measure of performa...

The paper "Single Layer Networks in Unsupervised Feature Learning" by Coates, Lee, Ng is one of the most interesting on this years NIPS in my opinion. [edit] It's now available online! ( pdf ) [/edit] It follows a very simple idea: Compare "shallow" unsupervised feature extraction on image data using classification. Datasets that are used are NORB and CIFAR 10, two of the most used datasets in the deep learning community. Filters are learned on image patches and then features are computed in a very simple pyramid over the image. These are then classified using a linear SVM. The approaches that are compared are: K-Means soft K-Means Sparse Autoencoder Sparse RBM Here soft K-Means is an ad-hoc method that the authors thought up as being a natural extension of K-Means. It is a local coding based on the k nearest neighbors of a point. A cross validation was performed to find the best patchsize, number of features and distance between sample points for f...

As promised, my group recently published our Restricted Boltzmann Machine implementation . It is based upon the CUV Library that is being developed here. The idea is to combine the ease of programming of Python with the computing power of the GPU. We used this implementation for several papers and it grew a lot over time. Here is a list of most of the features: Restricted Boltzmann Machine Training With n-step Contrastive Divergence With persistent Contrastive Divergence Weight decay, momentum, batch-learning Binary or gaussian visible nodes Restricted Boltzmann Machine Evaluation Sampling from the model Visualizing Filters Annealed Importance Sampling for approximating the partition function Calculating the partition function exactly Visualization and saving of hidden representations Stacking RBMs to Deep Belief Networks Sampling from DBNs Deep Boltzmann Machine Training With n-step Contrastive Divergence With persistent Contrastive Divergence Deep Boltzmann Ma...

The CUDA library from my working group was updated again today. Features of the library include matrix and vector operations on CPU and GPU using NVidia Cuda. Much work went into a clean design and ease of use. Beside the C++ interface there are also Python wrappers for very easy coding using the GPU. We mainly use this library for neural networks and restricted Boltzmann machines but it is quite generic and everybody who uses dense or diagonal matrix operations can benefit from it. New features include convenience functions in Python and Image Pyramids on the GPU. There are also some minor fixes. If everything goes as planned there will be another mayor update quite soon.

When I was on NIPS last year, I overheard Ruslan Salakhutdinov being asked whether training RBMs is more of an art than a science. I think this question is answered (at least for the moment) by a great paper by Asja Fischer and Christian Igel that will be presented tomorrow at the ICANN . They evaluate different training methods for RBMs on toy problems, where the partition function can be evaluated explicitly. What they find is that after an inital increase, the log-likelyhood of all models diverges. That is except the learn rate schedule or the weight decay parameter are choosen just right. Since it is impossible to evaluate the true log probability on a "real-world" dataset (see my older post ) this means that it seems impossible to know whether divergence occures and to choose the parameters accordingly. This paper evaluates CD, PCD and fast PCD but does not use parallel tampering (yet). It would be very interesting to see if parallel tampering might solve this...

On the connectionist mailing list Geoffrey Hinton , the father of the Restricted Boltzmann Machine, posted a guide on how to use and train them . Quite nice if you are interested in these kinds of models. But keep in mind: Restricted Boltzmann Machines are Hard to Approximately Evaluate or Simulate. No matter what the guys in Torronto say ;)