Peekaboo

The ICML2012 accepted papers are officially online . On twitter , Andrej Kaparthy complained that the list is a bit hard to browse through. I agree and even though this is probably not the nice visualization he had in mind, I felt like having topical reading lists would somehow mitigate this problem. Here is my reading list on deep learning and unsupervised feature extraction: A Generative Process for Contractive Auto-Encoders Salah Rifai, Yann Dauphin, Pascal Vincent, Yoshua Bengio – Accepted Abstract: The contractive auto-encoder learns a representation of the input data that captures the local manifold structure around each data point, through the leading singular vectors of the Jacobian of the transformation from input to representation. The corresponding singular values specify how much local variation is plausible in directions associated with the corresponding singular vectors, whi...

My colleague Hannes works in deep learning and started on a new feature extraction method this week. As with all feature extraction algorithms, it was obviously of utmost importance to be able to learn Gabor filters . Inspired by his work and Natural Image Statistics , a great book on the topic of feature extraction from images, I wanted to see how hard it is to learn Gabor filters with my beloved scikit-learn . I chose independent component analysis , since this is discusses to some depth in the book. Luckily mldata had some image patches that I could use for the task. Here goes: import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import fetch_mldata from sklearn.decomposition import FastICA # fetch natural image patches image_patches = fetch_mldata("natural scenes data") X = image_patches.data # 1000 patches a 32x32 # not so much data, reshape to 16000 patches a 8x8 X = X.reshape(1000, 4, 8, 4, 8) X = np.rollaxis(X, 3, 2).reshape(-1, 8 *...

John Langford published a great article on his blog today: http://hunch.net/?p=1822 Don't miss out on it ;)

The paper "Single Layer Networks in Unsupervised Feature Learning" by Coates, Lee, Ng, that I talked about in this post , is now available online ( pdf )! Thanks to Olivier Grisel from metasploit for pointing that out.

There was an interesting talk by Jitendra Malik about "Rich Representations for Learning Visual Recognition" and thereafter a panel discussion with Jitendra Malik, Yann LeCun, Geoff Hinton, Tomaso Poggio, Kai Yu, Yoshua Bengio and Andrew Ng. Many "deep" topics were touched but there is one or two ideas that I found the most noteworthy. The first is the idea by Malik to do "hyper supervision". This is his idea of doing the exact opposite than weak supervision: The training examples are labeled very precisely and with lots of extra information. This makes it possible to find more interesting intermediate representations. It also gives the learning algorithm more to work on. In his introduction he said: "Learning object recognition from bounding boxes is like learning language from a list of sentences." If I understand his ideas correctly, he thinks that is its necessary to have additional clues - like 3D information, tracking and time consist...

My colleague Hannes Schulz and me also had a paper in this years NIPS deep learning workshop : “ Investigating Convergence of Restricted Boltzmann Machine Learning “. It is about evaluation of RBM training. One problem one faces when training a RBM is that it is usually not possible to evaluate the actual objective function. This makes it hard to evaluate the training and find the right hyperparameters. This problem is even more severe since contrastive divergence and persistent contrastive divergence learning, which are the most popular learning algorithms for RBMs are know to diverge if the hyperparamters are not tuned well. In our work we train a small RBM for which we can compute the partition function and evaluate the objective function exactly. We trained RBMs with a minimum number of hyper parameters and computed exact learning curves. We confirm the divergence of the algorithms in some cases and we also confirm that the reconstruction error is not a good measure of performa...

The paper "Single Layer Networks in Unsupervised Feature Learning" by Coates, Lee, Ng is one of the most interesting on this years NIPS in my opinion. [edit] It's now available online! ( pdf ) [/edit] It follows a very simple idea: Compare "shallow" unsupervised feature extraction on image data using classification. Datasets that are used are NORB and CIFAR 10, two of the most used datasets in the deep learning community. Filters are learned on image patches and then features are computed in a very simple pyramid over the image. These are then classified using a linear SVM. The approaches that are compared are: K-Means soft K-Means Sparse Autoencoder Sparse RBM Here soft K-Means is an ad-hoc method that the authors thought up as being a natural extension of K-Means. It is a local coding based on the k nearest neighbors of a point. A cross validation was performed to find the best patchsize, number of features and distance between sample points for f...

As promised, my group recently published our Restricted Boltzmann Machine implementation . It is based upon the CUV Library that is being developed here. The idea is to combine the ease of programming of Python with the computing power of the GPU. We used this implementation for several papers and it grew a lot over time. Here is a list of most of the features: Restricted Boltzmann Machine Training With n-step Contrastive Divergence With persistent Contrastive Divergence Weight decay, momentum, batch-learning Binary or gaussian visible nodes Restricted Boltzmann Machine Evaluation Sampling from the model Visualizing Filters Annealed Importance Sampling for approximating the partition function Calculating the partition function exactly Visualization and saving of hidden representations Stacking RBMs to Deep Belief Networks Sampling from DBNs Deep Boltzmann Machine Training With n-step Contrastive Divergence With persistent Contrastive Divergence Deep Boltzmann Ma...

My colleague Hannes uploaded a simple Multi Layer Perceptron as a demo for our CUV library. It is written entirely in Python and classifies the MNIST dataset of handwritten digits. He also wrote a blog post explaining the design and use in detail. The code is very easy to understand and to expand. But it is very fast - of course, using CUDA - and can serve as the basis for many experiments. There are also convolution routines in the library and its even possible to extend the network to a convolutional neural network entirely in Python and on the GPU.

The CUDA library from my working group was updated again today. Features of the library include matrix and vector operations on CPU and GPU using NVidia Cuda. Much work went into a clean design and ease of use. Beside the C++ interface there are also Python wrappers for very easy coding using the GPU. We mainly use this library for neural networks and restricted Boltzmann machines but it is quite generic and everybody who uses dense or diagonal matrix operations can benefit from it. New features include convenience functions in Python and Image Pyramids on the GPU. There are also some minor fixes. If everything goes as planned there will be another mayor update quite soon.

On the connectionist mailing list Geoffrey Hinton , the father of the Restricted Boltzmann Machine, posted a guide on how to use and train them . Quite nice if you are interested in these kinds of models. But keep in mind: Restricted Boltzmann Machines are Hard to Approximately Evaluate or Simulate. No matter what the guys in Torronto say ;)