Peekaboo

Yesterday a week-long scikit-learn coding sprint in Paris ended. And let me just say: a week is pretty long for a sprint. I think most of us were pretty exhausted in the end. But we put together a release candidate for 0.14 that Gael Varoquaux tagged last night. You can install it via:   pip install -U https://github.com/scikit-learn/scikit-learn/archive/0.14a1.zip There are also tarballs on github and binaries on sourceforge . If you want the most current version, you can check out the release branch on github: https://github.com/scikit-learn/scikit-learn/tree/0.14.X The full list of changes can be found in what's new . The purpose of  the release candidate is to give users a chance to give us feedback before the release. So please try it out and report back if you have any issues.

As you hopefully have heard, we at scikit-learn are doing a user survey (which is still open by the way). One of the requests there was to provide some sort of flow chart on how to do machine learning. As this is clearly impossible, I went to work straight away. This is the result: [edit2] clarification: With ensemble classifiers and ensemble regressors I mean random forests , extremely randomized trees, gradient boosted trees , and the soon-to-be-come weight boosted trees (adaboost). [/edit2] Needless to say, this sheet is completely authoritative.

Recently we added another method for kernel approximation, the Nyström method, to scikit-learn , which will be featured in the upcoming 0.13 release. Kernel-approximations were my first somewhat bigger contribution to scikit-learn and I have been thinking about them for a while. To dive into kernel approximations, first recall the kernel-trick .

Today there was a workshop at my uni, organized by my Professor Sven Behnke, together with my colleagues Hannes Schulz, Nenard Birešev and me. The target group was a local graduate school with a general scientific background, but not much CS or machine learning. The workshop consisted of us explaining the methods and the students then playing around with them and answering some questions using IPython notebooks that we provided (if you still don't know about IPython Notebooks, watch this talk now ). Using the notebooks worked out great! There is only so much you can teach in a 5 hour workshop but I think we got across some basic concepts of machine learning and working with data in Python. We got some positive feedback and the students really went exploring. We covered PCA, k-means, linear regression, logistic regression and nearest neighbors, including some real-world examples. You can find all resources, including tex and notebooks for generating figures etc. on gith...

Recently Jake showed some pretty cool videos in his blog . This inspired me to go back to an idea I had some time ago, about visualizing high-dimensional data via random projections. I love to do exploratory data analysis with scikit-learn , using the manifold , decomposition and clustering module. But in the end, I can only look at two (or three) dimensions. And I really like to see what I am doing.

Recently I entered my first kaggle competition - for those who don't know it, it is a site running machine learning competitions. A data set and time frame is provided and the best submission gets a money prize, often something between 5000$ and 50000$. I found the approach quite interesting and could definitely use a new laptop, so I entered Detecting Insults in Social Commentary. My weapon of choice was Python with scikit-learn - for those who haven't read my blog before: I am one of the core devs of the project and never shut up about it.

Last night I uploaded the new version 0.12 of scikit-learn to pypi . Also the updated website is up and running and development now starts towards 0.13 . The new release has some nifty new features ( see whatsnew ): * Multidimensional scaling * Multi-Output random forests ( like these ) * Multi-task Lasso * More loss functions for ensemble methods and SGD * Better text feature extraction

I just pushed an update for my structured SVM in Python. This contains a bugfix in the dual formulation  and a subgradient descent version of the structured SVM.

[EDIT: If you are reading this now, have a look at pystruct.github.io . The project matured quit a bit in the meantime.]  Today I pushed some of my code to github that I use for experimenting with CRF learning. This goes along the lines of my recent posts on graphcut and I hope to post a full CRF learning framework for semantic image segmentation soon. This is a pretty standard setup in computer vision, but I really haven't found much code online. Actually I haven't found any code to learn loopy CRFs, so I hope my simple implementation can help to get a better understanding of these methods. It certainly helped me ;)

I just pushed some of my structured learning code to github and hope that some people might find it useful. Before describing my code here, I wanted to give a basic intro into structured prediction. I hope I can at least convey some intuition for this vast research area. So here goes... What is structured learning and prediction? Structured prediction is a generalization of the standard paradigms of supervised learning, classification and regression. All of these can be thought of finding a function that minimizes some loss over a training set. The differences are in the kind of functions that are used and the losses. In classification, the target domain are discrete class labels, and the loss is usually the 0-1 loss, i.e. counting the misclassifications. In regression, the target domain is the real numbers, and the loss is usually mean squared error. In structured prediction, both the target domain and the loss are more or less arbitrary. This means the goal is not to predict ...

The ICML2012 accepted papers are officially online . On twitter , Andrej Kaparthy complained that the list is a bit hard to browse through. I agree and even though this is probably not the nice visualization he had in mind, I felt like having topical reading lists would somehow mitigate this problem. Here is my reading list on deep learning and unsupervised feature extraction: A Generative Process for Contractive Auto-Encoders Salah Rifai, Yann Dauphin, Pascal Vincent, Yoshua Bengio – Accepted Abstract: The contractive auto-encoder learns a representation of the input data that captures the local manifold structure around each data point, through the leading singular vectors of the Jacobian of the transformation from input to representation. The corresponding singular values specify how much local variation is plausible in directions associated with the corresponding singular vectors, whi...

As I have been working on some clustering algorithms recently, I invested some time last weekend to refactor some code inside sklearn to generate some toy data sets to visualize the results of clustering algorithms. That looks something like this: While the first two are nice to show off that your algorithm can handle non-convex clusters, these data sets obviously look nothing like the data you'll see in practice. So I wanted to have some a bit more general data set generator. What I ended up doing is a nonparametric mixture of Gaussians. While Gaussians are a bit boring, combining them with a non-parametric prior makes them somewhat more general. As I didn't found some very easy to use package to do that (though David pointed out p ymc ) I went ahead and wrote the generative model down myself. It's a mixture of Gaussians with a Chinese restaurant process as prior for the mixture components and Wishard-Gaussian priors for mean and variance. You can find the ...

My favourite machine learning library has got a major upgrade (unless you're using the dev version like me ;) Important new features include manifold learning, Dirichlet process mixture models and dataset downloader / import functions . You can find the changelog here .

Some tips and facts that I took from the summer school. They are pretty random but may be usefull for pactitioners of vision. Many may seem obvious - but I just didn't see it before ... Gist doesn't work on cropped or rotated images. Since it does a kind of whole image template matching, this is pretty clear. And maybe it shouldn't - the scene layout is changed after all. For doing BoW Cordelia Schmid suggests (and I guess uses) 6x6 patches and a pyramid with scale factor 1.2. Scaling is done by Gaussian convolution. Ponce uses 10x10 patches to do sparse coding. If you combine multiple features using MKL or by just adding up kernels (which is the same as concatenating features), normalize each feature by it's variance and then search for a joint gamma. This heuristic get's you out of doing grid search over a huge space! Cordelia Schmid thinks that "clever clusters" don't help much in doing BoW. She thinks it's more important to work ...

John Langford published a great article on his blog today: http://hunch.net/?p=1822 Don't miss out on it ;)

While reading around MetaOptimize , I found a reference to the book " Bayesian Reasoning and Machine Learning" by David Barber. You can download the pdf for free and it will be published soon in Cambridge University Press. It comes with many Matlab examples and seems worth taking a look at :)

Two days ago, Josh Tenenbaum gave a very inspiring talk about how to "learn learning", about abstraction and structure in learning. In my opinion, it has been one of the best talks of this conference. It gave a broad overview over some new conceptual ideas and some experiments building upon these ideas. Josh Tenenbaum is a great speaker and I am sure I can not live up to giving a good summary of his talk. Nevertheless I will try and give a short summary of his ideas and some pointers to his work. I very much recommend you see this talk as a video lecture and I am sure I will see it again. The basic question that Tenenbaum asked is "How does the mind get so much from so little." In my words he is asking: How can we generalize from the little and very noisy data that we get? His main interest in this is from a cognitive science perspective but of course his ideas are also very applicable to machine learning. He formulates his approach as reverse engineering hu...

NIPS 2010 has been going on for half a week already and finally I have the time to write about it. So many inspiring posters and talks and so many great people to talk to. I'll start by giving you this years top 9 tips to get your paper accepted to NIPS: Write a strong abstract. A paper with the following abstract got rejected: "This paper is great." Hide the fact that the paper is recycled. One paper was submitted that was obviously resubmitted from ICML. How do we know? It had additional to the line numbers in the new NIPS layout also (and overlapping) the ICML line numbers on both sides. Carefully maintain anonymity. One of the reviewers remarked that he found the authors name in the pdf properties of a submission. Avoid certain reviewers - apparently some are quite harsh. Sadly I don't have the review in question but since you can't be sure to avoid the reviewer, it's better if you don't know it. Get your bid in early. Someone asked for a t...

Wow. So much to read today . While following link upon link, I found so many great toolkits that I think it is worth listing them here. One of the greatest sources was the GNU/Linux AI & Alife HOWTO . [edit] It's been a while since I wrote this blog post but still many people seem to find it, so here a quick update. After looking into many libraries, I started using scikit-learn and then using it exclusively. Now I am a regular contributor. It is a fast growing project with great documentation resources, many algorithms and it is just so easy to use. Also, working with Python and the Python crowd is fun. I heartly recommend it. [/edit] Here goes: Vowpal Wabbit - project on very fast online gradient descent by Yahoo research (C++) VFML (Very Fast Machine Learning) - library for very fast decision trees and Bayes networks (C++) Stochastic Gradient Descent - library for SVMs with stochastic gradient descent (C++) Maximum Entropy Modeling Toolkit for Python and C++ - ...

Today I stumbled upon metaoptimize . I am always looking for new blogs/communities/forums an this one seems very active and very in-depth. There is a stackoverflow like Q&A section with many many questions and comments on many machine learning topics. There is a lot on probabilistic modeling there which I find particularly interesting. I haven't seen any other community like this. The reason I found this community is actually that this blog was linked from there :) There is a thread on machine learning blog s that gives many more than I was aware of... I did not have the time to look into all of them but I will definitely try and follow them all :)