NIPS 2010 - Thinking dynamically

Apart from the presentations and posters, there is another great thing about NIPS: you can discuss machine learning with great researcher in person.

One of the people I talked to quite a lot is Jascha Sohl-Dickstein. We discussed some deep methods and training procedures at some length and he is an amazing person with a lot of energy and new ideas.
He recently wrote two papers that I quite liked:
Minimum Probability Flow Learning and An Unsupervised Algorithm For Learning Lie Group Transformations.
I like both of them for their quite unusual point of view. Jascha has a background in physics and his point of view focuses a lot on understanding the dynamics of learning and transformations.
It think "Minimum Probability Flow Learning" gives new insights into training probabilistic models and as far as I know it is used quite successfully for training Ising models.
Both works are not published yet but I find they are quite worth reading and so I'd like to draw a little more attention to them. He was actually quite surprised that I was so familiar with his work but I really like it a lot. Both papers have clear mathematical formulations of the underlying problems and feature beautiful formulations.
So if you are interested in nice math and probabilistic models, you should definitely read "Minimum Probability Flow Learning". If you are interested in videos and image transformations, "On Unsupervised Algorithm For Learning Lie Group" is for you.

Comments

  1. Yaroslav BulatovDecember 16, 2010 at 9:36 PM

    Minimum flow paper is interesting idea, but I wonder about the utility of learning the parameters of an intractable model....even sampling could be intractable, so why would you care about knowing the parameters of such model? (unless you are a physicist and the value of parameter is of interest in itself)

    ReplyDelete
  2. Andreas MuellerDecember 17, 2010 at 10:44 AM

    Are you talking about the Ising models? I think not only physicists are interested in those. Many people use them to learn about the structure of certain models. I think they are used in Biology quite a bit.
    And about intractable models: Depends on what you see as intractable. If you are talking about exact learning and inverence, isn't every interesting model intractable? RBMs are intractable and many people work on them. And every latent variable model is intractable in this sense. But they are used a lot nontheless, right?

    ReplyDelete

Post a Comment

Popular posts from this blog

Machine Learning Cheat Sheet (for scikit-learn)

Image
As you hopefully have heard, we at scikit-learn are doing a user survey (which is still open by the way). One of the requests there was to provide some sort of flow chart on how to do machine learning. As this is clearly impossible, I went to work straight away. This is the result: [edit2] clarification: With ensemble classifiers and ensemble regressors I mean random forests , extremely randomized trees, gradient boosted trees , and the soon-to-be-come weight boosted trees (adaboost). [/edit2] Needless to say, this sheet is completely authoritative.
Read more

A Wordcloud in Python

Image
Last week I was at Pycon DE , the German Python conference. After hacking on scikit-learn a lot last week, I decided to to something different on my way back, that I had planned for quite a while: doing a wordl -like word cloud . I know, word clouds are a bit out of style but I kind of like them any way. My motivation to think about word clouds was that I thought these could be combined with topic-models to give somewhat more interesting visualizations. So I looked around to find a nice open-source implementation of word-clouds ... only to find none. (This has been a while, maybe it has changed since). While I was bored in the train last week, I came up with this code . A little today-themed taste:
Read more

MNIST for ever....

Image
[update] This post is a bit old, but many people still seem interested. So just a short update: Nowadays I would use Python and scikit-learn to do this. Here is an example of how to do cross-validation for SVMs in scikit-learn. Scikit-learn even downloads MNIST for you. [/update] MNIST is, for better or worse, one of the standard benchmarks for machine learning and is also widely used in then neural networks community as a toy vision problem. Just for the unlikely case that anyone is not familiar with it: It is a dataset of handwritten digits, 0-9, in black on white background. It looks something like this: There are 60000 training and 10000 test images, each 28x28 gray scale. There are roughly the same number of examples of each category in the test and training datasets. I used it in some papers myself even though there are some reasons why it is a little weird. Some not-so-obvious (or maybe they are) facts are: - The images actually contain a 20x20 patch of digi
Read more

Python things you never need: Empty lambda functions

Ever wanted to create an empty Python function in-line? No? Me neither. But my coworker does... We found the answer: <pre> x = lambda : None </pre> If you ever needed this, please tell me ;) [Edit] Apparently some people do need it :) Btw, most people seem to try Most people try lambda: pass first. But you have to remember, the thing at the right side of the colon is the return value of the function, not the body! So an expression is needed, not a statement. [\Edit]
Read more