November 12, 2014

Some starting points for deep learning and RNNs

This version 2016-03-22 cross posted at www.aistartups.org, older versions: 2014-11-12,  2014-05-17.

Bengio, LeCun, Jordan, Hinton, Schmidhuber, Ng, de Freitas and OpenAI have done reddit AMA's.  These are nice places to start to get a Zeitgeist of the field.

Hinton and Ng lectures at Coursera, UFLDL, CS224d and CS231n at Stanford, the deep learning course at Udacity, and the summer school at IPAM have excellent tutorials, video lectures and programming exercises that should help you get started. (Update 2016-08-22: Unfortunately coursera closed down their old courses, but here are youtube links for Neural Nets by Geoffrey Hinton, Machine Learning by Andrew Ng, Reinforcement Learning by David Silver, Probabilistic Graphical Models by Daphne Koller, Convex Optimization by Stephen Boyd).

The online book by Nielsen, notes for CS231n, and blogs by Karpathy, Olah and Britz have clear explanations of MLPs, CNNs and RNNs.  The tutorials at UFLDL and deeplearning.net give equations and code. The encyclopaedic book by Goodfellow et al. is a good place to dive into details. I have a draft book in progress.

Theano, Torch, Caffe, ConvNet, TensorFlow, MXNet, CNTK, Veles, CGT, Neon, Chainer, Blocks and Fuel, Keras, Lasagne, Mocha.jl, Deeplearning4j, DeepLearnToolbox, Currennt, Project Oxford, Autograd (for Torch), Warp-CTC are some of the many deep learning software libraries and frameworks introduced in the last 10 years.  convnet-benchmarks and deepframeworks compare the performance of many existing packages. I am working on developing an alternative, Knet.jl, written in Julia supporting CNNs and RNNs on GPUs and supporting easy development of original architectures.  More software can be found at deeplearning.net.

Deeplearning.net and homepages of Bengio, Schmidhuber have further information, background and links.

13 comments:

Ahmet A. Akın said...

There is CURRENNT for RNN - LSTM training with GPUs.

Deniz Yuret said...

From Volkan: DeepLearnToolbox
Matlab/Octave toolbox for deep learning. Includes Deep Belief Nets, Stacked Autoencoders, Convolutional Neural Nets, Convolutional Autoencoders and vanilla Neural Nets. Each method has examples to get you started.

Deniz Yuret said...

A concise tutorial for CUDA.

Deniz Yuret said...

A new book Neural Networks and Deep Learning by Michael Nielsen.

Deniz Yuret said...

Andrej Karpathy has a fun, informative, and detailed blog post on RNNs: The Unreasonable Effectiveness of Recurrent Neural Networks.

Deniz Yuret said...

Richard Socher has a new course on Deep Learning for Natural Language Processing.

Deniz Yuret said...

Can we use vanilla RNNs with smart initialization instead of LSTM? A Simple Way to Initialize Recurrent Networks of Rectified Linear Units.

Ozan Arkan Can said...

Memory Networks & Question Answering: Memory Networks, Weakly Supervised Memory Networks, Demo

Ozan Arkan Can said...

A Critical Review of Recurrent Neural Networks for Sequence Learning

Deniz Yuret said...

There has been a flurry of new releases of deep learning packages. Here is a few:
TensorFlow by Google (blog post, Online course).
Autograd for Torch by Twitter.
Project Oxford by Microsoft.
The VELES by Samsung.

MXNet, and Visualizing CNN architectures side by side with mxnet

CNTK by Microsoft.

Lasagne is a lightweight library to build and train neural networks in Theano.

convnet-benchmarks compares the performance of many existing packages

Deepframeworks is another comparison effort.

Deniz Yuret said...

Check out Warp-CTC by Baidu.

Deniz Yuret said...

Some resources for learning about Convolutional Neural Networks:
* Karpathy's ecture notes with very good explanation of MLPs and CNNs: http://cs231n.github.io
* Colah's blog entries with very good explanation of CNNs: http://colah.github.io/posts/2014-07-Conv-Nets-Modular/
* There is a nice tutorial with programming examples (similar to UFLDL) at http://deeplearning.net/tutorial
* Nielsen's online book covering MLPs and CNNs: http://neuralnetworksanddeeplearning.com
* Goodfellow et al.'s book with encyclopaedic coverage: http://www.deeplearningbook.org
* Notes on convnet gradients: http://people.csail.mit.edu/jvb/papers/cnn_tutorial.pdf
* UFLDL and Hinton's coursera lecture (Lec 5) has good intros to CNNs.

Deniz Yuret said...

Yet another framework: Darknet.