Parsing with context embeddings

Ömer Kırnap, Berkay Furkan Önder and Deniz Yuret. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, Vancouver, 2017. (PDF, poster, presentation, related posts).

Abstract. We introduce context embeddings, dense vectors derived from a language model that represent the left/right context of a word instance, and demonstrate that context embeddings significantly improve the accuracy of our transition based parser. Our model consists of a bidirectional LSTM (BiLSTM) based language model that is pre-trained to predict words in plain text, and a multi-layer perceptron (MLP) decision model that uses features from the language model to predict the correct actions for an ArcHybrid transition based parser. We participated in the CoNLL 2017 UD Shared Task as the ``Koç University'' team and our system was ranked 7th out of 33 systems that parsed 81 treebanks in 49 languages.

Full post...

JuliaCon 2017, Berkeley, June 20-24

I gave a talk at JuliaCon introducing Knet on Wednesday, June 21, 2017, 4:16pm - 4:52pm, East Pauley Pauley Ballroom, Berkeley, CA. See these related posts.
Full post...

Congratulations to the Koç parsing team

Our neural net based dependency parser was number 7 overall out of 33 teams participating in the CoNLL 2017 Shared Task "Multilingual Parsing from Raw Text to Universal Dependencies" in which participating teams had to parse 68 corpora in 50 languages. I would like to thank Ömer Kırnap and Berkay Furkan Önder for their contributions and all-nighters.
Full post...

The third deep learning revolution

The first revolution took place 1958-1969.

• We figured out how to train perceptrons.
• We proved the perceptron convergence theorem.
• Interest waned after a book (Perceptrons) written by mathematicians.

The second revolution took place 1986-1995.

• We figured out how to train multi-layer perceptrons.
• We proved the universal approximation theorem.
• Interest waned after a book (SLT) written by mathematicians.

The third revolution started in 2012.

• We figured out how to train deep nets.
• A mathematician will write a book around 2022.
• The fourth revolution will not start until 2036 :)

Full post...

Overfitting, underfitting, regularization, dropout

Here is an IJulia notebook demonstrating overfitting, underfitting, regularization and dropout in Knet for my machine learning class.
Full post...