November 01, 2016

Transfer Learning for Low-Resource Neural Machine Translation

Zoph, Barret and Yuret, Deniz and May, Jonathan and Knight, Kevin. 2016. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing pp 1568--1575, Austin, Texas. (PDF)

Abstract

The encoder-decoder framework for neural machine translation (NMT) has been shown effective in large data scenarios, but is much less effective for low-resource languages. We present a transfer learning method that significantly improves BLEU scores across a range of low-resource languages. Our key idea is to first train a high-resource language pair (the parent model), then transfer some of the learned parameters to the low-resource pair (the child model) to initialize and constrain training. Using our transfer learning method we improve baseline NMT models by an average of 5.6 BLEU on four low-resource language pairs. Ensembling and unknown word replacement add another 2 BLEU which brings the NMT performance on low-resource machine translation close to a strong syntax based machine translation (SBMT) system, exceeding its performance on one language pair. Additionally, using the transfer learning model for re-scoring, we can improve the SBMT system by an average of 1.3 BLEU, improving the state-of-the-art on low-resource machine translation.


Full post...

Why Neural Translations are the Right Length

Shi, Xing and Knight, Kevin and Yuret, Deniz. 2016. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing pp 2278--2282, Austin, Texas. (PDF)

Abstract

We investigate how neural, encoder-decoder translation systems output target strings of appropriate lengths, finding that a collection of hidden units learns to explicitly implement this functionality.


Full post...