52 - Sequence-to-Sequence Learning as Beam-Search Optimization, with Sam Wiseman

NLP Highlights

English - March 15, 2018 21:56 - 23 minutes - 21 MB - ★★★★★ - 22 ratings
Science Homepage Download Apple Podcasts Google Podcasts Overcast Castro Pocket Casts RSS feed

Previous Episode: 51 - A Regularized Framework for Sparse and Structured Neural Attention, with Vlad Niculae

Next Episode: 53 - Classical Structured Prediction Losses for Sequence to Sequence Learning, with Sergey and Myle

EMNLP 2016 paper by Sam Wiseman and Sasha Rush.

In this episode we talk with Sam about a paper from a couple of years ago on bringing back some ideas from structured prediction into neural seq2seq models. We talk about the classic problems in structured prediction of exposure bias, label bias, and locally normalized models, how people used to solve these problems, and how we can apply those solutions to modern neural seq2seq architectures using a technique that Sam and Sasha call Beam Search Optimization.

(Note: while we said in the episode that BSO with beam size of 2 is equivalent to a token-level hinge loss, that's not quite accurate; it's close, but there are some subtle differences.)

https://www.semanticscholar.org/paper/Sequence-to-Sequence-Learning-as-Beam-Search-Optim-Wiseman-Rush/28703eef8fe505e8bd592ced3ce52a597097b031