Attention Is All You Need
In a nutshellThe Transformer replaces recurrence and convolution with attention. It connects words directly across a sequence, enabling more parallel training and strong machine translation performance.
Language models
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
THE PAPER, IN A NUTSHELL
BERT learns language representations using context from both sides of a word. A pretrained model can then be adapted to different language understanding tasks with small architectural changes.
Read the paper
The original PDF. Select a passage to explore it with Jenni.
Preparing your reading space…
The key takeaways
01
Bidirectional context improves language representations.
02
One pretrained model transfers to multiple downstream tasks.
03
The paper reports improvements on eleven language understanding tasks.
How the research works
Pretraining on unlabeled text is followed by supervised fine-tuning and evaluation on language understanding and question-answering benchmarks.
A thoughtful reading
Downstream fine-tuning still needs task-specific training data. These results concern language understanding, rather than open-ended text generation.
Editorial overview · Always refer to the original study.