ResearchInteractive previewWrite with Jenni Jakob Uszkoreit is a credited coauthor of “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale” and “Attention Is All You Need”. Explore the papers, research findings, and collaborators represented in this collection.
Read the original papers, explore an overview, and collect ideas for your own research.
In a nutshellVision Transformer treats an image as a sequence of patches. With large-scale pretraining, this straightforward adaptation of a Transformer performs strongly on image recognition.
In a nutshellThe Transformer replaces recurrence and convolution with attention. It connects words directly across a sequence, enabling more parallel training and strong machine translation performance.
Images are divided into fixed-size patches and encoded as sequences. The model is pretrained on large datasets and fine-tuned on image classification tasks.
Reading context: The strong results depend on the training-data regime. They should not be read as a universal advantage over convolutional models on small datasets.
Read the original on arXiv ↗The authors evaluate an attention-based encoder and decoder on WMT 2014 translation tasks, comparing translation quality and training cost with earlier architectures.
Reading context: The experiments focus primarily on translation. Results do not establish that attention is the best choice for every task or resource budget.
Read the original on arXiv ↗