Self-Attentive Model for Headline Generation

One line that carries the whole article

Every piece of news arrives with a title already attached, which makes headline generation the rare summarization task with practically unlimited supervision. No annotator sits in the loop: the training pair is the article and the headline its editor wrote. The recipe is also language-independent, because nothing in it requires a native speaker to mark anything up.

The compression involved is extreme. In our filtered New York Times training set the average article runs 707.6 words and the average headline 7.9, so the model must fold roughly ninety words into one while staying factual and fluent. That pressure is what makes the task a genuine test of whether an architecture captures the structure of a text rather than its surface.

Prior systems softened the problem in exactly that way. The classic baseline reprints the first sentence; the strongest neural approaches ran an encoder-decoder over that sentence, or over a summary pre-generated by an external statistical system. We train on the full text and the title, end to end, with no preprocessing beyond lowercasing.

Why we abandoned the recurrent bottleneck

We model the headline autoregressively: the network defines $P(y_t \mid y_1,\dots,y_{t-1}, X, \theta)$, the distribution of the next title token given the article $X$ and the tokens written so far, and we fit $\theta$ by maximizing likelihood over the corpus. At inference we decode with a beam of width 10, which consistently beats greedy decoding at the cost of extra compute.

The interesting choice is what computes that distribution. In a recurrent encoder, information from token $i$ reaches token $j$ only through $j-i$ sequential updates, so every long-range dependency fights the architecture's limited memory. Self-attention connects any pair of tokens in a single hop, and the Universal Transformer we adopt adds a second idea: one weight-shared block applied repeatedly, a recurrence over refinement steps in depth rather than over positions in the text.

Byte-pair encoding closes the vocabulary problem. Frequent words stay whole while rare ones split into subword units, so the model can emit words it never saw in training and we never need an unknown-word token. We train the tokenizer per corpus, with a 40,000-token vocabulary for English and 50,000 for Russian.

Two corpora, one recipe

We evaluate on two languages. From the New York Times Annotated Corpus we keep articles whose titles run 3 to 15 words and whose bodies run 20 to 2,000 words, skip obituaries, and end up with 1,444,919 articles. Rossiya Segodnya provided us a Russian news archive covering 2010 through 2014, 1,003,869 articles with a mean title of 9.5 words, which we released for research use as the RIA corpus.

Both models share one configuration: four layers in the encoder and decoder, eight attention heads, dropout $p=0.3$ before layer normalization, and Adam with the original Transformer's scaled learning-rate schedule, 4,000 warmup steps and $\beta=(0.9, 0.98)$. We hold out 20,000 random articles per corpus for testing and report means over five random seeds.

The first sentence is a recall trick, not a headline

The first-sentence baseline is a lesson in metric design. It posts the best recall on both corpora, 34.67 ROUGE-1 recall on NYT against our 25.33, because the opening sentence usually mentions everything the headline mentions. But it is roughly ten times longer than a headline, so precision, and with it F1, collapses to 11.64. Reprinting is not summarizing.

On F1, where length discipline counts, we lead every metric on both corpora: 26.86, 13.48 and 24.84 ROUGE-1, 2 and L on NYT, and 39.75, 22.15 and 36.81 on RIA, which had no prior published system. The strongest earlier NYT system reported only recall and leaned on an external summarizer; against its 8.17 ROUGE-2 recall we reach 13.01 while training end to end from raw text.

Readers still prefer the human headline

ROUGE has blind spots: a reasonable headline phrased unlike the reference scores poorly, and we kept meeting generations that read well yet scored low. So we ran a human study. Five annotators compared our headline with the published one on 100 random articles per corpus, choosing the human headline, the machine one, or calling a tie.

The flavor of the remaining gap shows in the outputs. For an article about Southwest defending its Chicago hub, the editor wrote "southwest is adding flights to protect its chicago hub"; we generated "southwest airlines to add 16 flights from chicago". Ours is accurate and fluent, and it misses the why. Closing that last editorial distance, the intent behind the facts, is the part we leave open.

What we take away from 2019: a general-purpose non-local architecture plus subword tokenization, with no task-specific machinery, no pointer networks and no external summarizer, was enough to pass every specialized pipeline of its time. And because every language writes headlines, the benchmark we released keeps paying rent as a testbed far beyond English.