The Augmented Social Scientist. Using Sequential Transfer Learning to Annotate Millions of Texts with Human-Level Accuracy

Here’s the big idea in one sentence: the paper shows that a single expert can fine‑tune a pre‑trained language model with a very small, carefully labeled sample and then let that model annotate an entire, decades‑spanning news corpus with near human accuracy. In other words, sequential transfer learning turns a few hours of expert labeling into a scalable annotator that performs on par with trained research assistants. Why does that matter for scalability? Traditional human annotation doesn’t scale: it’s slow, costly, and quality declines with fatigue or when work is outsourced to minimally trained annotators. Here, the authors fine‑tune CamemBERT, a French BERT model, on tiny fractions of the data—about 0.12% of the corpus for a sentence‑level policy‑versus‑politics classifier and 1% for a token‑level “off‑the‑record” detector (Table 1 on page 11). After roughly 8 hours of expert labeling for the first task and about 37 hours for the second, the resulting models achieve F1 scores of 0.78 and 0.82 respectively, essentially matching trained research assistants at 0.80 and 0.86 (Table 2 on page 20), and vastly outperforming models without pre‑training. Once trained, the model can label the entire corpus—over 60,000 articles and 39 million words from Le Monde (pages 7–8)—in minutes to hours, a job no feasible team of annotators could finish with consistent quality. Two visuals in the paper make the scalability payoff vivid. First, the sample‑efficiency curves in Figure 3 (page 23) show performance plateaus fast: for policy vs. politics, the model nears peak quality with only one‑third to one‑half of the already small training set, implying the expert could have annotated less and still reached the same accuracy. For the off‑the‑record task, performance also plateaus—about 50% of the annotated sample—despite the label being rare (2.75% in the test set), underscoring how pre‑training lets the model learn from sparse signals. Second, Figure 4 (page 25) demonstrates what scalable annotation buys substantively: year‑by‑year estimates show the long‑term rise of “horse‑race” coverage and clear spikes in “politics” during election years, aligning with established literature—evidence that the automated labels are coherent at scale. A few clarifying notes. Sequential transfer learning means the model arrives “pre‑loaded” with general language knowledge from massive corpora and needs only light, task‑specific tuning from the expert. Quality is reported with F1, a balance of precision and recall, which is crucial when labels are rare. And the study also surfaces a practical lesson about scaling the annotation phase itself: for tedious, simple tasks with many negatives, distributing the labeling load can mitigate expert fatigue without harming final model quality (see Figure 3 and the discussion on pages 22–23). The key takeaway: by pairing small, high‑quality expert labels with a pre‑trained language model, the approach collapses the marginal cost of annotation to near zero while preserving human‑level accuracy. This lets social scientists design nuanced indicators, validate them against a gold standard, and then apply them across entire corpora—achieving the depth of human coding and the breadth of computational scale.