Machine-assisted quantitizing designsaugmenting humanities and social sciences with artificial intelligence

OverviewTechnicalshimmer voice

Think of MAQD as a way to make large language models do serious inference work without handing them the keys to the lab. That's the pitch from Karjus: treat the model as a narrow assistant, wire its outputs to your theory through a codebook, and then do statistics that explicitly account for the assistant's mistakes. Human interpretation stays in the driver's seat. The value is scale and speed, not surrendering judgment. The pipeline is disciplined by design. You start with unitization: cut the corpus into comparable and meaningful units, such as sentences, story synopses, chapter-level scenes, or sentence pairs for translation. The units should be big enough to carry the signal you care about but small enough to be comparable. Next, translate your research question into a codebook with a fixed set of variables—categorical or numeric—that you want per unit. Now comes the operational bridge: prompts that turn theory into annotation. Use zero-shot when you can, with a single carefully framed instruction, and iterative refinement when you must. Calibrate everything against an expert-labeled test set, so when the model drifts, you see it and can correct course. Why the fuss about calibration? Because in MAQD, the model is an annotator with a measurable error structure. As Karjus showed, you estimate that structure on a held-out test set, bake it into a confusion matrix, and then propagate it forward. That way, your regression line or prevalence estimate knows how often the machine confuses class A for B and widens the interval accordingly. You get speed from the model, and you keep validity by carrying its uncertainty through to the end. Let's get concrete with error propagation, because this is the statistical core. You build a confusion matrix, where the rows are true classes from the expert labels and the columns are the model's predictions. Normalize each row. This gives you, for each true class i, a probability distribution di over what the model tends to predict when i is the truth. Bootstrapping then mimics annotation under that error: for each data point whose true class is i, you redraw a label from di—weighted random sampling over the class set. Do that across the dataset to create many synthetic replicates. Recompute your statistic on each replicate, such as counts, prevalence, logistic coefficients, and summarize the spread. A 95 percent interval is the estimate plus or minus 1.96 times the bootstrap standard error. If your annotator is messy, your intervals widen; if it's sharp, they tighten. Simple, and importantly, honest. Hold that in mind as we walk through cases. The flagship example is a Russian newsreel corpus: 12,707 story synopses from 1,745 reels, with an expert-coded nine-level topic variable and a year covariate. A small test set—100 stories—anchored the confusion matrix. In that calibration, one-shot prompting yielded an accuracy of 0.88 for GPT-3.5 with a Cohen's kappa of 0.85, and an accuracy of 0.84 for GPT-4 with kappa around 0.80. Applying the pipeline to the full corpus, a logistic regression of the Social topic against year from 1974 to 1989 returned a year coefficient of 0.064 with a p-value below 0.0001. Translate that: the odds of a Social topic story grew by a factor of about 1.07 per year. After bootstrapping through the confusion matrix, the 95 percent interval around the log-odds coefficient was about plus or minus 0.02, and the p-value remained effectively unchanged within a very tight window. The signal survived the machine's wobble. A second case—Estonian maritime wrecks—shows the same logic on a smaller label space. Here, the initial codebook had 54 fine-grained causes that were collapsed into four primary categories for inference. GPT-4 reached 88 percent accuracy with a kappa of 0.83 on the test set, and the model's recall for the mine category was reported as 100 percent. With that confusion profile in hand, you can bootstrap cause distributions and any downstream counts with intervals that reflect misclassification risk, rather than pretending the annotations are gold. Design choices matter upstream of all that. Unitization sets the stage. In the newsreels, the unit is a terse synopsis; in Eighteenth-century British ads, it's phrase-sized modifiers like "fine jewelry," carrying a subjectivity label; in translation analysis, it's sentence pairs aligned with an LLM-assisted tool. The codebook aligns with the question. Nine topics for propaganda drift. Two levels for subjectivity. An open-ended list of character interactions that later becomes a weighted, undirected network. Four broad causes for shipwrecks. The MAQD stance is permissive about multiple units and variables, as long as comparability is preserved and the variables tie back to hypotheses you'll actually model. Prompts sit between theory and annotation, and they're not an afterthought. In one replication, a single-example prompt answered per unit outperformed bulkier multi-example batches: GPT-3.5 hit an accuracy of 0.88 with a kappa of 0.85, while GPT-4 reached an accuracy of 0.84 and about 0.80, respectively. In multilingual settings, prompt language can trump the document language; English prompts sometimes edged out native-language prompts for Russian data. And when the domain brings noise—think optical character recognition—the pipeline accommodates a cleaning pass. In a replication of Kanger and colleagues, cleaning distorted passages before classification nudged GPT-3.5 from an accuracy of 0.79 to 0.82 and GPT-4 from 0.90 to 0.92, with kappas improving from 0.49 to 0.56 and from 0.77 to 0.82. Small moves numerically, but big moves for inferential confidence. The framework stretches across languages and genres without changing its spine. Karjus's cases span Estonian, Finnish, German, Italian, Japanese, Latin, Russian, Turkish, and English. In the modifier-subjectivity study on Eighteenth-century English ads, GPT-4's agreement with experts was high—0.94 accuracy and 0.89 kappa—despite historical spelling and optical character recognition artifacts. In a genre-detection replication of Sobchuk and Šeļa, a doc2vec plus cosine similarity baseline reached an adjusted Rand index around 0.70, while a zero-shot large language model classifier matched the spirit if not the machinery, with an accuracy of 0.80 and a kappa of 0.73, all without heavy preprocessing. Different road, same destination. There's translation work too, where the unit is a sentence pair. Alignments in a small English to Italian sample—47 pairs from Orwell's 1984—were assisted by a large language model and checked by native Italian experts. The point wasn't scale; it was that the same prompts-and-codebook logic can support fine-grained, expert-reviewed alignment. In a cross-script simulation with 100 English to Japanese pairs and four synthetic difference classes, GPT-4 landed at 96 percent accuracy with a kappa of 0.95. Even the aggregate count of differences was nearly exact: a predicted sum of 74 out of 100, off by just one. When the problem demands a crisp, symbolic output, the model is capable—and the error model tells you how to trust it. On the social side, MAQD handles open coding that later gets quantified. Take literary networks. You can ask the model to list interacting character pairs per chapter and then convert those lists into a weighted undirected graph. In that setup, GPT-3.5 produced a baseline network, and GPT-4 supplied gender estimates for nodes, with richer and more consistent attributes than the manual baseline. It wasn't flawless—entity resolution anomalies popped up—but, as in the other tasks, expert calibration and updates to the prompt or model version tightened things. Interview and stance detection show the same pattern at a different scale. In a 259-sentence newspaper interview test set, GPT-3.5 came in at 76 percent accuracy, while GPT-4 climbed to 95 percent with a kappa around 0.90 and a mean F1 near 0.90. That's the kind of lift that makes machine-assisted coding viable for qualitative teams facing thousands of sentences. And again, the MAQD instruction is to carry those error rates forward: compute prevalence of stances with intervals widened by the confusion matrix; let the data, and the model's known quirks, speak together. Two pragmatic notes round this out. First, time. In the newsreel case alone, twelve thousand seven hundred and seven synopses total about 281,000 words. At a modest 184 words per minute, that's more than 19 hours of reading before any annotation—never mind reliability checks and modeling. With a large language model, you can produce calibrated annotations and bootstrap intervals in hours. For a larger translation and adaptation corpus of roughly 558,000 words, manual reading and note-taking slides into weeks; a prompt-based pass collapses that to hours or even minutes, depending on the task and your tolerance for interactivity. Second, transparency. Karjus puts the data and code in the open—GitHub under andreskarjus slash Machine Assisted—so the practice can spread, not just the headline results. What does this add up to? A general recipe that's language-agnostic and task-agnostic. Zero-shot prompts can rival tailored pipelines in many settings; the modifier study at 0.94 accuracy and the genre replication at 0.73 kappa show that plainly. But the real contribution is methodological: you don't stop at accuracy. You estimate the annotator's confusion on a ground-truth test set, you sample from that matrix to simulate annotation error, and you propagate the uncertainty through aggregation and regression. In the newsreels, that meant a year-on-topic coefficient of 0.064 with a 95 percent interval of roughly plus or minus 0.02 after accounting for machine noise. In the wrecks data, it meant trustworthy cause distributions even when the original label space was messy. It's errors-in-variables thinking, brought to qualitative coding. There are guardrails. Biases and domain shifts don't vanish just because you measured a confusion matrix once; prompts age, and models change. The fix is procedural: expert audits, preregistration-friendly designs, open code, and test sets that travel with the project, so future replications can re-estimate error. Karjus is explicit on that point. Keep the artifacts open. Keep the interpretation human. If you're listening for the frontier, it's not hard to spot. Multimodal data—text plus image or text plus audio—slot naturally into the same pipeline once you define units and a codebook that spans modes. Cross-lingual work is already in play; expanding into low-resource scripts will lean even more on calibration and prompt language choice. But the center of gravity doesn't move. Unitize with care, code to theory, calibrate with experts, and let the confusion matrix set your confidence. That's MAQD in practice: speed from the machine, rigor from the method, and inference you can stand behind.

The pipeline is disciplined by design. You start with unitization: cut the corpus into comparable and meaningful units, such as sentences, story synopses, chapter-level scenes, or sentence pairs for translation. The units should be big enough to carry the signal you care about but small enough to be comparable.

Next, translate your research question into a codebook with a fixed set of variables—categorical or numeric—that you want per unit. Now comes the operational bridge: prompts that turn theory into annotation. Use zero-shot when you can, with a single carefully framed instruction, and iterative refinement when you must.

Calibrate everything against an expert-labeled test set, so when the model drifts, you see it and can correct course.

Why the fuss about calibration? Because in MAQD, the model is an annotator with a measurable error structure. As Karjus showed, you estimate that structure on a held-out test set, bake it into a confusion matrix, and then propagate it forward.

That way, your regression line or prevalence estimate knows how often the machine confuses class A for B and widens the interval accordingly. You get speed from the model, and you keep validity by carrying its uncertainty through to the end.

Let's get concrete with error propagation, because this is the statistical core. You build a confusion matrix, where the rows are true classes from the expert labels and the columns are the model's predictions. Normalize each row.

This gives you, for each true class i, a probability distribution di over what the model tends to predict when i is the truth. Bootstrapping then mimics annotation under that error: for each data point whose true class is i, you redraw a label from di—weighted random sampling over the class set. Do that across the dataset to create many synthetic replicates.

Recompute your statistic on each replicate, such as counts, prevalence, logistic coefficients, and summarize the spread. A 95 percent interval is the estimate plus or minus 1.96 times the bootstrap standard error. If your annotator is messy, your intervals widen; if it's sharp, they tighten. Simple, and importantly, honest.

Hold that in mind as we walk through cases. The flagship example is a Russian newsreel corpus: 12,707 story synopses from 1,745 reels, with an expert-coded nine-level topic variable and a year covariate. A small test set—100 stories—anchored the confusion matrix.

In that calibration, one-shot prompting yielded an accuracy of 0.88 for GPT-3.5 with a Cohen's kappa of 0.85, and an accuracy of 0.84 for GPT-4 with kappa around 0.80. Applying the pipeline to the full corpus, a logistic regression of the Social topic against year from 1974 to 1989 returned a year coefficient of 0.064 with a p-value below 0.0001. Translate that: the odds of a Social topic story grew by a factor of about 1.07 per year.

After bootstrapping through the confusion matrix, the 95 percent interval around the log-odds coefficient was about plus or minus 0.02, and the p-value remained effectively unchanged within a very tight window. The signal survived the machine's wobble.

A second case—Estonian maritime wrecks—shows the same logic on a smaller label space. Here, the initial codebook had 54 fine-grained causes that were collapsed into four primary categories for inference. GPT-4 reached 88 percent accuracy with a kappa of 0.83 on the test set, and the model's recall for the mine category was reported as 100 percent.

With that confusion profile in hand, you can bootstrap cause distributions and any downstream counts with intervals that reflect misclassification risk, rather than pretending the annotations are gold.

Design choices matter upstream of all that. Unitization sets the stage. In the newsreels, the unit is a terse synopsis; in Eighteenth-century British ads, it's phrase-sized modifiers like "fine jewelry," carrying a subjectivity label; in translation analysis, it's sentence pairs aligned with an LLM-assisted tool.

The codebook aligns with the question. Nine topics for propaganda drift. Two levels for subjectivity.

An open-ended list of character interactions that later becomes a weighted, undirected network. Four broad causes for shipwrecks. The MAQD stance is permissive about multiple units and variables, as long as comparability is preserved and the variables tie back to hypotheses you'll actually model.

Prompts sit between theory and annotation, and they're not an afterthought. In one replication, a single-example prompt answered per unit outperformed bulkier multi-example batches: GPT-3.5 hit an accuracy of 0.88 with a kappa of 0.85, while GPT-4 reached an accuracy of 0.84 and about 0.80, respectively. In multilingual settings, prompt language can trump the document language;

English prompts sometimes edged out native-language prompts for Russian data. And when the domain brings noise—think optical character recognition—the pipeline accommodates a cleaning pass. In a replication of Kanger and colleagues, cleaning distorted passages before classification nudged GPT-3.5 from an accuracy of 0.79 to 0.82 and GPT-4 from 0.90 to 0.92, with kappas improving from 0.49 to 0.56 and from 0.77 to 0.82. Small moves numerically, but big moves for inferential confidence.

The framework stretches across languages and genres without changing its spine. Karjus's cases span Estonian, Finnish, German, Italian, Japanese, Latin, Russian, Turkish, and English. In the modifier-subjectivity study on Eighteenth-century English ads, GPT-4's agreement with experts was high—0.94 accuracy and 0.89 kappa—despite historical spelling and optical character recognition artifacts.

In a genre-detection replication of Sobchuk and Šeļa, a doc2vec plus cosine similarity baseline reached an adjusted Rand index around 0.70, while a zero-shot large language model classifier matched the spirit if not the machinery, with an accuracy of 0.80 and a kappa of 0.73, all without heavy preprocessing. Different road, same destination.

There's translation work too, where the unit is a sentence pair. Alignments in a small English to Italian sample—47 pairs from Orwell's 1984—were assisted by a large language model and checked by native Italian experts. The point wasn't scale; it was that the same prompts-and-codebook logic can support fine-grained, expert-reviewed alignment.

In a cross-script simulation with 100 English to Japanese pairs and four synthetic difference classes, GPT-4 landed at 96 percent accuracy with a kappa of 0.95. Even the aggregate count of differences was nearly exact: a predicted sum of 74 out of 100, off by just one. When the problem demands a crisp, symbolic output, the model is capable—and the error model tells you how to trust it.

On the social side, MAQD handles open coding that later gets quantified. Take literary networks. You can ask the model to list interacting character pairs per chapter and then convert those lists into a weighted undirected graph.

In that setup, GPT-3.5 produced a baseline network, and GPT-4 supplied gender estimates for nodes, with richer and more consistent attributes than the manual baseline. It wasn't flawless—entity resolution anomalies popped up—but, as in the other tasks, expert calibration and updates to the prompt or model version tightened things.

Interview and stance detection show the same pattern at a different scale. In a 259-sentence newspaper interview test set, GPT-3.5 came in at 76 percent accuracy, while GPT-4 climbed to 95 percent with a kappa around 0.90 and a mean F1 near 0.90. That's the kind of lift that makes machine-assisted coding viable for qualitative teams facing thousands of sentences.

And again, the MAQD instruction is to carry those error rates forward: compute prevalence of stances with intervals widened by the confusion matrix; let the data, and the model's known quirks, speak together.

Two pragmatic notes round this out. First, time. In the newsreel case alone, twelve thousand seven hundred and seven synopses total about 281,000 words.

At a modest 184 words per minute, that's more than 19 hours of reading before any annotation—never mind reliability checks and modeling. With a large language model, you can produce calibrated annotations and bootstrap intervals in hours. For a larger translation and adaptation corpus of roughly 558,000 words, manual reading and note-taking slides into weeks; a prompt-based pass collapses that to hours or even minutes, depending on the task and your tolerance for interactivity.

Second, transparency. Karjus puts the data and code in the open—GitHub under andreskarjus slash Machine Assisted—so the practice can spread, not just the headline results.

What does this add up to? A general recipe that's language-agnostic and task-agnostic. Zero-shot prompts can rival tailored pipelines in many settings; the modifier study at 0.94 accuracy and the genre replication at 0.73 kappa show that plainly.

But the real contribution is methodological: you don't stop at accuracy. You estimate the annotator's confusion on a ground-truth test set, you sample from that matrix to simulate annotation error, and you propagate the uncertainty through aggregation and regression. In the newsreels, that meant a year-on-topic coefficient of 0.064 with a 95 percent interval of roughly plus or minus 0.02 after accounting for machine noise.

In the wrecks data, it meant trustworthy cause distributions even when the original label space was messy. It's errors-in-variables thinking, brought to qualitative coding.

There are guardrails. Biases and domain shifts don't vanish just because you measured a confusion matrix once; prompts age, and models change. The fix is procedural: expert audits, preregistration-friendly designs, open code, and test sets that travel with the project, so future replications can re-estimate error.

Karjus is explicit on that point. Keep the artifacts open. Keep the interpretation human.

If you're listening for the frontier, it's not hard to spot. Multimodal data—text plus image or text plus audio—slot naturally into the same pipeline once you define units and a codebook that spans modes. Cross-lingual work is already in play; expanding into low-resource scripts will lean even more on calibration and prompt language choice.

But the center of gravity doesn't move. Unitize with care, code to theory, calibrate with experts, and let the confusion matrix set your confidence. That's MAQD in practice: speed from the machine, rigor from the method, and inference you can stand behind.