A Latent Variable Model Approach to PMI-based Word Embeddings

Take the word "cat." Not the animal — the vector. A list of a few hundred numbers, sitting in a giant matrix, that somehow knows "cat" is closer to "dog" than to "democracy." Nobody programmed that relationship in. It emerged from counting which words appear near which other words billions of times across massive text corpora. The question this paper by Arora, Li, Liang, Ma, and Risteski is trying to answer isn't whether that works. It's why. By the early 2010s, the field had a striking mismatch on its hands. Word embeddings — dense, low-dimensional vectors representing words — were everywhere and they worked astonishingly well. Word2vec, GloVe, and older Pointwise Mutual Information, or PMI, based methods all produced vectors that captured meaning in a way that held up across benchmarks. But these methods were built from hand-tuned heuristics: nonlinear operations on co-occurrence counts, reweighting tricks like taking square roots or logarithms, hyperparameters chosen by feel. Nobody had a unified, principled account of why any of it worked. The poster child for what embeddings could do was the analogy task introduced by Mikolov and colleagues. Take the vector for "king," subtract the vector for "man," add the vector for "woman," and the nearest neighbor in the resulting space is "queen." Arora and colleagues call this phenomenon relations equal lines — semantic relations correspond to single directions in vector space. It was a striking empirical fact. And it was almost entirely unexplained. To build an explanation, Arora and colleagues introduce a generative model — a story about how text gets produced, from which the properties of word vectors can be derived. The key idea is a slowly drifting discourse vector. Imagine that at every moment, a conversation has a hidden direction in a high-dimensional space, a vector c sub t that moves gradually. The probability of a word being emitted at any moment is proportional to the exponential of how well that word's latent vector aligns with the current discourse direction. Put simply, a word is more likely when it points the same way as what's being talked about. This is a log-linear production model, the same family as energy-based models, and it connects naturally to the structure of both word2vec and GloVe. The immediate technical problem with log-linear models is the partition function — the normalizer that ensures all word probabilities sum to one. In general, this normalizer depends on the current discourse vector c, which makes everything complicated. Arora and colleagues sidestep this with a geometric insight they call isotropy. The isotropy assumption says that word vectors, in bulk, point in all directions roughly equally. No global direction in the embedding space is privileged. Formally, they model each latent word vector as a scalar magnitude times a uniformly random direction, where the scalar has expectation tau of order one and an absolute upper bound kappa. Under this prior, they prove a concentration result: for most discourse vectors, the partition function stays very close to a constant Z. The deviation is of order one over the square root of the vocabulary size n, and the probability of exceptional cases is exponentially tiny. That means you can treat the normalizer as essentially fixed, regardless of context. This is the self-normalization property that practitioners had been exploiting without a rigorous justification. Isotropy isn't just assumed — it's verified. Arora and colleagues report experimental support showing that real trained embeddings actually satisfy the condition. And intuitively, it makes sense: language has no single privileged global direction. There are many independent semantic axes — animacy, tense, geography, sentiment — and in a space of a few hundred dimensions, the bulk of word vectors really do spread out roughly uniformly. That geometric fact is what makes the whole theory tractable. With self-normalization in hand, Arora and colleagues can integrate out the drifting discourse vectors and derive closed-form expressions for observable co-occurrence statistics. The key result, Theorem two point two, says the dominant term controlling the log probability of two words appearing near each other is one over two d — where d is the embedding dimension — times the squared length of the sum of the two word vectors. That's a quadratic function of the latent vectors. And when you expand it out, the PMI between two words emerges as approximately the inner product of their vectors, up to a constant shift. PMI isn't a heuristic — it falls out of the model. From there, turning the likelihood into a training objective produces something recognizable. The weighted least squares loss that emerges from the model matches GloVe's loss function, including GloVe's specific weighting function — which truncates co-occurrence counts at 100 and applies a three-quarter exponent — and the need for bias terms. The model gives these choices a concrete interpretation: the bias for a word is its squared vector length, divided by two d. The earlier result by Levy and Goldberg, showing that skip-gram with negative sampling implicitly factorizes a shifted PMI matrix, also fits naturally here: the constant shift appears as a consequence of the model and the low-dimensional constraint, not as an ad hoc tweak. Even CBOW's averaging of context vectors emerges as the model's maximum a posteriori estimate of the discourse vector. These methods weren't independently discovered heuristics that happened to agree. Under the generative assumptions, they are approximately optimal, and they were always pointing at the same underlying structure. The longest-standing mystery — the one at the center of the paper — is relations equal lines. Why does subtracting "man" from "king" and adding "woman" land you at "queen"? Arora and colleagues show this follows from two things: the generative model, and a denoising effect of low-dimensional projection. The model predicts that for any semantic relation R — male to female, country to capital, present tense to past tense — there exists a fixed direction in embedding space, a relation vector mu sub R, such that the difference between any pair of word vectors satisfying that relation equals mu sub R plus noise. The noise comes from the fact that individual word vectors reflect all the contexts a word appears in, not just its role in a particular relation. But when you project to a lower-dimensional space, something useful happens: the noise gets suppressed while the signal — the consistent direction mu sub R — survives. Theorem four point one formalizes this, showing the relative error in the dimension-reduced space is smaller by a factor of the square root of d over n, where d is the dimension and n is the vocabulary size. The empirical support is specific. Arora and colleagues find that the average projection of word-pair differences onto the top singular vector for a relation — the best estimate of mu sub R — is zero point five one overall, with zero point five eight for semantic relations and zero point four six for syntactic ones. The projection onto the second singular vector averages just zero point zero three five. One direction dominates, which is exactly what the theory predicts. They then introduce the RD algorithm, which estimates mu sub R via rank-one singular value decomposition across many examples and uses it to solve analogies. On the GOOGLE analogy testbed, using the RD algorithm with thirty examples pushes accuracy from zero point seventy one to zero point seventy nine — roughly a ten percent improvement over the standard nearest neighbor approach. The singular values of the full word vector matrix are also telling: the quadratic mean of singular values is thirty four point three while the minimum nonzero singular value is eleven, giving an empirical constant of about one-third, consistent with the near-isotropic behavior the theory requires. What the paper delivers, in the end, is a unified generative framework that retroactively makes sense of a decade of empirical practice. PMI, word2vec, and GloVe aren't three different methods that happen to work similarly — they are approximately the same method, all aimed at the same underlying statistical structure, all derivable from a single generative story about how words are produced by a slowly drifting discourse. The hyperparameters practitioners spent years tuning by hand — the truncation constants, the exponents, the bias terms — have principled interpretations under the model. The authors point toward future work: building similar generative models that incorporate linguistic structure, and making those models provable and interpretable from the ground up rather than post-hoc. That's the longer arc. For now, the paper's contribution is a different kind of progress — not making word embeddings work better overnight, but explaining why they work at all. And that kind of understanding is exactly what eventually leads to building better systems on purpose, rather than by feel. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.