Improving Distributional Similarity with Lessons Learned from Word Embeddings

Everyone in natural language processing assumed word2vec was winning because of something deep in the math — a fundamentally better way of representing language, a new kind of algorithm that had simply outgrown the old count-based approaches. That assumption shaped how papers were written, how baselines were chosen, and how an entire field understood its own progress. Levy, Goldberg, and Dagan asked a simpler question: what if the algorithm was almost beside the point? To understand what they found, you need the landscape they were working in. On one side of the field sat the older count-based distributional models. These work by building an explicit word-by-context co-occurrence matrix. You count how often each word appears near each other word across a large corpus, apply pointwise mutual information weighting to amplify meaningful co-occurrences, and optionally compress the result using singular value decomposition, or SVD. On the other side sat the newer neural network-inspired embeddings: word2vec's skip-gram with negative sampling, known as SGNS, and GloVe. These produce continuous, dense vector representations learned through a prediction-style training objective. Across the community, the received wisdom was that the dense prediction-based methods had won. Baroni and colleagues ran systematic comparisons that appeared to confirm it. The neural models outperformed the count-based ones on word similarity and analogy detection tasks. Case closed, or so it seemed. There was one analytical result that should have given everyone pause. Levy and Goldberg had shown that SGNS can be interpreted as implicitly factorizing a word-context pointwise mutual information matrix — the very kind of structure that count-based methods build explicitly. In other words, the "neural" model and the "count" model were drawing on the same underlying information. That raised an obvious question: if the information source was shared, why were the results so different? The answer, it turned out, was sitting inside a handful of settings nobody was holding constant. Levy, Goldberg, and Dagan identified five hyperparameters that shipped with word2vec and GloVe as implementation defaults — not core algorithmic choices but just knobs on the machine — and showed that those knobs were doing enormous work. The first is dynamic context windows. Traditional count methods treat every token inside a fixed window identically. Word2vec and GloVe weight nearer contexts more heavily: a context three tokens away from the target word counts as only one-third of an occurrence. Levy and colleagues call this modification "dyn" and note it both emphasizes close contexts and reduces the number of co-occurrence updates during training. The second is subsampling of frequent words. Word2vec randomly drops very common tokens before forming context pairs, using a threshold of ten to the negative fifth power. This compresses the influence of words like "the" and effectively expands the reach of surviving tokens into slightly wider neighborhoods. The third is shifted pointwise mutual information. Through the SGNS analysis, Levy and Goldberg had shown that the model implicitly targets a pointwise mutual information score shifted downward by the logarithm of k, where k is the number of negative samples. Said plainly, it is pointwise mutual information between a word and a context, minus the log of k. Levy and colleagues expose this as a tunable option for count-based methods, testing k equal to one, five, and fifteen. The fourth knob is context distribution smoothing. Word2vec samples its negative examples from a unigram distribution where each context count is raised to the power of zero point seventy-five. Levy and colleagues port this into pointwise mutual information by replacing raw context probabilities with their smoothed equivalents. They report this modification alone increases performance by over three points per task on average. The fifth is what they call w plus c — using the sum of a word's word vector and its context vector as the final representation. The geometric effect is that the cosine similarity of these summed vectors captures both second-order distributional similarity and first-order word-to-context affinity. Here is the critical observation: these modifications were being applied to SGNS and GloVe by default, but not to the count-based baselines being used for comparison. When Baroni and colleagues found that prediction-based methods outperformed traditional ones, they were comparing word2vec with its full complement of tuning against a vanilla pointwise mutual information or SVD setup. The race was not fair. Nobody had announced it was unfair. Nobody had tried to make it fair. The hyperparameter choices and the algorithmic choices were completely entangled. Levy, Goldberg, and Dagan untangled them. They trained all four methods — pointwise mutual information, SVD, SGNS, and GloVe — on the same English Wikipedia corpus: one point five billion tokens, with vocabulary trimmed to one hundred eighty-nine thousand five hundred thirty-three terms by ignoring words appearing fewer than one hundred times. They evaluated on six word similarity datasets and two analogy sets, measuring cosine similarity ranked against human ratings via Spearman's rho, and analogy accuracy using the three cosine add and three cosine multiply formulas. Crucially, they applied the same hyperparameter toolkit across methods wherever applicable, generating six hundred seventy-two total trained representations across a systematic grid. They structured results in three ways. A vanilla scenario set all hyperparameters to neutral defaults. A word2vec-style scenario applied the recommended defaults from Mikolov and colleagues. And an oracle scenario found the best hyperparameter combination for each method and task independently, providing an upper bound on what tuning could achieve. The results are the story. The best hyperparameter combination improved performance by up to fifteen point seven percentage points over the vanilla defaults, and by more than six points on average. That is a gap large enough to separate methods that look completely different in the literature. And when hyperparameters were equalized and tuned across methods, the large consistent performance gaps reported in prior work largely disappeared. There was no single algorithm that dominated across tasks and window sizes. On word similarity tasks, SVD — a count-based method — attained higher average scores than SGNS at window sizes of two and five. SGNS never beat SVD by more than one point seven points in those conditions. On analogy tasks, the picture was more nuanced: SGNS and GloVe beat pointwise mutual information on Google analogies by only about three point seven points. The one genuine exception was MSR's syntactic analogies, where SGNS and GloVe substantially outperformed the count-based methods regardless of tuning. That is a real algorithmic difference worth noting. But it is narrow and task-specific, not the broad global superiority the field had been claiming. Levy and colleagues also tested whether more data could substitute for better tuning. Using a corpus of ten point five billion words — roughly seven times larger — they found that in three out of six word similarity tasks, exploring more hyperparameter combinations outperformed simply expanding the data. That is a striking result. Adding data is expensive. Hyperparameter search is cheap. The practical recommendations that follow are specific. Always apply context distribution smoothing at zero point seventy-five. Avoid the traditional SVD eigenvalue weighting of one point zero — it can degrade performance by more than fifteen points in some cases — and use symmetric SVD variants instead. Treat SGNS as a robust default: it is fast to train, light on memory, and does not significantly underperform in any scenario. With SGNS, prefer more negative samples. Try the w plus c representation, which can yield substantial gains, but be aware it can also hurt — check it before committing. The methodological lesson is harder to summarize in a rule of thumb but is more important. This paper is a case study in how benchmark comparisons mislead when confounds go uncontrolled. Levy, Goldberg, and Dagan release their code alongside the paper for exactly this reason: reproducibility is the fix. When one group's SGNS is being compared to another group's pointwise mutual information, and nobody specifies whether subsampling was on, whether context smoothing was applied, or whether shifted pointwise mutual information was used — the comparison measures something other than what it claims to measure. The authors are not accusatory about this. They note that inconsistencies appear in work by Baroni and colleagues and by Pennington and colleagues without framing it as misconduct. The field was moving fast, defaults felt natural, and nobody had done the work of separating the knobs from the core math. Levy, Goldberg, and Dagan did that work. So when you read a paper that claims method A beats method B, the first question is not whether the numbers are right. It is whether the comparison was fair — whether the same preprocessing ran on both sides, whether the hyperparameter choices were symmetric, whether the defaults one method enjoys were also given to the other. The algorithm is rarely the secret. The setup usually is. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.