A Lower Loss Gave My GloVe Model Worse Vectors

After word2vec, I built GloVe in NumPy and ran both models on the same text, with the same vocabulary and the same 100 dimensions.

This post has two parts. First comes the theory, kept short. The rest is experiments, because they taught me things I would not have got from reading the paper alone: GloVe is about twice as cheap per pass, it needs far more text than word2vec, and it depends heavily on one threshold. The result I did not expect is that in every experiment the training loss went down, even in the runs where the vectors got worse.

What GloVe is about

word2vec learns from the text one (center, context) pair at a time. GloVe never trains on the text directly.

It first builds a co-occurrence table $X$, where $X_{ij}$ adds up how often word $j$ appears within ten words of word $i$, with a neighbour $d$ positions away counting as $1/d$. Then it fits a vector and a bias for every word so that each non-empty cell is reproduced:

$$ w_i \cdot \tilde w_j + b_i + \tilde b_j \approx \log X_{ij} $$

The loss is the squared error of that fit, with every cell multiplied by a weight so that rare, noisy co-occurrences count for less than frequent ones:

$$ f(x) = \min\left( (x / x_{\max})^{0.75},\ 1 \right) $$

A cell at or above $x_{\max}$ gets the full weight of 1, and everything below it is scaled down. The paper uses $x_{\max} = 100$. My runs use $x_{\max} = 10$, because 100 did not work on my small corpus. The threshold section below shows why.

Setup

Everything runs on text8 with words seen fewer than five times removed, at three sizes: the first 30k, 100k and 2 million words. GloVe uses a window of 10 and AdaGrad at a learning rate of 0.05, as in the paper, and $x_{\max} = 10$. word2vec is the skip-gram model with hierarchical softmax from the last post. The two smaller sizes were run with three seeds each and the figures show the mean.

Quality is measured in two ways. WordSim-353 gives a rank correlation between the model’s similarities and human ratings, and the word2vec analogy questions give an accuracy. Only questions whose words are in the vocabulary are scored, which matters a lot at the small sizes: the 30k run covers 15 word pairs, the 100k run 81, and the 2M run 303.

A pass is cheaper

Two line charts of total seconds against epochs. At 30k words GloVe takes 1.7, 7.8 and 15.5 seconds for 1, 5 and 10 epochs and word2vec takes 3.6, 17.7 and 35.7. At 100k words GloVe takes 6.7 and 31.1 seconds for 1 and 5 epochs and word2vec takes 14.4 and 70.3.
Figure 1. Time to build the training data and train, mean of three seeds.

At both sizes GloVe takes a little under half the time of word2vec for the same number of epochs, and both grow linearly. GloVe pays slightly more before training starts, 0.58 s against 0.13 s at 100k words, because it has to build the table first.

The reason it is cheaper is the table itself. Every repeat of a word pair in the text lands in the same cell, so an epoch visits each distinct pair once, while word2vec visits every occurrence. On the 2M corpus, about 38 million (position, neighbour) pairs collapse into 8.15 million cells. That also means an epoch is not the same amount of learning in the two models, so this figure compares cost and nothing else.

It needs more text

Three line charts of WordSim-353 rank correlation against epochs. At 30k words GloVe falls from 0.01 to minus 0.37 at epoch 5 and minus 0.15 at epoch 10, while word2vec rises to 0.32. At 100k words GloVe falls from 0.17 to minus 0.08 while word2vec rises to 0.27. At 2 million words GloVe rises steadily from 0.02 to 0.25, with word2vec's single pass at 0.41.
Figure 2. Agreement with WordSim-353 as training goes on. Epoch 0 is the random initialisation. At 2M words, word2vec was trained for one pass only.

On 30k and 100k words, training makes GloVe worse on this test. It ends below its own random initialisation, while word2vec improves on exactly the same text. Its loss meanwhile behaves perfectly, falling from 0.169 to 0.089 over five epochs at 100k words. Only at 2 million words does GloVe start to climb.

I think the table explains this as well. Even at 2M words, 79.8% of the non-empty cells hold less than 1 and the median is 0.27. The cell for france and berlin holds 0.111, which is a single co-occurrence nine words apart. With less text nearly every cell is like that, so the model is fitting a table of coincidences very accurately. word2vec sees the same sparse evidence, but it is never asked to reproduce a count.

The small runs cover very few word pairs, so I would not read the exact values, only the direction.

The threshold has to fit the corpus

At 2M words I first used the paper’s $x_{\max} = 100$ and then 10, with the same table, seed and five epochs.

Two line charts against epochs for two GloVe runs at 2 million words. Training loss: with x_max = 100 it falls from 0.045 to 0.020, with x_max = 10 from 0.146 to 0.086. Analogy accuracy: with x_max = 10 it rises to 1.45 percent, with x_max = 100 it reaches 0.22 percent.
Figure 3. Two thresholds on the 2M corpus.

The run with $x_{\max} = 100$ has a loss four times lower and almost no usable vectors. Its nearest neighbours for france are election, biography, population, jackson and king, against russia, spain, england, germany and austria for $x_{\max} = 10$. It answers 20 of 8,995 analogies where the other answers 130.

The two losses are not comparable, though, because the weight is part of the loss. For the 98.6% of cells that hold less than 10, the weight under $x_{\max} = 100$ is exactly $10^{0.75} \approx 5.6$ times smaller, so the same errors produce a much smaller number. When I measured both finished models with the same weights, the order flipped:

Error measured with Trained with $x_{\max} = 100$ Trained with $x_{\max} = 10$
Weights at $x_{\max} = 100$ 0.0197 0.0160
Weights at $x_{\max} = 10$ 0.0997 0.0817
No weights 1.284 1.008

The $x_{\max} = 10$ model fits the table better by every measure, including the other run’s own objective. The smaller weights also shrink the gradients, so with $x_{\max} = 100$ almost the whole table was learning 5.6 times more slowly. After the first epoch, the unweighted error on cells below 10 was 2.29 with $x_{\max} = 100$, slightly worse than the 2.19 it started from, and 1.39 with $x_{\max} = 10$.

A threshold of 100 suits corpora of billions of tokens, where many cells pass it. On my table only 0.09% of cells do. I have not run enough epochs to say that 10 is the better setting in general, only that it was over five passes on this corpus.

Against word2vec at 2M words

Vectors Analogies correct WordSim-353
GloVe, W 1.45% 0.250
GloVe, W + Wc 1.23% 0.243
word2vec 3.36% 0.407

word2vec is ahead on both tests, after one pass against GloVe’s five. Almost all of the analogy signal for either model comes from one category, family relations such as man : woman :: king : queen, where GloVe gets 14.0% and word2vec 21.0%. No other category passes 6%. The paper recommends summing the two vector tables, and here that halved the family score to 7.0%.

GloVe was still improving at epoch 5 and the paper trains for 50, so this is where I stopped, and I can’t say where it would have ended up.

Takeaway

GloVe’s idea shows up clearly in the cost. Counting first makes every pass cheaper, because repeated pairs land in one cell. The price is that the counts have to be worth fitting. On a small corpus most of them are not, and the threshold can only partly make up for that. The loss cannot see the problem at all.

In three experiments the loss fell while the vectors got worse, stayed useless, or could not be compared. Each time I only found out by looking at nearest neighbours and benchmark scores. That is the habit I am taking into the next model.

Looking back, the one thing that has stayed the same from counting models through NPLM and word2vec is co-occurrence: which words appear near which. Those models used it locally, one window of neighbours at a time. GloVe collects the counts for the whole corpus first and then fits all of them together, which is why it is cheaper per pass and also why it needs a lot of text.

References