Word Embeddings — Word2Vec, GloVe, FastText
From Words to Vectors
Neural networks cannot process raw text. We need to convert words into dense numerical vectors that capture semantic meaning. Word embeddings map each word to a point in high-dimensional space where similar words are geometrically close. Famous property: king − man + woman ≈ queen
Word2Vec (Google, 2013)
Word2Vec trains a shallow neural network to predict context from a target word or vice versa. CBOW (Continuous Bag of Words): Predict center word from surrounding context words. Skip-gram: Predict surrounding context words from the center word. Works better for rare words. Training trick — Negative Sampling: Instead of softmax over all vocabulary (slow), train binary classifier to distinguish true context words from randomly sampled 'noise' words.
GloVe (Stanford, 2014)
GloVe (Global Vectors) uses global word co-occurrence statistics. Factorizes the log co-occurrence matrix: w_i · w̃_j + b_i + b̃_j = log(X_ij) Xij = count of word j appearing in context of word i. GloVe often outperforms Word2Vec on word analogy tasks.
FastText (Meta, 2016)
FastText represents each word as a bag of character n-grams. 'apple' = {ap, pp, pl, le, app, ppl, ple, apple}. Key advantage: handles OOV (out-of-vocabulary) words by composing subword embeddings. Great for morphologically rich languages and misspellings.
Using Embeddings with Gensim
For most modern NLP tasks, use contextual embeddings (BERT, GPT) instead of static embeddings. Static embeddings assign the same vector to 'bank' in 'river bank' and 'bank account'.
Finished reading? Mark it complete to earn your XP.