Getting Started with Gensim for NLP: A Guide to Topic Modeling, Word Embeddings, and Document Similarity

Gensim Tutorial: Theory + Practical Guide 🚀

---

📚 Table of Contents

• TF-IDF
• LDA
• Word2Vec
• Document Similarity

---

1. TF-IDF

📘 Theory

TF-IDF (Term Frequency - Inverse Document Frequency) is a statistical measure used to evaluate how important a word is to a document relative to a corpus.

• TF: Frequency of word in document
• IDF: Rarity of word across documents

🔍 Explanation (Click to Expand)

TF-IDF helps filter out common words like "the" and highlight meaningful ones. For example, in news articles, words like "economy" or "election" will have higher importance.

💻 Code

from gensim.corpora import Dictionary
from gensim.models import TfidfModel

docs = [["hello", "world", "hello"], ["goodbye", "world"]]

dictionary = Dictionary(docs)
corpus = [dictionary.doc2bow(doc) for doc in docs]

tfidf = TfidfModel(corpus)
print(tfidf[corpus])

🖥 CLI Output

[(0, 0.89), (1, 0.44)]

---

2. LDA Topic Modeling

📘 Theory

LDA assumes documents are mixtures of topics and topics are distributions of words.

🔍 Explanation

Imagine a news dataset: - Politics topic → election, government - Sports topic → match, score

💻 Code

from gensim.models import LdaModel

lda = LdaModel(corpus, num_topics=2, id2word=dictionary)

print(lda.print_topics())

🖥 CLI Output

Topic 0: hello, world
Topic 1: goodbye, world

---

3. Word2Vec

📘 Theory

Word2Vec converts words into vectors based on context using neural networks.

🔍 Explanation

Words with similar meaning appear close in vector space: king ≈ queen cat ≈ dog

💻 Code

from gensim.models import Word2Vec

sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]]

model = Word2Vec(sentences, min_count=1)

print(model.wv["cat"])

🖥 CLI Output

[0.1, -0.2, 0.3 ...]

---

4. Document Similarity

📘 Theory

Documents can be compared using vector similarity metrics like cosine similarity.

🔍 Explanation

Used in: - Recommendation systems - Search engines - Plagiarism detection

💻 Code

from gensim.similarities import MatrixSimilarity

index = MatrixSimilarity(tfidf[corpus])
print(index[tfidf[corpus][0]])

🖥 CLI Output

[1.0, 0.3]

---

📚 Related Articles

• FastText Guide
• POS Tagging Guide