Text_Representation_Tutorial
ipynb11 kBPython · 15 cellsExtra material of chapter 4
Tutorial: Feature Engineering for NLP Tasks
Using Scikit-Learn and Gensim
Feature engineering is a critical step in Natural Language Processing (NLP). It involves transforming raw text data into structured numerical features that machine learning algorithms can understand.
This tutorial guides you through two fundamental approaches to NLP feature engineering:
- Frequency-Based Methods (using
scikit-learn): Bag-of-Words, Stopword filtering, N-grams, and TF-IDF. - Prediction-Based Methods / Dense Word Embeddings (using
gensim): Word2Vec (Pre-trained and Custom) and FastText.
Part 1: Frequency-Based Vectorization (Scikit-Learn)
1. Count Vectorization (Bag-of-Words)
The Bag-of-Words (BoW) model represents text by counting the occurrences of each word within a document. It completely disregards grammar, word order, and context, focusing entirely on raw term frequency.
We use scikit-learn's CountVectorizer to construct a vocabulary of all unique tokens across our corpus and map each sentence to its corresponding frequency vector.
from sklearn.feature_extraction.text import CountVectorizer
sents = [
'The dog barked in the park on another dog',
'The owner of the dog put him on the leash since he barked.',
'My dog is barking and chasing its tail.'
]
cv = CountVectorizer()
X = cv.fit_transform(sents)
X_array = X.toarray()
print("Vocabulary Features:")
print(sorted(cv.vocabulary_.keys()))
print("\nDocument-Term Matrix Grid Layout:")
print(X_array)2. Count Vectorization with Stopword Filtering
Stopwords are highly frequent words (such as "the", "is", "in", "on") that provide grammatical structure but carry minimal semantic value for downstream tasks like classification or clustering. Filtering them out drastically reduces dimensionality and cleans up noisy features. We can pass stop_words='english' into the vectorizer to apply an internal English stopword list.
from sklearn.feature_extraction.text import CountVectorizer
sents = [
'The dog barked in the park on another dog',
'The owner of the dog put him on the leash since he barked.',
'My dog is barking and chasing its tail.'
]
cv_stopwords = CountVectorizer(stop_words='english')
X_stopwords = cv_stopwords.fit_transform(sents).toarray()
print("Filtered Vocabulary (Stopwords Removed):")
print(sorted(cv_stopwords.vocabulary_.keys()))
print("\nFiltered Document-Term Matrix:")
print(X_stopwords)3. N-Gram Count Vectorization
To counter the complete loss of local context inherent in a basic unigram model (1-gram), we introduce N-grams. N-grams group consecutive sequences of tokens together.
ngram_range=(1,1)extracts single words (unigrams).ngram_range=(1,2)extracts both single words and word pairs (bigrams), preserving expressions like "dog barked" or "chasing tail".
from sklearn.feature_extraction.text import CountVectorizer
sents = [
'The dog barked in the park on another dog',
'The owner of the dog put him on the leash since he barked.',
'My dog is barking and chasing its tail.'
]
cv_ngram = CountVectorizer(ngram_range=(1, 2), stop_words='english')
X_ngram = cv_ngram.fit_transform(sents).toarray()
print("Expanded N-Gram Vocabulary Space:")
print(sorted(cv_ngram.vocabulary_.keys()))
print("\nN-Gram Matrix Shape:", X_ngram.shape)
print(X_ngram)4. TF-IDF Vectorization
While count arrays track presence, they can over-emphasize words that are naturally common across an entire subject field. TF-IDF (Term Frequency-Inverse Document Frequency) fixes this by applying a balanced scaling calculation:
Where TF evaluates frequency inside a singular document, and IDF suppresses tokens that appear redundantly across the broader corpus. This accurately brings out structural signature terms.
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
sents = [
'The dog barked in the park on another dog',
'The owner of the dog put him on the leash since he barked.',
'My dog is barking and chasing its tail.'
]
tfidf_vec = TfidfVectorizer(stop_words='english')
X_tfidf = tfidf_vec.fit_transform(sents)
print("Raw Output Feature Weight Metrics Grid:")
print(X_tfidf.toarray())
feature_names = tfidf_vec.get_feature_names_out()
df_tfidf = pd.DataFrame(X_tfidf[0].T.todense(), index=feature_names, columns=["TF-IDF"])
df_tfidf = df_tfidf.sort_values('TF-IDF', ascending=False)
print("\nRanked Terms within Document 1:")
print(df_tfidf)Part 2: Prediction-Based Continuous Dense Vectors (Gensim)
Unlike sparse frequency configurations, vector embeddings capture semantic similarities using low-dimensional, continuous spatial real numbers.
1. Utilizing Pre-Trained Word2Vec Models
Gensim allows you to import heavy pre-trained models (such as the standard 300-dimension Google News corpus configuration) to map high-fidelity universal associations instantly without training overhead.
from gensim import models
# Code template structure showcasing how to extract relationships using massive binaries:
"""
w2v_pretrained = models.KeyedVectors.load_word2vec_format(
'./GoogleNews-vectors-negative300.bin',
binary=True
)
healthy_vector = w2v_pretrained['healthy']
print(w2v_pretrained.most_similar('happy'))
"""
print("[Info] Pre-trained blueprint successfully structured for external usage.")2. Training a Custom Word2Vec Model
When work covers hyper-specific vocabulary or unique domains, building specialized contexts becomes crucial. Gensim expects its input data pattern to follow a list of tokenized sentence lists.
from gensim import models
sents = [
'The dog barked in the park on another dog',
'The owner of the dog put him on the leash since he barked.',
'My dog is barking and chasing its tail.'
]
tokenized_corpus = [sent.lower().replace('.', '').split() for sent in sents]
custom_w2v = models.Word2Vec(
sentences=tokenized_corpus,
min_count=1,
vector_size=300,
workers=4
)
dog_vector = custom_w2v.wv['dog']
print("Sample continuous coordinates slice for token 'dog':")
print(dog_vector[:5])
print("\nTop similar terms to 'barked':")
print(custom_w2v.wv.most_similar('barked'))3. FastText (Subword-Level Handling)
A main flaw of Word2Vec is its inability to manage completely unseen, unknown text (Out-Of-Vocabulary / OOV words). FastText resolves this issue by decomposing words into internal Character N-Grams.
Because meaning is aggregated directly from smaller internal subword pieces, it can construct accurate fallback representations for modified or typo-ridden words based on their structural components.
from gensim import models
sents = [
'The dog barked in the park on another dog',
'The owner of the dog put him on the leash since he barked.',
'My dog is barking and chasing its tail.'
]
tokenized_corpus = [sent.lower().replace('.', '').split() for sent in sents]
fasttext_model = models.FastText(
sentences=tokenized_corpus,
vector_size=100,
window=5,
min_count=1,
workers=4,
sg=1
)
print("FastText vector for target 'dog' (first 5 elements):")
print(fasttext_model.wv['dog'][:5])
print("\nEvaluating semantics matching via subwords for 'barked':")
print(fasttext_model.wv.most_similar('barked'))