Text_Representation_Tutorial

ipynb11 kBPython · 15 cellsExtra material of chapter 4

Tutorial: Feature Engineering for NLP Tasks

Using Scikit-Learn and Gensim

Feature engineering is a critical step in Natural Language Processing (NLP). It involves transforming raw text data into structured numerical features that machine learning algorithms can understand.

This tutorial guides you through two fundamental approaches to NLP feature engineering:

  1. Frequency-Based Methods (using scikit-learn): Bag-of-Words, Stopword filtering, N-grams, and TF-IDF.
  2. Prediction-Based Methods / Dense Word Embeddings (using gensim): Word2Vec (Pre-trained and Custom) and FastText.

Part 1: Frequency-Based Vectorization (Scikit-Learn)

1. Count Vectorization (Bag-of-Words)

The Bag-of-Words (BoW) model represents text by counting the occurrences of each word within a document. It completely disregards grammar, word order, and context, focusing entirely on raw term frequency.

We use scikit-learn's CountVectorizer to construct a vocabulary of all unique tokens across our corpus and map each sentence to its corresponding frequency vector.

In [1]
from sklearn.feature_extraction.text import CountVectorizer

sents = [
    'The dog barked in the park on another dog',
    'The owner of the dog put him on the leash since he barked.',
    'My dog is barking and chasing its tail.'
]

cv = CountVectorizer()
X = cv.fit_transform(sents)
X_array = X.toarray()

print("Vocabulary Features:")
print(sorted(cv.vocabulary_.keys()))
print("\nDocument-Term Matrix Grid Layout:")
print(X_array)

2. Count Vectorization with Stopword Filtering

Stopwords are highly frequent words (such as "the", "is", "in", "on") that provide grammatical structure but carry minimal semantic value for downstream tasks like classification or clustering. Filtering them out drastically reduces dimensionality and cleans up noisy features. We can pass stop_words='english' into the vectorizer to apply an internal English stopword list.

In [2]
from sklearn.feature_extraction.text import CountVectorizer

sents = [
    'The dog barked in the park on another dog',
    'The owner of the dog put him on the leash since he barked.',
    'My dog is barking and chasing its tail.'
]

cv_stopwords = CountVectorizer(stop_words='english')
X_stopwords = cv_stopwords.fit_transform(sents).toarray()

print("Filtered Vocabulary (Stopwords Removed):")
print(sorted(cv_stopwords.vocabulary_.keys()))
print("\nFiltered Document-Term Matrix:")
print(X_stopwords)

3. N-Gram Count Vectorization

To counter the complete loss of local context inherent in a basic unigram model (1-gram), we introduce N-grams. N-grams group consecutive sequences of tokens together.

  • ngram_range=(1,1) extracts single words (unigrams).
  • ngram_range=(1,2) extracts both single words and word pairs (bigrams), preserving expressions like "dog barked" or "chasing tail".
In [3]
from sklearn.feature_extraction.text import CountVectorizer

sents = [
    'The dog barked in the park on another dog',
    'The owner of the dog put him on the leash since he barked.',
    'My dog is barking and chasing its tail.'
]

cv_ngram = CountVectorizer(ngram_range=(1, 2), stop_words='english')
X_ngram = cv_ngram.fit_transform(sents).toarray()

print("Expanded N-Gram Vocabulary Space:")
print(sorted(cv_ngram.vocabulary_.keys()))
print("\nN-Gram Matrix Shape:", X_ngram.shape)
print(X_ngram)

4. TF-IDF Vectorization

While count arrays track presence, they can over-emphasize words that are naturally common across an entire subject field. TF-IDF (Term Frequency-Inverse Document Frequency) fixes this by applying a balanced scaling calculation: TF-IDF(t,d)=TF(t,d)×IDF(t)\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \text{IDF}(t)

Where TF evaluates frequency inside a singular document, and IDF suppresses tokens that appear redundantly across the broader corpus. This accurately brings out structural signature terms.

In [4]
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

sents = [
    'The dog barked in the park on another dog',
    'The owner of the dog put him on the leash since he barked.',
    'My dog is barking and chasing its tail.'
]

tfidf_vec = TfidfVectorizer(stop_words='english')
X_tfidf = tfidf_vec.fit_transform(sents)

print("Raw Output Feature Weight Metrics Grid:")
print(X_tfidf.toarray())

feature_names = tfidf_vec.get_feature_names_out()
df_tfidf = pd.DataFrame(X_tfidf[0].T.todense(), index=feature_names, columns=["TF-IDF"])
df_tfidf = df_tfidf.sort_values('TF-IDF', ascending=False)

print("\nRanked Terms within Document 1:")
print(df_tfidf)

Part 2: Prediction-Based Continuous Dense Vectors (Gensim)

Unlike sparse frequency configurations, vector embeddings capture semantic similarities using low-dimensional, continuous spatial real numbers.

1. Utilizing Pre-Trained Word2Vec Models

Gensim allows you to import heavy pre-trained models (such as the standard 300-dimension Google News corpus configuration) to map high-fidelity universal associations instantly without training overhead.

In [5]
from gensim import models

# Code template structure showcasing how to extract relationships using massive binaries:
"""
w2v_pretrained = models.KeyedVectors.load_word2vec_format(
    './GoogleNews-vectors-negative300.bin', 
    binary=True
)
healthy_vector = w2v_pretrained['healthy']
print(w2v_pretrained.most_similar('happy'))
"""
print("[Info] Pre-trained blueprint successfully structured for external usage.")

2. Training a Custom Word2Vec Model

When work covers hyper-specific vocabulary or unique domains, building specialized contexts becomes crucial. Gensim expects its input data pattern to follow a list of tokenized sentence lists.

In [6]
from gensim import models

sents = [
    'The dog barked in the park on another dog',
    'The owner of the dog put him on the leash since he barked.',
    'My dog is barking and chasing its tail.'
]

tokenized_corpus = [sent.lower().replace('.', '').split() for sent in sents]

custom_w2v = models.Word2Vec(
    sentences=tokenized_corpus, 
    min_count=1, 
    vector_size=300, 
    workers=4
)

dog_vector = custom_w2v.wv['dog']
print("Sample continuous coordinates slice for token 'dog':")
print(dog_vector[:5])

print("\nTop similar terms to 'barked':")
print(custom_w2v.wv.most_similar('barked'))

3. FastText (Subword-Level Handling)

A main flaw of Word2Vec is its inability to manage completely unseen, unknown text (Out-Of-Vocabulary / OOV words). FastText resolves this issue by decomposing words into internal Character N-Grams.

Because meaning is aggregated directly from smaller internal subword pieces, it can construct accurate fallback representations for modified or typo-ridden words based on their structural components.

In [7]
from gensim import models

sents = [
    'The dog barked in the park on another dog',
    'The owner of the dog put him on the leash since he barked.',
    'My dog is barking and chasing its tail.'
]

tokenized_corpus = [sent.lower().replace('.', '').split() for sent in sents]

fasttext_model = models.FastText(
    sentences=tokenized_corpus, 
    vector_size=100, 
    window=5, 
    min_count=1, 
    workers=4, 
    sg=1
)

print("FastText vector for target 'dog' (first 5 elements):")
print(fasttext_model.wv['dog'][:5])

print("\nEvaluating semantics matching via subwords for 'barked':")
print(fasttext_model.wv.most_similar('barked'))