Machine Learning Course Summary
md13 min readSummary of IN402
1. Introduction to Machine Learning
Types of ML
🎯 Supervised Learning: Learning from inputs to output given labeled data
- Classification: Output is discrete (e.g., spam/not spam)
- Regression: Output is continuous (e.g., house prices)
🔍 Unsupervised Learning: Uses unlabeled data to find groups, clusters, or patterns
🔄 Semi-supervised Learning: Uses a small amount of labeled data + large amount of unlabeled data
🎮 Reinforcement Learning: Learning through trial and error (agent-state-action-reward-environment)
ML Algorithms & Workflow
Common Algorithms: Linear regression, logistic regression, decision trees, KNN, Naïve Bayes, Neural Networks
Workflow:
- Problem definition — What are we predicting?
- Data collection — Gather relevant data
- Data preprocessing — Clean, normalize, handle missing values
- Split dataset — Training (60-70%), Validation (10-20%), Testing (10-20%)
- Model selection — Choose appropriate algorithm
- Model training — Fit model to training data
- Evaluation — Measure performance on validation/test data
2. Linear Regression
Univariate Linear Regression
Model:
📊 Cost Function (MSE): Measures the average squared difference between predicted and actual values
⚡ Gradient Descent: Algorithm to minimize the cost function by repeating until convergence
- is the learning rate (controls step size)
- ⚠️ Update and simultaneously
Partial Derivatives:
Multivariate Linear Regression
Involves multiple independent variables (features)
Hypothesis:
Gradient Descent (Vectorized):
📏 Feature Scaling: Recommended so features are on the same scale for faster convergence
- Divide by max or range (max - min)
- Get features roughly in range
Mean Normalization: Replace with (where is mean and is standard deviation)
🎯 Normal Equation (Direct Solution):
| Gradient Descent | Normal Equation |
|---|---|
| Need to choose | No need for |
| Needs many iterations | No iterations |
| Works well for large | Slow if is very large |
| for matrix inversion |
Polynomial Regression
Models non-linear relationships as an -th degree polynomial
⚠️ Feature scaling becomes critical for polynomial regression!
3. Logistic Regression (Classification)
Used to predict the probability of a binary (0 or 1) or multi-class outcome
Binary Classification
🔵 Sigmoid Function: Maps output between 0 and 1
Key Properties:
Decision Rule:
- If ⟺ , predict class 1
- If ⟺ , predict class 0
📉 Cost Function (Log Loss):
Cross-Entropy Loss (dataset):
Gradient Descent Update:
⚠️ Same form as linear regression, but is different!
Multiclass Classification
For classes:
- Compute linear score for each class
- Exponentiate
- Compute probability (Softmax):
- Prediction is the class with max probability
Cross-Entropy Loss (Multiclass):
4. Model Evaluation & Challenges
Bias vs. Variance
🎯 Bias: Inability to capture the true relationship (Model too simple → Underfitting)
📊 Variance: Inconsistency when applied to different datasets (Model too complex → Overfitting)
Error Decomposition:
Overfitting Prevention
Data Manipulation:
- ✅ Splitting (Train/Val/Test)
- ✅ Resampling
- ✅ Augmenting (rotation, scaling, flipping)
- ✅ Reducing features (feature selection)
🔒 Regularization: Adds a penalty to the loss function
- L1 (Lasso): Adds → Encourages sparsity (some weights → 0)
- L2 (Ridge): Adds → Encourages small weights
Dropout: Randomly drop units (neurons) during training
Early Stopping: Stop when validation accuracy starts to decrease
Classification Metrics
Confusion Matrix:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | TP (True Positive) | FN (False Negative) |
| Actual Negative | FP (False Positive) | TN (True Negative) |
Key Metrics:
| Metric | Formula | Meaning |
|---|---|---|
| Accuracy | Overall correctness | |
| Recall (TPR) | Coverage of actual positives | |
| Specificity (TNR) | Coverage of actual negatives | |
| Precision | Accuracy of positive predictions | |
| F-Score | Harmonic mean of P & R |
F1-Score (when ):
📈 ROC Curve: Plots TPR vs FPR
- AUC ≈ 1 → Perfect classifier
- AUC ≈ 0.5 → Random guessing
- AUC < 0.5 → Worse than random
5. Decision Trees
Components
- 🌳 Root node: Starting point (top of tree)
- 🔀 Internal nodes: Decisions based on features
- 🍃 Leaf nodes: Final class labels/values (predictions)
Process
- Select best splitting attribute (using splitting criteria)
- Create branches for each value
- Recursively grow subtrees until stopping condition
- Prune if necessary to avoid overfitting
Splitting Criteria
ID3 (Information Gain/Entropy):
- Choose attribute with highest Information Gain (IG)
- Entropy:
- Low entropy → High purity (homogeneous)
- High entropy → Low purity (mixed)
C4.5 (Gain Ratio):
Normalizes Information Gain to handle bias towards attributes with many values
CART (Gini Index): Used for classification
- Measures impurity
- Range: [0, 0.5] (0 = pure, 0.5 = maximum impurity)
Regression Trees: Use Mean Squared Error (MSE) reduction to choose splits
| Criterion | Best For | Range |
|---|---|---|
| Entropy | ID3, C4.5 | [0, 1] |
| Gini Index | CART (faster) | [0, 0.5] |
| MSE | Regression Trees | [0, ∞) |
6. K-Nearest Neighbors (K-NN)
Predicts class/value based on the most similar stored samples
Steps
- Compute distance between new point and all training points
- Sort by distance and select top neighbors
- Aggregate:
- Classification → Majority vote
- Regression → Average (or weighted average)
Distance Metrics
| Metric | Formula | Use Case |
|---|---|---|
| Euclidean | Default, measures straight-line distance | |
| Manhattan | Grid-like paths, robust to outliers | |
| Minkowski | General form ( Manhattan, Euclidean) |
⚖️ Weighted K-NN: Weights neighbors by inverse distance ()
- Closer neighbors have more influence
Choosing K:
- ✅ Small → More sensitive to noise (overfitting)
- ✅ Large → Smoother boundaries (underfitting)
- 📊 Use cross-validation to find optimal
- 🔢 Often use odd for binary classification (avoid ties)
7. Naïve Bayes
A probabilistic classifier based on Bayes' Theorem, assuming independence between features
Bayes' Theorem
Components:
- : Posterior — Probability of hypothesis after seeing data
- : Likelihood — Probability of data given hypothesis
- : Prior — Initial probability of hypothesis
- : Evidence — Probability of data (normalizing constant)
🔑 Naïve Assumption: All features are conditionally independent given the class
Classification Rule:
Laplace Smoothing
Used if a feature value never appears in a class to avoid zero probability
- : Count of feature in class
- : Total count of all features in class
- : Number of distinct values for feature
Why needed? If , entire product becomes 0!
8. Neural Networks
Structure
- Input Layer: Represents features ()
- Hidden Layers: Between input and output (can be multiple)
- Output Layer: Produces prediction ()
Node Computation
Linear Combination:
Activation:
where is the activation function
Activation Functions
| Function | Formula | Range | Use Case |
|---|---|---|---|
| Sigmoid | (0, 1) | Binary classification (output) | |
| Tanh | (-1, 1) | Hidden layers (zero-centered) | |
| ReLU | [0, ∞) | Most common for hidden layers | |
| Leaky ReLU | (-∞, ∞) | Fixes "dying ReLU" problem | |
| Softmax | (0, 1), sum=1 | Multiclass classification |
Training
🔄 Backpropagation: Propagates error backwards to update weights
- Forward pass: Compute predictions
- Calculate loss: Compare with true labels
- Backward pass: Compute gradients using chain rule
- Update weights: Adjust parameters
Weight Update:
where is the learning rate
Training Strategies (Gradient Descent)
| Strategy | Batch Size | Speed | Memory | Convergence |
|---|---|---|---|---|
| Stochastic GD (SGD) | 1 | Fast per update | Low | Noisy |
| Batch GD | All samples | Slow per update | High | Smooth |
| Mini-batch GD | samples (32-256) | Balanced | Medium | Most common |
🎯 Mini-batch GD is the sweet spot — combines benefits of both!
9. Clustering
Unsupervised learning — Grouping similar data points based on similarity/distance
K-Means Clustering
Partitions points into clusters by minimizing variance within clusters
🎯 Objective: Minimize Within-Cluster Sum of Squares (WCSS)
- : Centroid of cluster
- : Set of points in cluster
Algorithm Steps:
- Initialize: Randomly select initial centroids
- Assignment: Assign each point to nearest centroid
- Update: Recalculate centroids as mean of assigned points
- Repeat: Steps 2-3 until assignments stabilize (or max iterations)
Choosing K:
📊 Elbow Method:
- Plot WCSS vs
- Look for "elbow" where decrease slows
- Trade-off between fit and complexity
📏 Silhouette Score:
- Measures cohesion (within-cluster) vs separation (between-cluster)
- Range: [-1, 1]
- 1 → Perfect clustering
- 0 → Overlapping clusters
- -1 → Misclassified points
- : Avg distance to points in same cluster
- : Avg distance to points in nearest other cluster
Hierarchical Clustering
Produces a tree-like diagram called a Dendrogram
🌳 Agglomerative (Bottom-up):
- Start with each point as its own cluster
- Merge closest two clusters based on distance matrix
- Update distance matrix using linkage criteria
- Repeat until one cluster remains (or desired clusters)
Linkage Criteria:
| Method | Distance Between Clusters | Characteristics |
|---|---|---|
| Single | Min distance between any two points | Sensitive to outliers, chain effect |
| Complete | Max distance between any two points | Less sensitive to outliers, compact clusters |
| Average | Average distance between all pairs | Balanced approach |
| Ward's | Minimizes within-cluster variance | Produces even-sized clusters |
Advantages:
- ✅ No need to specify in advance
- ✅ Visual dendrogram for interpretation
- ✅ Captures hierarchical structure
Disadvantages:
- ❌ Computationally expensive: or
- ❌ Cannot undo merges (greedy algorithm)
📋 Quick Reference Cheatsheet
Algorithms Comparison
| Algorithm | Type | Pros | Cons | Use Case |
|---|---|---|---|---|
| Linear Regression | Regression | Simple, interpretable, fast | Assumes linearity | Continuous output, linear relationships |
| Logistic Regression | Classification | Probabilistic output, interpretable | Linear decision boundary | Binary/multiclass classification |
| Decision Trees | Both | Non-linear, interpretable, no scaling | Overfitting, unstable | Complex decision boundaries |
| K-NN | Both | Simple, no training | Slow prediction, sensitive to | Small datasets, non-linear |
| Naïve Bayes | Classification | Fast, works with small data | Independence assumption | Text classification, spam detection |
| Neural Networks | Both | Powerful, learns complex patterns | Black box, needs large data | Image, text, complex patterns |
| K-Means | Clustering | Fast, scalable | Need to specify , sensitive to init | Customer segmentation |
| Hierarchical | Clustering | No need for , dendrogram | Slow, not scalable | Taxonomy, small datasets |
Key Formulas
| Concept | Formula | Notes |
|---|---|---|
| Linear Hypothesis | for bias | |
| MSE Cost | Factor of simplifies derivative | |
| Gradient Descent | Update simultaneously | |
| Normal Equation | Direct solution, | |
| Sigmoid | Maps to (0, 1) | |
| Cross-Entropy | Logistic regression loss | |
| Softmax | Multiclass probabilities | |
| Entropy | Decision tree splitting | |
| Gini Index | Alternative to entropy | |
| Euclidean Distance | K-NN default metric | |
| Bayes' Theorem | Naïve Bayes foundation |
Evaluation Metrics
| Metric | Formula | When to Use |
|---|---|---|
| Accuracy | Balanced classes | |
| Precision | Minimize false positives (spam detection) | |
| Recall | Minimize false negatives (disease detection) | |
| F1-Score | Imbalanced classes, balance P & R | |
| MSE | Regression | |
| MAE | Regression, robust to outliers | |
| R² | Regression, % variance explained |
Hyperparameters
| Algorithm | Hyperparameter | Typical Values | Impact |
|---|---|---|---|
| All GD | Learning rate | 0.001, 0.01, 0.1, 1 | Speed & convergence |
| Regularization | 0.01, 0.1, 1, 10 | Overfitting control | |
| K-NN | neighbors | 3, 5, 7, 11 (odd) | Bias-variance tradeoff |
| Decision Tree | Max depth | 3-10 | Overfitting control |
| Neural Network | Hidden layers | 1-5+ | Model complexity |
| Neural Network | Neurons per layer | 32, 64, 128, 256 | Representational power |
| K-Means | clusters | Use elbow/silhouette | Number of groups |
| Mini-batch GD | Batch size | 32, 64, 128, 256 | Memory vs convergence |
Feature Preprocessing
| Technique | Formula | Purpose |
|---|---|---|
| Min-Max Scaling | Scale to [0, 1] | |
| Standardization | Mean=0, Std=1 (preferred for GD) | |
| Normalization | $\frac{x}{\ | x\ |
Model Selection Guide
Choose based on:
- Data size: Small → K-NN, Naïve Bayes; Large → Neural Networks
- Interpretability needed: Decision Trees, Linear/Logistic Regression
- Output type: Continuous → Regression; Discrete → Classification
- Linearity: Linear → Linear models; Non-linear → Trees, NN, K-NN
- Training speed: Fast → Naïve Bayes, Linear; Slow → Neural Networks
- Prediction speed: Fast → Linear, Trees; Slow → K-NN
💡 Key Takeaways:
- Always split data (train/val/test) to avoid overfitting
- Feature scaling improves gradient descent convergence
- Regularization helps prevent overfitting
- Choose metrics based on problem (accuracy vs precision vs recall)
- Cross-validation for reliable model selection
- Start simple, add complexity only if needed