IN400-partial
ipynb63 kBPython · 66 cellsExtra material of IN400
IN400 - Linear Algebra & NumPy Reference Guide
Applied Mathematics & Statistics for Computer Science
A comprehensive guide covering vectors, matrices, and advanced linear algebra concepts with both mathematical theory and practical NumPy implementations. Essential reference for AI and machine learning applications.
📑 Table of Contents
- Vectors
- Vector Representation & Creation
- Vector Operations (Addition, Scalar Multiplication, Dot Product)
- Vector Magnitude & Normalization
- Cross Product & Vector Projection
- Matrices
- Matrix Representation & Creation
- Matrix Operations (Addition, Multiplication, Transpose)
- Determinant, Inverse, and Rank
- Eigenvalues & Eigenvectors
- Advanced Topics
- Principal Component Analysis (PCA)
- Singular Value Decomposition (SVD)
- Practical Applications
- Data Representation
- Linear Transformations
- Solving Linear Systems
- Quick Reference Cheat Sheet
# Import NumPy library
import numpy as np
print("NumPy version:", np.__version__)
print("Ready to go! 🚀")1. Vectors
A vector is an ordered collection of numbers representing:
- Points in space
- Directions and magnitudes
- Data features in machine learning
- Physical quantities (velocity, force, etc.)
1.1 Vector Representation
Mathematical Notation
Column vector:
Row vector:
Example
In 2D space: represents point or a direction from the origin.
NumPy Implementation
# Creating vectors in NumPy
v1 = np.array([3, 4]) # 2D vector
v2 = np.array([1, 2, 3]) # 3D vector
v3 = np.array([1, 2, 3, 4, 5]) # 5D vector
print("2D vector:", v1)
print("3D vector:", v2)
print("5D vector:", v3)
print("\nShape of v2:", v2.shape) # (3,) means 1D array with 3 elements1.2 Vector Operations
1.2.1 Vector Addition and Subtraction
Vectors can be added or subtracted element-wise if they have the same dimension.
Mathematical Definition
Example
NumPy Implementation
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
addition = a + b
subtraction = a - b
print("a + b =", addition) # [5 7 9]
print("a - b =", subtraction) # [-3 -3 -3]1.2.2 Scalar Multiplication
Multiplying a vector by a scalar (number) scales each component.
Mathematical definition:
Example:
v = np.array([1, 2, 3])
scalar = 3
result = scalar * v
print("3 * v =", result) # [3 6 9]1.2.3 Dot Product (Inner Product)
The dot product combines two vectors into a single number. It measures how much two vectors "align" with each other.
Mathematical definition:
Example:
Properties:
- If , the vectors are orthogonal (perpendicular)
- If , they point in similar directions
- If , they point in opposite directions
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
# Method 1: Using np.dot()
dot_product = np.dot(a, b)
print("a · b =", dot_product) # 32
# Method 2: Using @ operator
dot_product = a @ b
print("a · b =", dot_product) # 32
# Method 3: Element-wise multiply then sum
dot_product = np.sum(a * b)
print("a · b =", dot_product) # 321.2.4 Vector Magnitude (Norm)
The magnitude (or length) of a vector is the distance from the origin to the point.
Mathematical definition:
Example:
v = np.array([3, 4])
# Method 1: Using np.linalg.norm()
magnitude = np.linalg.norm(v)
print("||v|| =", magnitude) # 5.0
# Method 2: Manual calculation
magnitude = np.sqrt(np.sum(v**2))
print("||v|| =", magnitude) # 5.01.2.5 Unit Vector (Normalization)
A unit vector has magnitude 1 and indicates direction only.
Mathematical definition:
Example:
v = np.array([3, 4])
# Normalize the vector
unit_vector = v / np.linalg.norm(v)
print("Unit vector:", unit_vector) # [0.6 0.8]
print("Magnitude of unit vector:", np.linalg.norm(unit_vector)) # 1.01.2.6 Cross Product (Vector Product)
The cross product is an operation on two 3D vectors that produces a third vector perpendicular to both input vectors. It's only defined for 3D vectors.
Mathematical definition:
Properties:
- The result is perpendicular to both input vectors
- The magnitude is: , where is the angle between vectors
- Not commutative:
- If vectors are parallel:
Applications:
- Finding normal vectors (computer graphics, physics)
- Computing torque and angular momentum
- Determining the orientation of surfaces
Example:
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
# Cross product using np.cross()
cross_product = np.cross(a, b)
print("a × b =", cross_product) # [-3 6 -3]
# Verify: the result is perpendicular to both a and b
# Dot product with perpendicular vectors should be 0
print("\n(a × b) · a =", np.dot(cross_product, a)) # Should be 0
print("(a × b) · b =", np.dot(cross_product, b)) # Should be 0
# Example: Find a vector perpendicular to the xy-plane
x_axis = np.array([1, 0, 0])
y_axis = np.array([0, 1, 0])
z_direction = np.cross(x_axis, y_axis)
print("\nx_axis × y_axis =", z_direction) # [0 0 1] - points in z direction
# Example: Parallel vectors have zero cross product
parallel_a = np.array([2, 4, 6])
parallel_b = np.array([1, 2, 3]) # parallel_b = 0.5 * parallel_a
result = np.cross(parallel_b, parallel_a)
print("\nCross product of parallel vectors:", result) # [0 0 0]1.2.7 Vector Projection
Vector projection projects one vector onto another, finding the component of one vector in the direction of another.
Mathematical definition:
Projection of onto :
The scalar projection (length of the projection):
Geometric interpretation:
- The projection shows "how much" of vector points in the direction of
- If perpendicular: projection = 0
- If parallel: projection = (or )
Applications:
- Linear regression (projecting data onto a line/plane)
- Component analysis
- Shadow calculations in graphics
- Signal decomposition
Example:
# Example 1: Project vector a onto vector b
a = np.array([3, 4])
b = np.array([1, 0])
# Calculate projection
proj_b_a = (np.dot(a, b) / np.dot(b, b)) * b
print("Vector a:", a)
print("Vector b:", b)
print("Projection of a onto b:", proj_b_a) # [3 0]
# Scalar projection (length)
scalar_proj = np.dot(a, b) / np.linalg.norm(b)
print("Scalar projection (length):", scalar_proj) # 3.0
# Example 2: 3D projection
a_3d = np.array([1, 2, 3])
b_3d = np.array([1, 1, 0])
proj_3d = (np.dot(a_3d, b_3d) / np.dot(b_3d, b_3d)) * b_3d
print("\n3D Example:")
print("Vector a:", a_3d)
print("Vector b:", b_3d)
print("Projection of a onto b:", proj_3d)
# Example 3: Perpendicular vectors (projection = 0)
perpendicular_a = np.array([1, 0, 0])
perpendicular_b = np.array([0, 1, 0])
proj_perp = (
np.dot(perpendicular_a, perpendicular_b) / np.dot(perpendicular_b, perpendicular_b)
) * perpendicular_b
print("\nPerpendicular vectors:")
print("Projection:", proj_perp) # [0 0 0]
# Example 4: Decompose a vector into parallel and perpendicular components
a_decomp = np.array([3, 4])
b_decomp = np.array([1, 0])
parallel_component = (
np.dot(a_decomp, b_decomp) / np.dot(b_decomp, b_decomp)
) * b_decomp
perpendicular_component = a_decomp - parallel_component
print("\nVector decomposition:")
print("Original vector:", a_decomp)
print("Parallel component:", parallel_component)
print("Perpendicular component:", perpendicular_component)
print("Sum (should equal original):", parallel_component + perpendicular_component)2. Matrices
A matrix is a 2D array of numbers arranged in rows and columns. Matrices are essential in AI for representing datasets, transformations, and neural network weights.
2.1 Matrix Representation
Mathematical notation:
- An m × n matrix has m rows and n columns
- Element at row i, column j is denoted as or
Example: A 2×3 matrix:
# Creating matrices
A = np.array([[1, 2, 3], [4, 5, 6]]) # 2x3 matrix
B = np.array([[1, 2], [3, 4], [5, 6]]) # 3x2 matrix
print("Matrix A:")
print(A)
print("Shape:", A.shape) # (2, 3) means 2 rows, 3 columns
# Identity matrix (diagonal of 1s)
I = np.eye(3)
print("\nIdentity matrix:")
print(I)
# Zero matrix
Z = np.zeros((2, 3))
print("\nZero matrix:")
print(Z)
# Random matrix
R = np.random.rand(2, 3) # Random values between 0 and 1
print("\nRandom matrix:")
print(R)2.2 Matrix Operations
2.2.1 Matrix Addition and Subtraction
Matrices of the same dimensions can be added/subtracted element-wise.
Mathematical definition:
Example:
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
addition = A + B
subtraction = A - B
print("A + B =")
print(addition)
# [[6 8]
# [10 12]]
print("\nA - B =")
print(subtraction)
# [[-4 -4]
# [-4 -4]]2.2.2 Scalar Multiplication
Multiply each element by a scalar.
Mathematical definition:
A = np.array([[1, 2], [3, 4]])
result = 3 * A
print("3 * A =")
print(result)
# [[3 6]
# [9 12]]2.2.3 Matrix Multiplication
Key rule: To multiply (m×n) by (p×q), we need n = p. The result is an m×q matrix.
Mathematical definition:
The element at position (i,j) in the result is the dot product of row i of A with column j of B.
Example:
Important: Matrix multiplication is not commutative:
A = np.array([[1, 2], [3, 4]]) # 2x2
B = np.array([[5, 6], [7, 8]]) # 2x2
# Matrix multiplication
C = np.dot(A, B)
# Or using @ operator
C = A @ B
print("A × B =")
print(C)
# [[19 22]
# [43 50]]
# Matrix-vector multiplication
v = np.array([1, 2])
result = A @ v
print("\nA × v =")
print(result) # [5 11]2.2.4 Matrix Transpose
Transpose flips a matrix over its diagonal, converting rows to columns.
Mathematical definition:
Example:
A = np.array([[1, 2, 3], [4, 5, 6]]) # 2x3
A_transpose = A.T
print("A =")
print(A)
print("\nA^T =")
print(A_transpose) # 3x2 matrix2.2.5 Matrix Inverse
The inverse of a square matrix is denoted and satisfies:
Important: Not all matrices have an inverse! A matrix is invertible if its determinant is non-zero.
Example:
A = np.array([[4, 7], [2, 6]])
# Calculate inverse
A_inv = np.linalg.inv(A)
print("A =")
print(A)
print("\nA^(-1) =")
print(A_inv)
# Verify: A × A^(-1) = I
identity = A @ A_inv
print("\nA × A^(-1) =")
print(identity) # Should be close to [[1, 0], [0, 1]]2.2.6 Matrix Determinant
The determinant is a scalar value that can be computed from a square matrix. It provides important information about the matrix:
- If det(A) ≠ 0, the matrix is invertible
- If det(A) = 0, the matrix is singular (not invertible)
- The determinant represents the volume scaling factor of the linear transformation
Mathematical definition:
For a 2×2 matrix:
For a 3×3 matrix:
Examples:
2×2:
3×3:
# Example 1: Determinant of a 2x2 matrix
A_2x2 = np.array([[3, 8], [4, 6]])
det_A = np.linalg.det(A_2x2)
print("Matrix A (2x2):")
print(A_2x2)
print("Determinant of A:", det_A) # -14.0
# Example 2: Determinant of a 3x3 matrix
A_3x3 = np.array([[1, 2, 3], [0, 1, 4], [5, 6, 0]])
det_B = np.linalg.det(A_3x3)
print("\nMatrix B (3x3):")
print(A_3x3)
print("Determinant of B:", det_B) # 1.0
# Example 3: Check if a matrix is invertible
singular_matrix = np.array([[2, 4], [1, 2]])
det_singular = np.linalg.det(singular_matrix)
print("\nSingular Matrix:")
print(singular_matrix)
print("Determinant:", det_singular) # 0.0 (or very close to 0)
if abs(det_singular) < 1e-10:
print("This matrix is SINGULAR (not invertible)")
else:
print("This matrix is INVERTIBLE")
# Example 4: Manual calculation for 2x2 (to verify)
a, b = 3, 8
c, d = 4, 6
manual_det = a * d - b * c
print(f"\nManual calculation: {a}*{d} - {b}*{c} = {manual_det}")2.2.7 Eigenvalues and Eigenvectors
Eigenvalues and eigenvectors are fundamental concepts in linear algebra. For a square matrix , an eigenvector and its corresponding eigenvalue satisfy:
Interpretation:
- When matrix is applied to eigenvector , the result is the same vector scaled by
- The eigenvector's direction is preserved, only the magnitude changes
- (eigenvalue) is the scaling factor
Finding eigenvalues: Solve the characteristic equation:
Properties:
- An n×n matrix has n eigenvalues (counting multiplicities)
- Eigenvectors corresponding to different eigenvalues are linearly independent
- For symmetric matrices, eigenvalues are real and eigenvectors are orthogonal
Applications in AI/ML:
- Principal Component Analysis (PCA): Dimensionality reduction
- Google PageRank: Website ranking algorithm
- Markov Chains: State transitions and steady states
- Image Compression: Spectral decomposition
- Stability Analysis: Neural network dynamics
- Graph Analysis: Community detection
Example: For matrix :
- Eigenvalues:
- Eigenvector for :
- Verification: ✓
# Example 1: Basic eigenvalue and eigenvector calculation
A = np.array([[4, 2], [1, 3]])
# Calculate eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)
print("Matrix A:")
print(A)
print("\nEigenvalues:", eigenvalues)
print("\nEigenvectors:")
print(eigenvectors)
# Verify: A @ v = λ * v for the first eigenvector
v1 = eigenvectors[:, 0] # First eigenvector (column 0)
lambda1 = eigenvalues[0]
left_side = A @ v1
right_side = lambda1 * v1
print(f"\nVerification for λ₁ = {lambda1:.4f}:")
print(f"A @ v₁ = {left_side}")
print(f"λ₁ * v₁ = {right_side}")
print(f"Equal? {np.allclose(left_side, right_side)}")
# Example 2: Symmetric matrix (real eigenvalues, orthogonal eigenvectors)
symmetric_matrix = np.array([[2, 1], [1, 2]])
eig_vals, eig_vecs = np.linalg.eig(symmetric_matrix)
print("\n\nSymmetric Matrix:")
print(symmetric_matrix)
print("\nEigenvalues:", eig_vals)
print("\nEigenvectors:")
print(eig_vecs)
# Check orthogonality: dot product should be 0
dot_product = np.dot(eig_vecs[:, 0], eig_vecs[:, 1])
print(f"\nDot product of eigenvectors: {dot_product:.10f} (should be ~0)")
# Example 3: 3x3 matrix
A_3x3 = np.array([[6, -1, 0], [-1, 6, -1], [0, -1, 6]])
eig_vals_3d, eig_vecs_3d = np.linalg.eig(A_3x3)
print("\n\n3x3 Matrix:")
print(A_3x3)
print("\nEigenvalues:", eig_vals_3d)
# Example 4: Application - Power iteration to find dominant eigenvalue
def power_iteration(A, num_iterations=10):
"""Find the dominant eigenvalue and eigenvector using power iteration"""
n = A.shape[0]
v = np.random.rand(n) # Random initial vector
for i in range(num_iterations):
# Multiply by matrix
v = A @ v
# Normalize
v = v / np.linalg.norm(v)
# Calculate eigenvalue: λ = (v^T A v) / (v^T v)
eigenvalue = (v.T @ A @ v) / (v.T @ v)
return eigenvalue, v
dominant_eval, dominant_evec = power_iteration(A, num_iterations=20)
print(f"\n\nPower Iteration (finds dominant eigenvalue):")
print(f"Dominant eigenvalue: {dominant_eval:.4f}")
print(f"Compare with np.linalg.eig: {max(eigenvalues):.4f}")
# Example 5: Trace and determinant relation to eigenvalues
print(f"\n\nTrace-Eigenvalue relation:")
print(f"Trace of A: {np.trace(A):.4f}")
print(f"Sum of eigenvalues: {np.sum(eigenvalues):.4f}")
print(f"Determinant of A: {np.linalg.det(A):.4f}")
print(f"Product of eigenvalues: {np.prod(eigenvalues):.4f}")2.2.8 Element-wise Operations
Sometimes we need element-wise operations (Hadamard product), not matrix multiplication.
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])
# Element-wise multiplication
element_wise = A * B
print("A ⊙ B (element-wise) =")
print(element_wise)
# [[5 12]
# [21 32]]
# Element-wise division
element_div = A / B
print("\nA ⊘ B (element-wise) =")
print(element_div)
# Element-wise power
powered = A**2
print("\nA^2 (element-wise) =")
print(powered)
# [[1 4]
# [9 16]]2.3 Special Matrices
# Identity matrix (3x3)
I = np.eye(3)
print("Identity matrix:")
print(I)
# Diagonal matrix
diag = np.diag([1, 2, 3])
print("\nDiagonal matrix:")
print(diag)
# Upper triangular
upper = np.triu(np.ones((3, 3)))
print("\nUpper triangular:")
print(upper)
# Lower triangular
lower = np.tril(np.ones((3, 3)))
print("\nLower triangular:")
print(lower)3. Practical Applications in AI
3.1 Data Representation
- Vectors: Each data point (e.g., image, text) is represented as a vector
- Matrices: A dataset is a matrix where each row is a data sample
# Dataset: 5 students with 3 test scores each
grades = np.array(
[
[85, 90, 88], # Student 1
[78, 85, 80], # Student 2
[92, 95, 93], # Student 3
[70, 75, 72], # Student 4
[88, 87, 90],
]
) # Student 5
print("Dataset shape:", grades.shape) # (5, 3)
# Calculate mean score for each student
mean_per_student = np.mean(grades, axis=1)
print("Mean per student:", mean_per_student)
# Calculate mean score for each test
mean_per_test = np.mean(grades, axis=0)
print("Mean per test:", mean_per_test)3.2 Linear Transformations
Matrix multiplication applies transformations to vectors.
# Rotation matrix (90 degrees counterclockwise)
theta = np.pi / 2 # 90 degrees in radians
rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
# Original point
point = np.array([1, 0])
# Rotate the point
rotated_point = rotation @ point
print("Original point:", point)
print("Rotated point:", rotated_point) # Should be close to [0, 1]3.3 Solving Linear Systems
Systems of linear equations can be solved using matrices.
System:
Matrix form:
# Coefficient matrix
A = np.array([[2, 3], [4, 1]])
# Constants vector
b = np.array([8, 10])
# Solve for x
x = np.linalg.solve(A, b)
print("Solution:", x) # x = [2, 2] means x=2, y=2
# Verify solution
print("Verification A×x =", A @ x) # Should equal b2.2.9 Matrix Rank
The rank of a matrix is the maximum number of linearly independent rows or columns.
Mathematical Definition:
- Rank indicates the dimension of the vector space spanned by the matrix's rows/columns
- For an m×n matrix:
Properties:
- Full rank: rank equals the smaller dimension
- Rank-deficient: rank is less than the smaller dimension
NumPy Implementation
# Example matrices with different ranks
A_full_rank = np.array([[1, 2], [3, 4]])
A_rank_deficient = np.array([[1, 2], [2, 4]]) # Second row is 2x first row
print("Full rank matrix:")
print(A_full_rank)
print("Rank:", np.linalg.matrix_rank(A_full_rank)) # 2
print("\nRank-deficient matrix:")
print(A_rank_deficient)
print("Rank:", np.linalg.matrix_rank(A_rank_deficient)) # 13. Advanced Topics
3.1 Principal Component Analysis (PCA)
PCA is a dimensionality reduction technique that transforms data into a new coordinate system where the greatest variances lie on the first coordinates (principal components).
Mathematical Foundation
Given a dataset (n samples × m features):
Step 1: Center the data where is the mean of each feature.
Step 2: Compute covariance matrix
Step 3: Find eigenvectors and eigenvalues Solve:
Step 4: Sort by eigenvalues
- Larger eigenvalues → more variance captured
- Eigenvectors become principal components
Step 5: Project data where contains the top k eigenvectors.
Applications in ML/AI
- Dimensionality reduction — Reduce features while preserving variance
- Data visualization — Project high-D data to 2D/3D
- Noise reduction — Remove low-variance components
- Feature extraction — Create uncorrelated features
- Compression — Store data more efficiently
NumPy Implementation
# Generate sample data: 100 samples, 3 features
np.random.seed(42)
X = np.random.randn(100, 3)
X[:, 1] = X[:, 0] * 2 + np.random.randn(100) * 0.5 # Correlated features
print("Original data shape:", X.shape)
print("First 5 samples:")
print(X[:5])
# Step 1: Center the data (subtract mean)
X_mean = np.mean(X, axis=0)
X_centered = X - X_mean
print("\nData mean:", X_mean)
print("Centered data mean (should be ~0):", np.mean(X_centered, axis=0))# Step 2: Compute covariance matrix
cov_matrix = np.cov(X_centered.T) # Note: transpose to get feature covariance
print("Covariance matrix shape:", cov_matrix.shape)
print("\nCovariance matrix:")
print(cov_matrix)# Step 3: Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
print("Eigenvalues:", eigenvalues)
print("\nEigenvectors (principal components):")
print(eigenvectors)
# Step 4: Sort by eigenvalues (descending order)
idx = eigenvalues.argsort()[::-1] # Sort indices in descending order
eigenvalues_sorted = eigenvalues[idx]
eigenvectors_sorted = eigenvectors[:, idx]
print("\nSorted eigenvalues:", eigenvalues_sorted)
# Calculate explained variance ratio
explained_variance_ratio = eigenvalues_sorted / np.sum(eigenvalues_sorted)
print("\nExplained variance ratio:", explained_variance_ratio)
print("Cumulative variance:", np.cumsum(explained_variance_ratio))# Step 5: Project data onto principal components
# Let's reduce from 3D to 2D (keep first 2 components)
k = 2
principal_components = eigenvectors_sorted[:, :k]
X_pca = X_centered @ principal_components
print(f"Reduced data shape: {X_pca.shape}")
print("\nFirst 5 samples in PCA space:")
print(X_pca[:5])
print(f"\nVariance preserved: {np.sum(explained_variance_ratio[:k]):.2%}")# Complete PCA function
def pca_transform(X, n_components):
"""
Perform PCA dimensionality reduction
Parameters:
X: (n_samples, n_features) data matrix
n_components: number of principal components to keep
Returns:
X_pca: transformed data
components: principal components (eigenvectors)
explained_var: variance explained by each component
"""
# Center data
X_mean = np.mean(X, axis=0)
X_centered = X - X_mean
# Covariance matrix
cov_matrix = np.cov(X_centered.T)
# Eigendecomposition
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)
# Sort by eigenvalues
idx = eigenvalues.argsort()[::-1]
eigenvalues = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]
# Select top k components
components = eigenvectors[:, :n_components]
# Project data
X_pca = X_centered @ components
# Explained variance
explained_var = eigenvalues[:n_components] / np.sum(eigenvalues)
return X_pca, components, explained_var
# Test the function
X_pca, components, explained_var = pca_transform(X, n_components=2)
print("PCA result shape:", X_pca.shape)
print("Explained variance:", explained_var)
print(f"Total variance explained: {np.sum(explained_var):.2%}")3.2 Singular Value Decomposition (SVD)
SVD decomposes any matrix into three simpler matrices, revealing its fundamental structure.
Mathematical Definition
For any matrix (m×n):
Where:
- (m×m): Left singular vectors (orthonormal)
- (m×n): Diagonal matrix of singular values
- (n×n): Right singular vectors (orthonormal)
Key Properties
Relationship to eigendecomposition:
- Singular values of = square roots of eigenvalues of
- contains eigenvectors of
- contains eigenvectors of
Important facts:
- Every matrix has an SVD (even non-square!)
- Singular values are always non-negative
- Rank of = number of non-zero singular values
Low-Rank Approximation
Best rank-k approximation of :
Where we keep only the k largest singular values.
Applications in ML/AI
- Image compression — Store images with fewer bytes
- Recommender systems — Matrix factorization (Netflix, Amazon)
- Latent Semantic Analysis — Text mining and NLP
- Noise reduction — Remove small singular values
- Data visualization — Similar to PCA
- Pseudoinverse — Solve ill-conditioned linear systems
NumPy Implementation
# Create a sample matrix
A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])
print("Original matrix A:")
print(A)
print("Shape:", A.shape) # 4×3
# Perform SVD
U, S, Vt = np.linalg.svd(A, full_matrices=True)
print("\nU (left singular vectors):")
print(U)
print("Shape:", U.shape) # 4×4
print("\nSingular values:")
print(S)
print("Shape:", S.shape) # (3,) - returned as 1D array
print("\nV^T (right singular vectors transposed):")
print(Vt)
print("Shape:", Vt.shape) # 3×3# Reconstruct original matrix
# Need to create full Sigma matrix from singular values
Sigma = np.zeros((U.shape[0], Vt.shape[0]))
Sigma[: S.shape[0], : S.shape[0]] = np.diag(S)
print("Sigma (diagonal matrix):")
print(Sigma)
print("Shape:", Sigma.shape)
# Reconstruct A
A_reconstructed = U @ Sigma @ Vt
print("\nReconstructed A:")
print(A_reconstructed)
print("\nReconstruction error:")
print(np.linalg.norm(A - A_reconstructed)) # Should be very small# Low-rank approximation
k = 2 # Keep top 2 singular values
U_k = U[:, :k]
S_k = S[:k]
Vt_k = Vt[:k, :]
# Create k×k diagonal matrix
Sigma_k = np.diag(S_k)
# Low-rank approximation
A_k = U_k @ Sigma_k @ Vt_k
print(f"Rank-{k} approximation:")
print(A_k)
print(f"\nOriginal matrix:")
print(A)
print(f"\nApproximation error:")
error = np.linalg.norm(A - A_k)
print(f"Frobenius norm: {error:.4f}")
# Compression ratio
original_elements = A.shape[0] * A.shape[1]
compressed_elements = k * (U_k.shape[0] + Vt_k.shape[1] + 1)
print(
f"\nCompression ratio: {compressed_elements}/{original_elements} = {compressed_elements/original_elements:.2%}"
)# Example: Image compression using SVD
# Create a simple "image" (grayscale matrix)
np.random.seed(42)
image = np.random.rand(50, 50)
# Add some structure (diagonal pattern)
for i in range(50):
image[i, i] = 1.0
if i < 49:
image[i, i + 1] = 0.8
image[i + 1, i] = 0.8
print("Original image shape:", image.shape)
# Perform SVD
U, S, Vt = np.linalg.svd(image, full_matrices=False)
# Try different compression levels
for k in [5, 10, 20]:
# Reconstruct with k components
image_k = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]
# Calculate error and compression
error = np.linalg.norm(image - image_k)
compression = (k * (image.shape[0] + image.shape[1])) / (
image.shape[0] * image.shape[1]
)
print(f"\nk={k} singular values:")
print(f" Reconstruction error: {error:.4f}")
print(f" Storage needed: {compression:.2%} of original")
print(f" Variance explained: {np.sum(S[:k]**2) / np.sum(S**2):.2%}")# SVD vs PCA relationship
# They're closely related! For centered data:
# Generate centered data
np.random.seed(42)
X_data = np.random.randn(100, 5)
X_centered = X_data - np.mean(X_data, axis=0)
# Method 1: PCA via eigendecomposition
cov_matrix = np.cov(X_centered.T)
eigenvalues_pca, eigenvectors_pca = np.linalg.eig(cov_matrix)
idx = eigenvalues_pca.argsort()[::-1]
eigenvalues_pca = eigenvalues_pca[idx]
# Method 2: PCA via SVD
U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)
eigenvalues_svd = (S**2) / (X_centered.shape[0] - 1)
print("Eigenvalues from PCA:", eigenvalues_pca)
print("Eigenvalues from SVD:", eigenvalues_svd)
print("\nAre they equal?", np.allclose(eigenvalues_pca, eigenvalues_svd))
# The principal components are the same (up to sign)
print("\nV from SVD matches eigenvectors from covariance matrix:")
print("Close match?", np.allclose(np.abs(Vt.T), np.abs(eigenvectors_pca[:, idx])))4. Practical Applications in AI
4.1 Data Representation
Vectors and matrices are fundamental for representing data in machine learning:
- Vectors — Each data point (image pixels, word embeddings, sensor readings)
- Matrices — Datasets where rows = samples, columns = features
- Tensors — Higher-dimensional arrays (images, video, time series)
NumPy Implementation
# Dataset: 5 students with 3 test scores each
grades = np.array(
[
[85, 90, 88], # Student 1
[78, 85, 80], # Student 2
[92, 95, 93], # Student 3
[70, 75, 72], # Student 4
[88, 87, 90],
]
) # Student 5
print("Dataset shape:", grades.shape) # (5, 3)
# Calculate mean score for each student
mean_per_student = np.mean(grades, axis=1)
print("Mean per student:", mean_per_student)
# Calculate mean score for each test
mean_per_test = np.mean(grades, axis=0)
print("Mean per test:", mean_per_test)4.2 Linear Transformations
Matrix multiplication applies geometric transformations to vectors:
- Rotation — Rotate points/vectors
- Scaling — Stretch/compress along axes
- Reflection — Mirror across axes
- Shear — Skew shapes
NumPy Implementation
# Rotation matrix (90 degrees counterclockwise)
theta = np.pi / 2 # 90 degrees in radians
rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
# Original point
point = np.array([1, 0])
# Rotate the point
rotated_point = rotation @ point
print("Original point:", point)
print("Rotated point:", rotated_point) # Should be close to [0, 1]4.3 Solving Linear Systems
Systems of linear equations appear everywhere in ML/AI:
- Linear regression — Fit line/plane to data
- Neural networks — Each layer is a linear transformation
- Optimization — Find parameters that minimize loss
Mathematical Form
System of equations:
Matrix form:
NumPy Implementation
# Coefficient matrix
A = np.array([[2, 3], [4, 1]])
# Constants vector
b = np.array([8, 10])
# Solve for x
x = np.linalg.solve(A, b)
print("Solution:", x) # x = [2, 2] means x=2, y=2
# Verify solution
print("Verification A×x =", A @ x) # Should equal b5. Quick Reference Cheat Sheet
Vector Operations
| Operation | Math Notation | NumPy Code | Note (Manual Calculation) |
|---|---|---|---|
| Create vector | v = np.array([1, 2, 3]) |
List of numbers | |
| Vector addition | a + b |
Add each component: | |
| Scalar multiplication | c * v |
Multiply each element: | |
| Dot product | np.dot(a, b) or a @ b |
or np.sum(a * b) |
|
| Cross product (3D) | np.cross(a, b) |
||
| Magnitude (norm) | np.linalg.norm(v) |
or np.sqrt(np.sum(v**2)) |
|
| Normalize | v / np.linalg.norm(v) |
Divide by magnitude | |
| Projection | (a@b / b@b) * b |
Matrix Operations
| Operation | Math Notation | NumPy Code | Note (Manual Calculation) |
|---|---|---|---|
| Create matrix | A = np.array([[1,2],[3,4]]) |
Nested lists (rows) | |
| Identity matrix | np.eye(n) |
1s on diagonal, 0s elsewhere | |
| Zero matrix | np.zeros((m, n)) |
All elements = 0 | |
| Matrix addition | A + B |
Add element-wise: | |
| Matrix multiplication | A @ B or np.dot(A, B) |
(row × column) | |
| Element-wise mult. | A * B |
Multiply element-wise: | |
| Transpose | A.T |
Flip over diagonal: | |
| Inverse | np.linalg.inv(A) |
Satisfies | |
| Determinant (2×2) | np.linalg.det(A) |
for | |
| Determinant (3×3) | np.linalg.det(A) |
||
| Rank | np.linalg.matrix_rank(A) |
Number of linearly independent rows/columns |
Advanced Operations
| Operation | Math Notation | NumPy Code | Note (Manual Calculation) |
|---|---|---|---|
| Eigendecomposition | eigenval, eigenvec = np.linalg.eig(A) |
Solve | |
| SVD | U, S, Vt = np.linalg.svd(A) |
Factorize any matrix into 3 matrices | |
| Solve linear system | x = np.linalg.solve(A, b) |
(but use solver, not inverse!) | |
| Covariance matrix | np.cov(X.T) |
Measures feature correlations (center data first) | |
| Trace | np.trace(A) |
Sum of diagonal elements: | |
| Frobenius norm | np.linalg.norm(A, 'fro') |
Matrix Creation
| Function | Description | Example | Note |
|---|---|---|---|
np.array() |
Create from list | np.array([[1,2],[3,4]]) |
Most basic method |
np.eye(n) |
n×n identity matrix | np.eye(3) |
Can also use np.identity(n) |
np.zeros((m,n)) |
m×n matrix of zeros | np.zeros((2,3)) |
Useful for initialization |
np.ones((m,n)) |
m×n matrix of ones | np.ones((2,3)) |
Scale with * value if needed |
np.diag(v) |
Diagonal matrix from vector | np.diag([1,2,3]) |
Creates square matrix |
np.random.rand(m,n) |
Random values in [0,1) | np.random.rand(2,3) |
Uniform distribution |
np.random.randn(m,n) |
Random values from standard normal | np.random.randn(2,3) |
Mean=0, std=1 |
np.arange(start, stop, step) |
Array with evenly spaced values | np.arange(0, 10, 2) |
Like Python's range() |
np.linspace(start, stop, n) |
n evenly spaced values | np.linspace(0, 1, 5) |
Includes both endpoints |
Additional Resources
- NumPy Documentation: https://numpy.org/doc/
- Linear Algebra Visualized (3Blue1Brown): https://www.3blue1brown.com/topics/linear-algebra
- SciPy Lectures: https://scipy-lectures.org/intro/numpy/index.html
- NumPy Linear Algebra: https://numpy.org/doc/stable/reference/routines.linalg.html
- PCA Tutorial: https://builtin.com/data-science/step-step-explanation-principal-component-analysis
- SVD Tutorial: https://www.askpython.com/python/examples/singular-value-decomposition