IN400-partial

ipynb63 kBPython · 66 cellsExtra material of IN400

IN400 - Linear Algebra & NumPy Reference Guide

Applied Mathematics & Statistics for Computer Science

A comprehensive guide covering vectors, matrices, and advanced linear algebra concepts with both mathematical theory and practical NumPy implementations. Essential reference for AI and machine learning applications.


📑 Table of Contents

  1. Vectors
    • Vector Representation & Creation
    • Vector Operations (Addition, Scalar Multiplication, Dot Product)
    • Vector Magnitude & Normalization
    • Cross Product & Vector Projection
  2. Matrices
    • Matrix Representation & Creation
    • Matrix Operations (Addition, Multiplication, Transpose)
    • Determinant, Inverse, and Rank
    • Eigenvalues & Eigenvectors
  3. Advanced Topics
    • Principal Component Analysis (PCA)
    • Singular Value Decomposition (SVD)
  4. Practical Applications
    • Data Representation
    • Linear Transformations
    • Solving Linear Systems
  5. Quick Reference Cheat Sheet

In [ ]
# Import NumPy library
import numpy as np

print("NumPy version:", np.__version__)
print("Ready to go! 🚀")

1. Vectors

A vector is an ordered collection of numbers representing:

  • Points in space
  • Directions and magnitudes
  • Data features in machine learning
  • Physical quantities (velocity, force, etc.)

1.1 Vector Representation

Mathematical Notation

Column vector: v=[v1v2⋮vn]\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}

Row vector: vT=[v1,v2,…,vn]\mathbf{v}^T = [v_1, v_2, \ldots, v_n]

Example

In 2D space: v=[34]\mathbf{v} = \begin{bmatrix} 3 \\ 4 \end{bmatrix} represents point (3,4)(3, 4) or a direction from the origin.

NumPy Implementation

In [ ]
# Creating vectors in NumPy
v1 = np.array([3, 4])  # 2D vector
v2 = np.array([1, 2, 3])  # 3D vector
v3 = np.array([1, 2, 3, 4, 5])  # 5D vector

print("2D vector:", v1)
print("3D vector:", v2)
print("5D vector:", v3)
print("\nShape of v2:", v2.shape)  # (3,) means 1D array with 3 elements

1.2 Vector Operations

1.2.1 Vector Addition and Subtraction

Vectors can be added or subtracted element-wise if they have the same dimension.

Mathematical Definition

a+b=[a1a2⋮an]+[b1b2⋮bn]=[a1+b1a2+b2⋮an+bn]\mathbf{a} + \mathbf{b} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} + \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix} = \begin{bmatrix} a_1 + b_1 \\ a_2 + b_2 \\ \vdots \\ a_n + b_n \end{bmatrix}

Example

[123]+[456]=[579]\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} + \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = \begin{bmatrix} 5 \\ 7 \\ 9 \end{bmatrix}

NumPy Implementation
In [ ]
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

addition = a + b
subtraction = a - b

print("a + b =", addition)  # [5 7 9]
print("a - b =", subtraction)  # [-3 -3 -3]
1.2.2 Scalar Multiplication

Multiplying a vector by a scalar (number) scales each component.

Mathematical definition: c⋅v=c⋅[v1v2⋮vn]=[c⋅v1c⋅v2⋮c⋅vn]c \cdot \mathbf{v} = c \cdot \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} = \begin{bmatrix} c \cdot v_1 \\ c \cdot v_2 \\ \vdots \\ c \cdot v_n \end{bmatrix}

Example: 3⋅[12]=[36]3 \cdot \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 3 \\ 6 \end{bmatrix}

In [ ]
v = np.array([1, 2, 3])
scalar = 3

result = scalar * v
print("3 * v =", result)  # [3 6 9]
1.2.3 Dot Product (Inner Product)

The dot product combines two vectors into a single number. It measures how much two vectors "align" with each other.

Mathematical definition: a⋅b=a1b1+a2b2+…+anbn=∑i=1naibi\mathbf{a} \cdot \mathbf{b} = a_1b_1 + a_2b_2 + \ldots + a_nb_n = \sum_{i=1}^{n} a_i b_i

Example: [123]⋅[456]=1(4)+2(5)+3(6)=4+10+18=32\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = 1(4) + 2(5) + 3(6) = 4 + 10 + 18 = 32

Properties:

  • If a⋅b=0\mathbf{a} \cdot \mathbf{b} = 0, the vectors are orthogonal (perpendicular)
  • If a⋅b>0\mathbf{a} \cdot \mathbf{b} > 0, they point in similar directions
  • If a⋅b<0\mathbf{a} \cdot \mathbf{b} < 0, they point in opposite directions
In [ ]
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Method 1: Using np.dot()
dot_product = np.dot(a, b)
print("a · b =", dot_product)  # 32

# Method 2: Using @ operator
dot_product = a @ b
print("a · b =", dot_product)  # 32

# Method 3: Element-wise multiply then sum
dot_product = np.sum(a * b)
print("a · b =", dot_product)  # 32
1.2.4 Vector Magnitude (Norm)

The magnitude (or length) of a vector is the distance from the origin to the point.

Mathematical definition: ∥v∥=v12+v22+…+vn2=v⋅v\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \ldots + v_n^2} = \sqrt{\mathbf{v} \cdot \mathbf{v}}

Example: ∥[34]∥=32+42=9+16=25=5\left\|\begin{bmatrix} 3 \\ 4 \end{bmatrix}\right\| = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5

In [ ]
v = np.array([3, 4])

# Method 1: Using np.linalg.norm()
magnitude = np.linalg.norm(v)
print("||v|| =", magnitude)  # 5.0

# Method 2: Manual calculation
magnitude = np.sqrt(np.sum(v**2))
print("||v|| =", magnitude)  # 5.0
1.2.5 Unit Vector (Normalization)

A unit vector has magnitude 1 and indicates direction only.

Mathematical definition: v^=v∥v∥\hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|}

Example: v=[34],v^=15[34]=[0.60.8]\mathbf{v} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}, \quad \hat{\mathbf{v}} = \frac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 0.6 \\ 0.8 \end{bmatrix}

In [ ]
v = np.array([3, 4])

# Normalize the vector
unit_vector = v / np.linalg.norm(v)
print("Unit vector:", unit_vector)  # [0.6 0.8]
print("Magnitude of unit vector:", np.linalg.norm(unit_vector))  # 1.0
1.2.6 Cross Product (Vector Product)

The cross product is an operation on two 3D vectors that produces a third vector perpendicular to both input vectors. It's only defined for 3D vectors.

Mathematical definition: a×b=[a2b3−a3b2a3b1−a1b3a1b2−a2b1]\mathbf{a} \times \mathbf{b} = \begin{bmatrix} a_2b_3 - a_3b_2 \\ a_3b_1 - a_1b_3 \\ a_1b_2 - a_2b_1 \end{bmatrix}

Properties:

  • The result is perpendicular to both input vectors
  • The magnitude is: ∥a×b∥=∥a∥∥b∥sin⁡(θ)\|\mathbf{a} \times \mathbf{b}\| = \|\mathbf{a}\| \|\mathbf{b}\| \sin(\theta), where θ\theta is the angle between vectors
  • Not commutative: a×b=−(b×a)\mathbf{a} \times \mathbf{b} = -(\mathbf{b} \times \mathbf{a})
  • If vectors are parallel: a×b=0\mathbf{a} \times \mathbf{b} = \mathbf{0}

Applications:

  • Finding normal vectors (computer graphics, physics)
  • Computing torque and angular momentum
  • Determining the orientation of surfaces

Example: [123]×[456]=[2(6)−3(5)3(4)−1(6)1(5)−2(4)]=[−36−3]\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \times \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = \begin{bmatrix} 2(6) - 3(5) \\ 3(4) - 1(6) \\ 1(5) - 2(4) \end{bmatrix} = \begin{bmatrix} -3 \\ 6 \\ -3 \end{bmatrix}

In [ ]
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Cross product using np.cross()
cross_product = np.cross(a, b)
print("a × b =", cross_product)  # [-3  6 -3]

# Verify: the result is perpendicular to both a and b
# Dot product with perpendicular vectors should be 0
print("\n(a × b) · a =", np.dot(cross_product, a))  # Should be 0
print("(a × b) · b =", np.dot(cross_product, b))  # Should be 0

# Example: Find a vector perpendicular to the xy-plane
x_axis = np.array([1, 0, 0])
y_axis = np.array([0, 1, 0])
z_direction = np.cross(x_axis, y_axis)
print("\nx_axis × y_axis =", z_direction)  # [0 0 1] - points in z direction

# Example: Parallel vectors have zero cross product
parallel_a = np.array([2, 4, 6])
parallel_b = np.array([1, 2, 3])  # parallel_b = 0.5 * parallel_a
result = np.cross(parallel_b, parallel_a)
print("\nCross product of parallel vectors:", result)  # [0 0 0]
1.2.7 Vector Projection

Vector projection projects one vector onto another, finding the component of one vector in the direction of another.

Mathematical definition:

Projection of a\mathbf{a} onto b\mathbf{b}: projba=a⋅b∥b∥2b=a⋅bb⋅bb\text{proj}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{b}\|^2}\mathbf{b} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}}\mathbf{b}

The scalar projection (length of the projection): compba=a⋅b∥b∥\text{comp}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{b}\|}

Geometric interpretation:

  • The projection shows "how much" of vector a\mathbf{a} points in the direction of b\mathbf{b}
  • If perpendicular: projection = 0
  • If parallel: projection = a\mathbf{a} (or −a-\mathbf{a})

Applications:

  • Linear regression (projecting data onto a line/plane)
  • Component analysis
  • Shadow calculations in graphics
  • Signal decomposition

Example: a=[34],b=[10]\mathbf{a} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}, \quad \mathbf{b} = \begin{bmatrix} 1 \\ 0 \end{bmatrix} projba=3(1)+4(0)12+02[10]=3[10]=[30]\text{proj}_{\mathbf{b}}\mathbf{a} = \frac{3(1) + 4(0)}{1^2 + 0^2}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = 3\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 3 \\ 0 \end{bmatrix}

In [ ]
# Example 1: Project vector a onto vector b
a = np.array([3, 4])
b = np.array([1, 0])

# Calculate projection
proj_b_a = (np.dot(a, b) / np.dot(b, b)) * b
print("Vector a:", a)
print("Vector b:", b)
print("Projection of a onto b:", proj_b_a)  # [3 0]

# Scalar projection (length)
scalar_proj = np.dot(a, b) / np.linalg.norm(b)
print("Scalar projection (length):", scalar_proj)  # 3.0

# Example 2: 3D projection
a_3d = np.array([1, 2, 3])
b_3d = np.array([1, 1, 0])

proj_3d = (np.dot(a_3d, b_3d) / np.dot(b_3d, b_3d)) * b_3d
print("\n3D Example:")
print("Vector a:", a_3d)
print("Vector b:", b_3d)
print("Projection of a onto b:", proj_3d)

# Example 3: Perpendicular vectors (projection = 0)
perpendicular_a = np.array([1, 0, 0])
perpendicular_b = np.array([0, 1, 0])

proj_perp = (
    np.dot(perpendicular_a, perpendicular_b) / np.dot(perpendicular_b, perpendicular_b)
) * perpendicular_b
print("\nPerpendicular vectors:")
print("Projection:", proj_perp)  # [0 0 0]

# Example 4: Decompose a vector into parallel and perpendicular components
a_decomp = np.array([3, 4])
b_decomp = np.array([1, 0])

parallel_component = (
    np.dot(a_decomp, b_decomp) / np.dot(b_decomp, b_decomp)
) * b_decomp
perpendicular_component = a_decomp - parallel_component

print("\nVector decomposition:")
print("Original vector:", a_decomp)
print("Parallel component:", parallel_component)
print("Perpendicular component:", perpendicular_component)
print("Sum (should equal original):", parallel_component + perpendicular_component)

2. Matrices

A matrix is a 2D array of numbers arranged in rows and columns. Matrices are essential in AI for representing datasets, transformations, and neural network weights.

2.1 Matrix Representation

Mathematical notation: A=[a11a12⋯a1na21a22⋯a2n⋮⋮⋱⋮am1am2⋯amn]\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix}

  • An m × n matrix has m rows and n columns
  • Element at row i, column j is denoted as aija_{ij} or A[i,j]A[i,j]

Example: A 2×3 matrix: A=[123456]\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}

In [ ]
# Creating matrices
A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3 matrix

B = np.array([[1, 2], [3, 4], [5, 6]])  # 3x2 matrix

print("Matrix A:")
print(A)
print("Shape:", A.shape)  # (2, 3) means 2 rows, 3 columns

# Identity matrix (diagonal of 1s)
I = np.eye(3)
print("\nIdentity matrix:")
print(I)

# Zero matrix
Z = np.zeros((2, 3))
print("\nZero matrix:")
print(Z)

# Random matrix
R = np.random.rand(2, 3)  # Random values between 0 and 1
print("\nRandom matrix:")
print(R)

2.2 Matrix Operations

2.2.1 Matrix Addition and Subtraction

Matrices of the same dimensions can be added/subtracted element-wise.

Mathematical definition: A+B=[a11+b11a12+b12a21+b21a22+b22]\mathbf{A} + \mathbf{B} = \begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} \\ a_{21} + b_{21} & a_{22} + b_{22} \end{bmatrix}

Example: [1234]+[5678]=[681012]\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 6 & 8 \\ 10 & 12 \end{bmatrix}

In [ ]
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

addition = A + B
subtraction = A - B

print("A + B =")
print(addition)
# [[6  8]
#  [10 12]]

print("\nA - B =")
print(subtraction)
# [[-4 -4]
#  [-4 -4]]
2.2.2 Scalar Multiplication

Multiply each element by a scalar.

Mathematical definition: c⋅A=[c⋅a11c⋅a12c⋅a21c⋅a22]c \cdot \mathbf{A} = \begin{bmatrix} c \cdot a_{11} & c \cdot a_{12} \\ c \cdot a_{21} & c \cdot a_{22} \end{bmatrix}

In [ ]
A = np.array([[1, 2], [3, 4]])
result = 3 * A

print("3 * A =")
print(result)
# [[3  6]
#  [9 12]]
2.2.3 Matrix Multiplication

Key rule: To multiply A\mathbf{A} (m×n) by B\mathbf{B} (p×q), we need n = p. The result is an m×q matrix.

Mathematical definition: (AB)ij=∑k=1naikbkj(\mathbf{AB})_{ij} = \sum_{k=1}^{n} a_{ik} b_{kj}

The element at position (i,j) in the result is the dot product of row i of A with column j of B.

Example: [1234]×[5678]=[1(5)+2(7)1(6)+2(8)3(5)+4(7)3(6)+4(8)]=[19224350]\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 1(5)+2(7) & 1(6)+2(8) \\ 3(5)+4(7) & 3(6)+4(8) \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix}

Important: Matrix multiplication is not commutative: AB≠BA\mathbf{AB} \neq \mathbf{BA}

In [ ]
A = np.array([[1, 2], [3, 4]])  # 2x2

B = np.array([[5, 6], [7, 8]])  # 2x2

# Matrix multiplication
C = np.dot(A, B)
# Or using @ operator
C = A @ B

print("A × B =")
print(C)
# [[19 22]
#  [43 50]]

# Matrix-vector multiplication
v = np.array([1, 2])
result = A @ v

print("\nA × v =")
print(result)  # [5 11]
2.2.4 Matrix Transpose

Transpose flips a matrix over its diagonal, converting rows to columns.

Mathematical definition: (AT)ij=Aji(\mathbf{A}^T)_{ij} = A_{ji}

Example: A=[123456],AT=[142536]\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}, \quad \mathbf{A}^T = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}

In [ ]
A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3

A_transpose = A.T

print("A =")
print(A)
print("\nA^T =")
print(A_transpose)  # 3x2 matrix
2.2.5 Matrix Inverse

The inverse of a square matrix A\mathbf{A} is denoted A−1\mathbf{A}^{-1} and satisfies: AA−1=A−1A=I\mathbf{A} \mathbf{A}^{-1} = \mathbf{A}^{-1} \mathbf{A} = \mathbf{I}

Important: Not all matrices have an inverse! A matrix is invertible if its determinant is non-zero.

Example: A=[4726],A−1=[0.6−0.7−0.20.4]\mathbf{A} = \begin{bmatrix} 4 & 7 \\ 2 & 6 \end{bmatrix}, \quad \mathbf{A}^{-1} = \begin{bmatrix} 0.6 & -0.7 \\ -0.2 & 0.4 \end{bmatrix}

In [ ]
A = np.array([[4, 7], [2, 6]])

# Calculate inverse
A_inv = np.linalg.inv(A)

print("A =")
print(A)
print("\nA^(-1) =")
print(A_inv)

# Verify: A × A^(-1) = I
identity = A @ A_inv
print("\nA × A^(-1) =")
print(identity)  # Should be close to [[1, 0], [0, 1]]
2.2.6 Matrix Determinant

The determinant is a scalar value that can be computed from a square matrix. It provides important information about the matrix:

  • If det(A) ≠ 0, the matrix is invertible
  • If det(A) = 0, the matrix is singular (not invertible)
  • The determinant represents the volume scaling factor of the linear transformation

Mathematical definition:

For a 2×2 matrix: det[abcd]=ad−bc\text{det}\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc

For a 3×3 matrix: det[abcdefghi]=a(ei−fh)−b(di−fg)+c(dh−eg)\text{det}\begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix} = a(ei - fh) - b(di - fg) + c(dh - eg)

Examples:

2×2: det[3846]=3(6)−8(4)=18−32=−14\text{det}\begin{bmatrix} 3 & 8 \\ 4 & 6 \end{bmatrix} = 3(6) - 8(4) = 18 - 32 = -14

3×3: det[123014560]=1(1⋅0−4⋅6)−2(0⋅0−4⋅5)+3(0⋅6−1⋅5)=1(−24)−2(−20)+3(−5)=−24+40−15=1\text{det}\begin{bmatrix} 1 & 2 & 3 \\ 0 & 1 & 4 \\ 5 & 6 & 0 \end{bmatrix} = 1(1 \cdot 0 - 4 \cdot 6) - 2(0 \cdot 0 - 4 \cdot 5) + 3(0 \cdot 6 - 1 \cdot 5) = 1(-24) - 2(-20) + 3(-5) = -24 + 40 - 15 = 1

In [ ]
# Example 1: Determinant of a 2x2 matrix
A_2x2 = np.array([[3, 8], [4, 6]])

det_A = np.linalg.det(A_2x2)
print("Matrix A (2x2):")
print(A_2x2)
print("Determinant of A:", det_A)  # -14.0

# Example 2: Determinant of a 3x3 matrix
A_3x3 = np.array([[1, 2, 3], [0, 1, 4], [5, 6, 0]])

det_B = np.linalg.det(A_3x3)
print("\nMatrix B (3x3):")
print(A_3x3)
print("Determinant of B:", det_B)  # 1.0

# Example 3: Check if a matrix is invertible
singular_matrix = np.array([[2, 4], [1, 2]])

det_singular = np.linalg.det(singular_matrix)
print("\nSingular Matrix:")
print(singular_matrix)
print("Determinant:", det_singular)  # 0.0 (or very close to 0)

if abs(det_singular) < 1e-10:
    print("This matrix is SINGULAR (not invertible)")
else:
    print("This matrix is INVERTIBLE")

# Example 4: Manual calculation for 2x2 (to verify)
a, b = 3, 8
c, d = 4, 6
manual_det = a * d - b * c
print(f"\nManual calculation: {a}*{d} - {b}*{c} = {manual_det}")
2.2.7 Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are fundamental concepts in linear algebra. For a square matrix A\mathbf{A}, an eigenvector v\mathbf{v} and its corresponding eigenvalue λ\lambda satisfy:

Av=λv\mathbf{A}\mathbf{v} = \lambda\mathbf{v}

Interpretation:

  • When matrix A\mathbf{A} is applied to eigenvector v\mathbf{v}, the result is the same vector scaled by λ\lambda
  • The eigenvector's direction is preserved, only the magnitude changes
  • λ\lambda (eigenvalue) is the scaling factor

Finding eigenvalues: Solve the characteristic equation: det(A−λI)=0\text{det}(\mathbf{A} - \lambda\mathbf{I}) = 0

Properties:

  • An n×n matrix has n eigenvalues (counting multiplicities)
  • Eigenvectors corresponding to different eigenvalues are linearly independent
  • For symmetric matrices, eigenvalues are real and eigenvectors are orthogonal

Applications in AI/ML:

  • Principal Component Analysis (PCA): Dimensionality reduction
  • Google PageRank: Website ranking algorithm
  • Markov Chains: State transitions and steady states
  • Image Compression: Spectral decomposition
  • Stability Analysis: Neural network dynamics
  • Graph Analysis: Community detection

Example: For matrix A=[4213]\mathbf{A} = \begin{bmatrix} 4 & 2 \\ 1 & 3 \end{bmatrix}:

  • Eigenvalues: λ1=5,λ2=2\lambda_1 = 5, \lambda_2 = 2
  • Eigenvector for λ1=5\lambda_1 = 5: v1=[21]\mathbf{v}_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}
  • Verification: [4213][21]=[105]=5[21]\begin{bmatrix} 4 & 2 \\ 1 & 3 \end{bmatrix}\begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} = 5\begin{bmatrix} 2 \\ 1 \end{bmatrix} ✓
In [ ]
# Example 1: Basic eigenvalue and eigenvector calculation
A = np.array([[4, 2], [1, 3]])

# Calculate eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Matrix A:")
print(A)
print("\nEigenvalues:", eigenvalues)
print("\nEigenvectors:")
print(eigenvectors)

# Verify: A @ v = λ * v for the first eigenvector
v1 = eigenvectors[:, 0]  # First eigenvector (column 0)
lambda1 = eigenvalues[0]

left_side = A @ v1
right_side = lambda1 * v1

print(f"\nVerification for λ₁ = {lambda1:.4f}:")
print(f"A @ v₁ = {left_side}")
print(f"λ₁ * v₁ = {right_side}")
print(f"Equal? {np.allclose(left_side, right_side)}")

# Example 2: Symmetric matrix (real eigenvalues, orthogonal eigenvectors)
symmetric_matrix = np.array([[2, 1], [1, 2]])

eig_vals, eig_vecs = np.linalg.eig(symmetric_matrix)

print("\n\nSymmetric Matrix:")
print(symmetric_matrix)
print("\nEigenvalues:", eig_vals)
print("\nEigenvectors:")
print(eig_vecs)

# Check orthogonality: dot product should be 0
dot_product = np.dot(eig_vecs[:, 0], eig_vecs[:, 1])
print(f"\nDot product of eigenvectors: {dot_product:.10f} (should be ~0)")

# Example 3: 3x3 matrix
A_3x3 = np.array([[6, -1, 0], [-1, 6, -1], [0, -1, 6]])

eig_vals_3d, eig_vecs_3d = np.linalg.eig(A_3x3)

print("\n\n3x3 Matrix:")
print(A_3x3)
print("\nEigenvalues:", eig_vals_3d)


# Example 4: Application - Power iteration to find dominant eigenvalue
def power_iteration(A, num_iterations=10):
    """Find the dominant eigenvalue and eigenvector using power iteration"""
    n = A.shape[0]
    v = np.random.rand(n)  # Random initial vector

    for i in range(num_iterations):
        # Multiply by matrix
        v = A @ v
        # Normalize
        v = v / np.linalg.norm(v)

    # Calculate eigenvalue: λ = (v^T A v) / (v^T v)
    eigenvalue = (v.T @ A @ v) / (v.T @ v)
    return eigenvalue, v


dominant_eval, dominant_evec = power_iteration(A, num_iterations=20)
print(f"\n\nPower Iteration (finds dominant eigenvalue):")
print(f"Dominant eigenvalue: {dominant_eval:.4f}")
print(f"Compare with np.linalg.eig: {max(eigenvalues):.4f}")

# Example 5: Trace and determinant relation to eigenvalues
print(f"\n\nTrace-Eigenvalue relation:")
print(f"Trace of A: {np.trace(A):.4f}")
print(f"Sum of eigenvalues: {np.sum(eigenvalues):.4f}")
print(f"Determinant of A: {np.linalg.det(A):.4f}")
print(f"Product of eigenvalues: {np.prod(eigenvalues):.4f}")
2.2.8 Element-wise Operations

Sometimes we need element-wise operations (Hadamard product), not matrix multiplication.

In [ ]
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Element-wise multiplication
element_wise = A * B
print("A ⊙ B (element-wise) =")
print(element_wise)
# [[5  12]
#  [21 32]]

# Element-wise division
element_div = A / B
print("\nA ⊘ B (element-wise) =")
print(element_div)

# Element-wise power
powered = A**2
print("\nA^2 (element-wise) =")
print(powered)
# [[1  4]
#  [9 16]]

2.3 Special Matrices

In [ ]
# Identity matrix (3x3)
I = np.eye(3)
print("Identity matrix:")
print(I)

# Diagonal matrix
diag = np.diag([1, 2, 3])
print("\nDiagonal matrix:")
print(diag)

# Upper triangular
upper = np.triu(np.ones((3, 3)))
print("\nUpper triangular:")
print(upper)

# Lower triangular
lower = np.tril(np.ones((3, 3)))
print("\nLower triangular:")
print(lower)

3. Practical Applications in AI

3.1 Data Representation

  • Vectors: Each data point (e.g., image, text) is represented as a vector
  • Matrices: A dataset is a matrix where each row is a data sample
In [ ]
# Dataset: 5 students with 3 test scores each
grades = np.array(
    [
        [85, 90, 88],  # Student 1
        [78, 85, 80],  # Student 2
        [92, 95, 93],  # Student 3
        [70, 75, 72],  # Student 4
        [88, 87, 90],
    ]
)  # Student 5

print("Dataset shape:", grades.shape)  # (5, 3)

# Calculate mean score for each student
mean_per_student = np.mean(grades, axis=1)
print("Mean per student:", mean_per_student)

# Calculate mean score for each test
mean_per_test = np.mean(grades, axis=0)
print("Mean per test:", mean_per_test)

3.2 Linear Transformations

Matrix multiplication applies transformations to vectors.

In [ ]
# Rotation matrix (90 degrees counterclockwise)
theta = np.pi / 2  # 90 degrees in radians
rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])

# Original point
point = np.array([1, 0])

# Rotate the point
rotated_point = rotation @ point
print("Original point:", point)
print("Rotated point:", rotated_point)  # Should be close to [0, 1]

3.3 Solving Linear Systems

Systems of linear equations can be solved using matrices.

System: 2x+3y=82x + 3y = 8 4x+y=104x + y = 10

Matrix form: Ax=b\mathbf{Ax} = \mathbf{b}

In [ ]
# Coefficient matrix
A = np.array([[2, 3], [4, 1]])

# Constants vector
b = np.array([8, 10])

# Solve for x
x = np.linalg.solve(A, b)
print("Solution:", x)  # x = [2, 2] means x=2, y=2

# Verify solution
print("Verification A×x =", A @ x)  # Should equal b
2.2.9 Matrix Rank

The rank of a matrix is the maximum number of linearly independent rows or columns.

Mathematical Definition:

  • Rank indicates the dimension of the vector space spanned by the matrix's rows/columns
  • For an m×n matrix: 0≤rank(A)≤min⁡(m,n)0 \leq \text{rank}(\mathbf{A}) \leq \min(m, n)

Properties:

  • Full rank: rank equals the smaller dimension
  • Rank-deficient: rank is less than the smaller dimension
  • rank(A)=rank(AT)\text{rank}(\mathbf{A}) = \text{rank}(\mathbf{A}^T)
NumPy Implementation
In [ ]
# Example matrices with different ranks
A_full_rank = np.array([[1, 2], [3, 4]])
A_rank_deficient = np.array([[1, 2], [2, 4]])  # Second row is 2x first row

print("Full rank matrix:")
print(A_full_rank)
print("Rank:", np.linalg.matrix_rank(A_full_rank))  # 2

print("\nRank-deficient matrix:")
print(A_rank_deficient)
print("Rank:", np.linalg.matrix_rank(A_rank_deficient))  # 1

3. Advanced Topics

3.1 Principal Component Analysis (PCA)

PCA is a dimensionality reduction technique that transforms data into a new coordinate system where the greatest variances lie on the first coordinates (principal components).

Mathematical Foundation

Given a dataset X\mathbf{X} (n samples × m features):

Step 1: Center the data Xcentered=X−μ\mathbf{X}_{\text{centered}} = \mathbf{X} - \boldsymbol{\mu} where μ\boldsymbol{\mu} is the mean of each feature.

Step 2: Compute covariance matrix C=1n−1XcenteredTXcentered\mathbf{C} = \frac{1}{n-1}\mathbf{X}_{\text{centered}}^T \mathbf{X}_{\text{centered}}

Step 3: Find eigenvectors and eigenvalues Solve: Cv=λv\mathbf{C}\mathbf{v} = \lambda\mathbf{v}

Step 4: Sort by eigenvalues

  • Larger eigenvalues → more variance captured
  • Eigenvectors become principal components

Step 5: Project data XPCA=XcenteredVk\mathbf{X}_{\text{PCA}} = \mathbf{X}_{\text{centered}} \mathbf{V}_k where Vk\mathbf{V}_k contains the top k eigenvectors.

Applications in ML/AI

  • Dimensionality reduction — Reduce features while preserving variance
  • Data visualization — Project high-D data to 2D/3D
  • Noise reduction — Remove low-variance components
  • Feature extraction — Create uncorrelated features
  • Compression — Store data more efficiently

NumPy Implementation

In [ ]
# Generate sample data: 100 samples, 3 features
np.random.seed(42)
X = np.random.randn(100, 3)
X[:, 1] = X[:, 0] * 2 + np.random.randn(100) * 0.5  # Correlated features

print("Original data shape:", X.shape)
print("First 5 samples:")
print(X[:5])

# Step 1: Center the data (subtract mean)
X_mean = np.mean(X, axis=0)
X_centered = X - X_mean

print("\nData mean:", X_mean)
print("Centered data mean (should be ~0):", np.mean(X_centered, axis=0))
In [ ]
# Step 2: Compute covariance matrix
cov_matrix = np.cov(X_centered.T)  # Note: transpose to get feature covariance

print("Covariance matrix shape:", cov_matrix.shape)
print("\nCovariance matrix:")
print(cov_matrix)
In [ ]
# Step 3: Compute eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

print("Eigenvalues:", eigenvalues)
print("\nEigenvectors (principal components):")
print(eigenvectors)

# Step 4: Sort by eigenvalues (descending order)
idx = eigenvalues.argsort()[::-1]  # Sort indices in descending order
eigenvalues_sorted = eigenvalues[idx]
eigenvectors_sorted = eigenvectors[:, idx]

print("\nSorted eigenvalues:", eigenvalues_sorted)

# Calculate explained variance ratio
explained_variance_ratio = eigenvalues_sorted / np.sum(eigenvalues_sorted)
print("\nExplained variance ratio:", explained_variance_ratio)
print("Cumulative variance:", np.cumsum(explained_variance_ratio))
In [ ]
# Step 5: Project data onto principal components
# Let's reduce from 3D to 2D (keep first 2 components)
k = 2
principal_components = eigenvectors_sorted[:, :k]

X_pca = X_centered @ principal_components

print(f"Reduced data shape: {X_pca.shape}")
print("\nFirst 5 samples in PCA space:")
print(X_pca[:5])

print(f"\nVariance preserved: {np.sum(explained_variance_ratio[:k]):.2%}")
In [ ]
# Complete PCA function
def pca_transform(X, n_components):
    """
    Perform PCA dimensionality reduction

    Parameters:
    X: (n_samples, n_features) data matrix
    n_components: number of principal components to keep

    Returns:
    X_pca: transformed data
    components: principal components (eigenvectors)
    explained_var: variance explained by each component
    """
    # Center data
    X_mean = np.mean(X, axis=0)
    X_centered = X - X_mean

    # Covariance matrix
    cov_matrix = np.cov(X_centered.T)

    # Eigendecomposition
    eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

    # Sort by eigenvalues
    idx = eigenvalues.argsort()[::-1]
    eigenvalues = eigenvalues[idx]
    eigenvectors = eigenvectors[:, idx]

    # Select top k components
    components = eigenvectors[:, :n_components]

    # Project data
    X_pca = X_centered @ components

    # Explained variance
    explained_var = eigenvalues[:n_components] / np.sum(eigenvalues)

    return X_pca, components, explained_var


# Test the function
X_pca, components, explained_var = pca_transform(X, n_components=2)
print("PCA result shape:", X_pca.shape)
print("Explained variance:", explained_var)
print(f"Total variance explained: {np.sum(explained_var):.2%}")

3.2 Singular Value Decomposition (SVD)

SVD decomposes any matrix into three simpler matrices, revealing its fundamental structure.

Mathematical Definition

For any matrix A\mathbf{A} (m×n): A=UΣVT\mathbf{A} = \mathbf{U} \boldsymbol{\Sigma} \mathbf{V}^T

Where:

  • U\mathbf{U} (m×m): Left singular vectors (orthonormal)
  • Σ\boldsymbol{\Sigma} (m×n): Diagonal matrix of singular values σ1≥σ2≥…≥0\sigma_1 \geq \sigma_2 \geq \ldots \geq 0
  • VT\mathbf{V}^T (n×n): Right singular vectors (orthonormal)

Key Properties

Relationship to eigendecomposition:

  • Singular values of A\mathbf{A} = square roots of eigenvalues of ATA\mathbf{A}^T\mathbf{A}
  • V\mathbf{V} contains eigenvectors of ATA\mathbf{A}^T\mathbf{A}
  • U\mathbf{U} contains eigenvectors of AAT\mathbf{AA}^T

Important facts:

  • Every matrix has an SVD (even non-square!)
  • Singular values are always non-negative
  • Rank of A\mathbf{A} = number of non-zero singular values

Low-Rank Approximation

Best rank-k approximation of A\mathbf{A}: Ak=UkΣkVkT\mathbf{A}_k = \mathbf{U}_k \boldsymbol{\Sigma}_k \mathbf{V}_k^T

Where we keep only the k largest singular values.

Applications in ML/AI

  • Image compression — Store images with fewer bytes
  • Recommender systems — Matrix factorization (Netflix, Amazon)
  • Latent Semantic Analysis — Text mining and NLP
  • Noise reduction — Remove small singular values
  • Data visualization — Similar to PCA
  • Pseudoinverse — Solve ill-conditioned linear systems

NumPy Implementation

In [ ]
# Create a sample matrix
A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])

print("Original matrix A:")
print(A)
print("Shape:", A.shape)  # 4×3

# Perform SVD
U, S, Vt = np.linalg.svd(A, full_matrices=True)

print("\nU (left singular vectors):")
print(U)
print("Shape:", U.shape)  # 4×4

print("\nSingular values:")
print(S)
print("Shape:", S.shape)  # (3,) - returned as 1D array

print("\nV^T (right singular vectors transposed):")
print(Vt)
print("Shape:", Vt.shape)  # 3×3
In [ ]
# Reconstruct original matrix
# Need to create full Sigma matrix from singular values
Sigma = np.zeros((U.shape[0], Vt.shape[0]))
Sigma[: S.shape[0], : S.shape[0]] = np.diag(S)

print("Sigma (diagonal matrix):")
print(Sigma)
print("Shape:", Sigma.shape)

# Reconstruct A
A_reconstructed = U @ Sigma @ Vt

print("\nReconstructed A:")
print(A_reconstructed)

print("\nReconstruction error:")
print(np.linalg.norm(A - A_reconstructed))  # Should be very small
In [ ]
# Low-rank approximation
k = 2  # Keep top 2 singular values

U_k = U[:, :k]
S_k = S[:k]
Vt_k = Vt[:k, :]

# Create k×k diagonal matrix
Sigma_k = np.diag(S_k)

# Low-rank approximation
A_k = U_k @ Sigma_k @ Vt_k

print(f"Rank-{k} approximation:")
print(A_k)

print(f"\nOriginal matrix:")
print(A)

print(f"\nApproximation error:")
error = np.linalg.norm(A - A_k)
print(f"Frobenius norm: {error:.4f}")

# Compression ratio
original_elements = A.shape[0] * A.shape[1]
compressed_elements = k * (U_k.shape[0] + Vt_k.shape[1] + 1)
print(
    f"\nCompression ratio: {compressed_elements}/{original_elements} = {compressed_elements/original_elements:.2%}"
)
In [ ]
# Example: Image compression using SVD
# Create a simple "image" (grayscale matrix)
np.random.seed(42)
image = np.random.rand(50, 50)

# Add some structure (diagonal pattern)
for i in range(50):
    image[i, i] = 1.0
    if i < 49:
        image[i, i + 1] = 0.8
        image[i + 1, i] = 0.8

print("Original image shape:", image.shape)

# Perform SVD
U, S, Vt = np.linalg.svd(image, full_matrices=False)

# Try different compression levels
for k in [5, 10, 20]:
    # Reconstruct with k components
    image_k = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]

    # Calculate error and compression
    error = np.linalg.norm(image - image_k)
    compression = (k * (image.shape[0] + image.shape[1])) / (
        image.shape[0] * image.shape[1]
    )

    print(f"\nk={k} singular values:")
    print(f"  Reconstruction error: {error:.4f}")
    print(f"  Storage needed: {compression:.2%} of original")
    print(f"  Variance explained: {np.sum(S[:k]**2) / np.sum(S**2):.2%}")
In [ ]
# SVD vs PCA relationship
# They're closely related! For centered data:

# Generate centered data
np.random.seed(42)
X_data = np.random.randn(100, 5)
X_centered = X_data - np.mean(X_data, axis=0)

# Method 1: PCA via eigendecomposition
cov_matrix = np.cov(X_centered.T)
eigenvalues_pca, eigenvectors_pca = np.linalg.eig(cov_matrix)
idx = eigenvalues_pca.argsort()[::-1]
eigenvalues_pca = eigenvalues_pca[idx]

# Method 2: PCA via SVD
U, S, Vt = np.linalg.svd(X_centered, full_matrices=False)
eigenvalues_svd = (S**2) / (X_centered.shape[0] - 1)

print("Eigenvalues from PCA:", eigenvalues_pca)
print("Eigenvalues from SVD:", eigenvalues_svd)
print("\nAre they equal?", np.allclose(eigenvalues_pca, eigenvalues_svd))

# The principal components are the same (up to sign)
print("\nV from SVD matches eigenvectors from covariance matrix:")
print("Close match?", np.allclose(np.abs(Vt.T), np.abs(eigenvectors_pca[:, idx])))

4. Practical Applications in AI

4.1 Data Representation

Vectors and matrices are fundamental for representing data in machine learning:

  • Vectors — Each data point (image pixels, word embeddings, sensor readings)
  • Matrices — Datasets where rows = samples, columns = features
  • Tensors — Higher-dimensional arrays (images, video, time series)

NumPy Implementation

In [ ]
# Dataset: 5 students with 3 test scores each
grades = np.array(
    [
        [85, 90, 88],  # Student 1
        [78, 85, 80],  # Student 2
        [92, 95, 93],  # Student 3
        [70, 75, 72],  # Student 4
        [88, 87, 90],
    ]
)  # Student 5

print("Dataset shape:", grades.shape)  # (5, 3)

# Calculate mean score for each student
mean_per_student = np.mean(grades, axis=1)
print("Mean per student:", mean_per_student)

# Calculate mean score for each test
mean_per_test = np.mean(grades, axis=0)
print("Mean per test:", mean_per_test)

4.2 Linear Transformations

Matrix multiplication applies geometric transformations to vectors:

  • Rotation — Rotate points/vectors
  • Scaling — Stretch/compress along axes
  • Reflection — Mirror across axes
  • Shear — Skew shapes

NumPy Implementation

In [ ]
# Rotation matrix (90 degrees counterclockwise)
theta = np.pi / 2  # 90 degrees in radians
rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])

# Original point
point = np.array([1, 0])

# Rotate the point
rotated_point = rotation @ point
print("Original point:", point)
print("Rotated point:", rotated_point)  # Should be close to [0, 1]

4.3 Solving Linear Systems

Systems of linear equations appear everywhere in ML/AI:

  • Linear regression — Fit line/plane to data
  • Neural networks — Each layer is a linear transformation
  • Optimization — Find parameters that minimize loss

Mathematical Form

System of equations:

{2x+3y=84x+y=10\begin{cases} 2x + 3y = 8 \\ 4x + y = 10 \end{cases}

Matrix form: Ax=b\mathbf{Ax} = \mathbf{b}

[2341][xy]=[810]\begin{bmatrix} 2 & 3 \\ 4 & 1 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} 8 \\ 10 \end{bmatrix}

NumPy Implementation

In [ ]
# Coefficient matrix
A = np.array([[2, 3], [4, 1]])

# Constants vector
b = np.array([8, 10])

# Solve for x
x = np.linalg.solve(A, b)
print("Solution:", x)  # x = [2, 2] means x=2, y=2

# Verify solution
print("Verification A×x =", A @ x)  # Should equal b

5. Quick Reference Cheat Sheet

Vector Operations

Operation Math Notation NumPy Code Note (Manual Calculation)
Create vector v=[1,2,3]\mathbf{v} = [1, 2, 3] v = np.array([1, 2, 3]) List of numbers
Vector addition a+b\mathbf{a} + \mathbf{b} a + b Add each component: [a1+b1,a2+b2,…][a_1+b_1, a_2+b_2, \ldots]
Scalar multiplication c⋅vc \cdot \mathbf{v} c * v Multiply each element: [c⋅v1,c⋅v2,…][c \cdot v_1, c \cdot v_2, \ldots]
Dot product a⋅b\mathbf{a} \cdot \mathbf{b} np.dot(a, b) or a @ b ∑i=1naibi\sum_{i=1}^{n} a_i b_i or np.sum(a * b)
Cross product (3D) a×b\mathbf{a} \times \mathbf{b} np.cross(a, b) [a2b3−a3b2,a3b1−a1b3,a1b2−a2b1][a_2b_3-a_3b_2, a_3b_1-a_1b_3, a_1b_2-a_2b_1]
Magnitude (norm) ∥v∥\|\mathbf{v}\| np.linalg.norm(v) ∑i=1nvi2\sqrt{\sum_{i=1}^{n} v_i^2} or np.sqrt(np.sum(v**2))
Normalize v^=v∥v∥\hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|} v / np.linalg.norm(v) Divide by magnitude
Projection projba\text{proj}_{\mathbf{b}}\mathbf{a} (a@b / b@b) * b a⋅bb⋅bb\frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}}\mathbf{b}

Matrix Operations

Operation Math Notation NumPy Code Note (Manual Calculation)
Create matrix A=[1234]\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} A = np.array([[1,2],[3,4]]) Nested lists (rows)
Identity matrix In\mathbf{I}_n np.eye(n) 1s on diagonal, 0s elsewhere
Zero matrix 0m×n\mathbf{0}_{m \times n} np.zeros((m, n)) All elements = 0
Matrix addition A+B\mathbf{A} + \mathbf{B} A + B Add element-wise: cij=aij+bijc_{ij} = a_{ij} + b_{ij}
Matrix multiplication AB\mathbf{AB} A @ B or np.dot(A, B) cij=∑k=1naikbkjc_{ij} = \sum_{k=1}^{n} a_{ik} b_{kj} (row × column)
Element-wise mult. A⊙B\mathbf{A} \odot \mathbf{B} A * B Multiply element-wise: cij=aij⋅bijc_{ij} = a_{ij} \cdot b_{ij}
Transpose AT\mathbf{A}^T A.T Flip over diagonal: (AT)ij=Aji(A^T)_{ij} = A_{ji}
Inverse A−1\mathbf{A}^{-1} np.linalg.inv(A) Satisfies AA−1=I\mathbf{AA}^{-1} = \mathbf{I}
Determinant (2×2) det(A)\text{det}(\mathbf{A}) np.linalg.det(A) ad−bcad - bc for [abcd]\begin{bmatrix} a & b \\ c & d \end{bmatrix}
Determinant (3×3) det(A)\text{det}(\mathbf{A}) np.linalg.det(A) a(ei−fh)−b(di−fg)+c(dh−eg)a(ei-fh) - b(di-fg) + c(dh-eg)
Rank rank(A)\text{rank}(\mathbf{A}) np.linalg.matrix_rank(A) Number of linearly independent rows/columns

Advanced Operations

Operation Math Notation NumPy Code Note (Manual Calculation)
Eigendecomposition Av=λv\mathbf{Av} = \lambda\mathbf{v} eigenval, eigenvec = np.linalg.eig(A) Solve det(A−λI)=0\text{det}(\mathbf{A} - \lambda\mathbf{I}) = 0
SVD A=UΣVT\mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T U, S, Vt = np.linalg.svd(A) Factorize any matrix into 3 matrices
Solve linear system Ax=b\mathbf{Ax} = \mathbf{b} x = np.linalg.solve(A, b) x=A−1b\mathbf{x} = \mathbf{A}^{-1}\mathbf{b} (but use solver, not inverse!)
Covariance matrix C=1n−1XTX\mathbf{C} = \frac{1}{n-1}\mathbf{X}^T\mathbf{X} np.cov(X.T) Measures feature correlations (center data first)
Trace tr(A)\text{tr}(\mathbf{A}) np.trace(A) Sum of diagonal elements: ∑i=1naii\sum_{i=1}^{n} a_{ii}
Frobenius norm ∥A∥F\|\mathbf{A}\|_F np.linalg.norm(A, 'fro') ∑i,jaij2\sqrt{\sum_{i,j} a_{ij}^2}

Matrix Creation

Function Description Example Note
np.array() Create from list np.array([[1,2],[3,4]]) Most basic method
np.eye(n) n×n identity matrix np.eye(3) Can also use np.identity(n)
np.zeros((m,n)) m×n matrix of zeros np.zeros((2,3)) Useful for initialization
np.ones((m,n)) m×n matrix of ones np.ones((2,3)) Scale with * value if needed
np.diag(v) Diagonal matrix from vector np.diag([1,2,3]) Creates square matrix
np.random.rand(m,n) Random values in [0,1) np.random.rand(2,3) Uniform distribution
np.random.randn(m,n) Random values from standard normal np.random.randn(2,3) Mean=0, std=1
np.arange(start, stop, step) Array with evenly spaced values np.arange(0, 10, 2) Like Python's range()
np.linspace(start, stop, n) n evenly spaced values np.linspace(0, 1, 5) Includes both endpoints

Additional Resources