IN400- Vectors and Matrices

ipynb37 kBPython · 45 cellsExtra material of IN400

Revision for IN400 - Applied Mathematics & Statistics for Computer Science

Vectors and Matrices: Theory and Practice with NumPy

This notebook covers the fundamental concepts of vectors and matrices, essential for AI and machine learning.

In [ ]
# Import NumPy - run this cell first!
import numpy as np

print("NumPy version:", np.__version__)

1. Vectors

A vector is an ordered collection of numbers (called components or elements) that can represent points in space, directions, magnitudes, or other quantities. Vectors are fundamental in AI for representing data, features, and transformations.

1.1 Vector Representation

Mathematical notation:

  • Column vector: v=[v1v2⋮vn]\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}
  • Row vector: vT=[v1,v2,…,vn]\mathbf{v}^T = [v_1, v_2, \ldots, v_n]

Example: In 2D space, v=[34]\mathbf{v} = \begin{bmatrix} 3 \\ 4 \end{bmatrix} represents a point at coordinates (3, 4) or a direction from the origin.

In [ ]
# Creating vectors
v1 = np.array([3, 4])  # 2D vector
v2 = np.array([1, 2, 3])  # 3D vector
v3 = np.array([1, 2, 3, 4, 5])  # 5D vector

print("v1:", v1)
print("v2:", v2)
print("Shape of v2:", v2.shape)  # (3,) means 1D array with 3 elements

1.2 Vector Operations

1.2.1 Vector Addition and Subtraction

Two vectors can be added/subtracted element-wise if they have the same dimension.

Mathematical definition: a+b=[a1a2⋮an]+[b1b2⋮bn]=[a1+b1a2+b2⋮an+bn]\mathbf{a} + \mathbf{b} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} + \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix} = \begin{bmatrix} a_1 + b_1 \\ a_2 + b_2 \\ \vdots \\ a_n + b_n \end{bmatrix}

Example: [123]+[456]=[579]\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} + \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = \begin{bmatrix} 5 \\ 7 \\ 9 \end{bmatrix}

In [ ]
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

addition = a + b
subtraction = a - b

print("a + b =", addition)  # [5 7 9]
print("a - b =", subtraction)  # [-3 -3 -3]
1.2.2 Scalar Multiplication

Multiplying a vector by a scalar (number) scales each component.

Mathematical definition: c⋅v=c⋅[v1v2⋮vn]=[c⋅v1c⋅v2⋮c⋅vn]c \cdot \mathbf{v} = c \cdot \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} = \begin{bmatrix} c \cdot v_1 \\ c \cdot v_2 \\ \vdots \\ c \cdot v_n \end{bmatrix}

Example: 3⋅[12]=[36]3 \cdot \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 3 \\ 6 \end{bmatrix}

In [ ]
v = np.array([1, 2, 3])
scalar = 3

result = scalar * v
print("3 * v =", result)  # [3 6 9]
1.2.3 Dot Product (Inner Product)

The dot product combines two vectors into a single number. It measures how much two vectors "align" with each other.

Mathematical definition: a⋅b=a1b1+a2b2+…+anbn=∑i=1naibi\mathbf{a} \cdot \mathbf{b} = a_1b_1 + a_2b_2 + \ldots + a_nb_n = \sum_{i=1}^{n} a_i b_i

Example: [123]⋅[456]=1(4)+2(5)+3(6)=4+10+18=32\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \cdot \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = 1(4) + 2(5) + 3(6) = 4 + 10 + 18 = 32

Properties:

  • If a⋅b=0\mathbf{a} \cdot \mathbf{b} = 0, the vectors are orthogonal (perpendicular)
  • If a⋅b>0\mathbf{a} \cdot \mathbf{b} > 0, they point in similar directions
  • If a⋅b<0\mathbf{a} \cdot \mathbf{b} < 0, they point in opposite directions
In [ ]
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Method 1: Using np.dot()
dot_product = np.dot(a, b)
print("a · b =", dot_product)  # 32

# Method 2: Using @ operator
dot_product = a @ b
print("a · b =", dot_product)  # 32

# Method 3: Element-wise multiply then sum
dot_product = np.sum(a * b)
print("a · b =", dot_product)  # 32
1.2.4 Vector Magnitude (Norm)

The magnitude (or length) of a vector is the distance from the origin to the point.

Mathematical definition: ∥v∥=v12+v22+…+vn2=v⋅v\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \ldots + v_n^2} = \sqrt{\mathbf{v} \cdot \mathbf{v}}

Example: ∥[34]∥=32+42=9+16=25=5\left\|\begin{bmatrix} 3 \\ 4 \end{bmatrix}\right\| = \sqrt{3^2 + 4^2} = \sqrt{9 + 16} = \sqrt{25} = 5

In [ ]
v = np.array([3, 4])

# Method 1: Using np.linalg.norm()
magnitude = np.linalg.norm(v)
print("||v|| =", magnitude)  # 5.0

# Method 2: Manual calculation
magnitude = np.sqrt(np.sum(v**2))
print("||v|| =", magnitude)  # 5.0
1.2.5 Unit Vector (Normalization)

A unit vector has magnitude 1 and indicates direction only.

Mathematical definition: v^=v∥v∥\hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|}

Example: v=[34],v^=15[34]=[0.60.8]\mathbf{v} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}, \quad \hat{\mathbf{v}} = \frac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix} = \begin{bmatrix} 0.6 \\ 0.8 \end{bmatrix}

In [ ]
v = np.array([3, 4])

# Normalize the vector
unit_vector = v / np.linalg.norm(v)
print("Unit vector:", unit_vector)  # [0.6 0.8]
print("Magnitude of unit vector:", np.linalg.norm(unit_vector))  # 1.0
1.2.6 Cross Product (Vector Product)

The cross product is an operation on two 3D vectors that produces a third vector perpendicular to both input vectors. It's only defined for 3D vectors.

Mathematical definition: a×b=[a2b3−a3b2a3b1−a1b3a1b2−a2b1]\mathbf{a} \times \mathbf{b} = \begin{bmatrix} a_2b_3 - a_3b_2 \\ a_3b_1 - a_1b_3 \\ a_1b_2 - a_2b_1 \end{bmatrix}

Properties:

  • The result is perpendicular to both input vectors
  • The magnitude is: ∥a×b∥=∥a∥∥b∥sin⁡(θ)\|\mathbf{a} \times \mathbf{b}\| = \|\mathbf{a}\| \|\mathbf{b}\| \sin(\theta), where θ\theta is the angle between vectors
  • Not commutative: a×b=−(b×a)\mathbf{a} \times \mathbf{b} = -(\mathbf{b} \times \mathbf{a})
  • If vectors are parallel: a×b=0\mathbf{a} \times \mathbf{b} = \mathbf{0}

Applications:

  • Finding normal vectors (computer graphics, physics)
  • Computing torque and angular momentum
  • Determining the orientation of surfaces

Example: [123]×[456]=[2(6)−3(5)3(4)−1(6)1(5)−2(4)]=[−36−3]\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \times \begin{bmatrix} 4 \\ 5 \\ 6 \end{bmatrix} = \begin{bmatrix} 2(6) - 3(5) \\ 3(4) - 1(6) \\ 1(5) - 2(4) \end{bmatrix} = \begin{bmatrix} -3 \\ 6 \\ -3 \end{bmatrix}

In [ ]
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])

# Cross product using np.cross()
cross_product = np.cross(a, b)
print("a × b =", cross_product)  # [-3  6 -3]

# Verify: the result is perpendicular to both a and b
# Dot product with perpendicular vectors should be 0
print("\n(a × b) · a =", np.dot(cross_product, a))  # Should be 0
print("(a × b) · b =", np.dot(cross_product, b))  # Should be 0

# Example: Find a vector perpendicular to the xy-plane
x_axis = np.array([1, 0, 0])
y_axis = np.array([0, 1, 0])
z_direction = np.cross(x_axis, y_axis)
print("\nx_axis × y_axis =", z_direction)  # [0 0 1] - points in z direction

# Example: Parallel vectors have zero cross product
parallel_a = np.array([2, 4, 6])
parallel_b = np.array([1, 2, 3])  # parallel_b = 0.5 * parallel_a
result = np.cross(parallel_b, parallel_a)
print("\nCross product of parallel vectors:", result)  # [0 0 0]
1.2.7 Vector Projection

Vector projection projects one vector onto another, finding the component of one vector in the direction of another.

Mathematical definition:

Projection of a\mathbf{a} onto b\mathbf{b}: projba=a⋅b∥b∥2b=a⋅bb⋅bb\text{proj}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{b}\|^2}\mathbf{b} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{b} \cdot \mathbf{b}}\mathbf{b}

The scalar projection (length of the projection): compba=a⋅b∥b∥\text{comp}_{\mathbf{b}}\mathbf{a} = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{b}\|}

Geometric interpretation:

  • The projection shows "how much" of vector a\mathbf{a} points in the direction of b\mathbf{b}
  • If perpendicular: projection = 0
  • If parallel: projection = a\mathbf{a} (or −a-\mathbf{a})

Applications:

  • Linear regression (projecting data onto a line/plane)
  • Component analysis
  • Shadow calculations in graphics
  • Signal decomposition

Example: a=[34],b=[10]\mathbf{a} = \begin{bmatrix} 3 \\ 4 \end{bmatrix}, \quad \mathbf{b} = \begin{bmatrix} 1 \\ 0 \end{bmatrix} projba=3(1)+4(0)12+02[10]=3[10]=[30]\text{proj}_{\mathbf{b}}\mathbf{a} = \frac{3(1) + 4(0)}{1^2 + 0^2}\begin{bmatrix} 1 \\ 0 \end{bmatrix} = 3\begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 3 \\ 0 \end{bmatrix}

In [ ]
# Example 1: Project vector a onto vector b
a = np.array([3, 4])
b = np.array([1, 0])

# Calculate projection
proj_b_a = (np.dot(a, b) / np.dot(b, b)) * b
print("Vector a:", a)
print("Vector b:", b)
print("Projection of a onto b:", proj_b_a)  # [3 0]

# Scalar projection (length)
scalar_proj = np.dot(a, b) / np.linalg.norm(b)
print("Scalar projection (length):", scalar_proj)  # 3.0

# Example 2: 3D projection
a_3d = np.array([1, 2, 3])
b_3d = np.array([1, 1, 0])

proj_3d = (np.dot(a_3d, b_3d) / np.dot(b_3d, b_3d)) * b_3d
print("\n3D Example:")
print("Vector a:", a_3d)
print("Vector b:", b_3d)
print("Projection of a onto b:", proj_3d)

# Example 3: Perpendicular vectors (projection = 0)
perpendicular_a = np.array([1, 0, 0])
perpendicular_b = np.array([0, 1, 0])

proj_perp = (
    np.dot(perpendicular_a, perpendicular_b) / np.dot(perpendicular_b, perpendicular_b)
) * perpendicular_b
print("\nPerpendicular vectors:")
print("Projection:", proj_perp)  # [0 0 0]

# Example 4: Decompose a vector into parallel and perpendicular components
a_decomp = np.array([3, 4])
b_decomp = np.array([1, 0])

parallel_component = (
    np.dot(a_decomp, b_decomp) / np.dot(b_decomp, b_decomp)
) * b_decomp
perpendicular_component = a_decomp - parallel_component

print("\nVector decomposition:")
print("Original vector:", a_decomp)
print("Parallel component:", parallel_component)
print("Perpendicular component:", perpendicular_component)
print("Sum (should equal original):", parallel_component + perpendicular_component)

2. Matrices

A matrix is a 2D array of numbers arranged in rows and columns. Matrices are essential in AI for representing datasets, transformations, and neural network weights.

2.1 Matrix Representation

Mathematical notation: A=[a11a12⋯a1na21a22⋯a2n⋮⋮⋱⋮am1am2⋯amn]\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1n} \\ a_{21} & a_{22} & \cdots & a_{2n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & \cdots & a_{mn} \end{bmatrix}

  • An m × n matrix has m rows and n columns
  • Element at row i, column j is denoted as aija_{ij} or A[i,j]A[i,j]

Example: A 2×3 matrix: A=[123456]\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}

In [ ]
# Creating matrices
A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3 matrix

B = np.array([[1, 2], [3, 4], [5, 6]])  # 3x2 matrix

print("Matrix A:")
print(A)
print("Shape:", A.shape)  # (2, 3) means 2 rows, 3 columns

# Identity matrix (diagonal of 1s)
I = np.eye(3)
print("\nIdentity matrix:")
print(I)

# Zero matrix
Z = np.zeros((2, 3))
print("\nZero matrix:")
print(Z)

# Random matrix
R = np.random.rand(2, 3)  # Random values between 0 and 1
print("\nRandom matrix:")
print(R)

2.2 Matrix Operations

2.2.1 Matrix Addition and Subtraction

Matrices of the same dimensions can be added/subtracted element-wise.

Mathematical definition: A+B=[a11+b11a12+b12a21+b21a22+b22]\mathbf{A} + \mathbf{B} = \begin{bmatrix} a_{11} + b_{11} & a_{12} + b_{12} \\ a_{21} + b_{21} & a_{22} + b_{22} \end{bmatrix}

Example: [1234]+[5678]=[681012]\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} + \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 6 & 8 \\ 10 & 12 \end{bmatrix}

In [ ]
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

addition = A + B
subtraction = A - B

print("A + B =")
print(addition)
# [[6  8]
#  [10 12]]

print("\nA - B =")
print(subtraction)
# [[-4 -4]
#  [-4 -4]]
2.2.2 Scalar Multiplication

Multiply each element by a scalar.

Mathematical definition: c⋅A=[c⋅a11c⋅a12c⋅a21c⋅a22]c \cdot \mathbf{A} = \begin{bmatrix} c \cdot a_{11} & c \cdot a_{12} \\ c \cdot a_{21} & c \cdot a_{22} \end{bmatrix}

In [ ]
A = np.array([[1, 2], [3, 4]])
result = 3 * A

print("3 * A =")
print(result)
# [[3  6]
#  [9 12]]
2.2.3 Matrix Multiplication

Key rule: To multiply A\mathbf{A} (m×n) by B\mathbf{B} (p×q), we need n = p. The result is an m×q matrix.

Mathematical definition: (AB)ij=∑k=1naikbkj(\mathbf{AB})_{ij} = \sum_{k=1}^{n} a_{ik} b_{kj}

The element at position (i,j) in the result is the dot product of row i of A with column j of B.

Example: [1234]×[5678]=[1(5)+2(7)1(6)+2(8)3(5)+4(7)3(6)+4(8)]=[19224350]\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \times \begin{bmatrix} 5 & 6 \\ 7 & 8 \end{bmatrix} = \begin{bmatrix} 1(5)+2(7) & 1(6)+2(8) \\ 3(5)+4(7) & 3(6)+4(8) \end{bmatrix} = \begin{bmatrix} 19 & 22 \\ 43 & 50 \end{bmatrix}

Important: Matrix multiplication is not commutative: AB≠BA\mathbf{AB} \neq \mathbf{BA}

In [ ]
A = np.array([[1, 2], [3, 4]])  # 2x2

B = np.array([[5, 6], [7, 8]])  # 2x2

# Matrix multiplication
C = np.dot(A, B)
# Or using @ operator
C = A @ B

print("A × B =")
print(C)
# [[19 22]
#  [43 50]]

# Matrix-vector multiplication
v = np.array([1, 2])
result = A @ v

print("\nA × v =")
print(result)  # [5 11]
2.2.4 Matrix Transpose

Transpose flips a matrix over its diagonal, converting rows to columns.

Mathematical definition: (AT)ij=Aji(\mathbf{A}^T)_{ij} = A_{ji}

Example: A=[123456],AT=[142536]\mathbf{A} = \begin{bmatrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{bmatrix}, \quad \mathbf{A}^T = \begin{bmatrix} 1 & 4 \\ 2 & 5 \\ 3 & 6 \end{bmatrix}

In [ ]
A = np.array([[1, 2, 3], [4, 5, 6]])  # 2x3

A_transpose = A.T

print("A =")
print(A)
print("\nA^T =")
print(A_transpose)  # 3x2 matrix
2.2.5 Matrix Inverse

The inverse of a square matrix A\mathbf{A} is denoted A−1\mathbf{A}^{-1} and satisfies: AA−1=A−1A=I\mathbf{A} \mathbf{A}^{-1} = \mathbf{A}^{-1} \mathbf{A} = \mathbf{I}

Important: Not all matrices have an inverse! A matrix is invertible if its determinant is non-zero.

Example: A=[4726],A−1=[0.6−0.7−0.20.4]\mathbf{A} = \begin{bmatrix} 4 & 7 \\ 2 & 6 \end{bmatrix}, \quad \mathbf{A}^{-1} = \begin{bmatrix} 0.6 & -0.7 \\ -0.2 & 0.4 \end{bmatrix}

In [ ]
A = np.array([[4, 7], [2, 6]])

# Calculate inverse
A_inv = np.linalg.inv(A)

print("A =")
print(A)
print("\nA^(-1) =")
print(A_inv)

# Verify: A × A^(-1) = I
identity = A @ A_inv
print("\nA × A^(-1) =")
print(identity)  # Should be close to [[1, 0], [0, 1]]
2.2.6 Matrix Determinant

The determinant is a scalar value that can be computed from a square matrix. It provides important information about the matrix:

  • If det(A) ≠ 0, the matrix is invertible
  • If det(A) = 0, the matrix is singular (not invertible)
  • The determinant represents the volume scaling factor of the linear transformation

Mathematical definition:

For a 2×2 matrix: det[abcd]=ad−bc\text{det}\begin{bmatrix} a & b \\ c & d \end{bmatrix} = ad - bc

For a 3×3 matrix: det[abcdefghi]=a(ei−fh)−b(di−fg)+c(dh−eg)\text{det}\begin{bmatrix} a & b & c \\ d & e & f \\ g & h & i \end{bmatrix} = a(ei - fh) - b(di - fg) + c(dh - eg)

Examples:

2×2: det[3846]=3(6)−8(4)=18−32=−14\text{det}\begin{bmatrix} 3 & 8 \\ 4 & 6 \end{bmatrix} = 3(6) - 8(4) = 18 - 32 = -14

3×3: det[123014560]=1(1⋅0−4⋅6)−2(0⋅0−4⋅5)+3(0⋅6−1⋅5)=1(−24)−2(−20)+3(−5)=−24+40−15=1\text{det}\begin{bmatrix} 1 & 2 & 3 \\ 0 & 1 & 4 \\ 5 & 6 & 0 \end{bmatrix} = 1(1 \cdot 0 - 4 \cdot 6) - 2(0 \cdot 0 - 4 \cdot 5) + 3(0 \cdot 6 - 1 \cdot 5) = 1(-24) - 2(-20) + 3(-5) = -24 + 40 - 15 = 1

In [ ]
# Example 1: Determinant of a 2x2 matrix
A_2x2 = np.array([[3, 8], [4, 6]])

det_A = np.linalg.det(A_2x2)
print("Matrix A (2x2):")
print(A_2x2)
print("Determinant of A:", det_A)  # -14.0

# Example 2: Determinant of a 3x3 matrix
A_3x3 = np.array([[1, 2, 3], [0, 1, 4], [5, 6, 0]])

det_B = np.linalg.det(A_3x3)
print("\nMatrix B (3x3):")
print(A_3x3)
print("Determinant of B:", det_B)  # 1.0

# Example 3: Check if a matrix is invertible
singular_matrix = np.array([[2, 4], [1, 2]])

det_singular = np.linalg.det(singular_matrix)
print("\nSingular Matrix:")
print(singular_matrix)
print("Determinant:", det_singular)  # 0.0 (or very close to 0)

if abs(det_singular) < 1e-10:
    print("This matrix is SINGULAR (not invertible)")
else:
    print("This matrix is INVERTIBLE")

# Example 4: Manual calculation for 2x2 (to verify)
a, b = 3, 8
c, d = 4, 6
manual_det = a * d - b * c
print(f"\nManual calculation: {a}*{d} - {b}*{c} = {manual_det}")
2.2.7 Eigenvalues and Eigenvectors

Eigenvalues and eigenvectors are fundamental concepts in linear algebra. For a square matrix A\mathbf{A}, an eigenvector v\mathbf{v} and its corresponding eigenvalue λ\lambda satisfy:

Av=λv\mathbf{A}\mathbf{v} = \lambda\mathbf{v}

Interpretation:

  • When matrix A\mathbf{A} is applied to eigenvector v\mathbf{v}, the result is the same vector scaled by λ\lambda
  • The eigenvector's direction is preserved, only the magnitude changes
  • λ\lambda (eigenvalue) is the scaling factor

Finding eigenvalues: Solve the characteristic equation: det(A−λI)=0\text{det}(\mathbf{A} - \lambda\mathbf{I}) = 0

Properties:

  • An n×n matrix has n eigenvalues (counting multiplicities)
  • Eigenvectors corresponding to different eigenvalues are linearly independent
  • For symmetric matrices, eigenvalues are real and eigenvectors are orthogonal

Applications in AI/ML:

  • Principal Component Analysis (PCA): Dimensionality reduction
  • Google PageRank: Website ranking algorithm
  • Markov Chains: State transitions and steady states
  • Image Compression: Spectral decomposition
  • Stability Analysis: Neural network dynamics
  • Graph Analysis: Community detection

Example: For matrix A=[4213]\mathbf{A} = \begin{bmatrix} 4 & 2 \\ 1 & 3 \end{bmatrix}:

  • Eigenvalues: λ1=5,λ2=2\lambda_1 = 5, \lambda_2 = 2
  • Eigenvector for λ1=5\lambda_1 = 5: v1=[21]\mathbf{v}_1 = \begin{bmatrix} 2 \\ 1 \end{bmatrix}
  • Verification: [4213][21]=[105]=5[21]\begin{bmatrix} 4 & 2 \\ 1 & 3 \end{bmatrix}\begin{bmatrix} 2 \\ 1 \end{bmatrix} = \begin{bmatrix} 10 \\ 5 \end{bmatrix} = 5\begin{bmatrix} 2 \\ 1 \end{bmatrix} ✓
In [ ]
# Example 1: Basic eigenvalue and eigenvector calculation
A = np.array([[4, 2], [1, 3]])

# Calculate eigenvalues and eigenvectors
eigenvalues, eigenvectors = np.linalg.eig(A)

print("Matrix A:")
print(A)
print("\nEigenvalues:", eigenvalues)
print("\nEigenvectors:")
print(eigenvectors)

# Verify: A @ v = λ * v for the first eigenvector
v1 = eigenvectors[:, 0]  # First eigenvector (column 0)
lambda1 = eigenvalues[0]

left_side = A @ v1
right_side = lambda1 * v1

print(f"\nVerification for λ₁ = {lambda1:.4f}:")
print(f"A @ v₁ = {left_side}")
print(f"λ₁ * v₁ = {right_side}")
print(f"Equal? {np.allclose(left_side, right_side)}")

# Example 2: Symmetric matrix (real eigenvalues, orthogonal eigenvectors)
symmetric_matrix = np.array([[2, 1], [1, 2]])

eig_vals, eig_vecs = np.linalg.eig(symmetric_matrix)

print("\n\nSymmetric Matrix:")
print(symmetric_matrix)
print("\nEigenvalues:", eig_vals)
print("\nEigenvectors:")
print(eig_vecs)

# Check orthogonality: dot product should be 0
dot_product = np.dot(eig_vecs[:, 0], eig_vecs[:, 1])
print(f"\nDot product of eigenvectors: {dot_product:.10f} (should be ~0)")

# Example 3: 3x3 matrix
A_3x3 = np.array([[6, -1, 0], [-1, 6, -1], [0, -1, 6]])

eig_vals_3d, eig_vecs_3d = np.linalg.eig(A_3x3)

print("\n\n3x3 Matrix:")
print(A_3x3)
print("\nEigenvalues:", eig_vals_3d)


# Example 4: Application - Power iteration to find dominant eigenvalue
def power_iteration(A, num_iterations=10):
    """Find the dominant eigenvalue and eigenvector using power iteration"""
    n = A.shape[0]
    v = np.random.rand(n)  # Random initial vector

    for i in range(num_iterations):
        # Multiply by matrix
        v = A @ v
        # Normalize
        v = v / np.linalg.norm(v)

    # Calculate eigenvalue: λ = (v^T A v) / (v^T v)
    eigenvalue = (v.T @ A @ v) / (v.T @ v)
    return eigenvalue, v


dominant_eval, dominant_evec = power_iteration(A, num_iterations=20)
print(f"\n\nPower Iteration (finds dominant eigenvalue):")
print(f"Dominant eigenvalue: {dominant_eval:.4f}")
print(f"Compare with np.linalg.eig: {max(eigenvalues):.4f}")

# Example 5: Trace and determinant relation to eigenvalues
print(f"\n\nTrace-Eigenvalue relation:")
print(f"Trace of A: {np.trace(A):.4f}")
print(f"Sum of eigenvalues: {np.sum(eigenvalues):.4f}")
print(f"Determinant of A: {np.linalg.det(A):.4f}")
print(f"Product of eigenvalues: {np.prod(eigenvalues):.4f}")
2.2.8 Element-wise Operations

Sometimes we need element-wise operations (Hadamard product), not matrix multiplication.

In [ ]
A = np.array([[1, 2], [3, 4]])
B = np.array([[5, 6], [7, 8]])

# Element-wise multiplication
element_wise = A * B
print("A ⊙ B (element-wise) =")
print(element_wise)
# [[5  12]
#  [21 32]]

# Element-wise division
element_div = A / B
print("\nA ⊘ B (element-wise) =")
print(element_div)

# Element-wise power
powered = A**2
print("\nA^2 (element-wise) =")
print(powered)
# [[1  4]
#  [9 16]]

2.3 Special Matrices

In [ ]
# Identity matrix (3x3)
I = np.eye(3)
print("Identity matrix:")
print(I)

# Diagonal matrix
diag = np.diag([1, 2, 3])
print("\nDiagonal matrix:")
print(diag)

# Upper triangular
upper = np.triu(np.ones((3, 3)))
print("\nUpper triangular:")
print(upper)

# Lower triangular
lower = np.tril(np.ones((3, 3)))
print("\nLower triangular:")
print(lower)

3. Practical Applications in AI

3.1 Data Representation

  • Vectors: Each data point (e.g., image, text) is represented as a vector
  • Matrices: A dataset is a matrix where each row is a data sample
In [ ]
# Dataset: 5 students with 3 test scores each
grades = np.array(
    [
        [85, 90, 88],  # Student 1
        [78, 85, 80],  # Student 2
        [92, 95, 93],  # Student 3
        [70, 75, 72],  # Student 4
        [88, 87, 90],
    ]
)  # Student 5

print("Dataset shape:", grades.shape)  # (5, 3)

# Calculate mean score for each student
mean_per_student = np.mean(grades, axis=1)
print("Mean per student:", mean_per_student)

# Calculate mean score for each test
mean_per_test = np.mean(grades, axis=0)
print("Mean per test:", mean_per_test)

3.2 Linear Transformations

Matrix multiplication applies transformations to vectors.

In [ ]
# Rotation matrix (90 degrees counterclockwise)
theta = np.pi / 2  # 90 degrees in radians
rotation = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])

# Original point
point = np.array([1, 0])

# Rotate the point
rotated_point = rotation @ point
print("Original point:", point)
print("Rotated point:", rotated_point)  # Should be close to [0, 1]

3.3 Solving Linear Systems

Systems of linear equations can be solved using matrices.

System: 2x+3y=82x + 3y = 8 4x+y=104x + y = 10

Matrix form: Ax=b\mathbf{Ax} = \mathbf{b}

In [ ]
# Coefficient matrix
A = np.array([[2, 3], [4, 1]])

# Constants vector
b = np.array([8, 10])

# Solve for x
x = np.linalg.solve(A, b)
print("Solution:", x)  # x = [2, 2] means x=2, y=2

# Verify solution
print("Verification A×x =", A @ x)  # Should equal b

4. Summary Cheat Sheet

Operation Math Notation NumPy Code
Create vector v=[1,2,3]\mathbf{v} = [1, 2, 3] v = np.array([1, 2, 3])
Create matrix A=[1234]\mathbf{A} = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} A = np.array([[1,2],[3,4]])
Vector addition a+b\mathbf{a} + \mathbf{b} a + b
Scalar multiplication c⋅vc \cdot \mathbf{v} c * v
Dot product a⋅b\mathbf{a} \cdot \mathbf{b} np.dot(a, b) or a @ b
Cross product (3D) a×b\mathbf{a} \times \mathbf{b} np.cross(a, b)
Vector magnitude ∥v∥\|\mathbf{v}\| np.linalg.norm(v)
Vector projection projba\text{proj}_{\mathbf{b}}\mathbf{a} (a@b / b@b) * b
Matrix multiplication AB\mathbf{AB} A @ B or np.dot(A, B)
Element-wise mult. A⊙B\mathbf{A} \odot \mathbf{B} A * B
Transpose AT\mathbf{A}^T A.T
Determinant det(A)\text{det}(\mathbf{A}) or ∥A∥\|\mathbf{A}\| np.linalg.det(A)
Eigenvalues/vectors Av=λv\mathbf{Av} = \lambda\mathbf{v} np.linalg.eig(A)
Inverse A−1\mathbf{A}^{-1} np.linalg.inv(A)
Identity matrix In\mathbf{I}_n np.eye(n)