Gaussian Lecture 07 Practice
Two numbers. One hill.A carry's farm

Gaussian

N(x | μ, Σ) — the multivariate oneStatistical Learning · Lecture 7 · 90 min
2025 Final, Task 1(b) asks for it

Gold and XP rise together. One formula holds both numbers at once. You will write it on a blank page.

Start with the toolbox Open the lab Cheat sheet
By the end you canno prior knowledge needed
  • Write N(x | μ, Σ) from memory and say what each piece does.
  • Compute |Σ|, Σ−1, the distance (x − μ)T Σ−1 (x − μ) and the density by hand.
  • Look at a Σ and draw the shape of its hill. Round, stretched or tilted.
  • Prove that a diagonal Σ means independent features.
  • Get μ̂ and Σ̂ from data, and solve the 2025 final tasks.
Time neededabout 90 minutes
30minutes on the toolbox and the story
35minutes on the formula, shape and lab
25minutes on exam tasks and practice

Every part ends with a line to remember. Read it, cover it, say it out loud.

↓Route15 stops

Everything in order.

01 ToolboxGaussian · 01/15

Ten tools. From zero.

The formula looks scary because it uses matrix notation. Each tool below is short, and each one comes with a tiny try-it. Press it to see the answer. Do the sum on paper first.

xTool 1Vector

A vector is a list of numbers written as a column. One game of your carry is two numbers, so one game is a vector with two entries. x1 is the GPM (gold per minute). x2 is the XPM (experience per minute).

x = x1x2 = 600700

We write x ∈ Rd. This says: x holds d real numbers. Here d = 2. To add or subtract vectors, work entry by entry.

Try itMean game μ is 550650. What is x − μ for the game above?
600700 − 550650 = 5050. This game beat the average by 50 in both numbers.
TTool 2Transpose

The small T means transpose. It turns rows into columns. A column vector becomes a row vector. A matrix flips across its main diagonal, which is the line from top left to bottom right.

5050T = 5050     abcdT = acbd

A matrix that stays the same after transposing is called symmetric. Then top right equals bottom left. Σ is always symmetric.

Try itTranspose 1234. Is it symmetric?
1324. It changed, so the original is not symmetric. The entries 2 and 3 would have to match.
×Tool 3Matrix multiplication

A matrix is a table of numbers. Multiply with one rule: each row of the left one, against each column of the right one. Pair entries, multiply the pairs, add them up.

2113 12 = 2·1 + 1·21·1 + 3·2 = 47

Three shapes matter today:

ShapeExampleResult
row × column12 34 = 1·3 + 2·4one number: 11
matrix × column2113 12a column
column × row12 34 = 3468a matrix (used in MLE)

Order matters. AB and BA can differ.

Try itMultiply 2113 by itself.
Row 1 with column 1: 2·2 + 1·1 = 5. Row 1 with column 2: 2·1 + 1·3 = 5. Row 2 with column 1: 1·2 + 3·1 = 5. Row 2 with column 2: 1·1 + 3·3 = 10. Result: 55510.
| |Tool 4Determinant of a 2×2

The determinant squeezes a square matrix into one number. Straight bars around a matrix mean determinant.

|A| = |abcd| = ad − bc

Main diagonal product, minus the other diagonal product. It measures how much the matrix stretches area. A determinant of 3 means areas grow 3 times. A determinant of 0 means everything collapses onto a line.

Try itFind |2112|. This is the Σ from the course notes.
2·2 − 1·1 = 3.
⁻¹Tool 5Inverse of a 2×2

The identity I is the matrix version of the number 1: 1001. The inverse A−1 undoes A, so AA−1 = I. The 2025 final prints this formula. Know it anyway.

A−1 = 1ad − bc d−b−ca

Three moves. Swap the two diagonal entries. Flip the sign of the other two. Divide everything by the determinant. A determinant of 0 means no inverse exists.

Diagonal matrix shortcut: a00d−1 = 1a001d.

Try itInvert 2111. Then multiply to check.
Determinant: 2·1 − 1·1 = 1. Swap and flip: 1−1−12. Check: 2111 1−1−12 = 2−1−2+21−1−1+2 = 1001. Correct.
QTool 6Quadratic form

Sandwich a matrix between a vector: vT A v. A row, a matrix and a column multiply down to one number. This is a quadratic form, because each term has two entries of v multiplied together.

StepWhy
1vT A v = v1v2 abcd v1v2Write it out.
2= v1v2 av1 + bv2cv1 + dv2Do matrix × column first.
3= av12 + (b + c)v1v2 + dv22Row × column gives one number.

A symmetric matrix has b = c, so the middle term becomes 2bv1v2. If this number is above 0 for every non-zero v, the matrix is positive definite. The notes require this of Σ.

Try itv = 11 and A = 2−1−12. Find vT A v.
2·1 + 2·(−1)·1·1 + 2·1 = 2 − 2 + 2 = 2.
σTool 7Variance, covariance, correlation

A deviation is value minus mean. Then:

Variance σ2Average of squared deviations. How wide one feature spreads.
Standard deviation σSquare root of the variance. Same units as the data.
CovarianceAverage of (deviation 1 × deviation 2). Positive: they rise together. Negative: one rises while the other falls.
Correlation ρCovariance divided by σ1σ2. Always between −1 and 1. It has no units.

The covariance matrix Σ holds all of them. This is how the slides write it for two features. On the diagonal you read the variances. Off the diagonal you read the covariance.

Σ = σ12ρσ1σ2ρσ1σ2σ22
Try itΣ = 4229. Find σ1, σ2 and ρ.
σ1 = √4 = 2. σ2 = √9 = 3. ρ = 2 / (2·3) = 1/3.
eTool 8Powers, roots, exp, log
a1/2 = √aPower one half is a square root.
(2π)d/2With d = 2 this is just 2π. With d = 3 it is about 15.75.
exp(t) = ete ≈ 2.718. exp(0) = 1. exp of a big negative number is close to 0.
exp(a + b) = exp(a) exp(b)A sum in the exponent splits into a product.
log(ab) = log a + log bLog turns products into sums. Also log exp(t) = t and log ak = k log a.
Try itUse a calculator: exp(−0.78125).
About 0.4578. You will need this number in part 04.
μTool 9The 1D Gaussian and density

One feature, like last hits at 10 minutes, can follow a bell curve. The notes write it like this:

p(x | μ, σ2) = 1√(2πσ2) exp(−12σ2 (x − μ)2)

A density is the height of the curve. It is a height, so it can be above 1 when the curve is narrow. Probabilities come from area under the curve, and the total area is 1. In two dimensions the curve becomes a hill, and the total volume under it is 1.

Try itHeight of the curve at x = μ when σ = 1?
exp(0) = 1, so the height is 1/√(2π) ≈ 0.399.
∂Tool 10Tools for the MLE part

Careful with two signs. The big sum sign ∑ means add up. The letter Σ alone is the covariance matrix. They look alike. Context tells you: a sum always has limits like ∑i=1N.

The trace tr(A) adds the diagonal entries. Two facts: a single number equals its own trace, and tr(ABC) = tr(CAB), so you can rotate the order.

A derivative by a vector collects the derivatives by each entry. The notes use these three rules, for symmetric A:

∂/∂a  (x − a)T A (x − a) = −2A(x − a)
∂/∂A  log |A| = A−T
∂/∂A  tr(BA) = BT

The first rule looks like the 1D rule: the derivative of A(x − a)2 by a is −2A(x − a).

Try itFind the trace of 1600160016002500.
1600 + 2500 = 4100.

Remember itDet is ad minus bc. Inverse: swap, flip, divide. A row times a matrix times a column is one number.

02 StoryGaussian · 02/15

Four games. One carry.

You log your carry's farm

Every game ends with two numbers on the scoreboard. GPM, gold per minute. XPM, experience per minute.

Last hits give both. Kills give both. So a game with fat gold usually has fat XP too.

GPM, XPM
1 / 3Safe lane
One bell curve per number misses this

You could model GPM alone and XPM alone. Two separate bells. But then you lose the link between them.

You want one model for the pair. That model is the multivariate Gaussian, also called MVN, the Multivariate Normal.

One hill
2 / 3Over two axes
The question for this lecture

Game A: 600 GPM, 700 XPM.

Game B: 600 GPM, 600 XPM. Gold came from bounty runes and a Midas, and XP stayed low.

Which game is stranger for this carry? By the end, you will know exactly how many times stranger.

A or B?
3 / 3Answer in part 05

Step 1. The log.

GameGPM x1XPM x2
1510580
2510640
3590660
4590720
Mean2200 / 4 = 5502600 / 4 = 650
Four dots, one mean. The dots climb from bottom left to top right. That climb is what covariance measures.

The mean is a vector: μ = 550650. Four games is a very small log. It keeps the sums easy to follow.

Step 2. Deviations.

Subtract the mean from each game. Then square, and multiply the pair.

GameGPM − 550XPM − 650GPM dev²XPM dev²GPM dev × XPM dev
1−40−70160049002800
2−40−101600100400
340101600100400
44070160049002800
Sum006400100006400
Sum ÷ 4160025001600

Each product is positive. When GPM is below its mean, XPM is below its mean too. So covariance comes out positive.

Step 3. Build Σ.

StepWhy
1σ12 = 1600, so σ1 = 40GPM spreads by about 40.
2σ22 = 2500, so σ2 = 50XPM spreads by about 50.
3covariance = 1600The last column of the table.
4ρ = 160040 · 50 = 16002000 = 0.8Divide covariance by both standard deviations. Strong link.
5Σ = 1600160016002500Variances on the diagonal, covariance off it.

Our model from here on

x ∼ N(μ, Σ),   μ = 550650,   Σ = 1600160016002500

The sign ∼ reads "is distributed as". We use these numbers for the rest of the lecture.

Remember itThe mean says where the hill stands. Variances and covariance fill Σ, which says how the hill is shaped.

03 The formulaGaussian · 03/15

Five pieces. Learn them in order.

N(x | μ, Σ) = p(x | μ, Σ) = 1(2π)d/2 |Σ|1/2 exp(−12 (x − μ)T Σ−1 (x − μ))
SymbolMeaningCarry example
x ∈ RdThe point we ask about600700
dNumber of features2
μ ∈ RdThe mean vector550650
Σ ∈ Rd×dThe covariance matrix, symmetric and positive definite1600160016002500
|Σ|Determinant of Σ1,440,000
Σ−1Inverse of Σpart 04

Press through it. One piece at a time.

1(2π)d/2 |Σ|1/2 exp(−12 (x − μ)T Σ−1 (x − μ))

Where it comes from. Multiply two bells.

Start with the easy case. Suppose GPM and XPM had nothing to do with each other. Then the joint density is the product of two 1D Gaussians (Tool 9). Watch the matrix formula appear.

StepWhy
1p(x) = p(x1) p(x2)Independent features multiply.
2= 1√(2πσ12) exp(−(x1 − μ1)22σ12) · 1√(2πσ22) exp(−(x2 − μ2)22σ22)Write each 1D density in full.
3= 12π √(σ12σ22) exp(−12 [ (x1 − μ1)2σ12 + (x2 − μ2)2σ22 ])Roots multiply. exp(a) exp(b) = exp(a + b). Pull out the one half.
4Let Σ = σ1200σ22. Then |Σ| = σ12σ22 and Σ−1 = 1/σ12001/σ22Tool 4 and the diagonal shortcut of Tool 5.
5(x − μ)T Σ−1 (x − μ) = (x1 − μ1)2σ12 + (x2 − μ2)2σ22Tool 6 with b = c = 0. This is the square bracket from line 3.
6p(x) = 1(2π)2/2 |Σ|1/2 exp(−12 (x − μ)T Σ−1 (x − μ))That is the formula with d = 2.

The same multiplication with d features gives d factors of √(2π), which is (2π)d/2. Now allow Σ to have off-diagonal entries. The formula stays the same, and the off-diagonal entries tilt the hill. Part 06 shows this.

Check with 1D. Put d = 1 and Σ = σ2. Then (2π)1/2 (σ2)1/2 = √(2πσ2), and the sandwich becomes (x − μ)2 / σ2. You get Tool 9 back.

Remember itOne over two-pi to the d over two, times root of det. Then exp of minus one half, gap, inverse, gap.

04 By handGaussian · 04/15

Five steps. Any exam.

  • 1Determinant |Σ|.
  • 2Inverse Σ−1: swap, flip, divide.
  • 3Gap v = (x − μ).
  • 4Middle first: Σ−1 v. Then vT times that. You now have D2.
  • 5Density: exp(−D2/2) divided by (2π)d/2 |Σ|1/2.

Game A. 600 GPM, 700 XPM.

StepWhy
1|Σ| = 1600 · 2500 − 1600 · 1600 = 4,000,000 − 2,560,000 = 1,440,000Step 1. ad − bc.
2|Σ|1/2 = √1,440,000 = 1200We need the root in step 5.
3Σ−1 = 11,440,000 2500−1600−16001600Step 2. Swap 1600 and 2500, flip the signs of the two 1600s, divide by the determinant.
4v = x − μ = 600 − 550700 − 650 = 5050Step 3. Entry by entry.
5Σ−1 v = 11,440,000 2500·50 − 1600·50−1600·50 + 1600·50 = 11,440,000 450000 = 0.031250Step 4a. Matrix × column.
6vT Σ−1 v = 5050 0.031250 = 50 · 0.03125 + 50 · 0 = 1.5625Step 4b. Row × column gives one number.
7D2 = 1.5625This is the sandwich.
8exp(−D2/2) = exp(−0.78125) ≈ 0.4578Step 5a. The height, as a share of the top.
9(2π)2/2 |Σ|1/2 = 2π · 1200 ≈ 7539.8Step 5b. With d = 2 the first factor is 2π.
10N(x | μ, Σ) ≈ 0.4578 / 7539.8 ≈ 6.07 × 10−5Step 5c. Tiny, because the hill spreads over thousands of GPM·XPM squares.

Why the density is so small

Volume must equal 1. This hill covers a footprint of hundreds of GPM by hundreds of XPM. So its height is small. The top of the hill is 1/7539.8 ≈ 1.33 × 10−4. Only ratios of heights carry meaning here.

The example from the course notes. Same steps.

The notes evaluate a density at x = 23 with μ = 12 and Σ = 2112.

StepWhy
1|Σ| = 2·2 − 1·1 = 3Determinant.
2Σ−1 = 13 2−1−12Swap, flip, divide.
3v = 2 − 13 − 2 = 11Gap.
4Σ−1 v = 13 2 − 1−1 + 2 = 13 11Matrix × column.
5vT Σ−1 v = 11 · 13 11 = 23Row × column.
6p = 12π √3 exp(−13) ≈ 0.0658One half of 2/3 is 1/3. The notes print ≈ 0.065. The exact value rounds to 0.066.

Remember itMultiply the middle first. Your vector turns into one number before you touch exp.

05 DistanceGaussian · 05/15

Same ruler distance. Very different games.

The sandwich has a name. The notes call (x − μ)T Σ−1 (x − μ) the Mahalanobis distance term. The strict distance is its square root. On this page D2 = (x − μ)T Σ−1 (x − μ) and D = √D2.

Now the question from the story. Both games lie the same ruler distance from the mean: √(50² + 50²) ≈ 70.7. Game B needs its own calculation.

Game B. 600 GPM, 600 XPM.

StepWhy
1v = 600 − 550600 − 650 = 50−50The gap. XPM is below average.
2Σ−1 v = 11,440,000 2500·50 + 1600·50−1600·50 − 1600·50 = 11,440,000 205000−160000Same inverse as Game A. Watch the signs: (−1600)(−50) = +80000.
3vT Σ−1 v = 50·205000 + 50·1600001,440,000 = 18,250,0001,440,000 ≈ 12.674(−50)(−160000) is positive too.
4D2 ≈ 12.674, so D ≈ 3.56Game A had D = 1.25.
5p(Game B) ≈ exp(−6.337) / 7539.8 ≈ 2.35 × 10−7Same front constant as Game A.
6p(Game A)p(Game B) = exp(12.674 − 1.56252) ≈ 259The constants cancel. Only the exponents matter.
The dashed circle is the ruler: every point on it is 70.7 from the centre. The rings show D = 1, 2 and 3. Game A lies along the tilt, inside ring 2. Game B cuts across the tilt, outside ring 3.

The answer

Game B is about 259 times less dense than Game A. Gold without XP goes against how this carry farms. The ruler cannot see that. Mahalanobis distance can, because it uses Σ.

Open it up. The 2D formula.

Put Σ in its σ and ρ form from Tool 7 and do the algebra. Call the z-score z1 = (x1 − μ1)/σ1. It counts how many standard deviations you are from the mean. Same for z2.

StepWhy
1|Σ| = σ12σ22 − (ρσ1σ2)2ad − bc.
2= σ12σ22 (1 − ρ2)Factor out σ12σ22.
3Σ−1 = 1σ12σ22(1 − ρ2) σ22−ρσ1σ2−ρσ1σ2σ12Swap, flip, divide.
4= 11 − ρ2 1/σ12−ρ/(σ1σ2)−ρ/(σ1σ2)1/σ22Divide each entry by σ12σ22.
5Let a = x1 − μ1 and b = x2 − μ2. Then D2 = 11 − ρ2 ( a2σ12 − 2ρabσ1σ2 + b2σ22 )Tool 6 with a symmetric matrix.
6D2 = z12 − 2ρz1z2 + z221 − ρ2Substitute z-scores.
p(x1, x2) = 12π σ1σ2 √(1 − ρ2) exp(− z12 − 2ρz1z2 + z222(1 − ρ2))
StepWhy
1Game A: z1 = 50/40 = 1.25, z2 = 50/50 = 1Both positive.
2D2 = (1.5625 − 2·0.8·1.25·1 + 1) / (1 − 0.64) = 0.5625 / 0.36 = 1.5625Same as part 04. The cross term helps.
3Game B: z1 = 1.25, z2 = −1Opposite signs.
4D2 = (1.5625 + 2 + 1) / 0.36 = 4.5625 / 0.36 ≈ 12.674Same as above. The cross term hurts.

Read the cross term −2ρz1z2. With ρ > 0, deviations of the same sign reduce the distance. Deviations of opposite signs raise it. That is the whole story of Game A against Game B.

Remember itMahalanobis distance is distance measured in the hill's own units. Along the tilt is cheap. Across the tilt is expensive.

06 ShapeGaussian · 06/15

Σ draws the hill.

A contour is a line through all points of equal height. Height depends on x only through D2, so a contour is the set of points where D2 equals one fixed number. These sets are ellipses centred at μ. The figures in the notes show the first and third case below.

CircleΣ = 1001Equal spread, ρ = 0. This is the identity, called spherical. Features are independent.
Flat ellipseΣ = 3000.5Unequal spread, ρ = 0. Axes lie along x1 and x2. Still independent.
Tilted upΣ = 10.80.81ρ = 0.8. Long axis runs up to the right. They rise together, like GPM and XPM.
Tilted downΣ = 1−0.8−0.81ρ = −0.8. Long axis runs down to the right. Like your farm against the enemy carry's farm.

Why ellipses? The diagonal case.

StepWhy
1D2 = cPick a contour. c is a fixed number, like 1 or 4.
2(x1 − μ1)2σ12 + (x2 − μ2)2σ22 = cThe sandwich for a diagonal Σ, from part 03.
3(x1 − μ1)2(√c σ1)2 + (x2 − μ2)2(√c σ2)2 = 1Divide by c. This is the equation of an ellipse with half-widths √c σ1 and √c σ2.

With correlation, the cross term −2ρz1z2 rotates this ellipse. The contour still reaches exactly μ1 ± √c σ1 left and right, and μ2 ± √c σ2 up and down.

Change thisThe hill does this
μMoves. Shape stays the same.
σ1 upStretches along x1. Gets lower, because |Σ| grows.
ρ from 0 towards 1Tilts and gets thinner. |Σ| = σ12σ22(1 − ρ2) shrinks, so the peak gets higher.
ρ = ±1|Σ| = 0. The ellipse collapses to a line. No inverse, no density.

How to spot a valid Σ. Three checks.

For a 2×2 matrix: top right equals bottom left, both diagonal entries are above 0, and |Σ| > 0. These three together say symmetric and positive definite.

One feature alone. Read it off.

The slides say the marginal of one feature is obtained by projecting the hill onto its axis. For the Gaussian you just read it off: p(x1) = N(x1 | μ1, σ12). So our carry's GPM alone follows N(550, 1600), and XPM alone follows N(650, 2500).

Remember itDiagonal gives the stretch. Off-diagonal gives the tilt. Its sign gives the direction.

07 IndependenceGaussian · 07/15

Diagonal Σ means independent.

Two features are independent when knowing one tells you nothing about the other. In formulas, the joint density splits into a product: p(x1, x2) = p(x1) p(x2). Part 03 went from product to matrix. Here is the proof the other way, for any d.

StepWhy
1Σ = diag(σ12, …, σd2)Assume zeros off the diagonal. "diag" lists the diagonal entries.
2|Σ| = σ12 ⋯ σd2The determinant of a diagonal matrix is the product of its diagonal.
3Σ−1 = diag(1/σ12, …, 1/σd2)Invert each entry.
4(x − μ)T Σ−1 (x − μ) = ∑j=1d (xj − μj)2 / σj2All cross terms vanish.
5N(x | μ, Σ) = 1(2π)d/2 (σ12 ⋯ σd2)1/2 exp(−12 ∑j (xj − μj)2 / σj2)Substitute into the density.
6= ∏j=1d 1√(2πσj2) exp(−(xj − μj)22σj2)exp of a sum is a product of exps. The sign ∏ means multiply all.
7N(x | μ, Σ) = ∏j=1d N(xj | μj, σj2)A product of 1D densities. That is independence.

For jointly Gaussian features, covariance 0 and independence are the same thing. Other distributions only promise one direction: independent features have covariance 0, while covariance 0 alone can fail to give independence.

True model, ρ = 0.8. D2 is 1.5625 for A and 12.674 for B. The model sees the difference.
Pretend independent, ρ = 0. D2 is 1.25² + 1² = 2.5625 for both. The model cannot tell the games apart.

Same variances. Same two games. Throwing away the covariance throws away the information that separates them.

Remember itZeros off the diagonal let the exponent split, so the density splits, so the features are independent.

08 MLEGaussian · 08/15

Best μ and Σ from data.

In part 02 we averaged and got μ and Σ. Maximum likelihood explains why that is right. Likelihood means: how probable is my data, for given parameters? MLE (maximum likelihood estimate) picks the parameters with the highest likelihood. Take N games x1, …, xN, independent and from the same distribution.

μ̂ = 1N ∑i=1N xi = x̄      Σ̂ = 1N ∑i=1N (xi − x̄)(xi − x̄)T

The log-likelihood.

StepWhy
1ℓ(μ, Σ) = log ∏i=1N N(xi | μ, Σ)Independent games multiply.
2= ∑i=1N log N(xi | μ, Σ)Log turns the product into a sum.
3log N(xi | μ, Σ) = −d2 log(2π) − 12 log |Σ| − 12 (xi − μ)T Σ−1 (xi − μ)Log of one density: log(1/a) = −log a, and log exp cancels.
4ℓ(μ, Σ) = −Nd2 log(2π) − N2 log |Σ| − 12 ∑i=1N (xi − μ)T Σ−1 (xi − μ)Add N copies. This is eq. (145) in the notes.

Finding μ̂

StepWhy
1∂ℓ∂μ = −12 ∑i (−2 Σ−1 (xi − μ))First rule of Tool 10. The first two terms hold no μ, so they vanish.
2= Σ−1 ∑i (xi − μ) = 0−½ · −2 = 1. Set to zero to find the top.
3∑i (xi − μ) = 0Multiply both sides by Σ. ΣΣ−1 = I.
4∑i xi − Nμ = 0μ appears N times.
5μ̂ = 1N ∑i xiSolve. The MLE of the mean is the sample mean.

Finding Σ̂. The trace trick.

Derivatives by Σ are awkward, so the notes use the precision matrix Λ = Σ−1. Note log |Σ| = −log |Λ|.

StepWhy
1(xi − μ)T Λ (xi − μ) = tr((xi − μ)T Λ (xi − μ))One number equals its own trace.
2= tr((xi − μ)(xi − μ)T Λ)Rotate the order inside the trace.
3ℓ = const + N2 log |Λ| − 12 tr( [∑i (xi − μ)(xi − μ)T] Λ )Substitute. A sum of traces is the trace of the sum.
4∂ℓ∂Λ = N2 Λ−T − 12 ∑i (xi − μ)(xi − μ)T = 0Rules 2 and 3 of Tool 10. The summed matrix is symmetric, so its transpose changes nothing.
5N2 Σ = 12 ∑i (xi − μ)(xi − μ)TCovariance is symmetric: Λ−T = Λ−1 = Σ.
6Σ̂ = 1N ∑i (xi − μ̂)(xi − μ̂)TPut μ̂ for μ and solve.

The notes add one remark: this estimate divides by N, which makes it biased. Dividing by N − 1 gives the unbiased version, called Bessel's correction. The MLE itself uses N.

Numbers. The course example.

Notes example: three points 12, 34, 20. Sample mean: (1/3) 66 = 22.

StepWhy
1v1 = −10,   v1v1T = 1000Deviation, then column × row.
2v2 = 12,   v2v2T = 1224Same.
3v3 = 0−2,   v3v3T = 0004Same.
4Σ̂ = 13 2228 = 2/32/32/38/3Add the three matrices, divide by N = 3.

Numbers. Our carry.

The four games from part 02 again, this time with outer products. The answer must match Σ from the story.

StepWhy
1Game 1: −40−70 gives 1600280028004900Deviation from 550650, then column × row.
2Game 2: −40−10 gives 1600400400100
3Game 3: 4010 gives 1600400400100Minus signs cancel: same as game 2.
4Game 4: 4070 gives 1600280028004900Same as game 1.
5Σ̂ = 14 64006400640010000 = 1600160016002500Matches. Our table in part 02 was this MLE, entry by entry.

Remember itMean of the points. Mean of the outer products of the deviations. Divide by N.

09 LabGaussian · 09/15

Drag the game. Bend the hill.

The orange x is a game. Drag it, or tap the map. Sliders set μ, σ1, σ2 and ρ. Each ring is one contour: D = 1, 2, 3. The buttons load the exact numbers from this page, so you can check your hand work.

D2 sandwich
distance D
density
Covariance matrix Σ
Determinant, and its root
Inverse Σ−1
Gap v = (x − μ)
Σ−1 v
Row × column
Density
The same game, un-stretched. Undo the stretch and tilt, and all rings become circles. The plain distance from the centre here is exactly D.

Four things to try

1. Load Game A, then Game B. Check 1.5625 and 12.674 from parts 04 and 05.

2. Keep Game B and move ρ to 0. D2 falls to 2.5625. Part 07 predicted this.

3. Push ρ close to 1. Watch |Σ| shrink and the density at the centre climb.

4. Load Class 3 and see the exam's D2 = 1 appear.

Remember itDistance is counted in rings. The same ruler step crosses few rings along the tilt and many across it.

10 ExamGaussian · 10/15

The real 2025 final.

2025 Final, Task 1(b)Warm-up

Give the formula for the density of multivariate Gaussian distribution N(x | μ, Σ) = p(x | μ, Σ), for μ ∈ RD and Σ ∈ RD×D.

What you write:

N(x | μ, Σ) = 1(2π)D/2 |Σ|1/2 exp(−12 (x − μ)T Σ−1 (x − μ))

The exam writes D where the notes write d. Use the exam's letter. Check four things before moving on: the power D/2, the root of the determinant, the minus one half, and the inverse in the middle.

2025 Final, Task 2(c)Warm-up on its numbers

The task opens like this, and part (c) gives the numbers:

Suppose we want to classify points. We have K different classes, each Gaussian N(μc | Σc), where (μc)c are the means and (Σc)c are the different covariances.

(c) Suppose now you are given the following values

μ1 = [3, 0]T,   μ2 = [0, 3]T,   μ3 = [0, 0]T
Σ1 = 1001,   Σ2 = 1001,   Σ3 = 2111

Classify the point x = [1, 1]T, if the prior for each class is 1/3 = P(Y = 1) = P(Y = 2) = P(Y = 3).

The sheet also prints: if A = abcd, then det(A) = |A| = ad − bc and A−1 = 1det(A) d−b−ca.

Our warm-up: compute (x − μ)T Σ−1 (x − μ) for each class. The classification itself belongs to the Gaussian classifier lecture.

Line, as you would write itWhy
1.1x − μ1 = 1 − 31 − 0 = −21Class 1 gap.
1.2Σ1−1 = 1001,   |Σ1| = 1The identity is its own inverse.
1.3D2 = −21 −21 = 4 + 1 = 5With the identity, the sandwich is a sum of squares.
2.1x − μ2 = 1 − 01 − 3 = 1−2Class 2 gap.
2.2D2 = 1−2 1−2 = 1 + 4 = 5Same inverse as class 1.
3.1|Σ3| = 2·1 − 1·1 = 1ad − bc.
3.2Σ3−1 = 11 1−1−12 = 1−1−12Swap 2 and 1, flip the signs of the off-diagonal 1s.
3.3x − μ3 = 11Class 3 gap. Its mean is the origin.
3.4Σ3−1 11 = 1 − 1−1 + 2 = 01Middle first.
3.5D2 = 11 01 = 0 + 1 = 1Row × column.
Class 15Density: (1/2π) exp(−5/2) ≈ 0.0131
Class 25Density: (1/2π) exp(−5/2) ≈ 0.0131
Class 31Density: (1/2π) exp(−1/2) ≈ 0.0965

All three determinants equal 1 and D = 2, so the front constant is the same 1/(2π) for every class. Load Class 1, Class 2 and Class 3 in the lab to see the same numbers.

Remember itEach class gets its own gap and its own inverse. Never reuse them between classes.

11 TrapsGaussian · 11/15

Where marks get lost.

  • Using Σ in the sandwichThe middle matrix is always Σ−1. Using Σ makes rare games look normal.
  • Forgetting the root of |Σ|The front has |Σ|1/2. Carry: √1,440,000 = 1200. Writing 1,440,000 there makes the answer 1200 times too small.
  • Keeping √(2π) with two featuresThe power is d/2. With d = 2 the factor is 2π. √(2π) belongs to d = 1.
  • Losing the one halfThe exponent is −D2/2. Game A: exp(−0.78125). The value exp(−1.5625) is wrong.
  • Inverting entry by entryThe shortcut 1/entry works only for diagonal matrices. For anything else: swap, flip, divide.
  • Flipping signs on the diagonalDiagonal entries swap and keep their sign. Only the off-diagonal entries change sign.
  • Mixing up D and D²The exponent holds D2, the sandwich itself. The notes call the sandwich the Mahalanobis distance term, so write clearly which one you mean.
  • Mixing up ρ and covarianceOff-diagonal in Σ is the covariance ρσ1σ2. In the carry it is 1600, and ρ is 0.8.
  • Calling density a probabilityA density is a height. Its value can exceed 1. Probability is volume under the hill.
  • Row × column in the MLEΣ̂ needs column × row, vvT, which is a matrix. vTv is one number and cannot give a covariance matrix.
  • Dividing by N − 1 in the MLEThe MLE in this course divides by N.
  • Trusting the 68% rule in 2DWith two features, only about 39% of the volume lies inside D = 1, 86% inside D = 2, 99% inside D = 3. These come from 1 − exp(−D2/2).
12 Cheat sheetsave a screenshot

All of it on one sheet.

Memorise theseLecture 7

Density
N(x | μ, Σ) = 1(2π)d/2 |Σ|1/2 exp(−12 (x − μ)T Σ−1 (x − μ))
Mahalanobis
D2 = (x − μ)T Σ−1 (x − μ),   D = √D2
2×2 determinant and inverse
|A| = ad − bc,   A−1 = 1ad − bc d−b−ca
Quadratic form, symmetric
vTAv = av12 + 2bv1v2 + dv22
Σ for two features
Σ = σ12ρσ1σ2ρσ1σ2σ22,   ρ = covσ1σ2
Determinant in σ, ρ
|Σ| = σ12σ22(1 − ρ2)
Distance in z-scores
D2 = z12 − 2ρz1z2 + z221 − ρ2,   zj = xj − μjσj
Shape
ρ = 0, equal σ: circle. ρ = 0, unequal σ: ellipse along the axes. ρ > 0: tilted up. ρ < 0: tilted down.
Independence
Σ diagonal ⟺ N(x | μ, Σ) = ∏j N(xj | μj, σj2)
Marginal
p(x1) = N(x1 | μ1, σ12)
MLE
μ̂ = 1N ∑i xi,   Σ̂ = 1N ∑i (xi − μ̂)(xi − μ̂)T
Five steps
|Σ|, then Σ⁻¹, then the gap, then middle first, then exp over constant.
13 Explain it backGaussian · 13/15

Say it first. Then press.

Why is there an inverse in the middle of the formula?

In 1D we divide the squared gap by σ2. In many dimensions the spread is a matrix, and dividing by a matrix means multiplying by its inverse. So Σ−1 is the matrix version of 1/σ2.

Why does the sandwich give one number?

Sizes: row (1×d) times matrix (d×d) times column (d×1). The result is 1×1. The exp needs one number, and this is it.

What does the front constant do?

It makes the total volume under the hill equal 1. The bare exp part has volume (2π)d/2 |Σ|1/2, so we divide by that. It holds no x, so it never changes the shape.

Games A and B are the same ruler distance from μ. Why do they get such different densities?

With ρ = 0.8, gold and XP rise together. Game A goes along that direction, so D2 = 1.5625. Game B goes across it, so D2 ≈ 12.674. Density ratio: exp((12.674 − 1.5625)/2) ≈ 259.

What shape are the contours, and what sets each property?

Ellipses centred at μ. σ1 and σ2 set the stretch along each axis. ρ sets the tilt and thinness. Equal σ with ρ = 0 gives a circle.

Prove in three sentences: diagonal Σ means independent.

With a diagonal Σ the sandwich loses its cross terms and becomes a sum of (xj − μj)2/σj2. The exp of a sum is a product of exps, and the front constant also splits, since |Σ| is the product of the variances. So the density is a product of 1D Gaussians, and that is independence.

What happens to the density at the centre as ρ goes to 1?

The peak height is 1/(2πσ1σ2√(1 − ρ2)). As ρ nears 1 the root nears 0, so the peak grows without limit. The hill turns into a thin ridge. At ρ = 1 exactly, |Σ| = 0 and there is no density.

Why does the MLE use column × row and the distance use row × column?

Σ̂ must be a d×d matrix, so each game contributes the matrix vvT. The distance must be one number, so it ends in row × column. The trace trick is the bridge between the two.

14 Quick test6 questions

Six questions. Let's check.

15 Practice13 problems

Paper first. Then open.

Difficulty rises as you go. Problems 9 to 13 are written like exam tasks. Each has a hint and a full solution.

Problem 1Warm-up

A mid player's last hits and denies at 10 minutes: x = 626. Their average is μ = 605. Find v = (x − μ), its transpose, and vTv.

Hint

Tool 1, Tool 2, then row × column.

Full solution
StepWhy
1v = 62 − 606 − 5 = 21Entry by entry.
2vT = 21Column becomes row.
3vTv = 2·2 + 1·1 = 5Row × column. The ruler distance squared.
Problem 2Warm-up

Find the determinant and inverse of A = 3112. Check by multiplying. This is the kind of Σ you get for last hits and denies in small units.

Hint

Swap 3 and 2, flip the signs of the 1s, divide by the determinant.

Full solution
StepWhy
1|A| = 3·2 − 1·1 = 5ad − bc.
2A−1 = 15 2−1−13Swap, flip, divide.
3AA−1 = 15 3·2 − 1·1−3 + 32 − 2−1 + 6 = 15 5005 = ICheck. Correct.
Problem 3Easy

Your carry's GPM (x1) and the enemy carry's GPM (x2) have Σ = 900−540−5401600. Find σ1, σ2, the covariance and ρ. Which way do the contours tilt?

Hint

Diagonal gives variances. Divide the off-diagonal by both standard deviations.

Full solution
StepWhy
1σ1 = √900 = 30,   σ2 = √1600 = 40Roots of the diagonal.
2covariance = −540Off-diagonal.
3ρ = −540 / (30 · 40) = −0.45Tool 7.
4Tilt: long axis runs down to the rightNegative ρ. When you farm well, the enemy carry tends to farm less.
Problem 4Easy

Last hits at 10 minutes have σ1 = 6. Denies have σ2 = 2. Correlation ρ = 0.5. Build Σ and find |Σ| in two ways.

Hint

Off-diagonal is ρσ1σ2. Second way: |Σ| = σ12σ22(1 − ρ2).

Full solution
StepWhy
1ρσ1σ2 = 0.5 · 6 · 2 = 6Covariance.
2Σ = 36664Variances on the diagonal.
3|Σ| = 36·4 − 6·6 = 144 − 36 = 108ad − bc.
4|Σ| = 36·4·(1 − 0.25) = 144·0.75 = 108Second way. They agree.
Problem 5Medium

Pretend GPM and XPM are independent: μ = 550650, Σ = 1600002500. Find D2 and the density for a game with 630 GPM and 700 XPM.

Hint

A diagonal Σ means the sandwich is a sum of squared z-scores.

Full solution
StepWhy
1z1 = (630 − 550)/40 = 2,   z2 = (700 − 650)/50 = 1z-scores.
2D2 = 22 + 12 = 5Sum of squares.
3|Σ|1/2 = √(1600 · 2500) = 40 · 50 = 2000Product of the σ.
4p = 12π · 2000 exp(−2.5) ≈ 0.08208 / 12566.4 ≈ 6.53 × 10−6Front constant, then exp.
Problem 6Medium

A mid's last hits and denies at 10 minutes: μ = 605, Σ = 4222. Compute D2 and the density at x = 626.

Hint

Five steps. Determinant first. It comes out as 4.

Full solution
StepWhy
1|Σ| = 4·2 − 2·2 = 4,   |Σ|1/2 = 2Step 1.
2Σ−1 = 14 2−2−24 = 0.5−0.5−0.51Step 2.
3v = 21Step 3.
4Σ−1 v = 1 − 0.5−1 + 1 = 0.50Step 4a.
5D2 = 21 0.50 = 1Step 4b.
6p = 12π · 2 exp(−0.5) ≈ 0.6065 / 12.566 ≈ 0.0483Step 5.
Problem 7Medium

Same mid as problem 6. A second game has x = 624. Find D2 and the density. How many times less dense is it than the game in problem 6? Say why in one line.

Hint

The inverse stays the same. Only the gap changes. For the ratio, the constants cancel.

Full solution
StepWhy
1v = 2−1New gap.
2Σ−1 v = 1 + 0.5−1 − 1 = 1.5−2Same inverse as before.
3D2 = 2−1 1.5−2 = 3 + 2 = 5Row × column.
4p = 14π exp(−2.5) ≈ 0.00653Same front constant.
5Ratio = exp((5 − 1)/2) = exp(2) ≈ 7.39The constants cancel.

Why: last hits and denies are positively correlated. More last hits with fewer denies goes across the tilt.

Problem 8Medium+

Net worth at 20 minutes in thousands of gold (x1) and deaths (x2): μ = 205, Σ = 9−3−34. Compare D2 for game P, 236, and game Q, 234. Which fits the model better?

Hint

The determinant is 27. Use fractions and keep the signs: the off-diagonal of the inverse comes out positive.

Full solution
StepWhy
1|Σ| = 9·4 − (−3)(−3) = 36 − 9 = 27Careful: (−3)(−3) = +9.
2Σ−1 = 127 4339Swap 9 and 4. Flip −3 to +3.
3P: v = 31,   Σ−1v = 127 15184·3 + 3·1 = 15. 3·3 + 9·1 = 18.
4P: D2 = (3·15 + 1·18)/27 = 63/27 = 7/3 ≈ 2.33Row × column.
5Q: v = 3−1,   Σ−1v = 127 904·3 − 3 = 9. 9 − 9 = 0.
6Q: D2 = (3·9 + 0)/27 = 1Row × column.

Game Q fits better. Here ρ = −3/(3·2) = −0.5: more net worth comes with fewer deaths, and Q has exactly that.

Problem 9Exam style

Four games, as (last hits, denies): (50, 4), (60, 8), (70, 8), (60, 4). State the MLE formulas, then compute μ̂ and Σ̂. Give ρ.

Hint

Deviations from the mean are (−10, −2), (0, 2), (10, 2), (0, −2).

Full solution
StepWhy
1μ̂ = 14 24024 = 606Mean of the points.
2v1v1T = 10020204,   v2v2T = 0004Column × row.
3v3v3T = 10020204,   v4v4T = 0004Column × row.
4Σ̂ = 14 200404016 = 5010104Sum, divide by N = 4.
5ρ = 10 / (√50 · 2) = 10 / 14.14 ≈ 0.707Tool 7.
Problem 10Exam style

Write the density for D = 3 features with a diagonal Σ = diag(100, 4, 25). The features are last hits, denies and GPM, with μ = (60, 4, 500). Evaluate at x = (70, 6, 505).

Hint

The power of 2π is 3/2. The determinant of a diagonal matrix is the product of its entries.

Full solution
StepWhy
1|Σ| = 100·4·25 = 10000,   |Σ|1/2 = 100Product of the diagonal.
2(2π)3/2 ≈ 15.7496D = 3.
3D2 = (10)2/100 + (2)2/4 + (5)2/25 = 1 + 1 + 1 = 3Sum of squared z-scores.
4p = 115.7496 · 100 exp(−1.5) ≈ 0.22313 / 1574.96 ≈ 1.42 × 10−4Constant, then exp.
Problem 11Exam style

In the style of 2025 Final 2(c). Three roles, on standardised farm scores. Point x = [1, 1]T.
Mid: μ = [2, 0]T, Σ = 1001. Offlane: μ = [1, 0]T, Σ = 2−1−11. Jungle: μ = [0, 2]T, Σ = 2002.
Compute D2 and the density for each. Which has the smallest D2? Does it also have the largest density?

Hint

Jungle has |Σ| = 4, so its front constant differs from the other two.

Full solution
StepWhy
1Mid: v = −11, D2 = 1 + 1 = 2The identity gives a sum of squares.
2Mid density: 12π exp(−1) ≈ 0.0585|Σ| = 1.
3Offlane: |Σ| = 2 − 1 = 1,   Σ−1 = 1112Swap, flip. −(−1) = +1.
4Offlane: v = 01,   Σ−1v = 12,   D2 = 0·1 + 1·2 = 2Middle first.
5Offlane density: 12π exp(−1) ≈ 0.0585Same as mid.
6Jungle: v = 1−1,   Σ−1 = 1/2001/2Diagonal shortcut.
7Jungle: D2 = 1/2 + 1/2 = 1Sum.
8Jungle density: 12π · 2 exp(−1/2) ≈ 0.0483√4 = 2 in the constant.

Answer: jungle has the smallest D2 and the smallest density. Its wide hill stands lower. Always include |Σ|1/2 when the Σ differ.

Problem 12Exam style

Which of these can be a covariance matrix? Give a reason for each.
(a) 1221   (b) 4123   (c) 4221   (d) 4111

Hint

Three checks: symmetric, positive diagonal, determinant above 0.

Full solution
StepWhy
1(a) |Σ| = 1 − 4 = −3Fails. It would need ρ = 2.
2(b) top right 1, bottom left 2Fails. Not symmetric.
3(c) |Σ| = 4 − 4 = 0Fails. ρ = 1, no inverse, no density.
4(d) symmetric, diagonal 4 and 1, |Σ| = 4 − 1 = 3Valid. ρ = 1/(2·1) = 0.5.
Problem 13Derivation

On a blank page: take d = 2 and Σ = σ2I, meaning both features have the same spread and no correlation. Show that the density is 12πσ2 exp(−‖(x − μ)‖22σ2) and that its contours are circles. The bars ‖ ‖ mean ruler length.

Hint

Σ = σ200σ2. Compute its determinant and inverse first.

Full solution
StepWhy
1Σ = σ200σ2Write σ2I in full.
2|Σ| = σ4, so |Σ|1/2 = σ2ad − bc, then the root.
3Σ−1 = 1σ2 IDiagonal shortcut.
4(x − μ)T Σ−1 (x − μ) = 1σ2 (x − μ)T (x − μ) = 1σ2 ‖(x − μ)‖2Row × column of the same vector is the ruler length squared.
5p(x) = 12πσ2 exp(−‖(x − μ)‖22σ2)Put into the formula. (2π)2/2 = 2π.
6Same height means ‖(x − μ)‖ = constantThis is the equation of a circle with centre μ.