Pass the retake.

Five moves solve almost the whole exam.

Every task in both 2026 exams was one of five moves, applied to a different distribution. Learn the words and the moves here. Every later chapter is practice of these moves.

The six words you need

The five moves

1. MLE (maximum likelihood estimate). Take the log of the likelihood, differentiate, set it to 0, solve for \(\theta\).
   Hook: log, derivative, zero.
2. Posterior. Multiply likelihood by prior, drop constants, add the exponents, recognise the family (Beta or Gamma).
   Hook: multiply, drop, add, recognise.
3. Point estimates. The posterior mean (the average) or the MAP (the peak, which is the mode). Read them from the family’s formula.
4. Predictive. The probability of the next observation: average the model over the posterior, \(p(x_{\text{new}}\mid D)=\int p(x_{\text{new}}\mid\theta)\,p(\theta\mid D)\,d\theta\).
5. Classify. For each class compute score \(=\log(\text{prior}) + \log p(x\mid\text{class})\). The largest score wins.

See all five on one coin

You flip a coin \(n=10\) times and see \(S=7\) heads. One flip has probability \(\theta^{x}(1-\theta)^{1-x}\) with \(x\in\{0,1\}\).

Math toolbox

These are the only calculation rules the exams need.

Distributions to know by heart.

The midterm opens with “define this distribution”. The whole exam uses the same dozen formulas. Learn them here, then switch on quiz mode and test yourself.

The dozen

What the two priors look like

Which prior goes with which model

Probability basics that score easy points.

Warm-up questions: a conditional probability with dice, the memoryless property, Bayes’ theorem, and a few definitions. Each has one recipe.

Conditional probability by counting

Draw all \(3\times3=9\) equally likely outcomes. Blue cells contain at least one A. The green cell is AA.

1. Definition: \(P(E\mid F)=\dfrac{P(E\cap F)}{P(F)}\). “A twice” already implies “A at least once”, so \(P(E\cap F)=P(AA)\).
2. \(P(AA)=\frac13\cdot\frac13=\frac19\).
3. \(P(\text{at least one A})=1-P(\text{no A})=1-\left(\frac23\right)^2=\frac59\).
4. \(P(AA\mid\text{at least one A})=\dfrac{1/9}{5/9}=\dfrac15\).

Bayes’ theorem, marginal and conditional

The memoryless property

1. Why the sum is 1. It adds the probabilities of all possible values of \(X\). All probabilities together give 1. (Or: geometric series \(p\cdot\frac{1}{1-(1-p)}=1\).)
2. Tail. By the hint, \(P(X\ge L)=(1-p)^{L-1}\). In words: the first \(L-1\) tries all failed.
3. Intersection. \(\{X\ge n+m\}\) is inside \(\{X\ge n\}\), so the intersection is just \(\{X\ge n+m\}\).
4. Divide. \[P(X\ge n+m\mid X\ge n)=\frac{(1-p)^{n+m-1}}{(1-p)^{n-1}}=(1-p)^{m}.\]

Independent vs uncorrelated

Take \(X\) uniform on \([-1,1]\) and \(Y=X^2\). Then \(\mathrm{Cov}(X,Y)=E[X^3]-E[X]E[X^2]=0-0=0\), but \(Y\) is completely determined by \(X\). Correlation only measures straight-line dependence; a parabola has none.

• Independent \(\Rightarrow\) uncorrelated (always).
• Uncorrelated \(\Rightarrow\) independent only for jointly Gaussian variables.

Exponential family

Form: \(p(x\mid\theta)=h(x)\exp\big(\eta(\theta)\,T(x)-A(\theta)\big)\), and the support (where \(p\gt 0\)) must not depend on \(\theta\).

The posterior task. It comes every time.

Six exams in a row had the same task with a different distribution: Negative binomial, Poisson, Geometric, Exponential, even an unknown lifetime density. The steps never change. This is the most valuable chapter.

The template

1. Likelihood. Multiply over all data points. Collect the parameter into powers. Everything without the parameter is the constant.
2. MLE. \(\ell=\log L\). Solve \(\ell'=0\).
3. Posterior. Likelihood × prior. Add exponents. Recognise Beta or Gamma and read off the new parameters.
4. Mean / MAP / mode. Plug the new parameters into the family’s formulas.
5. Predictive. \(\int p(x\mid\theta)p(\theta\mid D)d\theta\) = model constant × ratio of normalising constants (chapter 4).

The kernel table

With \(n\) data points and \(S=\sum_i k_i\) (the sum of the data). This table is the whole task in one line per model.

Worked: Poisson + Gamma (Midterm 2026, Task 3)

Goals per match: \(p(k\mid\lambda)=e^{-\lambda}\frac{\lambda^k}{k!}\). Prior \(p(\lambda)=\frac{b^a}{\Gamma(a)}\lambda^{a-1}e^{-b\lambda}\).

Numbers: prior Gamma(2, 1), goals {3, 1, 4, 2}: \(n=4,\ S=10\). Posterior Gamma(12, 5). MLE \(2.5\), MAP \(\frac{11}{5}=2.2\), mean \(\frac{12}5=2.4\).

Worked: Negative binomial + Beta (Midterm 2026, Task 2)

\(f(k\mid p)=\binom{k+r-1}{k}p^{r}(1-p)^{k}\), \(r\) known. Hint: “things which don’t depend on \(p\) can be … ignored”.

Worked: Geometric “one more drink” (Final 2026, Task 2)

\(p(y_i\mid\lambda)=\lambda(1-\lambda)^{y_i-1}\), \(y_i\in\{1,2,\dots\}\). Choose a prior, compute the posterior.

Unknown density? Same moves.

Retake 2025 gave a density nobody had seen: the Gompertz lifetime \(f(x\mid\eta)=\eta\,e^{\eta+x-\eta e^{x}}\), \(x\gt0\), with a Gamma(\(a,\lambda\)) prior. Don’t panic. Collect \(\eta\).

Try it: the posterior machine

Pick a model, set the prior, type data. It writes every step the exam wants. Use it to check your own practice answers.

Predictions, bets and the two dice.

After the posterior, the exam asks what happens next: the mean, the MAP, the probability of the next observation, or whether to take a bet.

Three estimates, three meanings

As \(n\to\infty\) all three agree: the prior becomes negligible. With a uniform prior, MAP = MLE.

Predictive: average, never plug in

The predictive averages the model over every possible \(\theta\), weighted by the posterior. The integral always becomes a ratio of normalising constants.

Should you take the bet?

A bet question is expected value: probability of winning × win, minus probability of losing × loss.

The two dice (Midterm 2025, Task 2)

Die \(h_1\) has faces 1–4, die \(h_2\) has faces 1–8. You see throws, not the die. Only two hypotheses, so every sum has two terms.

1. Likelihood. \(p(D\mid h_1)=(1/4)^n\) if all throws are ≤ 4, else 0. \(p(D\mid h_2)=(1/8)^n\).
2. Posterior (uniform prior, all ≤ 4): \(p(h_1\mid D)=\frac{(1/4)^n}{(1/4)^n+(1/8)^n}=\frac{2^n}{2^n+1}\). Each fitting throw doubles the odds for the small die: the size principle.
3. Model averaging. \(p(x_{n+1}\mid D)=\sum_i p(x_{n+1}\mid h_i)\,p(h_i\mid D)\). For faces 1–4: \(\frac14 w_1+\frac18 w_2\); faces 5–8: \(\frac18 w_2\).
4. MAP with a strong prior. \(p(h_1)=10^{-6}\), data {3, 4}: \(h_1\): \(10^{-6}\cdot\frac1{16}\approx6\cdot10^{-8}\); \(h_2\): \(\approx\frac1{64}\approx0.016\). MAP picks \(h_2\), even though \(h_1\) is the MLE.
5. Expected next throw after 15 throws all ≤ 4: \(w_1=\frac{2^{15}}{2^{15}+1}\approx1\), so \(E[X\mid D]=2.5\,w_1+4.5\,w_2\approx2.5\).

Priors: small data, flat priors, narrow priors.

A pure reasoning task. Write the posterior parameters first, then ask: who is bigger, the prior numbers or the data numbers?

More points you can add: the zero-count (black swan) problem in Naive Bayes; prior knowledge from experts can be built in. Downside: with small data the answer depends a lot on the prior.

First the posterior: \(\mathrm{Gamma}(\alpha+n,\ \beta+\sum x_i)\). Huge variance \(\frac{\alpha}{\beta^2}\) means tiny \(\beta\): the prior says “I know almost nothing”.

Recipe for any prior true/false

1. Write the posterior parameters (prior numbers + data counts).
2. Compare sizes: prior numbers vs \(n\) and \(\sum x_i\).
3. Bigger data ⇒ posterior ≈ MLE. Flat prior ⇒ data wins. Narrow prior ⇒ prior wins for small \(n\).

The short theory questions.

The final opens with eight quick questions worth easy points. Each one has a two-line answer. Read the question, cover the answer, say it out loud, then check.

Final 2026, Task 1

Also asked in 2025 finals and the retake

Gaussian classifiers: QDA and LDA.

Every final had one: “classify the point \(x\)” with three Gaussian classes. It is pure arithmetic with 2×2 matrices once you know the score.

The idea

Each class is a Gaussian cloud with its own centre \(\mu_c\) and shape \(\Sigma_c\). A new point goes to the class whose cloud, weighted by the prior \(\pi_c\), is highest at that point:

Take the log and drop the \((2\pi)^{-D/2}\) that every class shares. That gives the score:

Four models, one formula

(a) The definitions to write

(b) Classify a point: the recipe

1. For each class compute \(v=x-\mu_c\).
2. Compute \(|\Sigma_c|=ad-b^2\) and \(\Sigma_c^{-1}=\frac{1}{ad-b^2}\begin{pmatrix}d & -b\\ -b & a\end{pmatrix}\).
3. Compute \(d_c^2=v^T\Sigma_c^{-1}v\). For a symmetric matrix: \(a'v_1^2+2b'v_1v_2+d'v_2^2\).
4. Score \(g_c=\log\pi_c-\frac12\log|\Sigma_c|-\frac12d_c^2\). Largest wins.
5. If asked for probabilities: \(P(c\mid x)=e^{g_c}/\sum_{c'}e^{g_{c'}}\).

Worked: Final 2026, 3(b)

\(|\Sigma_1|=|\Sigma_2|=2\), \(|\Sigma_3|=6-4=2\). All equal, priors equal, so compare \(d^2\) only. \(\Sigma_1^{-1}=\mathrm{diag}(\frac12,1)\), \(\Sigma_2^{-1}=\mathrm{diag}(1,\frac12)\), \(\Sigma_3^{-1}=\frac12\begin{pmatrix}2 & -2\\ -2 & 3\end{pmatrix}=\begin{pmatrix}1 & -1\\ -1 & 1.5\end{pmatrix}\).

Class 1. Posteriors \(\approx0.48,\ 0.14,\ 0.38\).

Estimating the parameters (MLE)

\(\hat\pi_c=\frac{N_c}{N}\), \(\hat\mu_c=\frac{1}{N_c}\sum_{i\in c}x_i\) (class average), QDA: \(\hat\Sigma_c=\frac1{N_c}\sum_{i\in c}(x_i-\hat\mu_c)(x_i-\hat\mu_c)^T\). LDA pools all classes into one \(\hat\Sigma\) and divides by \(N\).

Worked in 1D: cats vs dogs (Retake 2025, Task 5)

Running times: cats 7, 9, 11, 13; dogs 10, 12. \(p(x\mid h)=\mathcal N(\mu_h,\sigma_h^2)\). Classify \(x=10\). Hints: \(\sqrt5\approx2.24\), \(\sqrt e\approx1.65\).

(c) A classifier for non-Gaussian data

Naive Bayes: count, multiply, compare.

The naive assumption: given the class, the features are independent. That turns every classification into a product of simple counts.

The model

\(\theta_{jc}\) = probability that feature \(j\) is 1 in class \(c\). \(N_c\) = number of points in class \(c\). \(N_{jc}\) = how many of them have feature \(j\) equal to 1.

1. Count. Sort the data by class. Count \(N_c\) and \(N_{jc}\).
2. Table. For the new point use \(\theta\) where the feature is 1, and \(1-\theta\) where it is 0.
3. Multiply the factors and the prior \(\pi_c\).
4. Compare (and normalise if probabilities are asked).

Worked: Final 2025, Task 5

Data (class, attendance, homework): (F,1,1), (F,0,0), (P,1,1), (P,1,0), (F,0,1), (F,1,0), (P,1,1). Classify (0, 1).

Fail. Notice why Pass got exactly 0: every passing student attended, so “no attendance” has estimated probability 0. That is the zero-count problem.

Worked: Final 2026, 4(a)

\(\hat\theta_{\text{hw,pass}}=\frac12,\ \hat\theta_{\text{att,pass}}=\frac12,\ \hat\theta_{\text{q,pass}}=0\); \(\hat\theta_{\text{hw,fail}}=0,\ \hat\theta_{\text{att,fail}}=\frac12,\ \hat\theta_{\text{q,fail}}=0\); \(\hat\pi_{\text{pass}}=\frac35\). Classify (1, 1, 0).

(b) Why priors matter for small data

Advantages of the Bayes approach (Final 2025, 5c): no zero probabilities, sensible estimates on small data, uncertainty is kept, prior knowledge can enter, and it still trains fast by counting.

Other NBC facts they ask

• Can one feature influence another? No, not within a class: features are conditionally independent given the class. (They can be dependent overall, through the class.)
• Danger of MLE with fewer points than \(K\cdot D\)? Many counts are 0, so many \(\hat\theta=0\) or 1: zero probabilities and overfitting.
• Class prior Dir(1, …, 1) after \(N\) observations? Posterior \(\mathrm{Dir}(1+N_1,\dots,1+N_K)\).
• Many features? Add logs instead of multiplying, to avoid numbers too small for the computer (underflow).
• Continuous features? Use a 1D Gaussian per feature and class: Gaussian Naive Bayes (like the cats and dogs in chapter 7).

Mutual information for feature selection

Compute it for each feature with the class label. A feature with the same \(\theta\) in every class tells you nothing about the class and has \(I=0\). Keep the top features, drop the ones with small \(I\).

Logistic regression.

A straight line squeezed through an S-curve so the output is always a probability. The exams ask three things: its log-likelihood, why it has no formula, and how to fit it.

The model

Log-likelihood, gradient, Hessian

Why there is no formula

Set the gradient to zero: \(X^T(\mathrm{sigm}(Xw)-y)=0\). The unknown \(w\) is trapped inside the sigmoid, so the equation is non-linear. Compare linear regression, where \(X^TXw=X^Ty\) is linear and solvable by hand. So we iterate:

The NLL is convex, so any minimum the algorithm finds is the global one.

One step by hand

Linear regression and its relatives.

One model, many variations. The key sentence: Gaussian gives squares, Laplace gives absolute values. On the noise it is the loss; on the weights it is the penalty.

The model and its MLE

Maximising the Gaussian likelihood is the same as minimising the residual sum of squares \(\mathrm{RSS}(w)=\sum_i(y_i-w^Tx_i)^2\). So standard linear regression minimises squares, not absolute values (Retake 4a: false).

MLE of the noise variance (Final 2025, 4e)

True or false, with reasons

Fighting overfitting and outliers

Bayesian linear regression

Prior \(w\sim\mathcal N(0,\tau^2I)\), Gaussian noise. The posterior is Gaussian:

Any law “number × known function”

Duffing \(F=-kz+\alpha z^3\): columns \(z_i,\ z_i^3\). Drag \(F=c_1v+c_2v^2\): columns \(v_i,\ v_i^2\). \(y=a\sin t+bt^2\): columns \(\sin t_i,\ t_i^2\). Add a column of ones for an intercept. Then \(\hat\beta=(X^TX)^{-1}X^Ty\). Not linear if the parameter sits inside the function, like \(e^{kz}\).

Optimisation facts (Retake 2025 true/false)

• A twice-differentiable \(f\) is convex if \(f''\ge0\) everywhere (curves up). “\(f''\) always negative” describes a concave function: false.
• Newton’s method uses the second derivative (the Hessian). “Convenient because it does not need the second derivative” is false. Gradient descent is the one that needs only the gradient.
• Least squares is convex, with one global minimum.

Text models: bag of words and Markov chains.

Both 2025 finals ended with this task. The bag of words forgets the order. The Markov chain remembers the last word.

Bag of words (Final 2025, 6a)

Good for document classification (spam), bad for predicting the next word, because it ignores order.

Markov chain

The next word depends only on the current word. Parameters: start distribution \(\pi_i=P(X_0=i)\) and transition matrix \(A_{ij}=P(X_{t+1}=j\mid X_t=i)\). Row = from, column = to. Each row sums to 1.

• MLE = count and divide: \(\hat\pi_i=\frac{\#\text{sentences starting with }i}{\#\text{sentences}}\), \(\hat A_{ij}=\frac{N_{ij}}{\sum_{j'}N_{ij'}}\).
• One step: read \(A_{ij}\). Two steps: \((A^2)_{ij}=\sum_kA_{ik}A_{kj}\).
• A whole sequence: \(\pi(x_0)\prod_tA_{x_t,x_{t+1}}\). So \(\pi\) and \(A\) give the full distribution (Retake 4f: true).

Worked: Final 2025, 6(c–d)

Sentences: “I Love Math”, “I Love Love Math”, “Math I Love”.

Worked: weather chain (Retake 2025, Task 6)

Rows: sunny \((\frac13,\frac13,\frac13)\), cloudy \((\frac12,\frac14,\frac14)\), rainy \((0,\frac12,\frac12)\). Order: sunny, cloudy, rainy.

Gaussian extras.

Smaller lecture topics that can appear as a true/false or a one-line question.

Linear transformations

If \(X\sim\mathcal N(\mu,\Sigma)\), then \(Y=AX+b\sim\mathcal N(A\mu+b,\ A\Sigma A^T)\).

Example: \(X,Y\) independent \(\mathcal N(0,1)\). \((X+Y,X-Y)\) has covariance \(\begin{pmatrix}2 & 0\\ 0 & 2\end{pmatrix}\). \((X+Y,X+2Y)\) has covariance \(\begin{pmatrix}2 & 3\\ 3 & 5\end{pmatrix}\).

Marginals and conditionals

Split \(x=(x_1,x_2)\). The marginal of \(x_1\) is \(\mathcal N(\mu_1,\Sigma_{11})\): just read the block. The conditional is Gaussian too:

Learning a Gaussian mean (known \(\sigma^2\))

Prior \(\mu\sim\mathcal N(\mu_0,\tau_0^2)\), \(n\) data with average \(\bar x\). Precisions (1 / variance) add, and the mean is a precision-weighted average:

This is the same idea as sensor fusion: a more reliable measurement gets more weight.

Gaussian interpolation (Retake 2025, 4d)

The lecture puts a smoothness prior on the unknown function values that penalises second differences \(x_j-2x_{j+1}+x_{j+2}\) (curvature). With noise-free data the posterior mean passes exactly through the data points and, among all such curves, has the smallest total squared curvature. It is a smooth curve, not a set of straight lines. The uncertainty is 0 at the data points and grows between them.

True or false? Train the reflex.

Every exam has true/false with a justification. The recipe: circle the answer, give one reason, add a formula or a counterexample. Pick an answer on each card, then read the reason.

Mock midterm retake.

Same structure as April 2026, new numbers. For each part, say the first step out loud before you open the solution. That is exactly the skill the exam tests.

Task 1 · Warm-up

(a) Define the Gamma distribution by its density. Give the range of values and of the parameters.

(b) A fair die has four sides {A, B, C, D}. It is rolled twice. What is the probability that A occurs twice, given that A occurred at least once?

(c) Let \(X\sim\mathrm{Exp}(\lambda)\). Show \(P(X\gt t)=e^{-\lambda t}\) and compute \(P(X\gt s+t\mid X\gt s)\).

Task 2 · Binomial + Beta

Every day \(N=10\) people visit a shop; \(k_i\) of them buy. \(f(k\mid\theta)=\binom{N}{k}\theta^k(1-\theta)^{N-k}\). Data \(D=\{k_1,\dots,k_n\}\).

(a) MLE of \(\theta\). (b) Posterior with prior Beta(a, b). (c) Posterior mean and MAP. (d) Numbers: \(D=\{3,5,4\}\), \(a=b=2\).

Task 3 · Exponential + Gamma

Minutes between buses: \(p(x\mid\lambda)=\lambda e^{-\lambda x}\). Prior Gamma(a, b).

(a) Show \(p(D\mid\lambda)\propto\lambda^ne^{-\lambda T}\) with \(T=\sum x_i\); what is the constant? (b) Posterior. (c) Prior mode for \(a\gt1\). (d) MAP. (e) Numbers: \(a=2,\ b=4\), \(D=\{1,3,2,2\}\). (f) Probability the next wait is longer than 2 minutes.

Task 4 · Priors

(a) Explain MLE, MAP and posterior mean in one sentence each. When do they agree?

(b) A coin gets the prior Beta(1000, 1000). True or false? (1) After 5 flips, all heads, the posterior mean is still about 0.5. (2) The prior says “I am almost sure the coin is fair.” (3) After \(10^8\) flips the posterior mean ≈ MLE. (4) MAP = MLE for this prior.

Mock final retake.

Same structure as June 2026, with a Markov task added because both 2025 finals had one.

Task 1 · Short questions

(a) Write the density of \(\mathcal N(x\mid\mu,\Sigma)\) in \(\mathbb R^D\). (b) Which classifier for correlated Gaussian features with a different covariance per class? (c) Name the algorithm for logistic regression and write one step. (d) A drag law \(F=c_1v+c_2v^2+\varepsilon\): which model, which MLE? (e) Zellner’s g-prior: what happens to the posterior mean as \(g\to0\)?

Task 2 · An unfamiliar density

\(f(x\mid\eta)=\eta\,x^{\eta-1}\) for \(x\in(0,1)\), \(\eta\gt0\). Let \(T=-\sum_i\log x_i\) (a positive number).

(a) Compute \(f(D\mid\eta)\). (b) The MLE. (c) Choose a prior and compute the posterior. (d) Numbers: \(n=4,\ T=2\), prior Gamma(2, 2).

Task 3 · Gaussian classification

\(\mu_1=(1,1)\), \(\mu_2=(-1,0)\), \(\mu_3=(0,2)\); \(\Sigma_1=I\), \(\Sigma_2=\begin{pmatrix}2 & 0\\ 0 & \frac12\end{pmatrix}\), \(\Sigma_3=\begin{pmatrix}2 & 1\\ 1 & 1\end{pmatrix}\); \(x=(0,1)\).

(a) What is LDA short for? Can it be used here? (b) Classify \(x\) with equal priors. (c) Classify with priors 0.2, 0.2, 0.6. (d) Write the log-likelihood of logistic regression.

Task 4 · Naive Bayes

Emails (class, contains “free”, has a link): (S,1,1), (S,1,0), (S,0,1), (H,0,0), (H,1,0), (H,0,0), (H,0,1). S = spam, H = normal.

(a) Fit the NBC by MLE. (b) Classify (1, 1) and (0, 0). (c) Redo (1, 1) with Laplace smoothing (Beta(1,1) and Dir(1,1) priors). (d) Why use priors?

Task 5 · True or false

(1) Newton’s method needs only the gradient. (2) LDA with a shared covariance gives linear boundaries. (3) Laplace noise leads to minimising the sum of absolute errors. (4) For a Markov chain, \(P(X_2=j\mid X_0=i)=(A_{ij})^2\).

Task 6 · Markov chain

Weather sequences: (sun, sun, rain), (rain, sun, sun, sun), (sun, rain, rain).

(a) MLE of \(\pi\) and \(A\). (b) \(P(X_2=\text{rain}\mid X_0=\text{sun})\). (c) Probability of the sequence (sun, sun, rain).

Cheat sheet.

Everything on one screen. Read it once more just before you walk in.

Ten hooks