Skip to content
POLARISAI ACADEMY
R09-101 · Navigator

Linear Algebra, Probability and Autograd in Code

You can score a short note with a dot product, multiply a weight matrix by a feature vector, add an L2 penalty from the squared length of the weights, and turn two scores into a cross-entropy loss plus one chain-rule gradient.

Level
Navigator
Time
28 h (estimate)
Prerequisites
F01–F07, R02-201
Course 1 of 4
Jump to

After this course you can

The student can implement reverse-mode automatic differentiation, cross-entropy loss, and the AdamW optimiser from scratch in pure NumPy whose analytical gradients match numerical finite differences to < 1e-6.

Vectors and the dot product

A vector is an ordered list of numbers. In this lesson a feature vector holds three counts from a short note: how many times the word refund appears, how many times the word thanks appears, and how many tokens the note contains. The note refund thanks has the counts 1, 1 and 2, so its feature vector is [1, 1, 2]. A weight vector has one number per feature, in that same order. The billing weights [2, -1, 0] mean that refund raises the billing score, thanks lowers it, and the token count leaves that score unchanged.

The dot product multiplies matching entries and adds those products. For [2, -1, 0] and [1, 1, 2] the products are 2, -1 and 0, and the sum is 1. Swap the two vectors and the sum is still 1. A bias is one extra number added after the dot product. With bias -1 the billing score is 1 + (-1) = 0. The product has no value when the two lists have different lengths, because one entry then has nothing to meet. When you code the score, write w · x + b: one multiply per feature, then one addition of the bias.

Rows, columns, and the product

A matrix stores one weight row per class. With billing on row 0 and greeting on row 1, the weight matrix is [[2, -1, 0], [0, 2, -1]] and the bias vector is [-1, 1]. The matrix-vector product runs the dot product once per row. Row 0 with [1, 1, 2] gives 1, and the bias -1 brings the billing score to 0. Row 1 gives 0·1 + 2·1 + (-1)·2 = 0, and the bias 1 brings the greeting score to 1. The score vector is [0, 1]. The shape check is the inner length: a 2 by 3 matrix meets a vector of length 3 and returns a vector of length 2.

The product of two matrices uses that same row-against-column dot product. Take A = [[1, 2], [0, -1]] and B = [[3, 1], [4, 2]]. The top-left entry is the row [1, 2] dotted with the column [3, 4], which is 3 + 8 = 11. The top-right entry is [1, 2] dotted with [1, 2], which is 1 + 4 = 5. The bottom row [0, -1] gives -4 and -2. So AB = [[11, 5], [-4, -2]]. BA = [[3, 5], [4, 6]]. The two products have the same shape and different entries, so the order of the factors belongs to the arithmetic. In code the inner sizes must match. A 2 by 2 matrix times a 2 by 2 matrix is defined. A 2 by 3 matrix times a 2 by 2 matrix has no product, because 3 fails to match 2.

Length and the L2 penalty

The L2 norm is the straight-line length of a vector. For [3, 4] the squares are 9 and 16, the sum of squares is 25, and the length is the square root, 5. Dividing each entry by that length gives the unit vector [0.6, 0.8], and 0.6 squared plus 0.8 squared equals 1. The squared length equals the dot product of the vector with itself. Scaling by -2 sends [3, 4] to [-6, -8]. Those squares are 36 and 64, the sum is 100, and the length is 10, which equals 2 times 5. The sign of the scale drops out, because every entry is squared before the root. The L1 norm adds absolute values instead. For the billing weights [2, -1, 0] the L1 norm is 2 + 1 + 0 = 3.

An L2 penalty charges the squared length of the weights and is added to the data loss. For [2, -1, 0] the squares are 4, 1 and 0, so the squared length is 5. With the stated root, the square root of 5 is 2.236, and that is the L2 norm of those weights. The penalty in this lesson is (λ / 2) times the squared length. At λ = 0.1 the factor λ / 2 is 0.05, and 0.05 times 5 is 0.25. A larger λ makes the same squared length cost more, so the pull toward zero on that row is stronger. The greeting row [0, 2, -1] has squares 0, 4 and 1, so its squared length is also 5 and its penalty at λ = 0.1 is also 0.25. Charging both rows adds 0.25 + 0.25 = 0.50.

From scores to cross-entropy

Softmax turns scores into probabilities that add to 1. Exponentiate each score, then divide by the sum of those exponentials. Use the rounded values e to the 0 equals 1, and e to the 1 equals 2.718. For the scores [0, 1] the sum is 3.718. Then 1 / 3.718 rounds to 0.269, and 2.718 / 3.718 rounds to 0.731. Those two rounded probabilities add to 1.000. Adding the same constant to every score leaves the probabilities unchanged, because the extra factor e to the c sits in every mass and cancels in the division. The scores [1, 2] are the old scores plus 1. With e to the 2 equal to 7.389 the masses are 2.718 and 7.389, and the sum is 10.107. Then 2.718 / 10.107 rounds to 0.269, and 7.389 / 10.107 rounds to 0.731. The displayed probabilities match.

A one-hot label puts 1 on the correct class and 0 on the others. Cross-entropy then equals the negative natural log of the correct probability. Use ln 2 = 0.693 and ln 3 = 1.099. Masses 1 and 3 have the sum 4, so the probabilities are 0.25 and 0.75. When the correct class is the second, ln(0.75) = ln 3 minus ln 4 = ln 3 minus 2 ln 2 = 1.099 - 1.386 = -0.287, and the loss is 0.287. When the correct class is the first, the loss is ln 4 = 1.386. A calculator gives ln(0.269) = -1.313 and ln(0.731) = -0.313. A billing label on probabilities 0.269 and 0.731 therefore has loss 1.313, and a greeting label has loss 0.313. Call the exponential of a one-note loss the spread. It equals 1 divided by the correct probability. A correct probability of 1/4 has spread 4, the same uncertainty as a flat draw over 4 labels. For two notes with correct probabilities 1/2 and 1/4, the losses are ln 2 and ln 4. Their mean is 1.5 times 0.693 = 1.0395, which rounds to 1.040. The spread of that mean is the geometric mean of the single-note spreads 2 and 4, so it is the square root of 8. With the square root of 2 equal to 1.414, twice 1.414 is 2.828.

One step of the chain rule

The chain rule multiplies local slopes. Take one weight w = 3, one feature x = 2, and a target score of 5. The score is s = w times x = 6. The loss is L = (s - 5) squared = 1. The slope of L with respect to s is 2 times (s - 5) = 2. The slope of s with respect to w is x = 2. The product of those slopes is 4, and that product is the slope of L with respect to w. A small check uses w = 3.001. Then s = 6.002 and the gap s - 5 is 1.002, so L = 1.002 squared = 1.004004. The difference quotient is (1.004004 - 1) / 0.001 = 4.004. The extra 0.004 is the bend of the square over a step of 0.001. The slope 4 is the value the quotient approaches as the step shrinks.

After softmax, the slope of cross-entropy with respect to each score is the probability minus the label entry. For probabilities 0.25 and 0.75 and a label on the second class, the label vector is [0, 1], so the score gradient is [0.25, -0.25]. For a tie, the scores [0, 0] give the probabilities [0.5, 0.5]. A label on the first class gives the score gradient [-0.5, 0.5]. A calculator gives the loss 0.693147 at those tied scores and the loss 0.692647 after 0.001 is added to the first score. From those displayed losses the difference quotient is (0.692647 - 0.693147) / 0.001 = -0.500, the same as the first entry of the gradient. Each score gradient then multiplies the feature vector to reach the weights. The slope of the penalty (λ / 2) times the squared length, with respect to a weight entry, is λ times that entry. The weights stay fixed while you compute the slope. A later update would subtract a step size times the gradient from each weight. That subtraction is a separate line from the slope itself.

An example

One note, from counts to weight slopes

Take the note refund thanks, with the feature vector x = [1, 1, 2]. The billing row is [2, -1, 0] with bias -1. The products are 2, -1 and 0, the dot product is 1, and the billing score is 0. The greeting row is [0, 2, -1] with bias 1. The products are 0, 2 and -2, the dot product is 0, and the greeting score is 1. In matrix form the weight matrix times x, plus the bias vector [-1, 1], returns the score vector [0, 1].

Use e to the 0 equals 1 and e to the 1 equals 2.718. The masses sum to 3.718. The billing probability rounds to 0.269 and the greeting probability rounds to 0.731. The label is billing, so the one-hot vector is [1, 0]. A calculator gives ln(0.269) = -1.313, and the cross-entropy is 1.313. Each weight row has squared length 5. At λ = 0.1 each row pays a penalty of 0.25, and the two rows together pay 0.50. The total is 1.313 + 0.50 = 1.813.

The score gradient is probability minus label: [0.269 - 1, 0.731 - 0] = [-0.731, 0.731]. The billing bias has slope -0.731, because the bias enters the score with slope 1. The greeting bias has slope 0.731. Multiply the billing score slope by x to reach the billing weights: -0.731 times [1, 1, 2] equals [-0.731, -0.731, -1.462]. The penalty adds λ times the weight row. For billing that addition is 0.1 times [2, -1, 0] = [0.2, -0.1, 0]. Entry by entry, -0.731 + 0.2 = -0.531, -0.731 + (-0.1) = -0.831, and -1.462 + 0 = -1.462. The full billing gradient is [-0.531, -0.831, -1.462].

The greeting data gradient is 0.731 times [1, 1, 2] = [0.731, 0.731, 1.462]. The penalty addition is 0.1 times [0, 2, -1] = [0, 0.2, -0.1]. Entry by entry, 0.731 + 0 = 0.731, 0.731 + 0.2 = 0.931, and 1.462 + (-0.1) = 1.362. The full greeting gradient is [0.731, 0.931, 1.362]. The same four lines, a matrix-vector product, a softmax, a negative log, and a chain-rule pass, are enough to fill a gradient array by hand before any training loop runs.

Exercise

Run a second note on paper

Use x = [2, 0, 1], the billing row [1, 4, -2] with bias 1, and the greeting row [0, -1, 3] with bias -1. Write both scores. With e to the 1 equal to 2.718 and e to the 2 equal to 7.389, write both probabilities rounded to three decimals. The label is greeting. Write the cross-entropy from the stated log ln(0.731) = -0.313. Charge an L2 penalty on the billing row only, with λ = 0.02, and add it to the loss. Then write the score gradient, the billing data gradient, the billing penalty gradient, and the full billing gradient. Also multiply C = [[1, 0], [2, 1]] by D = [[4, -1], [0, 3]]. Separately, take w = 1, x = 4 and a target score of 3. The score is w times x, the loss is the square of score minus 3, and you want the slope with respect to w, plus the difference quotient at w = 1.001.

The scores are 1 and 2. The masses 2.718 and 7.389 sum to 10.107, and the probabilities round to 0.269 and 0.731. The greeting loss is 0.313. The billing squares are 1, 16 and 4, so the squared length is 21. The penalty is 0.01 times 21 = 0.21, and the total loss is 0.313 + 0.21 = 0.523. The score gradient is [0.269, -0.269]. The billing data gradient is 0.269 times [2, 0, 1] = [0.538, 0, 0.269]. The penalty gradient is 0.02 times [1, 4, -2] = [0.02, 0.08, -0.04]. The full billing gradient is [0.558, 0.08, 0.229]. The product CD is [[4, -1], [8, 1]]. For the scalar, the score is 4 and the loss is 1. The slope is 2 times 4 = 8. At w = 1.001 the score is 4.004, the loss is 1.004 squared = 1.008016, and the difference quotient is 8.016.

Check questions

Answer on your own first. Then open a question to see the answer.

For the note refund thanks, with billing weights [2, -1, 0] and bias -1, what is the billing score?

0. The products are 2, -1 and 0, their sum is 1, and the bias subtracts 1.

Using e to the 0 = 1, e to the 1 = 2.718 and e to the 2 = 7.389, what probabilities rounded to three decimals do the scores [0, 1] and the scores [1, 2] share?

Both pairs round to 0.269 and 0.731. Adding 1 to both scores leaves the displayed probabilities unchanged.

On the worked note the billing label has cross-entropy 1.313, and each of the two rows pays an L2 penalty of 0.25. What is the total, and what is the full billing weight gradient?

The total is 1.813. The full billing gradient is [-0.531, -0.831, -1.462].

For probabilities 0.25 and 0.75 with the label on the second class, what is the cross-entropy from ln 2 = 0.693 and ln 3 = 1.099, and what is the score gradient?

The loss is 0.287. The score gradient is [0.25, -0.25].

Take with you

A linear scorer is a matrix-vector product plus a bias. Softmax turns the scores into probabilities, cross-entropy reads the correct probability, and the chain rule carries probability minus label back onto each weight, with λ times the weight added when an L2 penalty is on.

Final task

Submit a pure-NumPy autograd and optimiser module (micrograd_np.py) that passes 30 fixed unit tests comparing forward outputs, backward gradients, and AdamW weight updates against reference tensors within 1e-6 tolerance.

The final task is marked against fixed answer keys and automated tests once the tests open.

Terms in this course

This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.

More lessons coming

The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist