## Chapter 4 -  Training Models

### Regularised Linear Models

Recall that the cost function for:

Least Squares

$$\begin{align}J(\Theta) = \text{RSS} &= \sum_{i=1}^n \begin{pmatrix} y_i -  \hat{y_i} \end{pmatrix}^2
\\&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2\end{align}$$

Ridge Regression ($l_2$ normalisation)

$$\begin{align}J(\Theta) = \text{RSS} + \lambda \sum_{j=1}^p\beta^2_j &=\sum_{i=1}^n\begin{pmatrix} y_i - \hat{y_i} \end{pmatrix}^2 + \lambda \sum_{j=1}^p \beta^2_j\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2 + \lambda \sum_{j=1}^p\beta^2_j\end{align}$$

Lasso Regression ($l_1$ normalisation)

$$\begin{align}J(\Theta) = \text{RSS} + \lambda \sum_{j=1}^p|\beta_j| &=\sum_{i=1}^n\begin{pmatrix} y_i - \hat{y_i} \end{pmatrix}^2 + \lambda \sum_{j=1}^p |\beta_j|\\
&= \sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2 + \lambda \sum_{j=1}^p|\beta_j|\end{align}$$

and the objective is to solve for $\beta_0, \cdots, \beta_p$ that minimises the cost function.

Another formulation is:

Ridge Regression

$$\underset{\beta_0, \cdots, \beta_p}{\text{Minimise}}\left\{\sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2\right\}\text{ s. t. } \sum_{j=1}^p\beta^2_j \leq s$$

Lasso Regression
$$\underset{\beta_0, \cdots, \beta_p}{\text{Minimise}}\left\{\sum_{i=1}^n \begin{pmatrix} y_i - \beta_0 - \sum_{j=1}^p \beta_j x_{ij}\end{pmatrix}^2\right\}\text{ s. t. } \sum_{j=1}^p |\beta_j| \leq s$$

In other words, for every value of $\lambda$, there is some $s$ that gives the same coefficients for both sets of equations (ridge vs ridge, lasso vs lasso).

Graphical Representation
Consider $p=2$. The ridge minimisation constraint is now a circle $\beta_1^2 + \beta_2^2 \leq s$. Similarly, the Lasso minimisation constraint is now a diamond $|\beta_1| + |\beta_2| \leq s$. The coefficient estimates must be a point in these constraint regions (diamond / circle).

That means the coefficient estimates now need to satisfy the constraint bounded by $s$. If $s$ is very large, there is no restriction and the coefficient estimates could be the same as the solution without regularisation. If $s$ is small, then the coefficient estimates will be optimised within the constraint.

Graphically, the red contours represent the cost function, $J(\Theta)$. The contour lines all represent the same value of $J(\Theta)$. Recall that it is convex. The black coordinate $\hat{\beta_0}$ represents the least squares estimate of $\beta_1, \beta_2$. The diamond and circle represent that constraints $|\beta_1| + |\beta_2| \leq s$ and $\beta_1^2 + \beta_2^2 \leq s$ respectively. The point (on the edge of the region that is closest to the least squares estimate) is hence the coefficient estimates of the regularised model.

From here, observe that a large $s$ will eventually result in the least squares estimate to be in the region. That corresponds to a small $\lambda$.

Also observe that the lasso constrains have corners on the axes. When the minimum value of $J(\Theta)$ lies on the corners, then the coefficient estimate is $0$. In this case $\hat{\beta_1}=0$ and the regularised model only contains $\hat{\beta_2}$.

When $p=3$ then the ridge regression constraint is a sphere and the lasso regression constraint is a polyhedron. These key ideas still hold as $p$ increases beyond geometric formulations.


```python
import sympy
```

# Item V

Implement in Jupyter Notebook the Lagrange Interpolation method with *sympy*.
Then, find the interpolation polynomial for the following points:
* $(0,1),(1,2),(2,4)$. Is a second degree polynomial? If not, why is this?
* $(0,1),(1,2),(2,3)$. Is a second degree polynomial? If not, why is this?
---

```python
def lagrange(xs,ys):
    assert(len(xs)==len(ys))
    n = len(xs)
    x = sympy.Symbol('x')

    poly = 0
    for j in range(0,n):
        lag = ys[j]
        for m in range(0,n):
            if j!=m:
                lag *= (x-xs[m])/(xs[j]-xs[m])
        poly += lag
    return sympy.simplify(poly)
```

```python
lagrange([0,1,2],[1,2,4])
```

$\displaystyle \frac{x^{2}}{2} + \frac{x}{2} + 1$

```python
lagrange([0,1,2],[1,2,3])
```

$\displaystyle x + 1$

* The first points have to be interpolated by a second degree polynomial as they are not collinear, so they cannot be interpolated by a line.
* The second set of points are collinear, so they are interpolated by a polynomial of degree 1.


## Expected value of random variable

Expected value of random variable is generalization of taking average of numbers. It is similar to taking weighted average, where each value of random variable is multiplied by it's probability.

$$\mathbb{E}[X] = \sum_{x \in \mathcal{X}} x \cdot p_X(x) $$

Also in terms of conditional probability,
$$\mathbb{E}[X \mid Y=y] = \sum_{x \in \mathcal{X}} x \cdot p_{X\mid Y}(x\mid y)$$

In general, let $f$ any function from $\mathbb{R}$ to $\mathbb{R}$, then

$$ \mathbb{E}[f(X)] = \sum_{x \in \mathcal{X}} f(x) \cdot p_X(x) $$

Thus expectection gives a single number associated with a probability table.

### Exercise: Expected Value

Suppose that a student's score on a test will be $100$ if she studies the week before, and $75$ if she does not. Suppose also that the student's probability of studying the week before is $0.8$. What is her expected score? (Please provide an exact answer.)

```python
X = {'S': 100, 'N':75}; p_X = {'S': 0.80, 'N': 0.20}
E_X = sum([X[i] * p_X[i] for i in X]); E_X
```

    95.0

Let's look at why the expected value of a random variable is in some sense a “good" average value. Let $X$ be the result of a single fair six-sided die with faces $1$ up through $6$.

Simulate 10,000 rolls of the die roll in Python and take the average of the faces that appeared. What do you get? (Just make a note of it. There's no answer box to enter this in.)

What is $\mathbb{E}[X]$? (Please provide an exact answer.)

```python
E_X = sum([i * 1/6 for i in range(1,7)]); E_X
```

    3.5

You should notice that the average you get in simulation should be very close to E[X], and in fact, if you increase the number of rolls, it will tend to get closer (it doesn't necessarily have to get closer when you do each additional roll but the trend is there as you just keep increasing the number of rolls).

```python
import sys
import numpy as np
sys.path.append('../comp_prob_inference')
import comp_prob_inference
p_X = {i: 1/6 for i in range(1, 7)}
num_samples = 10000
print(np.mean([comp_prob_inference.sample_from_finite_probability_space(p_X) for n in range(num_samples)]))
```

    3.497

```python
import matplotlib.pyplot as plt
plt.figure(figsize=(8, 4))

n = 5000
x = list(range(1, n+1))
y = []
for i in x:
    if i == 1:
        y.append(comp_prob_inference.sample_from_finite_probability_space(p_X))
    if i > 1:
        y.append((y[i-2] * (i-1) + comp_prob_inference.sample_from_finite_probability_space(p_X)) / i)

plt.xlabel('No of dice rolled')
plt.ylabel('Expected value')
plt.plot(x,y)
plt.show()
```

We can observe that as the no of dice roll increases the become closer to $3.5$.

## Variance

This exercise explores the important concept of variance, which measures how much a random variable deviates from its expectation. This can be thought of as a measure of uncertainty. Higher variance means more uncertainty.

The variance of a real-valued random variable $X$ is defined as

$$\text {var}(X) \triangleq \mathbb {E}[ (X - \mathbb {E}[X])^2 ].$$

Note that as we saw previously, $\mathbb{E}[X]$ is just a single number. To keep the variance of $X$, what you could do is first compute the expectation of $X$.

For example, if $X$ takes on each of the values $3$, $5$, and $10$ with equal probability $1/3$, then first we compute $\mathbb{E}[X]$ to get $6$, and then we compute $\mathbb{E}[(X−6)^2]$, where we remember to use the result that for a function $f$, if $f(X)$ is a real-valued random variable, then $\mathbb{E}[f(X)]=\sum_x xf(x)pX(x)$. Here, $f$ is given by $f(x)=(x−6)^2$. So

$$\text {var}(X) = (3 - 6)^2 \cdot \frac13 + (5 - 6)^2 \cdot \frac13 + (10 - 6)^2 \cdot \frac13 = \frac{26}{3}.$$

```python
def E(p_X):
    return sum([key * value for key, value in p_X.items()])

def VAR(p_X):
    avg  = E(p_X)
    p_Xt = {(key - avg)**2 : value for key, value in p_X.items()}
    return E(p_Xt)
```

### Exercise

Let's return to the three lotteries from earlier. Here, random variables $L_1$, $L_2$, and $L_3$ represent the amount won (accounting for having to pay \$1):

|$L_1$      |            $p$           |        $L_2$ |            $p$           |   $L_3$ |       $p$      |
|----------:|:------------------------:|-------------:|:------------------------:|--------:|:--------------:|
|        -1 | $\frac{999999}{1000000}$ |           -1 | $\frac{999999}{1000000}$ |      -1 | $\frac{9}{10}$ |
|   -1+1000 |    $\frac{1}{1000000}$   |   -1+1000000 |    $\frac{1}{1000000}$   |   -1+10 | $\frac{1}{10}$ |

Compute the variance for each of these three random variables. (Please provide the exact answer for each of these.)

- var($L_1$)= {{V_1}}
- var($L_2$)= {{V_2}}
- var($L_3$)= {{V_3}}

```python
p_L1 = {-1: 999999/1000000, 999   : 1/1000000}
p_L2 = {-1: 999999/1000000, 999999: 1/1000000}
p_L3 = {-1:      9/10     , 9     : 1/10     }

V_1 = VAR(p_L1)
V_2 = VAR(p_L2)
V_3 = VAR(p_L3)
```

What units is variance in? Notice that we started with dollars, and then variance is looking at the expectation of a dollar amount squared. Thus, specifically for the lottery example $\text {var}(L_1)$, $\text {var}(L_2)$, and $\text {var}(L_3)$ are each in squared dollars.

## Standard Deviation

Some times, people prefer keeping the units the same as the original units (i.e., without squaring), which you can get by computing what's called the standard deviation of a real-valued random variable $X$:

$$\text {std}(X) \triangleq \sqrt {\text {var}(X)}.$$

```python
def STD(p_X):
    from sympy import sqrt
    return sqrt(VAR(p_X))
```

### Exercise

Compute the following standard deviations, which are in units of dollars. (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

- std($L_1$) =  {{print(S_1)}}
- std($L_2$) =  {{print(S_2)}}
- std($L_3$) =  {{print(S_3)}}

```python
S_1 = STD(p_L1)
S_2 = STD(p_L2)
S_3 = STD(p_L3)
```

!Note
    When we first introduced the three lotteries and computed average winnings, we didn't account for the uncertainty in the average winnings. Here, it's clear that the third lottery has far smaller standard deviation and variance than the second lottery.<br>
    As a remark, often in financial applications (e.g., choosing a portfolio of stocks to invest in), accounting for uncertainty is extremely important. For example, you may want to maximize profit while ensuring that the amount of uncertainty is not too high as to not be reckless in investing.

In the case of the three lotteries, to decide between them, you could for example use a score that is of the form

$$\mathbb {E}[L_ i] - \lambda \cdot \text {std}(L_ i) \qquad \text {for }i = 1,2,3,$$

where $λ≥0$ is some parameter that you choose for how much you want to penalize uncertainty in the lottery outcome. Then you could choose the lottery with the highest score.

Finally, a quick sanity check (this is more for you to think about the definition of variance rather than to compute anything out):

**Question:** Can variance be negative? If yes, give a specific distribution as a Python dictionary for which the variance is negative. If no, enter the text "no" (all lowercase, one word, no spaces).

**Answer:** NO

## The Law of Total Expectation

Remember the law of total probability? For a set of events $\mathcal{B}_{1},\dots ,\mathcal{B}_{n}$ that partition the sample space $Ω$ (so the Bi's don't overlap and together they fully cover the full space of possible outcomes),

$$\mathbb {P}(\mathcal{A})=\sum _{i=1}^{n}\mathbb {P}(\mathcal{A}\cap \mathcal{B}_{i})=\sum _{i=1}^{n}\mathbb {P}(\mathcal{A}\mid \mathcal{B}_{i})\mathbb {P}(\mathcal{B}_{i}),$$

where the second equality uses the product rule.

A similar statement is true for the expected value of a random variable, called the law of total expectation: for a random variable $X$ (with alphabet $\mathcal{X}$) and a partition $\mathcal{B}_1,\dots ,\mathcal{B}_ n$ of the sample space,

$$\mathbb {E}[X]=\sum _{i=1}^{n}\mathbb {E}[X\mid \mathcal{B}_{i}]\mathbb {P}(\mathcal{B}_{i}),$$

where

$$\mathbb {E}[X\mid \mathcal{B}_{i}] = \sum _{x\in \mathcal{X}}xp_{X\mid \mathcal{B}_{i}}(x) = \sum _{x\in \mathcal{X}}x\frac{\mathbb {P}(X=x,\mathcal{B}_{i})}{\mathbb {P}(\mathcal{B}_{i})}.$$

We will be using this result in the section “Towards Infinity in Modeling Uncertainty".

Show that the law of total expectation is true.

**Solution:** There are different ways to prove the law of total expectation. We take a fairly direct approach here, first writing everything in terms of outcomes in the sample space.

The main technical hurdle is that the events $\mathcal{B}_1, \dots , \mathcal{B}_ n$ are specified directly in the sample space, whereas working with values that $X$ takes on requires mapping from the sample space to the alphabet of $X$.

We will derive the law of total expectation starting from the right-hand side of the equation above, i.e., $\sum _{i=1}^{n}\mathbb {E}[X\mid \mathcal{B}_{i}]\mathbb {P}(\mathcal{B}_{i})$.

We first write $\mathbb {E}[X\mid \mathcal{B}_{i}]$ in terms of a summation over outcomes in $\Omega$:

$$\begin{align}
\mathbb {E}[X\mid \mathcal{B}_{i}]	=& \sum _{x\in \mathcal{X}}x\frac{\mathbb {P}(X=x,\mathcal{B}_{i})}{\mathbb {P}(\mathcal{B}_{i})}\\
=& \sum _{x\in \mathcal{X}}x\frac{\mathbb {P}(\{ \omega \in \Omega \; :\; X(\omega )=x\} \cap \mathcal{B}_{i})}{\mathbb {P}(\mathcal{B}_{i})}\\
=& \sum _{x\in \mathcal{X}}x\frac{\mathbb {P}(\{ \omega \in \Omega \; :\; X(\omega )=x\text { and }\omega \in \mathcal{B}_{i}\} )}{\mathbb {P}(\mathcal{B}_{i})}\\
=& \sum _{x\in \mathcal{X}}x\frac{\mathbb {P}(\{ \omega \in \mathcal{B}_{i}\; :\; X(\omega )=x\} )}{\mathbb {P}(\mathcal{B}_{i})}\\
=& \sum _{x\in \mathcal{X}}x\cdot \frac{\sum _{\omega \in \mathcal{B}_{i}\text { such that }X(\omega )=x}\mathbb {P}(\{ \omega \} )}{\mathbb {P}(\mathcal{B}_{i})}	\\
=& \frac{1}{\mathbb {P}(\mathcal{B}_{i})}\sum _{x\in \mathcal{X}}x\sum _{\omega \in \mathcal{B}_{i}\text { such that }X(\omega )=x}\mathbb {P}(\{ \omega \} )\\
=& \frac{1}{\mathbb {P}(\mathcal{B}_{i})}\sum _{\omega \in \mathcal{B}_{i}}X(\omega )\mathbb {P}(\{ \omega \} ).
\end{align}$$

Thus,

$$\begin{align}
\sum _{i=1}^{n}\mathbb {E}[X\mid \mathcal{B}_{i}]\mathbb {P}(\mathcal{B}_{i})=& \sum _{i=1}^{n}\bigg(\frac{1}{\mathbb {P}(\mathcal{B}_{i})}\sum _{\omega \in \mathcal{B}_{i}}X(\omega )\mathbb {P}(\{ \omega \} )\bigg)\mathbb {P}(\mathcal{B}_{i})\\
=& \sum _{i=1}^{n}\sum _{\omega \in \mathcal{B}_{i}}X(\omega )\mathbb {P}(\{ \omega \} )\\
=& \sum _{\omega \in \Omega }X(\omega )\mathbb {P}(\{ w\} )\\
=& \sum _{x\in \mathcal{X}}x\mathbb {P}(\{ \omega \in \Omega \text { such that }X(\omega )=x\} )\\
=& \sum _{x\in \mathcal{X}}xp_{X}(x)\\
=&\mathbb {E}[X].
\end{align}$$

```python

```


```python
from sympy import *
init_printing(use_latex='mathjax')
n = symbols('n', integer=True)
x, y, z = symbols('x,y,z')
```

## Integrales

En la primera sección aprendimos diferenciación simbólica con `diff`. Aquí abordaremos la integración simbólica con `integrate`.

Así es cómo escribimos la integral indefinida

$$ \int x^2 dx = \frac{x^3}{3}$$

```python
# Integral indefinida
integrate(x**2, x)
```

y la integral definida

$$ \int_0^3 x^2 dx = \left.\frac{x^3}{3} \right|_0^3 = \frac{3^3}{3} - \frac{0^3}{3} = 9 $$

```python
# Integral definida
integrate(x**2, (x, 0, 3))
```

Como siempre, debido a que estamos usando símbolos, podríamos usar un símbolo donde sea que hayamos usado un número

$$ \int_y^z x^n dx $$

```python
integrate(x**n, (x, y, z))
```

### Ejercicio

Calcule las siguientes integrales:

$$ \int \sin(x) dx $$
$$ \int_0^{\pi} \sin(x) dx $$
$$ \int_0^y x^5 + 12x^3 - 2x + 1  $$
$$ \int e^{\frac{(x - \mu)^2}{\sigma^2}} $$

Siéntete libre de jugar con varios parámetros y configuraciones y ver cómo cambian los resultados.

```python
# Usa `integrate` para resolver las integrales anteriores

```

¿Hay algunas integrales que *SymPy* no puede hacer? Encuentra alguna.

```python
# Usa `integrate` en otras ecuaciones. La integración simbólica tiene sus límites, encuéntralos.
```


# Applications

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as la
```

## Polynomial Interpolation

[Polynomial interpolation](https://en.wikipedia.org/wiki/Polynomial_interpolation) finds the unique polynomial of degree $n$ which passes through $n+1$ points in the $xy$-plane. For example, two points in the $xy$-plane determine a line and three points determine a parabola.

### Formulation

Suppose we have $n + 1$ points in the $xy$-plane

$$
(x_0,y_0),(x_1,y_1),\dots,(x_n,y_n)
$$

such that all the $x$ values are distinct ($x_i \not= x_j$ for $i \not= j$). The general form of a degree $n$ polynomial is

$$
p(x) = a_0 + a_1 x + a_2x^2 + \cdots + a_n x^n
$$

If $p(x)$ is the unique degree $n$ polynomial which interpolates all the points, then the coefficients $a_0$, $a_1$, $\dots$, $a_n$ satisfy the following equations:

\begin{align}
a_0 + a_1x_0 + a_2x_0^2 + \cdots + a_n x_0^n &= y_0 \\\
a_0 + a_1x_1 + a_2x_1^2 + \cdots + a_n x_1^n &= y_1 \\\
& \ \ \vdots \\\
a_0 + a_1x_n + a_2x_n^2 + \cdots + a_n x_n^n &= y_n
\end{align}

Therefore the vector of coefficients

$$
\mathbf{a} =
\begin{bmatrix}
a_0 \\\
a_1 \\\
\vdots \\\
a_n
\end{bmatrix}
$$

is the unique the solution of the linear system of equations

$$
X \mathbf{a}=\mathbf{y}
$$

where $X$ is the [Vandermonde matrix](https://en.wikipedia.org/wiki/Vandermonde_matrix) and $\mathbf{y}$ is the vector of $y$ values

$$
X =
\begin{bmatrix}
1 & x_0 & x_0^2 & \dots & x_0^n \\\
1 & x_1 & x_1^2 & \dots & x_1^n \\\
 & \vdots & & & \vdots \\\
1 & x_n & x_n^2 & \dots & x_n^n \\\
\end{bmatrix}
\ \ \mathrm{and} \ \
\mathbf{y} =
\begin{bmatrix}
y_0 \\\
y_1 \\\
y_2 \\\
\vdots \\\
y_n
\end{bmatrix}
$$

### Examples

**Simple Parabola**

Let's do a simple example. We know that $y=x^2$ is the unique degree 2 polynomial that interpolates the points $(-1,1)$, $(0,0)$ and $(1,1)$. Let's compute the polynomial interpolation of these points and verify the expected result $a_0=0$, $a_1=0$ and $a_2=1$.

Create the Vandermonde matrix $X$ with the array of $x$ values:

```python
x = np.array([-1,0,1])
X = np.column_stack([[1,1,1],x,x**2])
print(X)
```

    [[ 1 -1  1]
     [ 1  0  0]
     [ 1  1  1]]

Create the vector $\mathbf{y}$ of $y$ values:

```python
y = np.array([1,0,1]).reshape(3,1)
print(y)
```

    [[1]
     [0]
     [1]]

We expect the solution $\mathbf{a} = [0,0,1]^T$:

```python
a = la.solve(X,y)
print(a)
```

    [[0.]
     [0.]
     [1.]]

Success!

**Another Parabola**

The polynomial interpolation of 3 points $(x_0,y_0)$, $(x_1,y_1)$ and $(x_2,y_2)$ is the parabola $p(x) = a_0 + a_1x + a_2x^2$ such that the coefficients satisfy

\begin{align}
a_0 + a_1x_0 + a_2x_0^2 = y_0 \\\
a_0 + a_1x_1 + a_2x_1^2 = y_1 \\\
a_0 + a_1x_2 + a_2x_2^2 = y_2
\end{align}

Let's find the polynomial interpolation of the points $(0,6)$, $(3,1)$ and $(8,2)$.

Create the Vandermonde matrix $X$:

```python
x = np.array([0,3,8])
X = np.column_stack([[1,1,1],x,x**2])
print(X)
```

    [[ 1  0  0]
     [ 1  3  9]
     [ 1  8 64]]

And the vector of $y$ values:

```python
y = np.array([6,1,2]).reshape(3,1)
print(y)
```

    [[6]
     [1]
     [2]]

Compute the vector $\mathbf{a}$ of coefficients:

```python
a = la.solve(X,y)
print(a)
```

    [[ 6.        ]
     [-2.36666667]
     [ 0.23333333]]

And plot the result:

```python
xs = np.linspace(0,8,20)
ys = a[0] + a[1]*xs + a[2]*xs**2
plt.plot(xs,ys,x,y,'b.',ms=20)
plt.show()
```

**Over Fitting 10 Random Points**

Now let's interpolate points with $x_i=i$, $i=0,\dots,9$, and 10 random integers sampled from $[0,10)$ as $y$ values:

```python
N = 10
x = np.arange(0,N)
y = np.random.randint(0,10,N)
plt.plot(x,y,'r.')
plt.show()
```

Create the Vandermonde matrix and verify the first 5 rows and columns:

```python
X = np.column_stack([x**k for k in range(0,N)])
print(X[:5,:5])
```

    [[  1   0   0   0   0]
     [  1   1   1   1   1]
     [  1   2   4   8  16]
     [  1   3   9  27  81]
     [  1   4  16  64 256]]

We could also use the NumPy function [`numpy.vander`](https://docs.scipy.org/doc/numpy-1.15.0/reference/generated/numpy.vander.html). We specify the option `increasing=True` so that powers of $x_i$ increase left-to-right:

```python
X = np.vander(x,increasing=True)
print(X[:5,:5])
```

    [[  1   0   0   0   0]
     [  1   1   1   1   1]
     [  1   2   4   8  16]
     [  1   3   9  27  81]
     [  1   4  16  64 256]]

Solve the linear system:

```python
a = la.solve(X,y)
```

Plot the interpolation:

```python
xs = np.linspace(0,N-1,200)
ys = sum([a[k]*xs**k for k in range(0,N)])
plt.plot(x,y,'r.',xs,ys)
plt.show()
```

Success! But notice how unstable the curve is. That's why it better to use a [cubic spline](https://en.wikipedia.org/wiki/Spline_%28mathematics%29) to interpolate a large number of points.

However real-life data is usually very noisy and interpolation is not the best tool to fit a line to data. Instead we would want to take a polynomial with smaller degree (like a line) and fit it as best we can without interpolating the points.

## Least Squares Linear Regression

Suppose we have $n+1$ points

$$
(x_0,y_0) , (x_1,y_1) , \dots , (x_n,y_n)
$$

in the $xy$-plane and we want to fit a line

$$
y=a_0 + a_1x
$$

that "best fits" the data. There are different ways to quantify what "best fit" means but the most common method is called [least squares linear regression](https://en.wikipedia.org/wiki/Linear_regression). In least squares linear regression, we want to minimize the sum of squared errors

$$
SSE = \sum_i (y_i - (a_0 + a_1 x_i))^2
$$

### Formulation

If we form matrices

$$
X =
\begin{bmatrix}
1 & x_0 \\\
1 & x_1 \\\
\vdots & \vdots \\\
1 & x_n
\end{bmatrix}
\ , \ \
\mathbf{y} =
\begin{bmatrix}
y_0 \\\
y_1 \\\
\vdots \\\
y_n
\end{bmatrix}
\ , \ \
\mathbf{a} =
\begin{bmatrix}
a_0 \\\ a_1
\end{bmatrix}
$$

then the sum of squared errors can be expressed as

$$
SSE = \Vert \mathbf{y} - X \mathbf{a} \Vert^2
$$

---

**Theorem.** (Least Squares Linear Regression) Consider $n+1$ points

$$
(x_0,y_0) , (x_1,y_1) , \dots , (x_n,y_n)
$$

in the $xy$-plane. The coefficients $\mathbf{a} = [a_0,a_1]^T$ which minimize the sum of squared errors

$$
SSE = \sum_i (y_i - (a_0 + a_1 x_i))^2
$$

is the unique solution of the system

$$
\left( X^T X \right) \mathbf{a} = X^T \mathbf{y}
$$

*Sketch of Proof.* The product $X\mathbf{a}$ is in the column space of $X$. The line connecting $\mathbf{y}$ to the nearest point in the column space of $X$ is perpendicluar to the column space of $X$. Therefore

$$
X^T \left( \mathbf{y} - X \mathbf{a} \right) = \mathbf{0}
$$

and so

$$
\left( X^T X \right) \mathbf{a} = X^T \mathbf{y}
$$

---

### Examples

**Fake Noisy Linear Data**

Let's do an example with some fake data. Let's build a set of random points based on the model

$$
y = a_0 + a_1x + \epsilon
$$

for some arbitrary choice of $a_0$ and $a_1$. The factor $\epsilon$ represents some random noise which we model using the [normal distribution](https://en.wikipedia.org/wiki/Normal_distribution). We can generate random numbers sampled from the standard normal distribution using the NumPy function [`numpy.random.rand`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.randn.html).

The goal is to demonstrate that we can use linear regression to retrieve the coefficeints $a_0$ and $a_1$ from the linear regression calculation.

```python
a0 = 2
a1 = 3
N = 100
x = np.random.rand(100)
noise = 0.1*np.random.randn(100)
y = a0 + a1*x + noise
plt.scatter(x,y);
plt.show()
```

Let's use linear regression to retrieve the coefficients $a_0$ and $a_1$. Construct the matrix $X$:

```python
X = np.column_stack([np.ones(N),x])
print(X.shape)
```

    (100, 2)

Let's look at the first 5 rows of $X$ to see that it is in the correct form:

```python
X[:5,:]
```

    array([[1.        , 0.92365627],
           [1.        , 0.78757973],
           [1.        , 0.51506055],
           [1.        , 0.51540875],
           [1.        , 0.86563343]])

Use `scipy.linalg.solve` to solve $\left(X^T X\right)\mathbf{a} = \left(X^T\right)\mathbf{y}$ for $\mathbf{a}$:

```python
a = la.solve(X.T @ X, X.T @ y)
print(a)
```

    [2.02783873 2.95308228]

We have retrieved the coefficients of the model almost exactly! Let's plot the random data points with the linear regression we just computed.

```python
xs = np.linspace(0,1,10)
ys = a[0] + a[1]*xs
plt.plot(xs,ys,'r',linewidth=4)
plt.scatter(x,y);
plt.show()
```

**Real Kobe Bryant Data**

Let's work with some real data. [Kobe Bryant](https://www.basketball-reference.com/players/b/bryanko01.html) retired in 2016 with 33643 total points which is the [third highest total points in NBA history](https://en.wikipedia.org/wiki/List_of_National_Basketball_Association_career_scoring_leaders). How many more years would Kobe Bryant have to had played to pass [Kareem Abdul-Jabbar's](https://en.wikipedia.org/wiki/Kareem_Abdul-Jabbar) record 38387 points?

Kobe Bryant's peak was the 2005-2006 NBA season. Let's look at Kobe Bryant's total games played and points per game from 2006 to 2016.

```python
years = np.array([2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016])
games = [80,77,82,82,73,82,58,78,6,35,66]
points = np.array([35.4,31.6,28.3,26.8,27,25.3,27.9,27.3,13.8,22.3,17.6])

fig = plt.figure(figsize=(12,10))
axs = fig.subplots(2,1,sharex=True)
axs[0].plot(years,points,'b.',ms=15)
axs[0].set_title('Kobe Bryant, Points per Game')
axs[0].set_ylim([0,40])
axs[0].grid(True)
axs[1].bar(years,games)
axs[1].set_title('Kobe Bryant, Games Played')
axs[1].set_ylim([0,100])
axs[1].grid(True)
plt.show()
```

Kobe was injured for most of the 2013-2014 NBA season and played only 6 games. This is an outlier and so we can drop this data point:

```python
years = np.array([2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2015, 2016])
games = np.array([80,77,82,82,73,82,58,78,35,66])
points = np.array([35.4,31.6,28.3,26.8,27,25.3,27.9,27.3,22.3,17.6])
```

Let's compute the average games played per season over this period:

```python
avg_games_per_year = np.mean(games)
print(avg_games_per_year)
```

    71.3

Compute the linear model for points per game:

```python
X = np.column_stack([np.ones(len(years)),years])
a = la.solve(X.T @ X, X.T @ points)
model = a[0] + a[1]*years

plt.plot(years,model,years,points,'b.',ms=15)
plt.title('Kobe Bryant, Points per Game')
plt.ylim([0,40])
plt.grid(True)
plt.show()
```

Now we can extrapolate to future years and multiply points per games by games per season and compute the cumulative sum to see Kobe's total points:

```python
future_years = np.array([2017,2018,2019,2020,2021])
future_points = (a[0] + a[1]*future_years)*avg_games_per_year
total_points = 33643 + np.cumsum(future_points)
kareem = 38387*np.ones(len(future_years))

plt.plot(future_years,total_points,future_years,kareem)
plt.grid(True)
plt.xticks(future_years)
plt.title('Kobe Bryant Total Points Prediction')
plt.show()
```

Only 4 more years!

## Polynomial Regression

### Formulation

The same idea works for fitting a degree $d$ polynomial model

$$
y = a_0 + a_1x + a_2x^2 + \cdots + a_dx^d
$$

to a set of $n+1$ data points

$$
(x_0,y_0), (x_1,y_1), \dots , (x_n,y_n)
$$

We form the matrices as before but now the Vandermonde matrix $X$ has $d+1$ columns

$$
X =
\begin{bmatrix}
1 & x_0 & x_0^2 & \cdots & x_0^d \\\
1 & x_1 & x_1^2 & \cdots & x_1^d \\\
 & \vdots & & & \vdots \\\
1 & x_n & x_n^2 & \cdots & x_n^d
\end{bmatrix}
\ , \ \
\mathbf{y} =
\begin{bmatrix}
y_0 \\\
y_1 \\\
\vdots \\\
y_n
\end{bmatrix}
\ , \ \
\mathbf{a} =
\begin{bmatrix}
a_0 \\\
a_1 \\\
a_2 \\\
\vdots \\\
a_d
\end{bmatrix}
$$

The coefficients $\mathbf{a} = [a_0,a_1,a_2,\dots,a_d]^T$ which minimize the sum of squared errors $SSE$ is the unique solution of the linear system

$$
\left( X^T X \right) \mathbf{a} = \left( X^T \right) \mathbf{y}
$$

### Example

**Fake Noisy Quadratic Data**

Let's build some fake data using a quadratic model $y = a_0 + a_1x + a_2x^2 + \epsilon$ and use linear regression to retrieve the coefficients $a_0$, $a_1$ and $a_2$.

```python
a0 = 3
a1 = 5
a2 = 8
N = 1000
x = 2*np.random.rand(N) - 1 # Random numbers in the interval (-1,1)
noise = np.random.randn(N)
y = a0 + a1*x + a2*x**2 + noise
plt.scatter(x,y,alpha=0.5,lw=0);
plt.show()
```

Construct the matrix $X$:

```python
X = np.column_stack([np.ones(N),x,x**2])
```

Use `scipy.linalg.solve` to solve $\left( X^T X \right) \mathbf{a} = \left( X^T \right) \mathbf{y}$:

```python
a = la.solve((X.T @ X),X.T @ y)
```

Plot the result:

```python
xs = np.linspace(-1,1,20)
ys = a[0] + a[1]*xs + a[2]*xs**2
plt.plot(xs,ys,'r',linewidth=4)
plt.scatter(x,y,alpha=0.5,lw=0)
plt.show()
```

## Graph Theory

A [graph](https://en.wikipedia.org/wiki/Graph_%28discrete_mathematics%29) is a set of vertices and a set of edges connecting some of the vertices. We will consider simple, undirected, connected graphs:

* a graph is [simple](https://en.wikipedia.org/wiki/Graph_%28discrete_mathematics%29#Simple_graph) if there are no loops or multiple edges between vertices
* a graph is [undirected](https://en.wikipedia.org/wiki/Graph_%28discrete_mathematics%29#Undirected_graph) if the edges do not have an orientation
* a graph is [connected](https://en.wikipedia.org/wiki/Graph_%28discrete_mathematics%29#Connected_graph) if each vertex is connected to every other vertex in the graph by a path

We can visualize a graph as a set of vertices and edges and answer questions about the graph just by looking at it. However this becomes much more difficult with a large graphs such as a [social network graph](https://en.wikipedia.org/wiki/Social_network_analysis). Instead, we construct matrices from the graph such as the [adjacency matrix](https://en.wikipedia.org/wiki/Adjacency_matrix) and the [Laplacian matrix](https://en.wikipedia.org/wiki/Laplacian_matrix) and study their properties.

[Spectral graph theory](https://en.wikipedia.org/wiki/Spectral_graph_theory) is the study of the eigenvalues of the adjacency matrix (and other associated matrices) and the relationships to the structure of $G$.

### NetworkX

Let's use the Python package [NetworkX](https://networkx.github.io/) to construct and visualize some simple graphs.

```python
import networkx as nx
```

### Adjacency Matrix

The [adjacency matrix](https://en.wikipedia.org/wiki/Adjacency_matrix) $A_G$ of a graph $G$ with $n$ vertices is the square matrix of size $n$ such that $A_{i,j} = 1$ if vertices $i$ and $j$ are connected by an edge, and $A_{i,j} = 0$ otherwise.

We can use `networkx` to create the adjacency matrix of a graph $G$. The function `nx.adjacency_matrix` returns a [sparse matrix](https://docs.scipy.org/doc/scipy/reference/sparse.html) and we convert it to a regular NumPy array using the `todense` method.

For example, plot the [complete graph](https://en.wikipedia.org/wiki/Complete_graph) with 5 vertices and compute the adjacency matrix:

```python
G = nx.complete_graph(5)
nx.draw(G,with_labels=True)
```

```python
A = nx.adjacency_matrix(G).todense()
print(A)
```

    [[0 1 1 1 1]
     [1 0 1 1 1]
     [1 1 0 1 1]
     [1 1 1 0 1]
     [1 1 1 1 0]]

### Length of the Shortest Path

The length of the [shortest path](https://en.wikipedia.org/wiki/Shortest_path_problem) between vertices in a simple, undirected graph $G$ can be easily computed from the adjacency matrix $A_G$. In particular, the length of shortest path from vertex $i$ to vertex $j$ ($i\not=j$) is the smallest positive integer $k$ such that $A^k_{i,j} \not= 0$.

Plot the [dodecahedral graph](https://en.wikipedia.org/wiki/Regular_dodecahedron#Dodecahedral_graph):

```python
G = nx.dodecahedral_graph()
nx.draw(G,with_labels=True)
```

```python
A = nx.adjacency_matrix(G).todense()
print(A)
```

    [[0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1]
     [1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0]
     [0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0]
     [0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
     [0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0]
     [0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0]
     [0 0 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0]
     [0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0]
     [1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0]
     [0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0]
     [0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 0 0 0]
     [0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0]
     [0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0]
     [0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0]
     [0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0]
     [0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1]
     [1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0]]

With this labelling, let's find the length of the shortest path from vertex $0$ to $15$:

```python
i = 0
j = 15
k = 1
Ak = A
while Ak[i,j] == 0:
    Ak = Ak @ A
    k = k + 1
print('Length of the shortest path is',k)
```

    Length of the shortest path is 5

### Triangles in a Graph

A simple result in spectral graph theory is the number of [triangles](https://en.wikipedia.org/wiki/Adjacency_matrix#Matrix_powers) in a graph $T(G)$ is given by:

$$
T(G) = \frac{1}{6} ( \lambda_1^3 + \lambda_2^3 + \cdots + \lambda_n^3)
$$

where $\lambda_1 \leq \lambda_2 \leq \cdots \leq \lambda_n$ are the eigenvalues of the adjacency matrix.

Let's verify this for the simplest case, the complete graph on 3 vertices:

```python
C3 = nx.complete_graph(3)
nx.draw(C3,with_labels=True)
```

```python
A3 = nx.adjacency_matrix(C3).todense()
eigvals, eigvecs = la.eig(A3)
int(np.round(np.sum(eigvals.real**3)/6,0))
```

    1

Let's compute the number of triangles in the complete graph 7 vertices:

```python
C7 = nx.complete_graph(7)
nx.draw(C7,with_labels=True)
```

```python
A7 = nx.adjacency_matrix(C7).todense()
eigvals, eigvecs = la.eig(A7)
int(np.round(np.sum(eigvals.real**3)/6,0))
```

    35

There are 35 triangles in the complete graph with 7 vertices!

Let's write a function called `triangles` which takes a square matrix `M` and return the sum

$$
\frac{1}{6} ( \lambda_1^3 + \lambda_2^3 + \cdots + \lambda_n^3)
$$

where $\lambda_i$ are the eigenvalues of the symmetric matrix $A = (M + M^T)/2$. Note that $M = A$ if $M$ is symmetric. The return value is the number of triangles in the graph $G$ if the input $M$ is the adjacency matrix.

```python
def triangles(M):
    A = (M + M.T)/2
    eigvals, eigvecs = la.eig(A)
    eigvals = eigvals.real
    return int(np.round(np.sum(eigvals**3)/6,0))
```

Next, let's try a [Turan graph](https://en.wikipedia.org/wiki/Tur%C3%A1n_graph).

```python
G = nx.turan_graph(10,5)
nx.draw(G,with_labels=True)
```

```python
A = nx.adjacency_matrix(G).todense()
print(A)
```

    [[0 0 1 1 1 1 1 1 1 1]
     [0 0 1 1 1 1 1 1 1 1]
     [1 1 0 0 1 1 1 1 1 1]
     [1 1 0 0 1 1 1 1 1 1]
     [1 1 1 1 0 0 1 1 1 1]
     [1 1 1 1 0 0 1 1 1 1]
     [1 1 1 1 1 1 0 0 1 1]
     [1 1 1 1 1 1 0 0 1 1]
     [1 1 1 1 1 1 1 1 0 0]
     [1 1 1 1 1 1 1 1 0 0]]

Find the number of triangles:

```python
triangles(A)
```

    80

Finally, let's compute the number of triangles in the dodecahedral graph:

```python
G = nx.dodecahedral_graph()
nx.draw(G,with_labels=True)
```

```python
A = nx.adjacency_matrix(G).todense()
print(A)
```

    [[0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1]
     [1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0]
     [0 1 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0]
     [0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]
     [0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0]
     [0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0]
     [0 0 1 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0]
     [0 1 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0 0 0 0]
     [1 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0]
     [0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0]
     [0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 0 0 0]
     [0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0 0 0 0]
     [0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 0 0 0]
     [0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 1 0 0 0]
     [0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 1 0 1 0 0]
     [0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0]
     [0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1]
     [1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0]]

```python
np.round(triangles(A),2)
```

    0

Python/3. Computational Sciences and Mathematics/Linear Algebra/Various Applications -- Polynomial Interpolation, Least Squares Linear Regression, Polynomial Regression, Graph Theory, Network Visualizations with NetworkX.ipynb


# Aula 3
## Ferramentas básicas de cálculo
A biblioteca Sympy disponibiliza algumas ferramentas para cálculo integral e diferencial, dentre outras.

```python
from sympy import *
import numpy as np
import matplotlib.pyplot as plt
```

A linha abaixo realiza modificações para que as expressões matemáticas fiquem em formato legível

```python
init_printing(use_unicode=False,
              wrap_line=False, no_global=True) # Faz com que as expressoes sejam exibidas e formato legivel
```

É necessário também declarar símbolos e constatens que serão utilizados.

```python
x, y, z, t, theta = symbols('x y z t theta') # Declaracao de variaveis
k1, k2, m, n, a = symbols('k1 k2 m n a', integer=True) # Declaracao de constantes
```

Para realizar uma integral indefinida utiliza-se a função integrate(formula, variavel de integracao)

```python
integrate(x**2,x)
```

No caso de integral definida, deve ser passado uma tupla no formato (variavel de integracao, lim. inferior, lim. superior)

```python
integrate(x**2,(x,-1, 1))
```

Integrais multivariadas funcionam de maneira similar:

```python
integrate(x*y, (x,-7,3),(y,-13,5))
```

É possível também definir funções como variáveis, desde que o parâmetro passado seja um símbolo definido anteriormente.

```python
f = x**3+x**2+x+5
f
```

É possível integrar,diferenciar ou simplesmente avaliar a expressão criada:

```python
integrate(f, (x,-10,10)) # Integracao

```

```python
f.diff(x) # Diferenciacao
```

```python
f.subs(x,1) # Avaliacao da expressao em x=1
```

Funções trigonométricas também são válidas

```python
h = cos(theta)
h.subs(theta, pi)
```

## Exercício 1: Calcule a Série de Taylor para a expressão $cos(\theta)$ com três termos em torno de um ponto $a$ arbitrário:

```python

```

## Exercício 2: Plote a fução de erro para a aproximação por Série de Taylor no intervalo entre -pi e pi:

Dica: você pode usar uma estratégia similar a utilizada na Aula 1 para plotar o gráfico de Força vs Distância.

```python

```

## Exercício 3: Crie uma função que receba uma expressão e um número inteiro $n$ e que calcule os $n$ primeiros elementos da Série de Taylor para aquela expressão em torno do ponto $a$ e retorne a expressão completa:

```python
def CalcularTaylor(expr, n, a):
    ## complete a funcao
```

### Material de consulta
https://en.wikipedia.org/wiki/Taylor_series - teoria de Série de Taylor

https://www.youtube.com/watch?v=3d6DsjIBzJ4 - vídeo explicativo

http://docs.sympy.org/latest/index.html - documentação da biblioteca sympy


```python
from scipy import linalg as la
from scipy import optimize

import sympy
sympy.init_printing()

import numpy as np

import matplotlib.pyplot as plt
```

```python
# Graphical solution

x = np.arange(-4, 2, 1)
x2 = np.arange(-2, 6, 1)

y1 = (4 - 2*x) / 3
y2 = (3 - 5*x) / 4

fig, ax = plt.subplots(figsize=(10, 5))

ax.set_xlabel("${x_1}$")
ax.set_ylabel("${x_2}$")

ax.plot(x, y2, 'r', label="$2{x_1}+3{x_2}-4=0$")
ax.plot(x, y1, 'b', label="$5{x_1}+4{x_2}-3=0$")

ax.plot(-1, 2, 'black', lw=5, marker='o')

ax.annotate("The intersection point\nof the two lines is the solution\nto the system of equations", fontsize=14, family="serif", xy=(-1, 2),
    xycoords="data", xytext=(-150, -80),
    textcoords="offset points", arrowprops=dict(arrowstyle="->", connectionstyle="arc3, rad=-.5"))

ax.set_xticks(x)
ax.set_yticks(x2)

ax.legend()
```

### Squared system

```python
A = sympy.Matrix([[2, 3], [5, 4]])
b = sympy.Matrix([4, 3])

A.rank()
```

```python
A.condition_number()
```

```python
sympy.N(_)
```

```python
A.norm()
```

```python
A = np.array([[2, 3], [5, 4]])
b = np.array([4, 3])
```

```python
np.linalg.matrix_rank(A)
```

    2

```python
np.linalg.cond(A)
```

```python
np.linalg.norm(A)
```

```python
# LU factorization

A = sympy.Matrix([[2, 3], [5, 4]])
b = sympy.Matrix([4, 3])

L, U, _ = A.LUdecomposition()
L
```

$\displaystyle \left[\begin{matrix}1 & 0\\\frac{5}{2} & 1\end{matrix}\right]$

```python
U
```

$\displaystyle \left[\begin{matrix}2 & 3\\0 & - \frac{7}{2}\end{matrix}\right]$

```python
L * U == A
```

    True

```python
x = A.solve(b)
x
```

$\displaystyle \left[\begin{matrix}-1\\2\end{matrix}\right]$

```python
A = np.array([[2, 3], [5, 4]])
b = np.array([4, 3])
P, L, U = la.lu(A)
L
```

    array([[1. , 0. ],
           [0.4, 1. ]])

```python
P.dot(L.dot(U))
```

    array([[2., 3.],
           [5., 4.]])

```python
la.solve(A, b)
```

    array([-1.,  2.])

```python
# Symbolic vs Numerical
p = sympy.symbols("p", positive=True)
A = sympy.Matrix([[1, sympy.sqrt(p)], [1, 1/sympy.sqrt(p)]])
b = sympy.Matrix([1, 2])
x = A.solve(b)
x
```

$\displaystyle \left[\begin{matrix}\frac{2 p - 1}{p - 1}\\\frac{1}{- \sqrt{p} + \frac{1}{\sqrt{p}}}\end{matrix}\right]$

```python
#Symbolic problem specification
p = sympy.symbols("p", positive=True)
A = sympy.Matrix([[1, sympy.sqrt(p)], [1, 1/sympy.sqrt(p)]])
b = sympy.Matrix([1, 2])

# Solve symbolically
x_sym_sol = A.solve(b)
Acond = A.condition_number().simplify()

# Numerical problem specification
AA = lambda p: np.array([[1, np.sqrt(p)], [1, 1/np.sqrt(p)]])
bb = np.array([1, 2])
x_num_sol = lambda p: np.linalg.solve(AA(p), bb)

# Graph the difference between the symbolic (exact) and numerical results.
fig, axes = plt.subplots(1, 2, figsize=(12, 4))

p_vec = np.linspace(0.9, 1.1, 200)

for n in range(2):
    x_sym = np.array([x_sym_sol[n].subs(p, pp).evalf() for pp in p_vec])
    x_num = np.array([x_num_sol(pp)[n] for pp in p_vec])

axes[0].plot(p_vec, (x_num - x_sym)/x_sym, 'k')
axes[0].set_title("Error in solution\n(numerical - symbolic)/symbolic")
axes[0].set_xlabel(r'$p$', fontsize=18)

axes[1].plot(p_vec, [Acond.subs(p, pp).evalf() for pp in p_vec])
axes[1].set_title("Condition number")
axes[1].set_xlabel(r'$p$', fontsize=18)

```

### Rectangular system

```python
x_vars = sympy.symbols("x_1, x_2, x_3")

A = sympy.Matrix([[1, 2, 3], [4, 5, 6]])
x = sympy.Matrix(x_vars)
b = sympy.Matrix([7, 8])

sympy.solve(A*x - b, x_vars)
```

### Least squares

```python
# define true model parameters

x = np.linspace(-1, 1, 100)
a, b, c = 1, 2, 3
y_exact = a + b * x + c * x**2

# Simulate noisy data
m = 100
X = 1 - 2 * np.random.rand(m)

Y = a + b * X + c * X**2 + np.random.randn(m)
# fit the data to the model using linear least square
A = np.vstack([X**0, X**1, X**2]) # see np.vander for alternative
```

```python
sol, r, rank, sv = la.lstsq(A.T, Y)

y_fit = sol[0] + sol[1] * x + sol[2] * x**2

fig, ax = plt.subplots(figsize=(12, 4))

ax.plot(X, Y, 'go', alpha=0.5, label='Simulated data')
ax.plot(x, y_exact, 'k', lw=2, label='True value $y = 1 + 2x + 3x^2$')
ax.plot(x, y_fit, 'b', lw=2, label='Least square fit')

ax.set_xlabel(r"$x$", fontsize=18)
ax.set_ylabel(r"$y$", fontsize=18)
ax.legend(loc=2)
```

```python
# fit the data to the model using linear least square:
# 1st order polynomial
A = np.vstack([X**n for n in range(2)])
sol, r, rank, sv = la.lstsq(A.T, Y)

y_fit1 = sum([s * x**n for n, s in enumerate(sol)])
```

```python
# 15th order polynomial
A = np.vstack([X**n for n in range(16)])

sol, r, rank, sv = la.lstsq(A.T, Y)

y_fit15 = sum([s * x**n for n, s in enumerate(sol)])

fig, ax = plt.subplots(figsize=(12, 4))

ax.plot(X, Y, 'go', alpha=0.5, label='Simulated data')

ax.plot(x, y_exact, 'k', lw=2, label='True value $y = 1 + 2x +3x^2$')

ax.plot(x, y_fit1, 'b', lw=2, label='Least square fit [1st order]')

ax.plot(x, y_fit15, 'm', lw=2, label='Least square fit [15th order]')

ax.set_xlabel(r"$x$", fontsize=18)
ax.set_ylabel(r"$y$", fontsize=18)

ax.legend(loc=2)

```

### Eigenvalues / Eigenvectors

```python
eps, delta = sympy.symbols("epsilon, Delta")
H = sympy.Matrix([[eps, delta], [delta, -eps]])
H
```

$\displaystyle \left[\begin{matrix}\epsilon & \Delta\\\Delta & - \epsilon\end{matrix}\right]$

```python
H.eigenvals()
```

```python
H.eigenvects()
```

$\displaystyle \left[ \left( - \sqrt{\Delta^{2} + \epsilon^{2}}, \  1, \  \left[ \left[\begin{matrix}- \frac{\Delta}{\epsilon + \sqrt{\Delta^{2} + \epsilon^{2}}}\\1\end{matrix}\right]\right]\right), \  \left( \sqrt{\Delta^{2} + \epsilon^{2}}, \  1, \  \left[ \left[\begin{matrix}- \frac{\Delta}{\epsilon - \sqrt{\Delta^{2} + \epsilon^{2}}}\\1\end{matrix}\right]\right]\right)\right]$

```python
(eval1, _, evec1), (eval2, _, evec2) = H.eigenvects()

# Orthogonality
sympy.simplify(evec1[0].T * evec2[0])
```

$\displaystyle \left[\begin{matrix}0\end{matrix}\right]$

```python
A = np.array([[1, 3, 5], [3, 5, 3], [5, 3, 9]])
evals, evecs = la.eig(A)

evals
```

    array([13.35310908+0.j, -1.75902942+0.j,  3.40592034+0.j])

```python
evecs
```

    array([[ 0.42663918,  0.90353276, -0.04009445],
           [ 0.43751227, -0.24498225, -0.8651975 ],
           [ 0.79155671, -0.35158534,  0.49982569]])

```python
la.eigvalsh(A)
```

    array([-1.75902942,  3.40592034, 13.35310908])

### Nonlinear equations

```python
x, a, b, c = sympy.symbols("x, a, b, c")
sympy.solve(a + b*x + c*x**2, x)
```

```python
sympy.solve(a * sympy.cos(x) - b * sympy.sin(x), x)
```

```python
x = np.linspace(-2, 2, 1000)
# four examples of nonlinear functions

f1 = x**2 - x - 1
f2 = x**3 - 3 * np.sin(x)
f3 = np.exp(x) - 2
f4 = 1 - x**2 + np.sin(50 / (1 + x**2))

# plot each function
fig, axes = plt.subplots(1, 4, figsize=(12, 3), sharey=True)

for n, f in enumerate([f1, f2, f3, f4]):
    axes[n].plot(x, f, lw=1.5)
    axes[n].axhline(0, ls=':', color='k')
    axes[n].set_ylim(-5, 5)
    axes[n].set_xticks([-2, -1, 0, 1, 2])
    axes[n].set_xlabel(r'$x$', fontsize=18)

axes[0].set_ylabel(r'$f(x)$', fontsize=18)

titles = [
    r'$f(x)=x^2-x-1$',
    r'$f(x)=x^3-3\sin(x)$',
    r'$f(x)=\exp(x)-2$',
    r'$f(x)=\sin\left(50/(1+x^2)\right)+1-x^2$'
]

for n, title in enumerate(titles):
    axes[n].set_title(title)

```


```
import math
import numpy as np
import matplotlib.pyplot as plt

from sympy import *
```

## Introduction

Derivatives are used whenever we need to deal with rates of change (like velocity and acceleration, biological and economic growth, heat transfer, etc.).

They are useful in processes of optimization in Operation Research and Computer Science and learning algorithms for Artificial Intelligence.

In Mathematics derivatives are widely used to locally approximate/estimate nonlinear functions (which are hard to use) with linear functions (which are easy to use). This is an important part of Calculus (eg, Taylor Series).

How programmers, namely in Python, deal with derivatives?

## Symbolic Differentiation

Python's module `sympy` includes function `Derivative` that performs symbolic differentiation, ie, it performs the algebraic manipulation and returns an expression with the derivative:

```
x, y = symbols('x y')
expr = y*x**2 + 2 * y + y**3
print("Expression : {} ".format(expr))

expr_diff = diff(expr, x)
print("derivative : {}".format(expr_diff))

value_diff = expr_diff.evalf(subs={x:1,y:4})
print("Value of derivative at (1,4): {0:3.2f}".format(value_diff))
```

    Expression : x**2*y + y**3 + 2*y
    derivative : 2*x*y
    Value of derivative at (1,4): 8.00

We can use it to compute a second derivative, in this case $$\frac{\partial^2 f}{\partial x \partial y}$$

```
second_diff = Derivative(expr_diff, y)
print("Value of the 2nd derivative : {} ".format(second_diff.doit()))
```

    Value of the derivative : 2*x

One potential problem is that differential expressions can increase expontentially making them, on those cases, prohibitively slow.

## Numerical Differentiation

This process uses the math formula that defines differentials

$$f'(a) = \lim_{h \rightarrow 0} \frac{f(a+h) - f(a)}{h}$$

There is more than one expression to approximate this value. Here is the _central difference_ formula:

$$f'(a) \approx \frac{1}{2} \left( \frac{f(a + h) - f(a)}{h} + \frac{f(a) - f(a - h)}{h} \right) = \frac{f(a + h) - f(a - h)}{2h}$$

We can implement this easily on Python:

```
def derivative(f, a, h=1e-7):
  return (f(a + h) - f(a - h))/(2*h)
```

What is the derivative of $f(x) = x^2 + 5$ at $x=2.5$? We know by the definition that $f'(x) = 2x$ so the result should be $5$.

Let's check the approximation:

```
def f(x):
  return x**2 + 5

print(derivative(f, 2.5))
```

    4.9999999873762135

Module `scipy` has a function that does this for us:

```
import scipy.misc as spm

spm.derivative(f, 2.5, dx=1e-3)
```

    4.999999999999893

And what about numerical differentiation on second order derivatives?

Can we compute, say, $\frac{\partial^2 f}{\partial x \partial y}$ numerically?

For second derivative over the same variable, there are again several approximations. This is one:

$$\frac{\partial^2 f}{\partial x^2}(a,b) \approx \frac{-f(a-h,b) + 2f(a,b)-f(a+h,b)}{h^2}$$

For mixed variables we can use the next formula [[ref](https://www.uio.no/studier/emner/matnat/math/MAT-INF1100/h07/undervisningsmateriale/kap7.pdf)]

$$\frac{\partial^2 f}{\partial x \partial y}(a,b) \approx \frac{f(a+h_1,b+h_2) - f(a+h_1,b-h_2) - f(a-h_1,b+h_2) + f(a-h_1,b-h_2)}{4h_1 h_2}$$

Module `numdifftools`  solves automatic numerical differentiation problems in one or more variables. [website](https://pypi.org/project/numdifftools/)

### Example: Taylor Expansion

Derivatives are essential to compute the Taylor expansion of functions. [[ref](https://www.math.ubc.ca/~pwalls/math-python/differentiation/differentiation/)]

The Taylor expansion is the infinite series

$$f(x) \approx f(a) +  f^{'}(a)(x - a) + \frac{1}{2!} f^{''}(a) (x - a)^{2} + \frac{1}{3!} f^{(3)}(a) (x - a)^{3} \ldots$$

Let's say we wish the approximate function

$$f(x) = \frac{3e^x}{x^2 + x + 1}$$

around $x=1$

```
xs = np.linspace(-4,4,100)
f  = lambda x : 3*np.exp(x) / (x**2 + x + 1)

fig, ax = plt.subplots(figsize=(10, 4))
ax.plot(xs, f(xs));
```

The Taylor expansion needs the coefficients $$a_n = \frac{1}{n!} f^{(n)}(1)$$

Let's make the approximation of degree 3

```
x = 1

a0 = f(x)
a1 = spm.derivative(f, x, dx=1e-3, n=1)
a2 = spm.derivative(f, x, dx=1e-3, n=2) / 2
a3 = spm.derivative(f, x, dx=1e-3, n=3, order=5) / 6
```

With the coefficients, we can plot the approximation:

```
taylor3 = a0 + a1*(xs-x) + a2*(xs-x)**2 + a3*(xs-x)**3

fig, ax = plt.subplots(figsize=(10, 4))
plt.xlim([-4,4])
plt.ylim([0,8])
ax.plot(xs,ys, xs,taylor3);
```

What happened is that, around $x=1$, we can use a much simpler function (a polinomial of degree 3) instead of the original function (which includes exponentiation and division by a quadratic polynomial). We can zoom around $x=1$ to check how good this approximation is:

```
fig, ax = plt.subplots(figsize=(10, 4))
ax.plot(xs,ys, xs,taylor3), plt.xlim([0.5,1.5]), plt.ylim([2.7,2.85]);
```

## Automatic Differentiation

Automatic differentiation (autodiff) is a tecnique capable of giving an exact answer to differential values in constant time.

### Dual Numbers

Dual numbers have form $a + b \epsilon$, where $\epsilon^2=0$.

We can compute the value of arithmetic expressions over dual numbers.

Eg:

* $(a+b\epsilon) + (c+d\epsilon) = (a+c) + (b+d) \epsilon$

* $(a+b\epsilon) \times (c+d\epsilon) = ac + (ad+bc) \epsilon$

Now consider the Taylor Expansion of a function $f(a)$ at value $a+\epsilon$:

$$f(a+\epsilon) = f(a) + \frac{f'(a)}{1!} \epsilon + \frac{f''(a)}{2!}\epsilon^2 + \ldots$$

but all $\epsilon^n$, with $n>1$, will be zero, so:

$$f(a+\epsilon) = f(a) + f'(a) \epsilon$$

This is an exact solution, not an approximation (!!)

Let's check an example, $f(x) = x^2+1$:

$f(x+\epsilon) = (x+\epsilon)^2 + 1 = x^2 + \epsilon^2 + 2x\epsilon + 1 = x^2 + 1 + 2x\epsilon$

This means that $2x$ is the derivative of $x^2+1$

However, for more complex expressions there is a more efficient algorithm.

### Reverse Mode Differentiation

The Jacobian Matrix is a matrix that takes the partial derivatives of each element of a function $f : \mathbb{R}^n \rightarrow \mathbb{R}^m$

$$J_f=\begin{bmatrix}
\frac{\partial y_1}{\partial x_1} & \ldots & \frac{\partial y_1}{\partial x_n} \\
\vdots & \ddots & \vdots \\
\frac{\partial y_m}{\partial x_1} & \ldots & \frac{\partial y_m}{\partial x_n}
\end{bmatrix}$$

where $x_i$ is the i-th input and $y_j$ the j-th output.

An example with `sympy`:

```
from IPython.display import display, Math, Latex
from ipywidgets      import interact, widgets
from sympy import sin, cos, Matrix, latex
from sympy.abc import rho, phi

X = Matrix([rho*cos(phi), rho*sin(phi), rho**2])
Y = Matrix([rho, phi])

display(Math('\\text{The Jacobian of } f(\\rho,\\phi) = ' + latex(X)
             + '\\text{ is }' + latex(X.jacobian(Y)) ))
```

$$\text{The Jacobian of } f(\rho,\phi) = \left[\begin{matrix}\rho \cos{\left (\phi \right )}\\\rho \sin{\left (\phi \right )}\\\rho^{2}\end{matrix}\right]\text{ is }\left[\begin{matrix}\cos{\left (\phi \right )} & - \rho \sin{\left (\phi \right )}\\\sin{\left (\phi \right )} & \rho \cos{\left (\phi \right )}\\2 \rho & 0\end{matrix}\right]$$

Let's apply autodiff to compute the Jacobian of a vector valued function using a technique denoted **reverse mode differentiation**.

Our example function will be $f(x_1, x_2) = x_1x_2 - \sin(x_2)$.

We would like to evaluate $f^´(2,3)$.

First we need the respective [computational graph](https://medium.com/tebs-lab/deep-neural-networks-as-computational-graphs-867fcaa56c9):

Then we compute the *primal trace*, which is the set of intermediate values at each node:

$$\begin{array}{ccccc}
v_1 &=& x_1 &=& 2 \\
v_2 &=& x_2 &=&3 \\
v_3 &=& v_1 \times v_2 &=& 6 \\
v_4 &=& \sin v_2 &\approx& 0.14 \\
v_5 &=& v_3 - v_4 &\approx& 5.86 \\
y &=& v_5 &\approx& 5.86
\end{array}$$

Since each node corresponds to a single operation, all derivatives are easily known.

The next step is to compute the dual trace.

The *dual trace* measures how much each intermediate node varies with respect to the input

$$\frac{\partial v_i}{\partial x}$$

To compute the dual trace we need to apply the chain rule:

> The chain rule is a technique to break apart a derivative we don’t know how to solve into derivatives that we do know how to solve.

Herein,

$$\frac{\partial y_j}{\partial v_i} = \frac{\partial y_j}{\partial v_k} \frac{\partial v_k}{\partial v_i}$$

where $v_k$ is the parent of $v_i$ in the computational graph. If $v_i$ has more than one parent, we sum all contributions,

$$\frac{\partial y_j}{\partial v_i} = \sum_{p \in \text{parents}(i)} \frac{\partial y_j}{\partial v_p} \frac{\partial v_p}{\partial v_i}$$

This expression is called the *adjoint* of $v_i$ and denoted as $\overline{v_i}$.

It's possible to define the adjoint of a node in terms of the adjoints of its parents (eg: $v_3$ is parent of $v_1$ and $v_2$)

$$\overline{v_i} = \sum_p \overline{v_p}\frac{\partial v_p}{\partial v_i}$$

which give us a recursive algorithm. [ref](https://stats.stackexchange.com/questions/224140/step-by-step-example-of-reverse-mode-automatic-differentiation)

The dual trace starts at the end, at node $y$, and propagates backwards to its dependencies.

So the seed is $\frac{\partial y}{\partial y} = 1$ which means a change in $y$ results in exactly the same change in $y$ (duh!).

Since $y = v_5$,

$$\overline{v_5} = \overline{y} \times \frac{\partial y}{\partial v_5} = 1 \times 1 = 1$$

Since $v_5 = v_3 - v_4$,

$$\overline{v_3} = \overline{v_5} \times \frac{\partial v_5}{\partial v_3} = 1 \times 1 = 1$$

$$\overline{v_4} = \overline{v_4} \times \frac{\partial v_5}{\partial v_4} = 1 \times -1 = -1$$

Since $v_3 = v_1 \times v_2$ and $v_4 = \sin v_2$, $v_2$ has parents $v_3$ and $v_4$,

$$\overline{v_2} = \overline{v_3} \times \frac{\partial v_3}{\partial v_2} + \overline{v_4} \times \frac{\partial v_4}{\partial v_2}  = 1 \times v_1 - \cos v_2 = v_1 - \cos v_2$$

Notice that we already know the values of $v_1$ and $v_2$ from the primal trace,

$$\overline{v_2} = 2 - \cos(3) \approx 2.99$$

For $v_1$ its only parent is $v_3$,

$$\overline{v_1} = \overline{v_3} \times \frac{\partial v_3}{\partial v_1} = 1 \times v_2 = v_2 = 3$$

And the dual trace is done.

We wish to evaluate $f^´(2,3)$, ie,

$$\frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}$$

these values are given by $\overline{v_1}$ and $\overline{v_2}$, that is,

$$\frac{\partial y}{\partial x_1} = 3, \frac{\partial y}{\partial x_2} \approx 2.99$$

Just to check the result against `sympy`:

```
from sympy import symbols
x1, x2 = symbols('x1 x2')

f = x1*x2 - sin(x2)
X = Matrix([f])
Y = Matrix([x1, x2])
jacobian = X.jacobian(Y)

display(Math('\\text{The Jacobian of } f(x_1,x_2) = ' + latex(f)
             + '\\text{ is }' + latex(jacobian) ))

diffValue = jacobian.evalf(subs={x1:2, x2:3})

print('\nDerivate of f at point (2,3): ', end=" ")
print(np.array(diffValue).astype(np.float64).round(2))
```

$$\text{The Jacobian of } f(x_1,x_2) = x_{1} x_{2} - \sin{\left (x_{2} \right )}\text{ is }\left[\begin{matrix}x_{2} & x_{1} - \cos{\left (x_{2} \right )}\end{matrix}\right]$$

    Derivate of f at point (2,3):  [[3.   2.99]]

The advantage of autodiff is that this process can be automated, while *keeping the complexity proportional to the number of nodes*.

Of course, there are modules that perform autodiff:

```
from autograd import grad
import autograd.numpy as np

help(grad)
```

    Help on function grad in module autograd.wrap_util:

    grad(fun, argnum=0, *nary_op_args, **nary_op_kwargs)
        Returns a function which computes the gradient of `fun` with respect to
        positional argument number `argnum`. The returned function takes the same
        arguments as `fun`, but returns the gradient instead. The function `fun`
        should be scalar-valued. The gradient has the same type as the argument.

```
def f(x1, x2):
  return x1*x2 - np.sin(x2)

gradF = grad(f, [0,1])
diffValue = gradF(2.0,3.0)
print("Gradient of f(2,3) is", np.array(diffValue).astype(np.float64).round(2) )
```

    Gradient of f(2,3) is [3.   2.99]

An important point is that autodiff is able to find the derivative/gradient of a Python function even if the function includes conditionals, while loops and even recursion (!)

One immediate application for autodiff would be finding minimums of functions using [gradient descent](https://tillbe.github.io/python-gradient-descent.html).

For those readers that think this reminds you of backpropagation, the classical learning algorithm for Neural Networks, it is not a coincidence: [backpropagation is a subset of reverse mode differentiation](https://stackoverflow.com/questions/49926192).

## References

* Mark Saroufim, [Automatic Differentiation Step by Step](https://medium.com/@marksaroufim/automatic-differentiation-step-by-step-24240f97a6e6) (2019)

* [Automatic Differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation) @ wikipedia

* [Step-by-step example of reverse-mode automatic differentiation](https://stats.stackexchange.com/questions/224140) @ StackExchange

* Peter Sharpe, [Autograd tutorial](https://github.com/HIPS/autograd/blob/master/docs/tutorial.md) (2019)


```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
sns.set()
```

# 1. Fair coin from biased coin

$P(h) = p, \ P(t) = q = 1 - p$

$P(ht) = P(th) = pq \ \rightarrow$ use for fair generation $\rightarrow \ P(\text{dif}) = 2pq, \ P(\text{same}) = 1 - 2pq$

$P(hh) = p^2, \ P(tt) = q^2$

**Idea:** Flip twice -> if different outcomes, report the first one, otherwise repeat.

$
\begin{align}
\mathbb{E}[\text{flips}] = e &= \texttt{succeed_first_try} \ \lor \ \texttt{repeat} \\
e &= 2 \cdot P(\text{dif}) + P(\text{same}) \cdot (2 + e) \\
e &= 2 \cdot 2pq + (1-2pq) \cdot (2 + e) \\
e (1 - 1 + 2pq) &= 4pq + 2 - 4pq \\
e \cdot 2pq &= 2 \\
e &= \frac{1}{pq} = \frac{1}{p \cdot (1-p)}
\end{align}
$

```python
x = 0.05
ps = np.linspace(x, 1-x, 100)
es = 1 / (ps * (1 - ps))
plt.plot(ps, es)
plt.xlabel('$p$', fontsize=14)
plt.ylabel('expected', fontsize=14)
pass
```

# 2. `rand10` from `rand7` (1)

**Idea:** call `rand7` twice -> if $x \leq 40$, ok, otherwise repeat

$
\begin{align}
\mathbb{E}[\text{calls}] = e &= \texttt{succeed_first_try} \ \lor \ \texttt{repeat} \\
e &= 2 \cdot P(x \leq 40) + P(x > 40) \cdot (2 + e)\\
e &= 2 \cdot \frac{40}{49} + 2\cdot\frac{9}{49} + e \cdot \frac{9}{49} \\
e \cdot \frac{40}{49} &= 2 \\
e &= \frac{49}{20} = 2.45
\end{align}
$

### Ref
- https://leetcode.com/problems/implement-rand10-using-rand7/

```python
rand7 = lambda: np.random.randint(low=1, high=8)

def rand10():
    # idea: use [1,40], discard and repeat for [41,49]

    n = ((rand7() - 1) * 7) + rand7()

    if n <= 40:
        return 1 + n % 10
    else:
        return rand10()
```

```python
xs = np.array([rand10() for _ in range(100000)])
support = np.unique(xs)
plt.figure(figsize=(8,3))
bins = np.arange(support.max() + 2) - 0.5
plt.hist(xs, bins=bins, rwidth=0.25, color='b', density=True)
plt.xticks(support)
pass
```

# 3. Random points in a circle

```python
PI = np.arccos(-1)
rand = lambda: np.random.uniform()

def show(gen_func, circle, N=10_000):
    xs, ys = zip(*[gen_func(circle) for _ in range(N)])
    plt.figure(figsize=(5,5))
    plt.scatter(xs, ys, s=0.2)
    plt.grid(False)
    pass
```

### 3.1. Using polar coordinates

```python
def polar_coordinates(circle):
    """
    see: https://stackoverflow.com/questions/5837572/generate-a-random-point-within-a-circle-uniformly
    """
    cx, cy, r = circle

    l = np.sqrt(rand()) * r
    a = 2 * PI * rand()

    x = cx + l * np.cos(a)
    y = cy + l * np.sin(a)

    return x, y

show(polar_coordinates, circle=(0,0,1), N=10_000)
```

## 3.2. Using rejection sampling

- Given a circle $\mathcal{C}$ centered in $(c_x, c_y)$ of radius $r$, we can enclose it in a square $\mathcal{S}$ of size $2r$
- One possible method for generating random points in $\mathcal{C}$ is to generate random points in the square and discard those outside the circle
- For each step we need to generate 2 points $(x, y)$
- Thus, the expected number of trials for getting $(x, y) \in \mathcal{C}$ is $e = 2 \cdot p + (1-p) \cdot (2 + e)$, where $p = P((x,y) \in \mathcal{C}) = \mathcal{A}_{\mathcal{C}} \ / \ \mathcal{A}_{\mathcal{S}} = \pi \ / \ 4$
- We find that $e = 8 \ / \ \pi$

```python
def rejection_sampling(circle):
    cx, cy, r = circle

    xm, xM = cx-r, cx+r
    ym, yM = cy-r, cy+r

    rx = xm + rand() * (xM - xm)
    ry = ym + rand() * (yM - ym)

    dist = (cx - rx) ** 2 + (cy - ry) ** 2
    if dist <= r ** 2:
        return rx, ry

    return rejection_sampling(circle) # discard + repeat

show(rejection_sampling, circle=(0,0,1), N=10_000)
```

# What we notice

- [Law of total expectation](https://en.wikipedia.org/wiki/Law_of_total_expectation)
- [Finding expected value with recursion](https://math.stackexchange.com/q/521609).

$
\begin{align}
e &= \texttt{success}_1 \ \lor \ \texttt{repeat} \\
e &= \texttt{success}_1 \ \lor \ (\texttt{fail}_1 \ \land \ (\texttt{success}_2 \ \lor \ \texttt{repeat})) \\
e &= \texttt{success}_1 \ \lor \ (\texttt{fail}_1 \ \land \ (\texttt{fail}_2 \ \land \ (\texttt{success}_3 \ \lor \ \texttt{repeat}))) \\
e &= \texttt{success}_1 \ \lor \ (\texttt{fail}_1 \ \land \ (\texttt{fail}_2 \ \land \ (\texttt{fail}_3 \ \lor \ (\texttt{success}_4 \ \lor \ \texttt{repeat}))))
\end{align}
$

$\vdots$

Therefore, if we let $n =$ number of samples in one step (e.g. calls to `rand7`, coin tosses), and $p = P(\texttt{success})$, then:

$
\begin{align}
e &= p \cdot n + (1-p) \cdot (n + e) \\
e &= p \cdot n + n + e - p \cdot n - p \cdot e \\
e \cdot p &= n \Rightarrow e = n \ / \ p
\end{align}
$

Thus, $\mathbb{E}[\texttt{trials}] = n \ / \ P(\texttt{success})$

This essentially forms a Markov Chain, so we are interested in expected number of transitions until we reach a certain state. See [this answer](https://math.stackexchange.com/a/947339).

```python

```


# Quadrature

```python
%matplotlib inline

import os
import glob
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_context('notebook', font_scale=1.5)
```

## Numerical integration (Quadrature)

You may recall from Calculus that integrals can be numerically evaluated using quadrature methods such as Trapezoid and Simpson's's rules. This is easy to do in Python, but has the drawback of the complexity growing as $O(n^d)$ where $d$ is the dimensionality of the data, and hence infeasible once $d$ grows beyond a modest number.

### Integrating functions

```python
from scipy.integrate import quad
```

```python
def f(x):
    return x * np.cos(71*x) + np.sin(13*x)
```

```python
x = np.linspace(0, 1, 100)
plt.plot(x, f(x))
pass
```

#### Exact solution

```python
from sympy import sin, cos, symbols, integrate

x = symbols('x')
integrate(x * cos(71*x) + sin(13*x), (x, 0,1)).evalf(6)
```

    0.0202549

#### Using quadrature

```python
y, err = quad(f, 0, 1.0)
y
```

    0.02025493910239419

#### Multiple integration

Following the `scipy.integrate` [documentation](http://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html), we integrate

$$
I=\int_{y=0}^{1/2}\int_{x=0}^{1-2y} x y \, dx\, dy
$$

```python
x, y = symbols('x y')
integrate(x*y, (x, 0, 1-2*y), (y, 0, 0.5))
```

    0.0104166666666667

```python
from scipy.integrate import nquad

def f(x, y):
    return x*y

def bounds_y():
    return [0, 0.5]

def bounds_x(y):
    return [0, 1-2*y]

y, err = nquad(f, [bounds_x, bounds_y])
y
```

    0.010416666666666668


# Quadrature

```python
%matplotlib inline

import os
import glob
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_context('notebook', font_scale=1.5)
```

## Numerical integration (Quadrature)

You may recall from Calculus that integrals can be numerically evaluated using quadrature methods such as Trapezoid and Simpson's's rules. This is easy to do in Python, but has the drawback of the complexity growing as $O(n^d)$ where $d$ is the dimensionality of the data, and hence infeasible once $d$ grows beyond a modest number.

### Integrating functions

```python
from scipy.integrate import quad
```

```python
def f(x):
    return x * np.cos(71*x) + np.sin(13*x)
```

```python
x = np.linspace(0, 1, 100)
plt.plot(x, f(x))
pass
```

#### Exact solution

```python
from sympy import sin, cos, symbols, integrate

x = symbols('x')
integrate(x * cos(71*x) + sin(13*x), (x, 0,1)).evalf(6)
```

$\displaystyle 0.0202549$

#### Using quadrature

```python
y, err = quad(f, 0, 1.0)
y
```

    0.02025493910239419

#### Multiple integration

Following the `scipy.integrate` [documentation](http://docs.scipy.org/doc/scipy/reference/tutorial/integrate.html), we integrate

$$
I=\int_{y=0}^{1/2}\int_{x=0}^{1-2y} x y \, dx\, dy
$$

```python
x, y = symbols('x y')
integrate(x*y, (x, 0, 1-2*y), (y, 0, 0.5))
```

$\displaystyle 0.0104166666666667$

```python
from scipy.integrate import nquad

def f(x, y):
    return x*y

def bounds_y():
    return [0, 0.5]

def bounds_x(y):
    return [0, 1-2*y]

y, err = nquad(f, [bounds_x, bounds_y])
y
```

    0.010416666666666668


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's define an example function whose Taylor series we'd like to compute.

```python
var( 'x' )
formula = exp( 2*x + 1 )
formula
```

$\displaystyle e^{2 x + 1}$

Let's ask for a degree-5 Taylor series centered at $x=2$.  From the code below,
you can tell that the third parameter is the center point and the fourth
parameter is the degree.

```python
series( formula, x, 2, 5 )
```

$\displaystyle e^{5} + 2 \left(x - 2\right) e^{5} + 2 \left(x - 2\right)^{2} e^{5} + \frac{4 \left(x - 2\right)^{3} e^{5}}{3} + \frac{2 \left(x - 2\right)^{4} e^{5}}{3} + O\left(\left(x - 2\right)^{5}; x\rightarrow 2\right)$

The final term (starting with O---oh, not zero) means that there are more terms
in the infinite Taylor series not shown in this finite approximation.
If you want to show just the approximation, you can tell it to remove the O term.

```python
series( formula, x, 2, 5 ).removeO()
```

$\displaystyle \frac{2 \left(x - 2\right)^{4} e^{5}}{3} + \frac{4 \left(x - 2\right)^{3} e^{5}}{3} + 2 \left(x - 2\right)^{2} e^{5} + 2 \left(x - 2\right) e^{5} + e^{5}$

You can also compute individual coefficients in a Taylor series
by remembering the formula for the $n^\text{th}$ term in the series and applying it,
as follows.  The formula for a series centered on $x=a$ is $\frac{f^{(n)}(a)}{n!}$.

From the answer above, we can see that the coefficient on the $n=3$ term is $\frac43e^5$.

```python
n = 3
a = 2
diff( formula, x, n ).subs( x, a ) / factorial( n )
```

$\displaystyle \frac{4 e^{5}}{3}$

database/tasks/How to compute the Taylor series for a function/Python, using SymPy.ipynb


# Automatic Differentiation (AD)

In short, the promise of AD is

```julia
f(x) = 4x + x^2

df(x) = derivative(f, x)
```

such that

```julia
df(3) = 4 + 2*3 = 10
```

### What AD is not

**Symbolic rewriting:**
$$ f(x) = 4x + x^2 \quad \rightarrow \quad df(x) = 4 + 2x $$

**Numerical differentiation:**
$$ \frac{df}{dx} \approx \frac{f(x+h) - f(x)}{\Delta h} $$

## Forward mode AD

Key to AD is the application of the chain rule
$$\dfrac{d}{dx} f(g(x)) = \dfrac{df}{dg} \dfrac{dg}{dx}$$

Consider the function $f(a,b) = \ln(ab + \sin(a))$.

```julia
f(a,b) = log(a*b + sin(a))
```

```julia
f_derivative(a,b) = 1/(a*b + sin(a)) * (b + cos(a))
```

```julia
a = 3.1
b = 2.4
f_derivative(a,b)
```

Dividing the function into the elementary steps, it corresponds to the following "*computational graph*":

```julia
function f_graph(a,b)
    c1 = a*b
    c2 = sin(a)
    c3 = c1 + c2
    c4 = log(c3)
end
```

```julia
f(a,b) == f_graph(a,b)
```

To calculate $\frac{\partial f}{\partial a}$ we have to apply the chain rule multiple times.

$\dfrac{\partial f}{\partial a} = \dfrac{\partial f}{\partial c_4} \dfrac{\partial c_4}{\partial a} = \dfrac{\partial f}{\partial c_4} \left( \dfrac{\partial c_4}{\partial c_3} \dfrac{\partial c_3}{\partial a}  \right) = \dfrac{\partial f}{\partial c_4} \left( \dfrac{\partial c_4}{\partial c_3} \left( \dfrac{\partial c_3}{\partial c_2} \dfrac{\partial c_2}{\partial a} + \dfrac{\partial c_3}{\partial c_1} \dfrac{\partial c_1}{\partial a}\right)  \right)$

```julia
function f_graph_derivative(a,b)
    c1 = a*b
    c1_ϵ = b

    c2 = sin(a)
    c2_ϵ = cos(a)

    c3 = c1 + c2
    c3_ϵ = c1_ϵ + c2_ϵ

    c4 = log(c3)
    c4_ϵ = 1/c3 * c3_ϵ

    c4, c4_ϵ
end
```

```julia
f_graph_derivative(a,b)[2] == f_derivative(a,b)
```

**How can we automate this?**

```julia
# D for "dual number", invented by Clifford in 1873.
struct D <: Number
    x::Float64 # value
    ϵ::Float64 # derivative
end
```

```julia
import Base: +, *, /, -, sin, log, convert, promote_rule

a::D + b::D = D(a.x + b.x, a.ϵ + b.ϵ) # sum rule
a::D - b::D = D(a.x - b.x, a.ϵ - b.ϵ)
a::D * b::D = D(a.x * b.x, a.x * b.ϵ + a.ϵ * b.x) # product rule
a::D / b::D = D(a.x / b.x, (b.x * a.ϵ - a.x * b.ϵ)/b.x^2) # quotient rule

sin(a::D) = D(sin(a.x), cos(a.x) * a.ϵ)
log(a::D) = D(log(a.x), 1/a.x * a.ϵ)

Base.convert(::Type{D}, x::Real) = D(x, zero(x))
Base.promote_rule(::Type{D}, ::Type{<:Number}) = D
```

```julia
f(D(a,1), b)
```

Boom! That was easy!

```julia
f_derivative(a,b)
```

```julia
f(D(a,1), b).ϵ ≈ f_derivative(a,b)
```

**How does this work?!**

The trick of forward mode AD is to let Julia implicitly perform the mapping `f -> f_graph_derivative` for you and then let the compiler optimize the resulting code structure (that's what compilers do!).

```julia
@code_typed f(D(a,1), b)
```

While this is somewhat hard to parse, plugging these operations manually into each other we find that this code equals

```julia
D.x = log(a.x*b + sin(a.x))
D.ϵ = 1/(a.x*b + sin(a.x)) * (a.x*0 + (a.ϵ*b) + cos(a.x)*a.ϵ)
```

which, if we drop `a.x*0`, set `a.ϵ = 1`, and rename `a.x` $\rightarrow$ `a`, reads

```julia
D.x = log(a*b + sin(a))
D.ϵ = 1/(a*b + sin(a)) * (b + cos(a)
```

This precisely matches our definitions from above:

```julia
f(a,b) = log(a*b + sin(a))

f_derivative(a,b) = 1/(a*b + sin(a)) * (b + cos(a))
```

Importantly, the compiler sees the entire "rewritten" code and can therefore apply optimizations. In this simple example, we find that the code produced by our simple Forward mode AD is essentially identical to the explicit implementation.

```julia
@code_llvm debuginfo=:none f_graph_derivative(a,b)
```

```julia
@code_llvm debuginfo=:none f(D(a,1), b)
```

Our AD is alreadly pretty powerful and general. Let's define the promised function `derivative`:

```julia
derivative(f::Function, x::Number) = f(D(x, one(x))).ϵ
```

```julia
g(x) = x + x^2
```

```julia
derivative(g, 3.0)
```

Anonymous function oft come in handy here:

```julia
derivative(x->3*x^2+4x+5, 2)
```

```julia
derivative(x->sin(x)*log(x), 3)
```

We can also define the partial derivative $\frac{df(a,b)}{da}$ from above:

```julia
df(x) = derivative(a->f(a,b),x)
```

Here, `b` is "wrapped into a closure".

```julia
df(1.23)
```

## Taking the derivative of *code*

> Repeat $t \leftarrow (t + x/2)/2$ until $t$ converges to $\sqrt{x}$.

```julia
@inline function Babylonian(x; N = 10)
    t = (1+x)/2
    for i = 2:N
        t = (t + x/t)/2
    end
    t
end
```

```julia
Babylonian(2)
```

```julia
sqrt(2)
```

Using our forward mode AD, that is our dual numbers, we can compute the derivative of `Babylonian` **with no rewrite at all**.

```julia
Babylonian(D(5, 1))
```

```julia
sqrt(5)
```

```julia
1 / (2*sqrt(5))
```

**It just works and is efficient!**

```julia
@code_native debuginfo=:none Babylonian(D(5, 1))
```

Recursion? Works as well...

```julia
function power(x, n)
    if n <= 0
        return 1
    else
        return x*power(x, n-1)
    end
end
```

```julia
4.0^3
```

```julia
derivative(x -> power(x,3), 4.0)
```

```julia
3*4.0^2 # 3*x^2
```

Deriving our Vandermonde matrix from yesterday?

```julia
function vander_generic(x::AbstractVector{T}) where T
    m = length(x)
    V = Matrix{T}(undef, m, m)
    for j = 1:m
        V[j,1] = one(x[j])
    end
    for i= 2:m
        for j = 1:m
            V[j,i] = x[j] * V[j,i-1]
            end
        end
    return V
end
```

\begin{align}V=\begin{bmatrix}1&a&a^{2} &a^3\\1&b&b^{2} &b^3\\1&c&c^{2} &c^3\\1&d&d^{2} &d^3\end{bmatrix}\end{align}

\begin{align}\frac{dV}{da}=\begin{bmatrix}0&1&2a &3a^2\\0&0&0 &0\\0&0&0 &0\\0&0&0 &0\end{bmatrix}\end{align}

```julia
a, b, c, d = 2, 3, 4, 5
V = vander_generic([D(a,1), D(b,0), D(c,0), D(d,0)])
```

```julia
[V[i,j].ϵ for i in axes(V,1), j in axes(V,2)]
```

## Symbolically (because we can)

The below is mathematically equivalent, **though not exactly what the computation is doing**. Our AD isn't performing symbolic manipulations.

```julia
using SymPy
```

```julia
@vars x
```

```julia
Babylonian(x; N=1)
```

```julia
diff(Babylonian(x; N=1))
```

```julia
simplify(Babylonian(x; N=5))
```

```julia
simplify(diff(simplify(Babylonian(x; N=5)), x))
```

## Don't reinvent the wheel: ForwardDiff.jl

Now that we have understood how forward AD works, we can use the more feature complete package [ForwardDiff.jl](https://github.com/JuliaDiff/ForwardDiff.jl).

```julia
using ForwardDiff
```

```julia
ForwardDiff.derivative(Babylonian, 2)
```

```julia
@edit ForwardDiff.derivative(Babylonian, 2)
```

(Note: [DiffRules.jl](https://github.com/JuliaDiff/DiffRules.jl))

## If time permits: Reverse mode AD

Forward mode:
$\dfrac{\partial f}{\partial x} = \dfrac{\partial f}{\partial c_4} \dfrac{\partial c_4}{\partial x} = \dfrac{\partial f}{\partial c_4} \left( \dfrac{\partial c_4}{\partial c_3} \dfrac{\partial c_3}{\partial x}  \right) = \dfrac{\partial f}{\partial c_4} \left( \dfrac{\partial c_4}{\partial c_3} \left( \dfrac{\partial c_3}{\partial c_2} \dfrac{\partial c_2}{\partial x} + \dfrac{\partial c_3}{\partial c_1} \dfrac{\partial c_1}{\partial x}\right)  \right)$

Reverse mode:
$\dfrac{\partial f}{\partial x} = \dfrac{\partial f}{\partial c_4} \dfrac{\partial c_4}{\partial x} = \left( \dfrac{\partial f}{\partial c_3}\dfrac{\partial c_3}{\partial c_4}   \right) \dfrac{\partial c_4}{\partial x} = \left( \left( \dfrac{\partial f}{\partial c_2} \dfrac{\partial c_2}{\partial c_3} + \dfrac{\partial f}{\partial c_1} \dfrac{\partial c_1}{\partial c_3} \right) \dfrac{\partial c_3}{\partial c_4} \right) \dfrac{\partial c_4}{\partial x}$

Forward mode AD requires $n$ passes in order to compute an $n$-dimensional
gradient.

Reverse mode AD requires only a single run in order to compute a complete gradient but requires two passes through the graph: a forward pass during which necessary intermediate values are computed and a backward pass which computes the gradient.

*Rule of thumb:*

Forward mode is good for $\mathbb{R} \rightarrow \mathbb{R}^n$ while reverse mode is good for $\mathbb{R}^n \rightarrow \mathbb{R}$.

An efficient source-to-source reverse mode AD is implemented in [Zygote.jl](https://github.com/FluxML/Zygote.jl), the AD underlying [Flux.jl](https://fluxml.ai/) (since version 0.10).

```julia
using Zygote
```

```julia
f(x) = 5*x + 3
```

```julia
gradient(f, 5)
```

```julia
@code_llvm debuginfo=:none gradient(f,5)
```

```julia
@code_llvm debuginfo=:none derivative(f,5)
```

## Some nice reads

Papers:
* https://www.jmlr.org/papers/volume18/17-468/17-468.pdf

Lectures:

* https://mitmath.github.io/18337/lecture8/automatic_differentiation.html

Blog posts:

* ML in Julia: https://julialang.org/blog/2018/12/ml-language-compiler

* Nice example: https://fluxml.ai/2019/03/05/dp-vs-rl.html

* Nice interactive examples: https://fluxml.ai/experiments/

* Why Julia for ML? https://julialang.org/blog/2017/12/ml&pl

* Neural networks with differential equation layers: https://julialang.org/blog/2019/01/fluxdiffeq

* Implement Your Own Automatic Differentiation with Julia in ONE day : http://blog.rogerluo.me/2018/10/23/write-an-ad-in-one-day/

* Implement Your Own Source To Source AD in ONE day!: http://blog.rogerluo.me/2019/07/27/yassad/

Repositories:

* AD flavors, like forward and reverse mode AD: https://github.com/MikeInnes/diff-zoo (Mike is one of the smartest Julia ML heads)

Talks:

* AD is a compiler problem: https://juliacomputing.com/assets/pdf/CGO_C4ML_talk.pdf


# Linear Regression with Normal Equation and Gradient Descent

$X \in \mathfrak{M}_{m\times n}(\mathbb{R}), \mathbf{y} \in \mathbb{R}^m, \theta \in \mathbb{R}^n$
\begin{equation}
\mathcal{L(\theta) = || \mathbf{X}\theta - \mathbf{y} ||^2}
\end{equation}

We want to find $\theta$ which minizes the error function $\mathcal{L}(\theta)$.

```
import numpy as np
from numpy.linalg import inv
```

```
num_features = 5
X = np.random.randn(100, num_features)
Y = np.random.randn(100)
theta = np.random.randn(num_features)

```

```
error = np.sum((Y - np.dot(X, theta))**2, axis=0)
error
```

    441.1478116816043

## Normal Equation

 It has closed-form solution as follows, which is Normal Equation :
\begin{align*}
\text{argmin}_{\theta} \mathcal{L}(\theta) =  (X^tX)^{-1}X^t\mathbf{y}
\end{align*}

```
theta_hat = np.dot(inv(X.T @ X) @ X.T, Y)
final_loss = np.sum((Y - np.dot(X, theta_hat))**2, axis=0)
final_loss
```

    88.15259780536041

## Gradient Descent

Since $\mathcal{L}(\theta)$ is differentiable function, we can find find the parameter $\theta$ by gradient descent as follows, where $\eta$ is learning rate:
\begin{align*}
\nabla_\theta \mathcal{L}(\theta) := (\frac{\partial \mathcal{L}(\theta)}{\partial \theta_1}, \ldots, \frac{\partial \mathcal{L}(\theta)}{\partial \theta_n}, )
\end{align*}
\begin{equation}
\theta^{(t+1)} = \theta^{(t)} - \eta \nabla_{\theta}\mathcal{L}(\theta^{(t)})
\end{equation}

Let $\mathbf{x_i}$ the ith row vector of $X$.
\begin{align*}
\frac{\partial \mathcal{L}(\theta)}{\partial \theta_j} &= \sum\limits_{i=1}^m (\mathbf{x}_i\cdot \theta - y_i)^2 \\
&= \sum\limits_{i=1}^m \frac{\partial (\mathbf{x}_i \cdot \theta - y_i)^2}{\partial \theta_j} \\
&= \sum\limits_{i=1}^m 2(\mathbf{x}_i \cdot \theta - y_i) \frac{\partial (\mathbf{x}_i \cdot \theta - y_i)}{\partial \theta_j} \\
&=  \sum\limits_{i=1}^m 2(\mathbf{x}_i \cdot \theta - y_i) X_{ij} \:\: \text{ where } X_{ij} \text{ is } (i,j) \text{ entry of } X \\
&= 2 [X]^j \cdot (X\theta - \mathbf{y}) \text{ where } [X]^j \text{is the jth column vector of }X \\
\end{align*}

\begin{equation}\therefore \nabla_{\theta}\mathcal{L}(\theta) = 2X^t(X\theta - \mathbf{y})
\end{equation}

```
class Regression(object):
    def __init__(self, num_features, lr=0.001):
        self.weight = np.random.randn(num_features)
        self.lr = lr

    def forward(self, x):
        self.logits = np.dot(x, self.weight)

        return self.logits

    def backward(self, x, y):
        grad = 2 * x.T @ (self.logits - y)
        self.weight = self.weight - self.lr * grad
        return grad

```

```
net = Regression(num_features, lr=0.001)

for _ in range(100):
    logits = net.forward(X)
    loss = np.sum((logits - Y) ** 2, 0)
    grad = net.backward(X, Y)

logits = net.forward(X)
sgd_loss = np.sum((logits - Y)**2, axis=0)
print(sgd_loss)
```

    88.15259780539971

```
abs(final_loss - sgd_loss)
```

    3.929301328753354e-11