# Automatic differentiation

Automatic differentiation (AD) is a set of techniques to calculate **exact** derivatives, numerically, in an automatic way. It is neither symbolic differentiation, nor something like finite differences.

There are two main methods: forward-mode AD and reverse-mode AD.
Each has its strengths and weaknesses. Forward mode is significantly easier to implement.

# Forward-mode AD

Let's start by thinking about univariate functions $f: \mathbb{R} \to \mathbb{R}$. We would like to calculate the derivative $f'(a)$ at some point $a \in \mathbb{R}$.

We know various rules about how to calculate such derivatives. For example, if we have already managed to calculate $f'(a)$ and $g'(a)$, we can calculate $(f+g)'(a)$ and $(f.g)'(a)$ as

\begin{align}
(f+g)'(a) &= f'(a) + g'(a)\\
(f.g)'(a) &= f'(a) \, g(a) + f(a) \, g'(a)
\end{align}

We also have the chain rule, which plays a crucial role:

$$(f \circ g)'(a) = f'(g(a)) \, g'(a)$$

We see that in general we will need, for each function $f$, both the value $f(a)$ and the derivative $f'(a)$, and this is the *only* information that we require in order to calculate the first derivative of any combination of functions.

## Jets

Formally, we can think of a first-order Taylor polynomial of $f$, called the
[**jet** of $f$](https://en.wikipedia.org/wiki/Jet_(mathematics) at $a$, denoted $J_a(f)$:

$$(J_a(f))(x) := f(a) + x f'(a)$$

[This can be thought of as representing the **set of all functions** with the same data $f(a)$ and $f'(a)$.]

Formally, it is common to think of this as a "dual number", $f + \epsilon f'$, that we can manipulate, following the rule that $\epsilon^2 = 0$. (Cf. complex numbers, which have the same structure, but with $\epsilon^2 = -1$.) E.g.

$$(f + \epsilon f') \times (g + \epsilon g') = f \, g + \epsilon (f' g + f g')$$

shows how to define the multiplication of two jets.

## Computer representation

As usual, we can represent a polynomial just by its degree and its coefficients, so we can define a Julia object as follows. We will leave the evaluation point $(a)$ as being implicit, although we could, of course, include it if desired.

```julia
immutable Jet{T} <: Real
    val::T  # value
    der::T  # derivative   # type \prime<TAB> to get ′
end
```

```julia
import Base: +, *, -, convert, promote_rule
```

```julia
+(f::Jet, g::Jet) = Jet(f.val + g.val, f.der + g.der)
-(f::Jet, g::Jet) = Jet(f.val - g.val, f.der - g.der)

*(f::Jet, g::Jet) = Jet(f.val*g.val, f.der*g.val + f.val*g.der)
```

We can now define `Jet`s and manipulate them:

```julia
f = Jet(3, 4)  # any function f such that f(a) = 3 and f'(a) = 4, or the set of all such functions
g = Jet(5, 6)  # any function g such that g(a) = 5 and g'(a) = 6

f + g   # calculate the value and derivative of (f + g) for any f and g in these sets
```

```julia
f * g
```

```julia
f * (g + g)
```

## Performance

It seems like we must have introduced quite a lot of computational overhead by creating a relatively complex data structure, and associated methods, to manipulate pairs of numbers. Let's see how the performance is:

```julia
add(a1, a2, b1, b2) = (a1+b1, a2+b2)
```

```julia
add(1, 2, 3, 4)
@time add(1, 2, 3, 4)
```

```julia
a = Jet(1, 2)
b = Jet(3, 4)

add2(j1, j2) = j1 + j2
add2(a, b)
@time add2(a, b)
```

```julia

```

```julia
@code_native add(1, 2, 3, 4)
```

```julia
@code_native add2(a, b)
```

We see that there is only a slight overhead to do with moving the data around. The data structure itself has disappeared, and we basically have a standard Julia tuple.

## Functions on jets: chain rule

We can also define functions of these objects using the chain rule. For example, if `f` is a jet representing the function $f$, then we would like `exp(f)` to be a jet representing the function $\exp \circ f$, i.e. with value $\exp(f(a))$ and derivative $(\exp \circ f)'(a) = \exp(f(a)) \, f'(a)$:

```julia
import Base: exp
```

```julia
exp(f::Jet) = Jet(exp(f.val), exp(f.val) * f.der)
```

```julia
f
```

```julia
exp(f)
```

## Conversion and promotion

However, we can't do e.g. the following:

```julia
# 3 * f
```

In order to get this to work, we need to hook into Julia's type promotion and conversion machinery.
First, we specify how to promote a number and a `Jet`:

```julia
promote_rule{T<:Real,S}(::Type{Jet{S}}, ::Type{T}) = Jet{S}
```

Second, we specify how to `convert` a (constant) number to a `Jet`. By e.g. $g = f+3$, we mean the function such that $g(x) = f(x) + 3$ for all $x$, i.e. $g = f + 3.\mathbb{1}$, where $\mathbb{1}$ is the constant function $\mathbb{1}: x \mapsto 1$.

Thus we think of a constant $c$ as the constant function $c \, \mathbb{1}$, with $c(a) = c$ and $c'(a) = 0$, which we encode as the following conversion:

```julia
convert{T<:Union{AbstractFloat, Integer, Rational},S}(::Type{Jet{S}}, x::T) = Jet{S}(x, 0)
```

```julia
convert(Jet{Float64}, 3.1)
```

```julia
promote(Jet(1,2), 3.0)
```

```julia
promote(Jet(1,2), 3.1)
```

```julia
convert(Jet{Float64}, 3.0)
```

Julia's machinery now enables us to do what we wanted:

```julia
Jet(1.1, 2.3) + 3
```

## Calculating derivatives of arbitrary functions

How can we use this to calculate the derivative of an arbitrary function? For example, we wish to differentiate the function

```julia
h(x) = x^2 - 2
```

at $a = 3$.

We think of this as a function of $x$, which itself we think of as the identity function $\iota: x \mapsto x$, so that

$$h = \iota^2 - 2.\mathbb{1}$$

We represent the identity function as follows:

```julia
a = 3
x = Jet(a, 1)
```

since $\iota'(a) = 1$ for any $a$.

Now we simply evaluate the function `h` at `x`:

```julia
h(x)
```

The first component of the resulting `Jet` is the value $h(a)$, and the second component is the derivative, $h'(a)$.

We can codify this into a function as follows:

```julia
derivative(f, x) = f(Jet(x, one(x))).der
```

```julia
derivative(x -> 3x^5 + 2, 2)
```

This is capable of differentiating any function that involves functions whose derivatives we have specified by defining corresponding rules on `Jet` objects. For example,

```julia
y = [1.,2]
k(x) = (y'* [x 2; 3 4] * y)[]
```

```julia
k(3)
```

```julia
derivative(x->k(x), 10)
```

This works since Julia is constructing the following object:

```julia
[Jet(3.0, 1.0) 2; 3 4]
```

# Higher dimensions

How can we extend this to higher dimensions? For example, we wish to differentiate the following function $f: \mathbb{R}^2 \to \mathbb{R}$:

```julia
f1(x, y) = x^2 + x*y
```

As we learn in calculus, the partial derivative $\partial f/\partial x$ is the function obtained by fixing $y$, thinking of the resulting function as a function only of $x$, and then differentiating.

Suppose that we wish to differentiate $f$ at $(a, b)$:

```julia
a, b = 3.0, 4.0

f1_x(x) = f1(x, b)  # single-variable function
```

Since we now have a single-variable function, we can differentiate it:

```julia
derivative(f1_x, a)
```

Under the hood this is doing

```julia
f1(Jet(a, one(a)), b)
```

Similarly, we can differentiate with respect to $y$ by doing

```julia
f1(a, Jet(b, one(b)))
```

Note that we must do **two separate calculations** to get the two partial derivatives. To calculate a gradient of a function $f:\mathbb{R}^n \to \mathbb{R}$ thus requires $n$ separate calculations.

Forward-mode AD is implemented in a clean and efficient way in the `ForwardDiff.jl` package.

# Syntax trees

## Forward-mode

To understand what forward-mode AD is doing, and its name, it is useful to think of an expression as a **syntax tree**; cf. [this notebook](Syntax trees in Julia.ipynb).

If we label the nodes in the tree as $v_i$, then forward differentiation fixes a variable, e.g. $y$, and calculates $\partial v_i / \partial y$ for each $i$. If e.g. $v_1 = v_2 + v_3$, then we have

$$\frac{\partial v_1}{\partial y} = \frac{\partial v_2}{\partial y} + \frac{\partial v_3}{\partial y}.$$

Denoting $v_1' := \frac{\partial v_1}{\partial y}$, we have $v_1' = v_2' + v_3'$, so we need to calculate the derivatives and nodes lower down in the graph first, and propagate the information up. We start at $v_x' = 0$, since $\frac{\partial x}{\partial y} = 0$, and $v_y' = 1$.

## Reverse mode

An alternative method to calculate derivatives is to fix not the variable with which to differentiate, but *what it is* that we differentiate, i.e. to calculate the **adjoint**, $\bar{v_i} := \frac{\partial f}{\partial v_i}$, for each $i$.

If $f = v_1 + v_2$, with $v_1 = v_3 + v_4$ and $v_2 = v_3 + v_5$, then

$$\frac{\partial f}{\partial v_3} = \frac{\partial f}{\partial v_1} \frac{\partial v_1}{\partial v_3} + \frac{\partial f}{\partial v_2} \frac{\partial v_2}{\partial v_3},$$

i.e.

$$\bar{v_3} = \alpha_{13} \, \bar{v_1} + \alpha_{2,3} \, \bar{v_2},$$

where $\alpha_{ij}$ are the coefficients specifying the relationship between the different terms. Thus, the adjoint information propagates **down** the graph, in **reverse** order, hence the name "reverse-mode".

For this reason, reverse mode is much harder to implement. However, it has the advantage that all derivatives $\partial f / \partial x_i$ are calculated in a *single pass* of the tree.

Julia has en efficient implementation of reverse-mode AD in https://github.com/JuliaDiff/ReverseDiff.jl

## Example of reverse mode

Reverse mode is difficult to implement in a general way, but easy to do by hand. e.g. consider the function

$$f(x,y,z) = x \, y - \sin(z)$$

We decompose this into its tree with labelled nodes, corresponding to the following sequence of elementary operations:

```julia
ff(x, y, z) = x*y - 2*sin(x*z)

x, y, z = 1, 2, 3

v₁ = x
v₂ = y
v₃ = z
v₄ = v₁ * v₂
v₅ = v₁ * v₃
v₆ = sin(v₅)
v₇ = v₄ - 2v₆  # f
```

```julia
ff(x, y, z)
```

We have decomposed the **forward pass** into elementary operations. We now proceed to calculate the adjoints. The difficulty is to *find which variables depend on the current variable under question*.

```julia
v̄₇ = 1
v̄₆ = -2 # ∂f/∂v₆ = ∂v₇/∂v₆
v̄₅ = v̄₆ * cos(v₅)  # ∂v₇/∂v₆ * ∂v₆/∂v₅
v̄₄ = 1
v̄₃ = v̄₅ * v₁  # ∂f/∂v₃ = ∂f/∂v₅ . ∂v₅/∂v₃. # This gives ∂f/∂z
v̄₂ = v̄₄ * v₁
v̄₁ = v̄₅*v₃ + v̄₄*v₂
```

Thus, in a single pass we have calculated the gradient $\nabla f(1, 2, 3)$:

```julia
(v̄₁, v̄₂, v̄₃)
```

Let's check that it's correct:

```julia
ForwardDiff.gradient(x->ff(x...), [x,y,z])
```

# Example: optimization

As an example of the use of AD, consider the following function that we wish to optimize:

```julia
x = rand(3)
y = rand(3)

distance(W) = W*x - y
```

```julia
using ForwardDiff
```

```julia
ForwardDiff.jacobian(distance, rand(3,3))
```

```julia
objective(W) = (a = distance(W); dot(a, a))
```

```julia
W0 = rand(3, 3)
grad = ForwardDiff.gradient(objective, W0)
```

```julia
2*(W0*x-y)*x' == grad  # LHS is the analytical derivative
```

# Example: Interval arithmetic

How can we find roots of a function?

```julia
f2(x) = x^2 - 2
```

## Exclusion of domains

An idea is to *exclude* regions of $\mathbb{R}$ by showing that they *cannot* contain a zero, by calculating the image (range) of the function over a given domain.

This is, in general, a difficult problem, but **interval arithmetic** provides a partial solution, by calculating an **enclosure** of the range, i.e. and interval that is guaranteed to contain the range.

```julia
using ValidatedNumerics
```

```julia
X = 3..4
```

```julia
typeof(X)
```

This is a representation of the set $X = [3, 4] := \{x\in \mathbb{R}: 3 \le x \le 4\}$.

We can evaluate a Julia function on an `Interval` object `X`. The result is a new `Interval`, which is **guaranteed to contain the true image** $\mathrm{range}(f; X) := \{f(x): x \in X \}$.  This is achieved by defining arithmetic operations on intervals in the correct way, e.g.

$$X + Y = [x_1, x_2] + [y_1, y_2] = [x_1+y_1, x_2+y_2].$$

```julia
f2(X)
```

Since this result does not contain $0$, we have *proved* that $f$ has no zero in the domain $[3,4]$. We can even use semi-infinite intervals:

```julia
X1 = 3..∞  # type \infty<TAB>
```

```julia
f2(X1)
```

```julia
X2 = -∞.. -3   # space is required
```

```julia
f2(X2)
```

We have thus exclued two semi-infinite regions, and have proved that any root *must* lie in $[-3,3]$, by two simple calculations. However,

```julia
f2(-3..3)
```

We cannot conclude anything from this, since the result is, in general, an over-estimate of the true range, which thus may or may not contain zero. We can proceed by bisecting the interval. E.g. after two bisections, we find

```julia
f2(-3.. -1.5)
```

so we have excluded another piece.

## Proving existence of roots

To prove that there *does* exist a root, we need a different approach. It is a standard method to evaluate the function at two end-points of an interval:

```julia
f2(1), f2(2)
```

Since there is a sign change, there exists at least one root $x^*$ in the interval $[1,2]$, i.e. a point such that $f(x^*) = 0$.

To prove that it is unique, one method is to prove that $f_2$ is *monotone* in that interval, i.e. that the derivative has a unique sign. To do so, we need to evaluate the derivative *at every point in the interval*, which seems impossible.

Again, however, interval arithmetic easily gives an *enclosure* of this image. To show this, we need to evaluate the derivative using interval arithmetic.

Thanks to Julia's parametric types, we get **composability for free**: we can just substitute in an interval to `ForwardDiff` or `Jet`, and it works:

```julia
ForwardDiff.derivative(f2, 1..2)
```

Again, the reason for this is that Julia creates the object

```julia
Jet(x, one(x))
```

Since an enclosure of the derivative is the interval $[2, 4]$ (and, in fact, in this case this is the true image, but there is no way to know this other than with an analytical calculation), we have **proved** that the image of the derivative function $f'$ over the interval $X = [1,2]$ does *not* contain zero, and hence that the image is monotone.

To actually find the root within this interval, we can use the [Newton interval method](Interval Newton.ipynb). In general, we should not expect to be able to use intervals in standard numerical methods designed for floats; rather, we will need to modify the numerical method to take *advantage* of intervals.

The Newton interval method can find, in a guaranteed way, *all* roots of a function in a given interval (or tell you if when it is unable to to so, for example if there are double roots). Although people think that finding roots of a general function is difficult, this is basically a solved problem using these methods.


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from ipywidgets import interact, fixed
from matplotlib import cm
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['ytick.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['xtick.top'] = False
```

###  Regularized optimization problems
In practice, we often encounter situations where we would like to estimate lots of parameters but where we don't really have enough data. One possible solution is then to use prior knowledge to favor some parameter combinations over others. For example, if we know that many parameters are likely to be zero, then one possibility is to add a regularization term to our objective function that promotes sparse solutions. However, some commonly used regularization terms are not smooth functions, and consequently, we can no longer use simple gradient based methods for finding the optimal parameters. Here, we therefore introduce a method called proximal gradient, which is used for minimizing non-differentiable functions that can be divided into a sum of two convex functions: one smooth and differentiable, and one non-differentiable.

### Objective function with $L^1$ regularization
Building upon our previous Poisson regression example, we note that we can promote sparse solutions by adding an $L^1$ regularization term (also known as lasso-regularization as it shrinks dimensions to zero one at a time). Our new objective function thus takes the form:
\begin{align*}
 nll(w_0, w_1) &= \sum_i^N f(z_i) - y_i\log(f(z_i)) + \lambda ||\mathbf{w}||_{L^1},\\
 ||\mathbf{w}||_{L^1} &= \sum_j |w_j| = |w_1| + |w_2|,
\end{align*}
where $\lambda$ is a regularization parameter that adjust how heavily we penalize non-zero parameter values. Observe that we still refer to this objective function as the negative log-likelihood ($nll$), as the additional regularization term only corresponds to adding a Laplacean zero mean prior on our parameters. However, the objective function is now composed of two parts, which we can highlight by writing it as:
\begin{align*}
 nll(\mathbf{w}) &= g(\mathbf{w}) + h(\mathbf{w}),\\
 g(\mathbf{w})   &=\sum_i^N f(z_i) - y_i\log(f(z_i)),\\
 h(\mathbf{w})   &= ||\mathbf{w}||_{L^1}.
\end{align*}
The first part ($g$) is convex and differentiable, whereas the second part ($h$) is convex (as it is a norm) but non-differentiable.

```python
# Poisson regression, rectified inverse link function
def invLinkFun(X, w):
    z = np.dot(X, w)
    mu = np.zeros(z.shape)
    mu[z<0] = np.exp(z[z<0])
    mu[z>=0] = z[z>=0] + 1
    return mu

# Poisson regression, negative log-likelihood
gFun = lambda X, y, w: np.sum(invLinkFun(X, w) - y*np.log(invLinkFun(X, w)))

# L1 regularization term
hFun = lambda w, reg_lambda: reg_lambda*np.linalg.norm(w)

# Negative log-likelihood
negLogLikFun = lambda X, y, w, reg_lambda: gFun(X, y, w) + hFun(w, reg_lambda)

# Poisson regression, gradient of the negative log-likelihood
def gDerFun(X, y, w):
    z = np.dot(X, w)
    der = np.dot(np.exp(z[z<0])-y[z<0], X[z<0, :])
    der += np.dot(1-y[z>=0]/(z[z>=0]+1), X[z>=0, :])
    return der
```

### Proximal gradient
To introduce the logic behind the proximal gradient method, we begin by performing a quadratic expansion of $g$ around $\mathbf{w}_k$:
$$
g(\mathbf{w}_k)+\nabla g(\mathbf{w}_k)^T(\mathbf{w}-\mathbf{w}_k)+\frac{1}{2\eta}||\mathbf{w}-\mathbf{w}_k||^2_2
$$
where the Hessian is approximated by $\frac{1}{\eta}\mathbf{I}$. Differentiating the expansion with respect to $\mathbf{w}$ gives:
$$
\nabla g(\mathbf{w}_k) + \frac{1}{\eta}(\mathbf{w}-\mathbf{w}_k),
$$
which results in $\mathbf{w}_{k+1}=\mathbf{w}=\mathbf{w}_k - \eta\nabla g(\mathbf{w}_k)$ when set to zero. That is, we see that the normal gradient descent rule corresponds to minimizing a quadratic expansion around our current parameter values when the Hessian is approximated as $\frac{1}{\eta}\mathbf{I}$. A logical continuation is then to ask what kind of update rule we get if we seek to minimize the quadratic approximation plus the regularization term:
$$
\underset{\mathbf{w}}{\operatorname{argmin}} g(\mathbf{w}_k)+\nabla g(\mathbf{w}_k)^T(\mathbf{w}-\mathbf{w}_k)+\frac{1}{2\eta}||\mathbf{w}-\mathbf{w}_k||^2_2 + h(\mathbf{w}).
$$
This minimization problem can further be simplified as:
$$
\underset{\mathbf{w}}{\operatorname{argmin}} \frac{1}{2\eta}||\mathbf{w}-(\mathbf{w}_k-\eta\nabla g(\mathbf{w}_k))||^2_2 + h(\mathbf{w}).
$$
We therefore define the proximal operator to be:
$$
\operatorname{prox}_{\eta}(\mathbf{x}):=\underset{\mathbf{\mathbf{w}}}{\operatorname{argmin}} \frac{1}{2\eta}||\mathbf{w}-\mathbf{x}||_2^2+h(\mathbf{w})
$$
and the proximal gradient update rule as:
$$
\mathbf{w}_{k+1}=\operatorname{prox}_{\eta}(\mathbf{w}_{k}-\eta\nabla g(\mathbf{w}_{k}))
$$
At first sight, it might seem like we have just complicated things by introducing a new optimization problem that will have to be solved at each iteration of the proximal gradient method. However, as we shall see, this new optimization problem has closed form solutions for commonly used regularization functions.

### Proximal mapping for $L^1$ regularization

By expanding the ${L^1}$ regularization term, we see that the expression for the proximal operator corresponds to:
$$
\operatorname{prox}_{\eta}(\mathbf{x})=\underset{\mathbf{w}}{\operatorname{argmin}} \frac{1}{2\eta}||\mathbf{w}-\mathbf{x}||_{L^2}^2+\lambda ||\mathbf{w}||_{L^1}=\underset{\mathbf{w}}{\operatorname{argmin}} \frac{1}{2}||\mathbf{w}-\mathbf{x}||_{L^2}^2+\lambda \eta||\mathbf{w}||_{L^1}.
$$
This optimization problem can be directly solved using subradient, by enforcing the condition that the zero vector ($\mathbf{0}$) have to be in the set of subgradients:
$$
\mathbf{0}\in \partial_\mathbf{z} \left( \frac{1}{2} ||\mathbf{w}-\mathbf{x}||_{L^2}^2+\lambda \eta ||\mathbf{w}||_{L^1} \right)=\mathbf{w}-\mathbf{x}+\lambda \eta \ \partial ||\mathbf{w}||_{L^1} \iff \mathbf{x}-\mathbf{w} \in \lambda \eta\ \partial||\mathbf{w}||_{L^1}.
$$
The subgradient of the $L^1$-norm can be easily computed component-wise as:
$$
[\partial||\mathbf{z}||_{L^1}]_i=\begin{cases} 1 &\mbox{if } z_i>0 \\
[-1,1]&\mbox{if } z_i=0\\
-1 & \mbox{if } z_i<0
\end{cases}
$$
whereupon we find that the proximal mapping is given by:
$$
\operatorname{prox}(\mathbf{x}) = S(\mathbf{x}, \eta \lambda),
$$
where $S(\mathbf{x},\lambda)$ is the soft-thresholding operator, defined component-wise as:
$$
[S(\mathbf{x},\eta \lambda)]_i:=\begin{cases} x_i-\eta \lambda &\mbox{if } x_i> \eta \lambda \\
0&\mbox{if } x_i\in[-\eta \lambda, \eta \lambda]\\
x_i+\eta \lambda & \mbox{if } x_i<-\eta \lambda
\end{cases}
$$
Our final update step in each iteration of the proximal gradient method is thus given by:
$$
\mathbf{w}_{k+1}=S(\mathbf{w}_{k}-\eta_k\nabla g(\mathbf{w}_{k}), \eta_k \lambda).
$$

```python
def softThresholdingFun(w, th):
    w_th = np.zeros(np.size(w))
    w_th[w > th] = w[w > th] - th
    w_th[w < -th] = w[w < -th] + th
    return w_th
```

### Example case
We generate simulated data using the same example case as in Part 1.

```python
# Parameters
n = 200;
wTrue = [1, 0.5]
regLambda = 100

# Generate example data
x = np.random.rand(n)*30 - 15
x = np.sort(x)
X = np.vstack([np.ones(n), x]).T
mu = invLinkFun(X, wTrue)
y = np.random.poisson(mu)

# Evaluate the objective function over a grid
nGrid = 51
W0, W1 = np.meshgrid(np.linspace(wTrue[0]-2, wTrue[0]+2, nGrid), np.linspace(wTrue[1]-1, wTrue[1]+1, nGrid))
# Get the log-likelihood for each parameter combination
negLogLikVals = np.zeros([nGrid, nGrid])
for i in range(nGrid):
    for j in range(nGrid):
        wTmp = np.array([W0[i, j], W1[i, j]])
        negLogLikVals[i, j] = negLogLikFun(X, y, wTmp, regLambda)

# Plotting
def plotFun(W0, W1, funVals, X, y):
    fig = plt.figure(figsize=(15, 5))
    # Objective function surface
    ax_left = fig.add_subplot(1, 2, 1)
    contourHandle = ax_left.contourf(W0, W1, funVals, 150, cmap=cm.coolwarm)
    ax_left.set_xlabel('$w_0$')
    ax_left.set_ylabel('$w_1$')
    cBarHandle = plt.colorbar(contourHandle)
    cBarHandle.set_label('Objective function value');
    # Data
    ax_right = fig.add_subplot(1, 2, 2)
    ax_right.plot(x, y, 'k.', label='data')
    ax_right.set_ylabel('Spike count')
    ax_right.set_xlabel('x')
    ax_right.set_xticks([-10, 0, 10]);
    ax_right.set_ylim([-1, 11]);
    return ax_left, ax_right

plotFun(W0, W1, negLogLikVals, X, y);
```

### Vanilla proximal gradient

```python
# Proximal gradient parameters
eta = 1e-4
nIterations = 20
wInit = np.array([1.8, 0.85])

wUpdates = np.zeros([nIterations, 2])
wUpdates[0, :] = wInit
objFunVals=np.zeros(nIterations)
objFunVals[0] = negLogLikFun(X, y, wUpdates[0, :], regLambda)

# Proximal gradient loop
for i in range(1, nIterations):
    gradTmp = gDerFun(X, y, wUpdates[i-1, :])
    wUpdates[i, :] = softThresholdingFun(wUpdates[i-1, :]- eta*gradTmp, regLambda*eta)
    objFunVals[i] = negLogLikFun(X, y, wUpdates[i, :], regLambda)

# Plotting
ax_left, ax_right = plotFun(W0, W1, negLogLikVals, X, y);
ax_left.plot(wUpdates[:, 0], wUpdates[:, 1], '-o', ms=6, color='white', lw=3)
ax_right.plot(X[:, 1], invLinkFun(X, wUpdates[i, :]), '-', color=0.5*np.ones(3));
```

### Acceleration and backtracking for proximal gradient
Acceleration and backtracking can be implemented as for normal gradient descent, but with a slight modification to the backtracking criteria. That is, acceleration is implemented by first taking a step along the previous update direction ($v_{k+1}$), and then a corrective proximal step ($w_{k+1}$):
\begin{align}
 \mathbf{v}_{k+1} &= \mathbf{w}_k + \frac{k-2}{k+1}(\mathbf{w}_k - \mathbf{w}_{k-1}), \\
 \mathbf{w}_{k+1} &= \mathbf{v}_{k+1} - \eta \nabla nll(\mathbf{v}_{k+1}).
\end{align}
During the corrective proximal step, we again set $\eta$ through a backtracking procedure, where $\eta$ is decreased until the following condition is met:
$$
g(\mathbf{w}_{k+1})\leq g(\mathbf{v}_{k+1})+\nabla g(\mathbf{v}_{k+1})^T(\mathbf{w}_{k+1}-\mathbf{v}_{k+1})+\frac{1}{2\eta} ||\mathbf{w}_{k+1}-\mathbf{v}_{k+1}||^2
$$
The intuition behind this condition is that we do... FIX THIS STILL!!! and it simplifies to the condition used for gradient descent if $h$ is removed from the objective function.

```python
# Proximal gradient parameters
eta = 1e-1
beta = 0.8

wUpdates = np.zeros([nIterations, 2])
wUpdates[0:2, :] = wInit
wTmp = wUpdates[0, :]
objFunVals=np.zeros(nIterations)
objFunVals[0:2] = negLogLikFun(X, y, wUpdates[0, :], regLambda)

# Proximal gradient with backtracking and accerlation
for i in range(2, nIterations):
    gradTmp = gDerFun(X, y, wTmp)
    gTmp = gFun(X, y, wTmp)
    while True:
        wUpdates[i,:] = softThresholdingFun(wTmp - eta*gradTmp, regLambda*eta)
        diff = wUpdates[i,:] - wTmp
        if gFun(X, y, wUpdates[i,:]) > gTmp + np.dot(gradTmp, diff) + np.dot(diff, diff)/(2*eta):
            eta *= beta
        else:
            break
    objFunVals[i] = negLogLikFun(X, y, wUpdates[i,:], regLambda)
    wTmp = wUpdates[i,:] + (i-2)/(i+1)*(wUpdates[i,:] - wUpdates[i-1,:])

# Plotting
ax_left, ax_right = plotFun(W0, W1, negLogLikVals, X, y);
ax_left.plot(wUpdates[:, 0], wUpdates[:, 1], '-o', ms=6, color='white', lw=3)
ax_right.plot(X[:, 1], invLinkFun(X, wUpdates[i, :]), '-', color=0.5*np.ones(3));
```

ConvexOptimization/Part 2, proximal gradient and L1 regularisation.ipynb


```python
#  A regression attempts to fit a of function to observed data to make predictions on new data.
# A linear regression fits a straight line to observed data, attempting to demonstrate a linear relationship
# between variables and make predictions on new data yet to be observed.
```

```python
# Scikit-Learn to perform a basic, unvalidated linear regression on the sample of 10 dogs.
from matplotlib import *
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Import points
df = pd.read_csv('https://bit.ly/3goOAnt', delimiter=",")

# Extract input variables (all rows, all columns but last column)
X = df.values[:, :-1]

# Extract output column (all rows, last column)
Y = df.values[:, -1]

# Fit a line to the points
fit = LinearRegression().fit(X, Y)

# m = 1.7867224, b = -16.51923513
m = fit.coef_.flatten()
b = fit.intercept_.flatten()
print("m = {0}".format(m))
print("b = {0}".format(b))

# show in chart
plt.plot(X, Y, 'o') # scatterplot
plt.plot(X, m*X+b) # line
plt.show()
```

```python
# #  The residual is the numeric difference between the line and the points

#  Another name for residuals are errors, because they reflect how wrong our line is in predicting the data.

# Calculating the residuals for a given line and data

# Import points
points = pd.read_csv('https://bit.ly/3goOAnt', delimiter=",").itertuples()

# Test with a given line
m = 1.93939
b = 4.73333

# calculate sum of squares
for p in points:
    y_actual = p.y
    y_predict = m*p.x + b
    residual = y_actual - y_predict
    print(residual)
```

    -1.67272
    1.3878900000000005
    -0.5515000000000008
    2.5091099999999997
    -0.4302799999999998
    -1.3696699999999993
    0.6909400000000012
    -2.2484499999999983
    2.812160000000002
    -1.1272299999999973

```python
# If we are fitting a straight line through our 10 data points, we likely want to minimize these residuals
# in total so there is as little of a gap as possible between the line and points.
# But how do we measure the “total”? The best approach is to take the sum of squares,
# which simply squares each residual, or multiplies each residual by itself,
# and sums them. We take each actual y value and subtract from it the predicted y value taken from the line,
# then square and sum all those differences.
```

```python
# You might wonder why we have to square the residuals before summing them.
# Why not just add them up without squaring?
# That will not work because the negatives will cancel out the positives.
# What if we add the absolute values, where we turn all negative values into positive values?
# That sounds promising but absolute values are mathematically inconvenient.
# More specifically, absolute values do not work well with Calculus derivatives
# which we are going to use later for gradient descent.
# This is why we choose the squared residuals as our way of totaling the loss.
```

```python
# Calculating the sum of squares for a given line and data

points = pd.read_csv("https://bit.ly/2KF29Bd").itertuples()

# Test with a given line
m = 1.93939
b = 4.73333

sum_of_squares = 0.0

# calculate sum of squares
for p in points:
    y_actual = p.y
    y_predict = m*p.x + b
    residual_squared = (y_predict - y_actual)**2
    sum_of_squares += residual_squared

print("sum of squares = {}".format(sum_of_squares))
```

    sum of squares = 28.096969704500005

```python
# Calculating m and b for a simple linear regression

# Load the data
points = list(pd.read_csv('https://bit.ly/2KF29Bd', delimiter=",").itertuples())

n = len(points)

m = (n*sum(p.x*p.y for p in points) - sum(p.x for p in points) *
    sum(p.y for p in points)) / (n*sum(p.x**2 for p in points) -
    sum(p.x for p in points)**2)

b = (sum(p.y for p in points) / n) - m * sum(p.x for p in points) / n

print(m, b)
```

    1.9393939393939394 4.7333333333333325

```python
# Using inverse and transposed matrices to fit a linear regression

import pandas as pd
from numpy.linalg import inv,qr
import numpy as np

# Import points
df = pd.read_csv('https://bit.ly/3goOAnt', delimiter=",")

# Extract input variables (all rows, all columns but last column)
X = df.values[:, :-1].flatten()

# Add placeholder "1" column to generate intercept
X_1 = np.vstack([X, np.ones(len(X))]).T

# Extract output column (all rows, last column)
Y = df.values[:, -1]

# Calculate coefficents for slope and intercept
b = inv(X_1.transpose() @ X_1) @ (X_1.transpose() @ Y)
print(b) # [1.93939394 4.73333333]

# Predict against the y-values
y_predict = X_1.dot(b)

print (y_predict)
```

    [1.93939394 4.73333333]
    [ 6.67272727  8.61212121 10.55151515 12.49090909 14.43030303 16.36969697
     18.30909091 20.24848485 22.18787879 24.12727273]

```python
# Using QR decomposition to perform a linear regression

# Import points
df = pd.read_csv('https://bit.ly/3goOAnt', delimiter=",")

# Extract input variables (all rows, all columns but last column)
X = df.values[:, :-1].flatten()

# Add placeholder "1" column to generate intercept
X_1 = np.vstack([X, np.ones(len(X))]).transpose()

# Extract output column (all rows, last column)
Y = df.values[:, -1]

# calculate coefficents for slope and intercept
# using QR decomposition
Q, R = qr(X_1)
b = inv(R).dot(Q.transpose()).dot(Y)

print(b)

```

    [1.93939394 4.73333333]

```python
# Gradient descent is an optimization technique that uses derivatives and
# iterations to minimize/maximize a set of parameters against an objective.

# Using gradient descent to find the minimum of a parabola

import random

def f(x):
    return (x - 3) ** 2 + 4

def dx_f(x):
    return 2*(x - 3)

# The learning rate
L = 0.001

# The number of iterations to perform gradient descent
iterations = 100_000

 # start at a random x
x = random.randint(-15,15)

for i in range(iterations):

    # get slope
    d_x = dx_f(x)

    # update x by subtracting the (learning rate) * (slope)
    x -= L * d_x

print(x, f(x))
```

    3.000000000000111 4.0

```python
# Performing gradient descent for a linear regression

# Import points from CSV
points = list(pd.read_csv("https://bit.ly/2KF29Bd").itertuples())

# Building the model
m = 0.0
b = 0.0

# The learning Rate
L = .001

# The number of iterations
iterations = 100_000

n = float(len(points))  # Number of elements in X

# Perform Gradient Descent
for i in range(iterations):

    # slope with respect to m
    D_m = sum(2 * p.x * ((m * p.x + b) - p.y) for p in points)

    # slope with respect to b
    D_b = sum(2 * ((m * p.x + b) - p.y) for p in points)

    # update m and b
    m -= L * D_m
    b -= L * D_b

print("y = {0}x + {1}".format(m, b))
```

    y = 1.9393939393939548x + 4.733333333333227

```python
# Calculating partial derivatives for m and b

from sympy import *

m, b, i, n = symbols('m b i n')
x, y = symbols('x y', cls=Function)

sum_of_squares = Sum((m*x(i) + b - y(i)) ** 2, (i, 0, n))

d_m = diff(sum_of_squares, m)
d_b = diff(sum_of_squares, b)
print(d_m)
print(d_b)
```

    Sum(2*(b + m*x(i) - y(i))*x(i), (i, 0, n))
    Sum(2*b + 2*m*x(i) - 2*y(i), (i, 0, n))

```python
# Performing stochastic gradient descent for a linear regression

# Input data
data = pd.read_csv('https://bit.ly/2KF29Bd', header=0)

X = data.iloc[:, 0].values
Y = data.iloc[:, 1].values

n = data.shape[0]  # rows

# Building the model
m = 0.0
b = 0.0

sample_size = 1  # sample size
L = .0001  # The learning Rate
epochs = 1_000_000  # The number of iterations to perform gradient descent

# Performing Stochastic Gradient Descent
for i in range(epochs):
    idx = np.random.choice(n, sample_size, replace=False)
    x_sample = X[idx]
    y_sample = Y[idx]

    # The current predicted value of Y
    Y_pred = m * x_sample + b

    # d/dm derivative of loss function
    D_m = (-2 / sample_size) * sum(x_sample * (y_sample - Y_pred))

    # d/db derivative of loss function
    D_b = (-2 / sample_size) * sum(y_sample - Y_pred)
    m = m - L * D_m  # Update m
    b = b - L * D_b  # Update b

    # print progress
    if i % 10000 == 0:
        print(i, m, b)

print("y = {0}x + {1}".format(m, b))
```

    0 0.006 0.002
    10000 2.36326760682638 1.8835518624103165
    20000 2.208478682839455 2.860757272083633
    30000 2.0990271430034566 3.5015104981731477
    40000 2.059718240314351 3.918670691928773
    50000 2.0254319787495616 4.228272179662717
    60000 2.0128706983709828 4.3987557816635325
    70000 1.9588699155518574 4.511397889311558
    80000 1.9572242882073074 4.573105411192914
    90000 1.9603079930214902 4.646560110817617
    100000 1.915286401953193 4.679197268853813
    110000 1.9403467582673453 4.694118062444541
    120000 1.9599274086298113 4.725943861094068
    130000 1.9368722399981075 4.7143432848607825
    140000 1.963520934610125 4.71478629302077
    150000 1.9360986672130869 4.728305205329085
    160000 1.9324433155776284 4.741555567620757
    170000 1.926229705774608 4.7507334297890695
    180000 1.9432750003962185 4.731457564467172
    190000 1.9652083911680416 4.750695486563088
    200000 1.924857106440728 4.741023799177265
    210000 1.949168382522107 4.742302871466551
    220000 1.9482919821971325 4.732498280457749
    230000 1.9436648255769589 4.744293925905611
    240000 1.9705030140252286 4.742529291867208
    250000 1.9353429418031627 4.739497162984329
    260000 1.9562758398526634 4.738933645814807
    270000 1.9382884118185948 4.728545301878929
    280000 1.93255522206419 4.720959425792094
    290000 1.9410867423424232 4.719485579197047
    300000 1.928348444769958 4.709229998935416
    310000 1.944704859029115 4.734245656426094
    320000 1.9315489900068277 4.7164345649351676
    330000 1.939995611167781 4.709338225607842
    340000 1.961142019124275 4.705429525526002
    350000 1.9216025853518108 4.723954582852459
    360000 1.944900362530313 4.7252435418819125
    370000 1.9399721319259282 4.734938065834019
    380000 1.9497781564660994 4.742384445451385
    390000 1.9805381395468729 4.731931988681261
    400000 1.9454482978428054 4.730618875951833
    410000 1.9703383827976337 4.733665784535996
    420000 1.981650785600547 4.742295189046688
    430000 1.972530745570693 4.74071158972351
    440000 1.9111024810843704 4.726256650760091
    450000 1.9082781943786304 4.731152175860961
    460000 1.9204477114790428 4.739821573687611
    470000 1.9644981269863893 4.745855875675362
    480000 1.9263137432887223 4.743643371721059
    490000 1.91711757641405 4.74294635046912
    500000 1.930934583192176 4.74212184317665
    510000 1.967479133941378 4.736673843592278
    520000 1.9505228605459906 4.728075419769489
    530000 1.9516554472517902 4.728522913632032
    540000 1.9417919849569742 4.749508598760058
    550000 1.9611385059437294 4.766887931996634
    560000 1.9267487723441967 4.745625753029247
    570000 1.9160704416577639 4.735761047329838
    580000 1.929617903629193 4.721682821898547
    590000 1.9149289269251695 4.728326533479536
    600000 1.931097863515108 4.706517870443267
    610000 1.938513500360612 4.72158595083187
    620000 1.9654752270010292 4.727653810731125
    630000 1.9124274374663552 4.714167893880229
    640000 1.9460836587073391 4.721447016138792
    650000 1.9548197285647888 4.714751503819112
    660000 1.946696243788322 4.7234709202203105
    670000 1.943371459630136 4.722904788603818
    680000 1.9573802094954733 4.7256764624636
    690000 1.931115907669294 4.718575794959121
    700000 1.9262178442406925 4.7242553744409905
    710000 1.9584890714298269 4.707751870061937
    720000 1.91096125496631 4.732834014230203
    730000 1.9638450307901556 4.730820586569408
    740000 1.9376318945047226 4.742484193616451
    750000 1.9764575310410655 4.721173739823862
    760000 1.959201460128207 4.728725027441086
    770000 1.9616063379508497 4.745386968448955
    780000 1.9303406405244505 4.7324681138365285
    790000 1.9424228463604172 4.728831976971964
    800000 1.9043294280432401 4.69159602664981
    810000 1.9336889001186668 4.7071650308872695
    820000 1.9441607414771653 4.712772819341543
    830000 1.959822283009035 4.73382719635445
    840000 1.9453503711741207 4.724613103589071
    850000 1.9587792297040645 4.737822839830986
    860000 1.9157005293366143 4.722802115138588
    870000 1.9464146899956396 4.719103639727119
    880000 1.9111757048916815 4.714647715026774
    890000 1.9257806715010586 4.731873835908689
    900000 1.9534173041959169 4.722614847513641
    910000 1.9505340945508731 4.73813717689114
    920000 1.9330475833082221 4.73912605385538
    930000 1.9594103111947985 4.752444875710609
    940000 1.9415876754302852 4.754499178310029
    950000 1.9306788443386962 4.752825434666956
    960000 1.921548945614283 4.76041767073736
    970000 1.962487828671824 4.772474403778124
    980000 1.9865264430013605 4.754946770865421
    990000 1.9421834301828678 4.7535420839260265
    y = 1.9502363279513883x + 4.748196684882565

```python
# correlation coefficient, also called the Pearson correlation,
# which measures the strength of the relationship between two variables as a value
# between -1 and 1. A correlation coefficient closer to 0 indicates there is no correlation.
# A correlation coefficient closer to 1 indicates a strong positive correlation,
# meaning when one variable increases the other proportionally increases.
# If it is closer to -1 then it indicates a strong negative correlation,
# which means as one variable increases the other proportionally decreases.
```

```python
# Using Pandas to see the correlation coefficent between every pair of variables

# Read data into Pandas dataframe
df = pd.read_csv('https://bit.ly/2KF29Bd', delimiter=",")

# Print correlations between variables
correlations = df.corr(method='pearson')
print(correlations)

```

              x         y
    x  1.000000  0.957586
    y  0.957586  1.000000

```python
# Calculating correlation coefficient from scratch in Python

from math import sqrt

# Import points from CSV
points = list(pd.read_csv("https://bit.ly/2KF29Bd").itertuples())
n = len(points)

numerator = n * sum(p.x * p.y for p in points) - \
            sum(p.x for p in points) * sum(p.y for p in points)

denominator = sqrt(n*sum(p.x**2 for p in points) - sum(p.x for p in points)**2) \
              * sqrt(n*sum(p.y**2 for p in points) - sum(p.y for p in points)**2)

corr = numerator / denominator

print(corr)
```

    0.9575860952087218

```python
#  Calculating the critical value from a T-distribution

from scipy.stats import t

n = 10
lower_cv = t(n-1).ppf(.025)
upper_cv = t(n-1).ppf(.975)

print(lower_cv, upper_cv)
```

    -2.262157162740992 2.2621571627409915

```python
# Testing significance for linear-looking data

from scipy.stats import t
from math import sqrt

# sample size
n = 10

lower_cv = t(n-1).ppf(.025)
upper_cv = t(n-1).ppf(.975)

# correlation coefficient
# derived from data https://bit.ly/2KF29Bd
r = 0.957586

# Perform the test
test_value = r / sqrt((1-r**2) / (n-2))

print("TEST VALUE: {}".format(test_value))
print("CRITICAL RANGE: {}, {}".format(lower_cv, upper_cv))

if test_value < lower_cv or test_value > upper_cv:
    print("CORRELATION PROVEN, REJECT H0")
else:
    print("CORRELATION NOT PROVEN, FAILED TO REJECT H0 ")

# Calculate p-value
if test_value > 0:
    p_value = 1.0 - t(n-1).cdf(test_value)
else:
    p_value = t(n-1).cdf(test_value)

# Two-tailed, so multiply by 2
p_value = p_value * 2
print("P-VALUE: {}".format(p_value))
```

    TEST VALUE: 9.399564671312076
    CRITICAL RANGE: -2.262157162740992, 2.2621571627409915
    CORRELATION PROVEN, REJECT H0
    P-VALUE: 5.9763860877914965e-06

```python
# Creating a correlation matrix in Pandas

# Read data into Pandas dataframe
df = pd.read_csv('https://bit.ly/2KF29Bd', delimiter=",")

# Print correlations between variables
coeff_determination = df.corr(method='pearson') ** 2
print(coeff_determination)
```

              x         y
    x  1.000000  0.916971
    y  0.916971  1.000000

```python
# Calculating a prediction interval of vet visits for a dog that’s 8.5 years old

from math import sqrt

# Load the data
points = list(pd.read_csv('https://bit.ly/2KF29Bd', delimiter=",").itertuples())

n = len(points)

# Linear Regression Line
m = 1.939
b = 4.733

# Calculate Prediction Interval for x = 8.5
x_0 = 8.5
x_mean = sum(p.x for p in points) / len(points)

t_value = t(n - 2).ppf(.975)

standard_error = sqrt(sum((p.y - (m * p.x + b)) ** 2 for p in points) / (n - 2))

margin_of_error = t_value * standard_error * \
                  sqrt(1 + (1 / n) + (n * (x_0 - x_mean) ** 2) / \
                       (n * sum(p.x ** 2 for p in points) - sum(p.x for p in points) ** 2))

predicted_y = m*x_0 + b

# Calculate prediction interval
print(predicted_y - margin_of_error, predicted_y + margin_of_error)
```

    16.462516875955465 25.966483124044537

```python
# Doing a train/test split on linear regression

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Load the data
df = pd.read_csv('https://bit.ly/3cIH97A', delimiter=",")

# Extract input variables (all rows, all columns but last column)
X = df.values[:, :-1]

# Extract output column (all rows, last column)
Y = df.values[:, -1]

# Separate training and testing data
# This leaves a third of the data out for testing
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=1/3)

model = LinearRegression()
model.fit(X_train, Y_train)
result = model.score(X_test, Y_test)
print("R^2: %.3f" % result)
```

    R^2: 0.993

```python
# Using 3-fold cross validation for a linear regression

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score

df = pd.read_csv('https://bit.ly/3cIH97A', delimiter=",")

# Extract input variables (all rows, all columns but last column)
X = df.values[:, :-1]

# Extract output column (all rows, last column)\
Y = df.values[:, -1]

# Perform a simple linear regression
kfold = KFold(n_splits=3, random_state=7, shuffle=True)
model = LinearRegression()
results = cross_val_score(model, X, Y, cv=kfold)
print(results)
print("MSE: mean=%.3f (stdev-%.3f)" % (results.mean(), results.std()))
```

    [0.99337354 0.99345032 0.99251425]
    MSE: mean=0.993 (stdev-0.000)

```python
# Using a random-fold validation for a linear regression

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score

df = pd.read_csv('https://bit.ly/3cIH97A', delimiter=",")

# Extract input variables (all rows, all columns but last column)
X = df.values[:, :-1]

# Extract output column (all rows, last column)\
Y = df.values[:, -1]

# Perform a simple linear regression
kfold = KFold(n_splits=3, random_state=7, shuffle=True)
model = LinearRegression()
results = cross_val_score(model, X, Y, cv=kfold)
print(results)
print("MSE: mean=%.3f (stdev-%.3f)" % (results.mean(), results.std()))
```

    [0.99337354 0.99345032 0.99251425]
    MSE: mean=0.993 (stdev-0.000)

```python
#  A linear regressoin with two input variables

# Load the data
df = pd.read_csv('https://bit.ly/2X1HWH7', delimiter=",")

# Extract input variables (all rows, all columns but last column)
X = df.values[:, :-1]

# Extract output column (all rows, last column)\
Y = df.values[:, -1]

# Training
fit = LinearRegression().fit(X, Y)

# Print coefficients
print("Coefficients = {0}".format(fit.coef_))
print("Intercept = {0}".format(fit.intercept_))
print("z = {0} + {1}x + {2}y".format(fit.intercept_, fit.coef_[0], fit.coef_[1]))
```

    Coefficients = [2.00672647 3.00203798]
    Intercept = 20.109432820035963
    z = 20.109432820035963 + 2.0067264725128062x + 3.002037976646693y

```python

```


# Linear Vs. Non-Linear Functions
** October 2017 **

** Andrew Riberio @ [AndrewRib.com](http://www.andrewrib.com) **

Resources
* https://en.wikipedia.org/wiki/Linear_function
* https://www.montereyinstitute.org/courses/Algebra1/COURSE_TEXT_RESOURCE/U03_L2_T5_text_final.html
* https://en.wikipedia.org/wiki/Linear_combination

## Libraries

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

## Linear Functions

A linear function of one variable has the geometric interpretation as a line when we plot the space of inputs against the output space: y = f(x). In three dimensions, in 3d space, we can do a similar thing z = f(x,y), but we arrive at the generalization of a line in 3d space called a *plane.*

```python
# There are no functions in one dimension since we cannot deliniate inputs/outputs.
# A single dimension is just a constant value of existence with no notion of difference.

# Two dimensional funciton
def f2(x):
    return x

# Three dimensional funciton. Output is z.
def f3(x,y):
    return x + y

# Four dimensional function. We cannot visualize this easily.
def f4(x,y,z):
    return x + y + z

t1 = np.arange(0.0, 5.0, 0.1)

plt.figure(1)
plt.subplot(211)
plt.plot(t1, f2(t1))

X,Y = np.meshgrid(t1,t1)

plt.subplot(212,projection="3d")

plt.plot(X,Y, f3(t1,t1))

plt.show()
```

**Theorem:** The derivitive of every linear function is a constant.

```python
x_sym, y_sym = sp.symbols('x y')
print("dy/dx f2 = {0}".format(sp.diff(f2(x_sym))))
print("∂z/∂x f3 = {0}".format(sp.diff(f3(x_sym,y_sym),x_sym)))
```

    dy/dx f2 = 1
    ∂z/∂x f3 = 1

**Definition:** A *linear combination* of varibles (x1, x2, ... , xn) = ∂1x1 + ∂2x2+ ... + ∂nxn where ∂ are constants.

**Theorem:** All linear functions can be represented as a linear combination.

## Non-Linear Functions

Non-linear functions are more unpredictable than linear functions because the derivitive of a non-linear function is always a function, not a constant. The geometric interpretation of non-linear functions encompasses the diverse space of curves.

```python
# Two dimensional funciton
def f2(x):
    return x**2

# Three dimensional funciton. Output is z.
def f3(x,y):
    return x * y

t1 = np.arange(0.0, 5.0, 0.1)

plt.figure(1)
plt.subplot(211)
plt.plot(t1, f2(t1))

plt.subplot(212,projection="3d")
X,Y = np.meshgrid(t1,t1)
plt.plot(X,Y, f3(t1,t1))
plt.show()
```

**Theorem:** The derivitive of every non-linear function is **not a constant**.

```python
x_sym, y_sym = sp.symbols('x y')
print("dy/dx f2 = {0}".format(sp.diff(f2(x_sym))))
print("∂z/∂x f3 = {0}".format(sp.diff(f3(x_sym,y_sym),x_sym)))
```

    dy/dx f2 = 2*x
    ∂z/∂x f3 = y

**Theorem:** Non-linear functions cannot be represented by a linear combination.

Notebooks/Linear Vs. Non-Linear Functions.ipynb


# Examples of image reconstruction using PCA

Data classification in high dimensional spaces can be challenging and the results often lack robustness.
This well-known problem has its own name; the curse of dimensionality. Principal Component
Analysis is a popular method for dimensionality reduction. It can also be used as a
visualisation tool to project data into lower dimensional spaces, which improves data exploration and comprehension.

In this script, we will use PCA to study a few characteristics of the MNIST test dataset. It contains 10 K images of
hand-written digits from 0 to 9. It is a good introduction to PCA in image analysis.

## The PCA transform
This linear transform allows us to move from the natural/original space $\cal{X}$ into the component space $\cal{Z}$ and is defined as

  $\bf{z} = \bf{W}^{T}(\bf{x}-\bf{\mu})$

with

$\begin{align}
\bf{x} &= [x_{1} x_{2} \cdots  x_{N}]^\top \\
\bf{\mu} &= [\mu_{1} \mu_{2} \cdots \mu_{N}]^\top  \\
\bf{z} &= [z_{1} z_{2} \cdots z_{N}]^\top  \\
\end{align}$

where N is the dimension of space and $\bf{\mu}$ is the mean of the N-dimensional data X.

The $W$ matrix if made of the N eigenvectors $\bf{w}_{i}$ of the covariance matrix $\Sigma$. They are stacked together along columns

$\begin{align}
\bf{W} &= \begin{pmatrix} \bf{w}_{1} & \bf{w}_{2} & \dotsb & \bf{w}_{N}  \end{pmatrix} \\
&= \begin{pmatrix} w_{1,1} & w_{2,1} & \dotsb & w_{N,1} \\
w_{1,2} & w_{2,2} & \dotsb & w_{N,2} \\
\vdots &  \vdots & \dotsb & \vdots \\
w_{1,N} & w_{2,N} & \dotsb & w_{N,N} \end{pmatrix}
\end{align}$

As for the covariance matrix $\Sigma$, it is computed from the N-dimensional data distribution X

$
\begin{align}
\bf{\Sigma} &= E\{(\bf{x}-\bf{\mu})(\bf{x}-\bf{\mu})^{T} \} \\
&= \begin{pmatrix} \sigma_{1}^2 & \sigma_{1,2} & \cdots & \sigma_{1,N} \\
\sigma_{1,2} & \sigma_{2}^2 & \cdots & \sigma_{2,N} \\
\vdots & \vdots & \ddots & \vdots \\
\sigma_{1,N} & \sigma_{2,N} & \cdots & \sigma_{N}^2 \end{pmatrix}
\end{align}
$

The $\it{inverse}$ PCA transform allows us to move from the component space $\cal{Z}$ back to the natural/original space $\cal{X}$
and is defined as

  $\bf{x} = \bf{W}\bf{z} + \bf{\mu}$

The principal components z are sorted in decreasing order of variance $var(z_{i})=\lambda_{i}$ where the $\lambda_{i}$ are
the eigenvalues of the covariance matrix $\bf{\Sigma}$.

N.B. The principal components usually come in one of the two popular notations: $z_{i}$ or $PCA_{i}$.

## PCA as a tool for dimensionality reduction

The last equation shows that we can reconstruct the vector $\bf{x}$ exactly. We usually drop the
least significant principal components (the last elements in z and the last columns in $\bf{W}$)
because they only contain noise. This produces an approximate reconstruction of x

  $\bf{x} \approx \bf{\tilde{W}}\bf{\tilde{z}} + \bf{\mu}$

with the modified arrays

  $\bf{\tilde{z}} = \begin{pmatrix} z_{1} \ z_{2} \ \dotsb \ z_{M} \end{pmatrix}^{T} $

and

$
\begin{align}
\bf{\tilde{W}} &= \begin{pmatrix} \bf{w}_{1} & \bf{w}_{2} & \dotsb & \bf{w}_{M}  \end{pmatrix} \\
&= \begin{pmatrix} w_{1,1} & w_{2,1} & \dotsb & w_{M,1} \\
w_{1,2} & w_{2,2} & \dotsb & w_{M,2} \\
\vdots &  \vdots & \ddots & \vdots \\
w_{1,N} & w_{2,N} & \dotsb & w_{M,N} \end{pmatrix}
\end{align}
$

The scree plot, which displays $\lambda$ $\it{versus}$ the component number, is a practical tool for finding
the number of relevant components, i.e. those that contain information rather than noise.

## PCA as a tool for visualisation

The MNIST dataset contains 10 K images. A PCA analysis will generate as many eigenvectors $\bf{w}_{i}$
that can be reshaped into images. Each original images is a linear combinations of the eigenvectors. As
we will see below, the first principal components $z_{i}$ are the most important ones. If we keep only
the first two, each image $\bf{x}$ can be approximated as

$
\begin{align}
\bf{x} &= \sum_{i=1}^N z_{i} \bf{w}_{i} + \bf{\mu} \\
& \approx z_{1} \bf{w}_{1} + z_{2} \bf{w}_{2} + \bf{\mu}
\end{align}
$

This means that each image can be 'summarized' by only two numbers $z_{1}$ and $z_{2}$. Thus, the
10 K images can be represented by as many points in the 2-D PCA space of
coordinates ($z_{1}$, $z_{2}$) or equivalently ($PCA_{1}$, $PCA_{2}$). This representation
makes image comparison easy as we will see below.

```python
print(__doc__)

# Author: Pierre Gravel <pierre.gravel@iid.ulaval.ca>
# License: BSD

%matplotlib inline

import os
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
from matplotlib.offsetbox import OffsetImage, AnnotationBbox

from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

import seaborn as sns
sns.set(color_codes=True)

# Used for reproductibility of the results
np.random.seed(43)
```

    Automatically created module for IPython interactive environment

# Data preprocessing

### Load the MNIST test dataset.

The dataset contains 10 K images of size 28 x 28, each stored into a line of 784 elements. The X data is an array of
shape 10K x 784; each line corresponds to an image (an observation) and each column corresponds to a feature.
The y data is an array of 10 K elements that contains the image labels $[0,\cdots ,9]$

The test dataset can be downloaded from the Kaggle website: https://www.kaggle.com/oddrationale/mnist-in-csv

In what follows, we assume that you downloaded the dataset into the current directory.

```python
filename = 'mnist_test.csv'
df = pd.read_csv(filename)

# Extract from the Panda dataframe the features X and the labels y
X = df.drop(['label'], axis=1).values
y = df[['label']].values

```

Display one (inverted) image for each class. N.B. The original images are white on a black background.

```python
fig, ax = plt.subplots(2,5, figsize=(10,6))
for i in range(10):
    idx = np.where(y==i)
    idx = idx[0][0]
    plt.subplot(2,5, i + 1)
    plt.imshow(255 - X[idx,:].reshape(28, 28), cmap='gray', interpolation='nearest')
    plt.xticks(())
    plt.yticks(())
fig.tight_layout()

plt.savefig('13.1.1_Examples_from_MNIST_dataset.png')
plt.savefig('13.1.1_Examples_from_MNIST_dataset.pdf')
```

### Normalise the data

```python
sc = StandardScaler().fit(X)
X_s = sc.transform(X)
```

# PCA analysis

### Compute the PCA transform using all the images

```python
pca = PCA()
pca.fit(X_s);
```

### Display the PCA scree plot

The scree plot shows the eigenvalues $\lambda$ in a decreasing order. The most relevant eigenvectors are on the left and
the least relevant ones on the right.

The second panel shows the fraction of the image variance already explained by the first n principal components.
For instance, the first 50 principal components explain
about 60% of the information in the dataset, whereas the first 200 account for 90% of it.

```python
(n,m) = X_s.shape
n_components = np.arange(1,m+1)

fig, ax = plt.subplots(2,1, figsize=(10,6))
ax[0].plot(n_components, pca.singular_values_)
ax[0].set_xlabel('Eigenvectors', fontsize=16)
ax[0].set_ylabel('Eigenvalues', fontsize=16)
ax[0].set_title('MNIST Scree plot', fontsize=16)

ax[1].plot(n_components, 100*np.cumsum(pca.explained_variance_ratio_))
ax[1].set_xlabel('Eigenvectors', fontsize=16)
ax[1].set_ylabel('Ratio (%)', fontsize=16)
ax[1].set_title('Proportion of variance explained', fontsize=16)

fig.tight_layout()

plt.savefig('13.1.2_MNIST_scree_plot.png')
plt.savefig('13.1.2_MNIST_scree_plot.pdf')
```

### Show examples of eigenvectors (reshaped as images)

The first eigenvectors (first row) are the most important ones as they contain coherent structures.
The last eigenvectors (second row) usually contain only noise and are generally discarded.

```python
indx = [1, 2, 3, 4, 300, 400, 500, 600]

fig, ax = plt.subplots(2,4, figsize=(10,6))
for i in range(8):
    im = pca.components_[indx[i]-1,:].reshape(28, 28)
    plt.subplot(2,4, i + 1)
    plt.imshow(im, cmap='gray', interpolation='nearest')
    plt.title('$Eigenvector_{%d}$' % (indx[i]), fontsize=16)
    plt.xticks(())
    plt.yticks(())
fig.tight_layout()

plt.savefig('13.1.3_Examples_of_eigenvectors.png')
plt.savefig('13.1.3_Examples_of_eigenvectors.pdf')
```

# Image similarities

### Project the image data into a 2-D space defined by the first two principal components.

Compute the principal components of the image data and make a plot where each image is represented as a
point in 2-D PCA space of coordinates ($PCA_{1}$, $PCA_{2}$) or equivalently ($z_{1}$, $z_{2}$).
Superpose on it the labels for five images in each class.

Notice how the classes are clustered; the '1' images are on the left, the '0' images are on the right, the '7' on the top, etc.
This is one of the reasons why PCA is so much used in data analysis.

The classes are not perfectly separated however since there is some visible overlap between them.

```python
Z = pca.transform(X_s)
```

```python
sns.set_style('white')
fig, ax = plt.subplots(figsize = (10, 10))
ax.scatter(Z[:,0],Z[:,1], c = 'c', s=1)
ax.set_xlabel('$PCA_{1}$', fontsize=18)
ax.set_ylabel('$PCA_{2}$', fontsize=18)
ax.set_title('2-D PCA space for MNIST', fontsize=16)

for i in range(10):
    idx = np.where(y==i)
    idx = idx[0][0:5]
    ax.scatter(Z[idx,0], Z[idx,1],c='k', marker=r"$ {} $".format(i), edgecolors='none', s=150 )

ax.grid(color='k', linestyle='--', linewidth=.2)
sns.despine()

plt.savefig('13.1.4_2D_PCA_space_for_MNIST.png')
plt.savefig('13.1.4_2D_PCA_space_for_MNIST.pdf')
```

In the next figure, we replace the label markers with their corresponding images. Different labels overlap because
their handwritten images share similarities. For instance, a curved '7' may look like a '9', a squashed '3' may
looked like an '8', etc.

```python
sns.set_style('white')
image_shape = (28, 28)
fig, ax = plt.subplots(figsize = (10, 10))
ax.scatter(Z[:,0],Z[:,1], c = 'b', s=1)
ax.set_xlabel('$PCA_{1}$', fontsize=18)
ax.set_ylabel('$PCA_{2}$', fontsize=18)
ax.set_title('2-D PCA space for MNIST', fontsize=16)

z = np.zeros((28, 28))

for i in range(10):
    idx = np.where(y==i)
    idx = idx[0]

    for j in range(5):
        J = 1.-X[idx[j],:].reshape(image_shape)
        I = (np.dstack((J,J,z)) * 255.999) .astype(np.uint8)

        imagebox = OffsetImage(I, zoom=.5);
        ab = AnnotationBbox(imagebox, (Z[idx[j],0], Z[idx[j],1]), frameon=True, pad=0);
        ax.add_artist(ab);

ax.grid(color='k', linestyle='--', linewidth=.2)
sns.despine()

plt.savefig('13.1.5_2D_PCA_space_for_MNIST_with_images.png')
plt.savefig('13.1.5_2D_PCA_space_for_MNIST_with_images.pdf')
```

# Find the most similar image classes

If images from two different classes look alike, chances are, we will make classification errors when looking at them. The
example of '1' and '7' images is well known. This is why we often put an horizontal bar in the middle of the '7' to
make differences between them more visible. Do classifiers make the same errors?

In what follows, we will split the dataset into a training and a test datasets. A K-Nearest-Neighbor
classifier (KNN) will first be trained on the training dataset. Then, it will be used to make class predictions on the test dataset.

The confusion matrix between the predicted and the true test labels will help to identify the most
common errors. This will tell us what are the most lookalike image classes from the classifier standpoint.

### Split the dataset into a training and a test datasets

```python
X_train, X_test, y_train, y_test = train_test_split(X_s, y, random_state=0,train_size=0.8)
y_train = y_train.ravel()
y_test = y_test.ravel()

```

Train a KNN classifier on the training dataset (using 5 neighbors) and use it to classify the test
dataset.

Warning: the next cell may take a few seconds to a minute to compute. In a few years from now, this warning will be pointless
given the increasing speeds of hardware and software!

```python
clf = KNeighborsClassifier(n_neighbors=5)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)

```

Compute and display the confusion matrix between the true and the predicted labels for the test dataset. The confusion matrix
tells us what are the most common classification mistakes. For instance, images of '8' were confused 13 times
with images of '5'. The most common mistakes were found between (8,5) and (7,9) pairs. Surprisingly, the (1,7)
pair was not the most prevalent source of confusion.

```python
fig, ax = plt.subplots(figsize = (7, 7))

cm = confusion_matrix(y_test, y_pred)
ConfusionMatrixDisplay(cm).plot(ax=ax)

fig.savefig('13.1.6_confusion_matrix.png')
fig.savefig('13.1.6_confusion_matrix.pdf')
```

# Examples of image reconstructions and comparisons

The following section will explain several observations we made about the image class distributions in the 2-D PCA space.
It is important to mention that the results could have been different if we had trained the KNN classifier with more
neighbors (5) and more principal components per image (2).

### A few useful functions

We define functions that will be used to

Show the reconstruction performances with increasing number of principal components

Compare the images reconstructed from the first two components $PCA_{1}$ and $PCA_{2}$

The first function computes PCA approximations of images of labels i and j using the first n principal components.

```python
def PCA_approximations(n_comp, X_s, y, sc, image_shape, i, j):

    # Find one image with label i and one image with label j
    idx = np.where(y==i)
    idx = idx[0][0]
    image_i = X_s[idx,:].reshape(1, -1)

    idx = np.where(y==j)
    idx = idx[0][0]
    image_j = X_s[idx,:].reshape(1, -1)

    # Compute the PCA transform using only the first n principal components
    pca = PCA(n_components=n_comp)
    pca.fit(X_s)

    # Transform and reconstruct the image i using the first n principal components
    Z = pca.transform(image_i)
    x_i = pca.inverse_transform(Z)

    # Remove the normalisation transform
    x_i = sc.inverse_transform(x_i)
    x_i = x_i.reshape(image_shape)

    # Transform and reconstruct the image  using the first n principal components
    Z = pca.transform(image_j)
    x_j = pca.inverse_transform(Z)

    # Remove the normalisation transform
    x_j = sc.inverse_transform(x_j)
    x_j = x_j.reshape(image_shape)

    # Remove the normalisation transform from the corresponding original images
    X_i = sc.inverse_transform(image_i)
    X_i = X_i.reshape(image_shape)
    X_j = sc.inverse_transform(image_j)
    X_j = X_j.reshape(image_shape)

    return (x_i, x_j, X_i, X_j)
```

The second function displays a mosaic of the original images with their PCA approximations with 1, 2, 20
and 200 principal components.

```python
def display_PCA_reconstructions(X_s, y, sc, image_shape, i, j):
    # Number of principal components used for the reconstruction of each image
    ncomp = [1, 2, 20, 200]

    fig, ax = plt.subplots(2,len(ncomp)+1, figsize=(10,6))
    for k in range(len(ncomp)):
        # Reconstruct both images with the same number of components
        (x_i, x_j, X_i, X_j) = PCA_approximations(ncomp[k], X_s, y, sc, image_shape, i, j)

        plt.subplot(2,5, k+2)
        plt.imshow(255-x_i, cmap='gray', interpolation='nearest')
        plt.title('M = %d' % ncomp[k], fontsize=16)
        plt.xticks(())
        plt.yticks(())

        plt.subplot(2,5, k + 7)
        plt.imshow(255-x_j, cmap='gray', interpolation='nearest')
        plt.xticks(())
        plt.yticks(())

    # Original images for reference
    plt.subplot(2,5, 1)
    plt.imshow(255-X_i, cmap='gray', interpolation='nearest')
    plt.title('Original', fontsize=16)
    plt.xticks(())
    plt.yticks(())

    plt.subplot(2,5, 6)
    plt.imshow(255-X_j, cmap='gray', interpolation='nearest')
    plt.xticks(())
    plt.yticks(())

    fig.tight_layout()

```

## Example I: 7 versus 9

The figure below shows image reconstructions for an increasing number of principal components.
Notice how the reconstructions of '7' and '9' images become easily recognizable when 200 principal components are used.
This is not too surprising since, as mentionned before, the first 200 components account for 90% of the total image variance.

Notice also how the first two reconstructions (M = 1,2) are very similar for both '7' and '9' images. Hence, they share
similar values of $z_{1}$ and $z_{2}$. As a result, the '7' and '9' images should be neighbors in the 2-D PCA space
($z_{1}$, $z_{2}$) or equivalently ($PCA_{1}$, $PCA_{2}$). This is the case; their distributions overlap in the
top of the 2-D PCA space (see corresponding figures above).

```python
i = 7
j = 9

display_PCA_reconstructions(X_s, y, sc, image_shape, i, j)

plt.savefig('13.1.7_Examples_image_reconstructions_7_and_9.png')
plt.savefig('13.1.7_Examples_image_reconstructions_7_and_9.pdf')
```

## Example II: 1 versus 7

This new example is counter intuitive. The reconstructions with M = 2 are now quite different; the '1' and '7' images
do not share similar values of $z_{1}$ and $z_{2}$. As a result, the '1' and '7' images are not close neighbors in
the 2-D PCA space. The '1' are found on the left of the 2-D PCA space whereas the '7' are found at the top. Their
distributions barely overlap.

```python
i = 1
j = 7
display_PCA_reconstructions(X_s, y, sc, image_shape, i, j)

plt.savefig('13.1.8_Examples_image_reconstructions_1_and_7.png')
plt.savefig('13.1.8_Examples_image_reconstructions_1_and_7.pdf')
```

```python

```


# Special Series

```python
%matplotlib inline
from sympy import *
init_printing()
```

```python
x, t = symbols('x, t')
```

SymPy can compute special series like formal power series and fourier series. This is a new feature released in SymPy 1.0

Let's try computing formal power series of some basic functions.

```python
exp_series = fps(exp(x), x)
exp_series
```

This looks very similar to what ``series`` has to offer, but unlike series a formal power series object returns an infinite expansion.

```python
exp_series.infinite # Infinite representation
```

We can easily find out any term of the expansion (no need to recompute the expansion).

```python
exp_series.term(51) # equivalent to exp_series[51]
```

```python
exp_series.truncate(10) # return a truncated series expansion
```

# Exercise

Try computing the formal power series of $\log(1 + x)$. Try to look at the infinite representation. What is the 51st term in this case? Compute the expansion about 1.

```python
log_series = fps(?)
log_series
```

```python
# infinite representation

```

```python
# 51st term
```

```python
# expansion about 1
```

# Fourier Series

Fourier series for functions can be computed using ``fourier_series`` function.

A sawtooth wave is defined as:
   1. $$ s(x) = x/\pi \in (-\pi, \pi) $$
   2. $$ s(x + 2k\pi) = s(x) \in (-\infty, \infty) $$

Let's compute the fourier series of the above defined wave.

```python
sawtooth_series = fourier_series(x / pi, (x, -pi, pi))
sawtooth_series
```

```python
plot(sawtooth_series.truncate(50))
```

See https://en.wikipedia.org/wiki/Gibbs_phenomenon for why the fourier series has peculiar behavior near jump discontinuties.

Just like formal power series we can index fourier series as well.

```python
sawtooth_series[51]
```

It is easy to shift and scale the series using ``shift`` and ``scale`` methods.

```python
sawtooth_series.shift(10).truncate(5)
```

```python
sawtooth_series.scale(10).truncate(5)
```

# Exercise

Consider a square wave defined over the range of (0, 1) as:
   1. $$ f(t) = 1 \in (0, 1/2] $$
   2. $$ f(t) = -1 \in (1/2, 1) $$
   3. $$ f(t + 1) = f(t) \in (-\infty, \infty) $$

Try computing the fourier series of the above defined function. Also, plot the computed fourier series.

```python
square_wave = Piecewise(?)
```

```python
square_series = fourier_series(?)
square_series
```

```python
plot(?)
```

# What next?

Try some basic operations like addition, subtraction, etc on formal power series, fourier series and see what happens.

tutorial_exercises/Advanced - Special Series.ipynb


# Linear Equations
The equations in the previous lab included one variable, for which you solved the equation to find its value. Now let's look at equations with multiple variables. For reasons that will become apparent, equations with two variables are known as linear equations.

## Solving a Linear Equation
Consider the following equation:

\begin{equation}2y + 3 = 3x - 1 \end{equation}

This equation includes two different variables, **x** and **y**. These variables depend on one another; the value of x is determined in part by the value of y and vice-versa; so we can't solve the equation and find absolute values for both x and y. However, we *can* solve the equation for one of the variables and obtain a result that describes a relative relationship between the variables.

For example, let's solve this equation for y. First, we'll get rid of the constant on the right by adding 1 to both sides:

\begin{equation}2y + 4 = 3x \end{equation}

Then we'll use the same technique to move the constant on the left to the right to isolate the y term by subtracting 4 from both sides:

\begin{equation}2y = 3x - 4 \end{equation}

Now we can deal with the coefficient for y by dividing both sides by 2:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Our equation is now solved. We've isolated **y** and defined it as 3x-4/2

While we can't express **y** as a particular value, we can calculate it for any value of **x**. For example, if **x** has a value of 6, then **y** can be calculated as:

\begin{equation}y = \frac{3\cdot6 - 4}{2} \end{equation}

This gives the result 14/2 which can be simplified to 7.

You can view the values of **y** for a range of **x** values by applying the equation to them using the following Python code:

```python
import pandas as pd

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Add a y column by applying the solved equation to x
df['y'] = (3*df['x'] - 4) / 2

#Display the dataframe
df
```

      x
      y

      0
      -10
      -17.0

      1
      -9
      -15.5

      2
      -8
      -14.0

      3
      -7
      -12.5

      4
      -6
      -11.0

      5
      -5
      -9.5

      6
      -4
      -8.0

      7
      -3
      -6.5

      8
      -2
      -5.0

      9
      -1
      -3.5

      10
      0
      -2.0

      11
      1
      -0.5

      12
      2
      1.0

      13
      3
      2.5

      14
      4
      4.0

      15
      5
      5.5

      16
      6
      7.0

      17
      7
      8.5

      18
      8
      10.0

      19
      9
      11.5

      20
      10
      13.0

We can also plot these values to visualize the relationship between x and y as a line. For this reason, equations that describe a relative relationship between two variables are known as *linear equations*:

```python
%matplotlib inline
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey", marker = "o")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.show()
```

In a linear equation, a valid solution is described by an ordered pair of x and y values. For example, valid solutions to the linear equation above include:
- (-10, -17)
- (0, -2)
- (9, 11.5)

The cool thing about linear equations is that we can plot the points for some specific ordered pair solutions to create the line, and then interpolate the x value for any y value (or vice-versa) along the line.

## Intercepts
When we use a linear equation to plot a line, we can easily see where the line intersects the X and Y axes of the plot. These points are known as *intercepts*. The *x-intercept* is where the line intersects the X (horizontal) axis, and the *y-intercept* is where the line intersects the Y (horizontal) axis.

Let's take a look at the line from our linear equation with the X and Y axis shown through the origin (0,0).

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.show()
```

The x-intercept is the point where the line crosses the X axis, and at this point, the **y** value is always 0. Similarly, the y-intercept is where the line crosses the Y axis, at which point the **x** value is 0. So to find the intercepts, we need to solve the equation for **x** when **y** is 0.

For the x-intercept, our equation looks like this:

\begin{equation}0 = \frac{3x - 4}{2} \end{equation}

Which can be reversed to make it look more familar with the x expression on the left:

\begin{equation}\frac{3x - 4}{2} = 0 \end{equation}

We can multiply both sides by 2 to get rid of the fraction:

\begin{equation}3x - 4 = 0 \end{equation}

Then we can add 4 to both sides to get rid of the constant on the left:

\begin{equation}3x = 4 \end{equation}

And finally we can divide both sides by 3 to get the value for x:

\begin{equation}x = \frac{4}{3} \end{equation}

Which simplifies to:

\begin{equation}x = 1\frac{1}{3} \end{equation}

So the x-intercept is 11/3 (approximately 1.333).

To get the y-intercept, we solve the equation for y when x is 0:

\begin{equation}y = \frac{3\cdot0 - 4}{2} \end{equation}

Since 3 x 0 is 0, this can be simplified to:

\begin{equation}y = \frac{-4}{2} \end{equation}

-4 divided by 2 is -2, so:

\begin{equation}y = -2 \end{equation}

This gives us our y-intercept, so we can plot both intercepts on the graph:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()

## add axis lines for 0,0
plt.axhline()
plt.axvline()
plt.annotate('x-intercept',(1.333, 0))
plt.annotate('y-intercept',(0,-2))
plt.show()
```

The ability to calculate the intercepts for a linear equation is useful, because you can calculate only these two points and then draw a straight line through them to create the entire line for the equation.

## Slope
It's clear from the graph that the line from our linear equation describes a slope in which values increase as we travel up and to the right along the line. It can be useful to quantify the slope in terms of how much **x** increases (or decreases) for a given change in **y**. In the notation for this, we use the greek letter Δ (*delta*) to represent change:

\begin{equation}slope = \frac{\Delta{y}}{\Delta{x}} \end{equation}

Sometimes slope is represented by the variable ***m***, and the equation is written as:

\begin{equation}m = \frac{y_{2} - y_{1}}{x_{2} - x_{1}} \end{equation}

Although this form of the equation is a little more verbose, it gives us a clue as to how we calculate slope. What we need is any two ordered pairs of x,y values for the line - for example, we know that our line passes through the following two points:
- (0,-2)
- (6,7)

We can take the x and y values from the first pair, and label them x1 and y1; and then take the x and y values from the second point and label them x2 and y2. Then we can plug those into our slope equation:

\begin{equation}m = \frac{7 - -2}{6 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{7 + 2}{6 - 0} \end{equation}

That gives us the result 9/6 which is 11/2 or 1.5 .

So what does that actually mean? Well, it tells us that for every change of **1** in x, **y** changes by 11/2 or 1.5. So if we start from any point on the line and move one unit to the right (along the X axis), we'll need to move 1.5 units up (along the Y axis) to get back to the line.

You can plot the slope onto the original line with the following Python code to verify it fits:

```python
plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# set the slope
m = 1.5

# get the y-intercept
yInt = -2

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

### Slope-Intercept Form
One of the great things about algebraic expressions is that you can write the same equation in multiple ways, or *forms*. The *slope-intercept form* is a specific way of writing a 2-variable linear equation so that the equation definition includes the slope and y-intercept. The generalised slope-intercept form looks like this:

\begin{equation}y = mx + b \end{equation}

In this notation, ***m*** is the slope and ***b*** is the y-intercept.

For example, let's look at the solved linear equation we've been working with so far in this section:

\begin{equation}y = \frac{3x - 4}{2} \end{equation}

Now that we know the slope and y-intercept for the line that this equation defines, we can rewrite the equation as:

\begin{equation}y = 1\frac{1}{2}x + -2 \end{equation}

You can see intuitively that this is true. In our original form of the equation, to find y we multiply x by three, subtract 4, and divide by two - in other words, x is half of 3x - 4; which is 1.5x - 2. So these equations are equivalent, but the slope-intercept form has the advantages of being simpler, and including two key pieces of information we need to plot the line represented by the equation. We know the y-intecept that the line passes through (0, -2), and we know the slope of the line (for every x, we add 1.5 to y.

Let's recreate our set of test x and y values using the slope-intercept form of the equation, and plot them to prove that this  describes the same line:

```python
%matplotlib inline

import pandas as pd
from matplotlib import pyplot as plt

# Create a dataframe with an x column containing values from -10 to 10
df = pd.DataFrame ({'x': range(-10, 11)})

# Define slope and y-intercept
m = 1.5
yInt = -2

# Add a y column by applying the slope-intercept equation to x
df['y'] = m*df['x'] + yInt

# Plot the line
from matplotlib import pyplot as plt

plt.plot(df.x, df.y, color="grey")
plt.xlabel('x')
plt.ylabel('y')
plt.grid()
plt.axhline()
plt.axvline()

# label the y-intercept
plt.annotate('y-intercept',(0,yInt))

# plot the slope from the y-intercept for 1x
mx = [0, 1]
my = [yInt, yInt + m]
plt.plot(mx,my, color='red', lw=5)

plt.show()
```

```python

```

MathsToML/Module01-Equations, Graphs, and Functions/01-02-Linear Equations.ipynb


## Product Rule for Random Variables

We introduced inference in the context of random variables, where there was a simple way to visualize what was going on in terms of joint probability tables. Marginalization referred to summing out rows or columns. Conditioning referred to taking a slice of the table and renormalizing so entries within that slice summed to 1. We then saw a more general story in terms of events. In fact, we saw that for many inference problems, using random variables to solve the problem is not necessary – reasoning with events was enough! A powerful tool we saw was Bayes' theorem.

We now return to random variables and build up to Bayes' theorem for random variables. This machinery will be extremely important as it will be how we automate inference for much larger problems in the later sections of the course, where we can have a large number of random variables at play, and a large amount of observations that we need to incorporate into our inference.

## Product Rule for Random Variables

We know that product rule for event is
$$\mathbb {P}(\mathcal{A} \cap \mathcal{B}) = \mathbb {P}(\mathcal{A}) \mathbb {P}(\mathcal{A} | \mathcal{B})$$
Similarly the product rule for random varaible will be given by follwing formula if $p_Y(y) \neq 0$.
$$\begin{align}
    p_{x|y}(x|y) &= \frac{p_{X,Y}(x,y)}{p_Y(y)} \\
                 &\Downarrow      \\
    p_{X,Y}(x,y) &= p_Y(y) \, p_{x|y}(x|y)
\end{align}$$

<!-- Note: $p_Y(y) = 0 \Rightarrow p_{x|y}(x|y)=0$ -->

In general the formula for joint probabiliy distribution is given by

$$  p_{X,Y}(x,y) = \begin{cases}
       p_Y(y) \, p_{x|y}(x|y) & \mbox{if } p_Y(y) > 0 \\
       0                      & \mbox{if } p_Y(y) = 0
       \end{cases}
$$

### More than 2 random variable

Suppose we have have three random variable then we can think any two as one random variable, for example trat last two as one random variable we get

$$\begin{align}p_{X_1, X_2, X_3}(x_1, x_2, x_3)
&= p_{X_1}(x_1)p_{X_2, X_3|X_1}(x_2, x_3|x_1) \\
&= p_{X_1}(x_1)p_{X2|X_1}(x_2|x_1)p_{X_3|X_1,X_2}(x_3|x_1,x_2)
\end{align}$$

We can genrealize the formula as follows,

$$\begin{align}p_{X_1, X_2,\ldots, X_N}(x_1, x_2,\ldots,x_N)
&= p_{X_1}(x_1) p_{X_2,\ldots, X_N|X_1}(x_2,\ldots, x_N|x_1) \\
&= p_{X_1}(x_1) p_{X_2|X_1}(x_2|x_1) p_{X_3,\ldots,X_N|X_1, X_2}(x_3,\ldots,x_N|x_1,x_2) \\
&= p_{X_1}(x_1) p_{X_2|X_1}(x_2|x_1) \cdots p_{X_N|X_1, \ldots, X_{N-1}}(x_n|x_1,\ldots,x_{N-1})
\end{align}$$

### Exercise: The Product Rule for Random Variables - Medical Diagnosis Revisited

Let's revisit the medical diagnosis problem we saw earlier. We now use random variables to construct a joint probability table.

Let random variable $X$ represent the patient's condition — whether “healthy" or “infected", with the following distribution for $X$:

Meanwhile, the test outcome $Y$ for whether the patient is infected is either “positive" (for the disease) or “negative". As before, the test is $99\%$ accurate, which means that the conditional probability table for $Y$ given $X$ is as follows (note that we also show how to write things out as a single table):

Using the product rule for random variables, what are the four entries for the joint probability table? Please provide the exact answer for these four quantities.

$p_{X,Y}(\text {healthy}, \text {positive}) = p_X(\text {healthy})~p_{Y|X}(\text {positive}~|~\text {healthy})$

```python
prior_or_p_X = {
    "healthy" : 0.999,
    "infected": 0.001
}

p_Y_given_X = {
    ('positive', 'healthy' ): 0.01,
    ('positive', 'infected'): 0.99,
    ('negative', 'healthy' ): 0.99,
    ('negative', 'infected'): 0.01
}

# p_X_Y stores the joint probability dist. of X and Y
p_X_Y = {}
for key, values in p_Y_given_X.items():
    p_X_Y[key[::-1]] = values * prior_or_p_X[key[1]]

p_X_Y
```

    {('healthy', 'negative'): 0.98901,
     ('healthy', 'positive'): 0.00999,
     ('infected', 'negative'): 1e-05,
     ('infected', 'positive'): 0.00099}

```python
print("{0:.5f}".format(p_X_Y[('healthy', 'positive')]))
```

    0.00999

$p_{X,Y}(\text {healthy}, \text {negative}) = $

```python
print("{0:.5f}".format(p_X_Y[('healthy', 'negative')]))
```

    0.98901

$p_{X,Y}(\text {infected}, \text {positive}) =$

```python
print("{0:.5f}".format(p_X_Y[('infected', 'positive')]))
```

    0.00099

$p_{X,Y}(\text {infected}, \text {negative}) =$

```python
print("{0:.5f}".format(p_X_Y[('infected', 'negative')]))
```

    0.00001

## Baye's Rule for Random Variable

In inference, what we want to reason about is some unknown random variable $X$, where we get to observe some other random variable $Y$, and we have some model for how $X$ and $Y$ relate. Specifically, suppose that we have some “prior" distribution $p_X$ for $X$; this prior distribution encodes what we believe to be likely or unlikely values that $X$ takes on, before we actually have any observations. We also suppose we have a “likelihood" distribution $p_{Y∣X}$.

After observing that $Y$ takes on a specific value $y$, our “belief" of what $X$ given $Y=y$ is now given by what's called the “posterior" distribution $p_{X∣Y}(⋅∣y)$. Put another way, we keep track of a probability distribution that tells us how plausible we think different values $X$ can take on are. When we observe data $Y$ that can help us reason about $X$, we proceed to either upweight or downweight how plausible we think different values $X$ can take on are, making sure that we end up with a probability distribution giving us our updated belief of what $X$ can be.

Thus, once we have observed $Y=y$, our belief of what $X$ is changes from the prior $p_X$ to the posterior $p_{X∣Y}(⋅∣y)$.

Bayes' theorem (also called Bayes' rule or Bayes' law) for random variables explicitly tells us how to compute the posterior distribution $p_{X∣Y}(⋅∣y)$, i.e., how to weight each possible value that random variable $X$ can take on, once we've observed $Y=y$. Bayes' theorem is the main workhorse of numerous inference algorithms and will show up many times throughout the course.

**Bayes' theorem:** Suppose that $y$ is a value that random variable $Y$ can take on, and $p_Y(y)>0$. Then

$$p_{X\mid Y}(x\mid y)=\frac{p_{X}(x)p_{Y\mid X}(y\mid x)}{\sum _{ x'}p_{X}( x')p_{Y\mid X}(y\mid x')}$$

for all values $x$ that random variable $X$ can take on.

!!!important
    Remember that $p_{Y∣X}(⋅∣x)$ could be undefined but this isn't an issue since this happens precisely when     $p_X(x)=0$, and we know that $p_{X,Y}(x,y)=0$ (for every $y$) whenever $p_X(x)=0$.

    Proof: We have

    $$p_{X\mid Y}(x\mid y)\overset {(a)}{=}\frac{p_{X,Y}(x,y)}{p_{Y}(y)}\overset {(b)}{=}\frac{p_{X}(x)p_{Y\mid X}  (y\mid x)}{p_{Y}(y)}\overset {(c)}{=}\frac{p_{X}(x)p_{Y\mid X}(y\mid x)}{\sum _{ x'}p_{X,Y}( x',y)}\overset {(d)}  {=}\frac{p_{X}(x)p_{Y\mid X}(y\mid x)}{\sum _{ x'}p_{X}( x')p_{Y\mid X}(y\mid x')},$$

    where step (a) uses the definition of conditional probability (this step requires $p_Y(y)>0$, step (b) uses the  product rule (recall that for notational convenience we're not separately writing out the case when $p_X(x)=0$, step (c) uses the formula for marginalization, and step (d) uses the product rule (again, for notational convenience, we're not separately writing out the case when $p_X(x′)=0$.

## BAYES' THEOREM FOR RANDOM VARIABLES: A COMPUTATIONAL VIEW

Computationally, Bayes' theorem can be thought of as a two-step procedure. Once we have observed $Y=y$:

1. For each value $x$ that random variable $X$ can take on, initially we believed that $X=x$ with a score of $p_X(x)$, which could be thought of as how plausible we thought ahead of time that $X=x$. However now that we have observed $Y=y$, we weight the score $p_X(x)$ by a factor $p_{Y∣X}(y∣x)$, so

    $$\text {new belief for how plausible }X=x\text { is:}\quad \alpha (x\mid y)\triangleq p_{X}(x)p_{Y\mid X}(y\mid x),$$

    where we have defined a new table $α(⋅∣y)$ which is not a probability table, since when we put in the weights, the new beliefs are no longer guaranteed to sum to $1$ (i.e., $\sum _{x}\alpha (x\mid y)$ might not equal $1$)! $α(⋅∣y)$ is an unnormalized posterior distribution!

    Also, if $p_X(x)$ is already $0$, then as we already mentioned a few times, $p_{Y∣X}(y∣x)$ is undefined, but this case isn't a problem: no weighting is needed since an impossible outcome stays impossible.

    To make things concrete, here is an example from the medical diagnosis problem where we observe $Y = \text {positive}$:

2. We fix the fact that the unnormalized posterior table $α(⋅∣y)$ isn't guaranteed to sum to $1$ by renormalizing:

    $$p_{X\mid Y}(x\mid y)=\frac{\alpha (x\mid y)}{\sum _{ x'}\alpha ( x'\mid y)}=\frac{p_{X}(x)p_{Y\mid X}(y\mid x)}{\sum _{ x'}p_{X}( x')p_{Y\mid X}(y\mid x')}.$$

!!! Note
    Some times we won't actually care about doing this second renormalization step because we will only be   interested in what value that $X$ takes on is more plausible relative to others; while we could always do the renormalization, if we just want to see which value of $x$ yields the highest entry in the unnormalized table $α(⋅∣y)$, we could find this value of x without renormalizing!

### MAXIMUM A POSTERIORI (MAP) ESTIMATION

For a hidden random variable $X$ that we are inferring, and given observation $Y=y$, we have been talking about computing the posterior distribution $p_{X∣Y}(⋅|y)$ using Bayes' rule. ``The posterior is a distribution for what we are inferring``. Often times, we want to report which particular value of $X$ actually achieves the highest posterior probability, i.e., the most probable value $x$ that $X$ can take on given that we have observed $Y=y$.

The value that $X$ can take on that maximizes the posterior distribution is called the maximum a posteriori (MAP) estimate of $X$ given $Y=y$. We denote the MAP estimate by $\widehat{x}_{\text {MAP}}(y)$, where we make it clear that it depends on what the observed $y$ is. Mathematically, we write

$$\widehat{x}_{\text {MAP}}(y) = \arg \max _ x p_{X \mid Y}(x | y).$$

Note that if we didn't include the “arg" before the “max", then we would just be finding the highest posterior probability rather than which value–or “argument"–x actually achieves the highest posterior probability.

In general, there could be ties, i.e., multiple values that $X$ can take on are able to achieve the best possible posterior probability.

### Exercise: Bayes' Theorem for Random Variables - Medical Diagnosis, Continued

Recall the medical diagnosis setup from before, summarized in these tables:

Recall that Bayes' theorem is given by

$$p_{X\mid Y}(x\mid y)=\frac{p_{X}(x)p_{Y\mid X}(y\mid x)}{\sum _{ x'}p_{X}( x')p_{Y\mid X}(y\mid x')}$$

for all values $x$ that random variable $X$ can take on.

Use Bayes' theorem to compute the following probabilities: (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

```python
prior_or_p_X = {
    "healthy" : 0.999,
    "infected": 0.001
}

p_Y_given_X = {
    ('positive', 'healthy' ): 0.01,
    ('positive', 'infected'): 0.99,
    ('negative', 'healthy' ): 0.99,
    ('negative', 'infected'): 0.01
}

# p_X_Y stores the joint probabilty distribution of X and Y
p_X_Y = {}
for key, values in p_Y_given_X.items():
    p_X_Y[key[::-1]] = values * prior_or_p_X[key[1]]

# p_Y stores the marginal probabilty distribution Y
p_Y = {}
for key, values in p_X_Y.items():
    if key[1] in p_Y:
        p_Y[key[1]] += values

    else:
        p_Y[key[1]] = values

# p_X_given_Y stores the conditional probability dist. of X given Y.
p_X_given_Y = {}
for key, values in p_X_Y.items():
    p_X_given_Y[key] = values / p_Y[key[1]]

p_X_given_Y
```

    {('healthy', 'negative'): 0.9999898889810116,
     ('healthy', 'positive'): 0.9098360655737705,
     ('infected', 'negative'): 1.0111018988493663e-05,
     ('infected', 'positive'): 0.09016393442622951}

$p_{X\mid Y}(\text {healthy}\mid \text {positive}) = $

```python
print("{0:.5f}".format(p_X_given_Y [('healthy', 'positive')]))
```

    0.90984

$p_{X\mid Y}(\text {healthy}\mid \text {negative}) =$

```python
print("{0:.5f}".format(p_X_given_Y [('healthy', 'negative')]))
```

    0.99999

What is the MAP estimate for $X$ given $Y = \text{positive}$?

```python
comp = 0
MAP = ""
for key, val in p_X_given_Y.items():
    if 'positive' in key and comp < val:
        comp = val
        MAP = key[0]

print(MAP)
```

    healthy

What is the MAP estimate for $X$ given $Y=\text{negative}$?

```python
comp = 0
MAP = ""
for key, val in p_X_given_Y.items():
    if 'negative' in key and comp < val:
        comp = val
        MAP = key[0]

print(MAP)
```

    healthy

### Exercise: Complexity of Computing Bayes' Theorem for Random Variables

This exercise is extremely important and gets at how expensive it is to compute a posterior distribution when we have many quantities we want to infer.

Consider when we have $N$ random variables $X_1, \dots , X_ N$ with joint probability distribution $p_{X_1, \dots , X_ N}$, and where we have an observation $Y$ related to $X_1, \dots , X_ N$ through the known conditional probability table $p_{Y\mid X_1, \dots , X_ N}$. Treating $X=(X_1, \dots , X_ N)$ as one big random variable, we can apply Bayes' theorem to get

$$\begin{eqnarray}
&& p_{X_1, X_2, \dots, X_N \mid Y}(x_1, x_2, \dots, x_N \mid y) \\
&&
= \frac{p_{X_1, X_2, \dots, X_N}(x_1, x_2, \dots, x_N)
        p_{Y\mid X_1, X_2, \dots, X_N}(y\mid x_1, x_2, \dots, x_N)}
       {\sum_{x_1'}
        \sum_{x_2'}
        \cdots
        \sum_{x_N'}
          p_{X}(x_1',
                x_2',
                \dots,
                x_N')
          p_{Y\mid X_1, X_2, \dots, X_N}(y\mid x_1',
                x_2',
                \dots,
                x_N')}.
\end{eqnarray}$$

Suppose each $X_i$ takes on one of $k$ values. In the denominator, how many terms are we summing together? Express your answer in terms of $k$ and $N$.

In this part, please provide your answer as a mathematical formula (and not as Python code). Use "$\hat{}$" for exponentiation, e.g., $x\hat{}2$ to denotes $x^2$. Explicitly include multiplication using $*$, e.g. $x*y$ is $xy$.

**Answer:** Here we have $k$ choices of $X_1$ and $k$ choices of $X_2$ and so on. Hence total number of terms will be $k^N$.

```python

```

week03/01 Product Rule for Random Variables.ipynb