# SciPy

_Numpy_ provides a high-performance multidimensional array and basic tools to compute with and manipulate these arrays. **SciPy** builds on this, and provides a large number of functions that operate on numpy arrays and are useful for different types of scientific and engineering applications.

The best way to get familiar with SciPy is to browse the documentation (found at [https://docs.scipy.org/doc/scipy-1.1.0/reference/tutorial/index.html]). We will highlight some parts of SciPy that you might find useful for this class.

SciPy is a collection of mathematical algorithms and convenience functions built on the Numpy extension of Python. It adds significant power to the interactive Python session by providing the user with high-level commands and classes for manipulating and visualizing data. With SciPy an interactive Python session becomes a data-processing and system-prototyping environment rivaling systems such as MATLAB, IDL, Octave, R-Lab, and SciLab.

The additional benefit of basing SciPy on Python is that this also makes a powerful programming language available for use in developing sophisticated programs and specialized applications. Scientific applications using SciPy benefit from the development of additional modules in numerous niches of the software landscape by developers across the world. Everything from parallel programming to web and data-base subroutines and classes have been made available to the Python programmer. All of this power is available in addition to the mathematical libraries in SciPy.

This tutorial will acquaint the first-time user of SciPy with some of its most important features. It assumes that the user has already installed the SciPy package. Some general Python facility is also assumed, such as could be acquired by working through the Python distribution’s Tutorial. For further introductory help the user is directed to the Numpy documentation.

For brevity and convenience, we will often assume that the main packages (numpy, scipy, and matplotlib) have been imported as:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## SciPy Organization

SciPy is organized into subpackages covering different scientific computing domains. These are summarized in the following table:

| Subpackage | Description |
| ------ | ---- |
| `cluster` | Clustering algorithms |
| `constants` | Physical and mathematical constants |
| `fftpack` | Fast Fourier Transform routines |
| `integrate` | Integration and ordinary differential equation solvers |
| `interpolate` | Interpolation and smoothing splines |
| `io` | Input and Output |
| `linalg` | Linear algebra |
| `ndimage` | N-dimensional image processing |
| `odr` | Orthogonal distance regression |
| `optimize` | Optimization and root-finding routines |
| `signal` | Signal processing |
| `sparse` | Sparse matrices and associated routines |
| `spatial` | Spatial data structures and algorithms |
| `special` | Special functions |
| `stats` | Statistical distributions and functions |

We will barely scratch the surface in terms of the huge expanse of libraries that **SciPy** offers, but it is recommended that each SciPy sub-package is imported separately, for example:

```python
from scipy import linalg, optimize
```

## Integration

The `scipy.integrate` sub-package provides several integration techniques including an ordinary differential equation integrator. An overview of the module is provided by the `help` command:

```python
from scipy import integrate
```

## General Integration (quad)

The function quad is provided to integrate a function of one variable between two points. The points can be ($\pm \infty$) to indicate infinite limits. For example, let's say you wish to integrate:

$$
I=\int_0^{\frac{\pi}{2}}cos(x)dx
$$

This can be trivially computed using `integrate.quad()`:

```python
result = integrate.quad(lambda x: np.cos(x), 0, np.pi/2)
result
```

The first value represents the *integral*, as we would expect it is extremely close to $1$. The second value represents the *absolute error* estimate within the result, as SciPy computes the integral **numerically**.

If the function to integrate takes *additional parameters*, this can be provided for in the **args** argument. These parameters must be considered *constants*. Suppose that the following integral shall be calculated:

$$
I(a,b)=\int_0^1 ax^2 + b dx
$$

This is implemented as follows:

```python
def integrand(x, a, b):
    return a*x**2 + b

a = 2
b = 1
I = integrate.quad(integrand, 0, 1, args=(a,b))
I
```

## General multiple integration

The mechanics for double and triple integration have been wrapped up into the functions `dblquad` and `tplquad`. These functions take the function to integrate and four, or six arguments, respectively. The limits of all inner integrals need to be defined as functions.

An example of using double integration to compute several values of $I_n$ is shown below:

```python
def I(n):
    return integrate.dblquad(lambda t, x: np.exp(-x*t)/t**n, 0, np.inf, lambda x: 1, lambda x: np.inf)

print(I(2))
print(I(3))
print(I(4))
```

## Integration using samples

If we are working with data samples across some space, we can approximate an integral of both equally-spaced and arbitrarily-spaced samples using a variety of different methods. Two of the most common are `trapz` and `simps`:

```python
x = np.array([1,3,4])
y = x**2
integrate.simps(y, x)
```

This corresponds exactly to:

$$
\int_1^4 x^2 dx=21
$$

whereas integrating the following:

```python
y2 = x**3
integrate.simps(y2, x)
```

Doesn't correspond to:

$$
\int_1^4 x^3 dx = 63.75
$$

This is because Simpson's rule approximates the function between adjacent point as a parabola, as long as the function is a polynomial of order 2 or less with unequal spacing. Simpson's rule is more accurate than `trapz`, but `trapz` is considerably more reliable, as it interpolates *linearly* by integrating in small trapezoid parts along the sample space.

## Ordinary differential equations (ODEs)

Integrating a set of ordinary differential equations (ODEs) given initial conditions is another useful example. The function `odeint` is available in SciPy for integrating a first-order vector differential equation:

$$
\frac{d\dot{y}}{dt}=f(\dot{y},t)
$$

given initial conditions $\dot{y}(0)=y_0$, where $\dot{y}$ is a length $N$ vector and $f$ is a mapping from $\mathcal{R}^N$ to $\mathcal{R}^N$. A higher-order ordinary differential equation can always be reduced to a differential equation of this type by introducing intermediate derivatives into the $\dot{y}$ vector.

### Example

The second order differential equation for the angle theta of a pendulum acted on by gravity with friction can be written:

$$
\theta''(t) + b \theta'(t) + c \sin(\theta(t)) = 0
$$

where $b$ and $c$ are care positive constants, and a prime $'$ denotes a derivative. To solve this equation with `odeint`, we first convert it to a system of first-order equations. By defining angular velocity $\omega(t)=\theta'(t)$, we obtain the system:

$$
\begin{equation}
\theta'(t)=\omega(t) \\
\omega'(t)=-b \omega(t) - c \sin(\theta(t))
\end{equation}
$$

Let $y$ be the vector $[\theta, \omega]$. We implement this system in Python as:

```python
def pend(y, t, b, c):
    theta, omega = y
    dydt = [omega, -b*omega - c*np.sin(theta)]
    return dydt
```

We assume for the initial conditions, the pendulum is nearly vertical with $\theta(0)=\pi - 0.1$, and is initially at rest, so $\omega(0)=0$. Then the vector of initial conditions, with constants $b=0.25$ and $c=5.0$, is:

```python
b = 0.25
c = 5.0
y0 = [np.pi - 0.1, 0.0]
```

Now we generate a solution over a uniform-space sample set in the interval $t \in [0, 10]$:

```python
t = np.linspace(0, 10, 101)
```

Calling `odeint` to generate the solution. We pass $b$ and $c$ to `odeint` using the *args* argument:

```python
sol = integrate.odeint(pend, y0, t, args=(b,c))
```

In our solution, we have a $[101,2]$ array, whereby the first column is $\theta(t)$ and the second is $\omega(t)$. We plot as:

```python
plt.plot(t, sol[:,0], label=r"$\theta(t)$")
plt.plot(t, sol[:,1], label=r"$\omega(t)$")
plt.xlabel(r"$t$")
plt.legend()
plt.show()
```

## Interpolation

There are several general interpolation facilities available in SciPy, for data in 1, 2, and higher dimensions.

The `interp1d` class in `scipy.interpolate` is a convenient method to create a function based on fixed data points which can be evaluated anywhere within the domain defined by the given data using linear interpolation.

```python
from scipy import interpolate
```

```python
x = np.linspace(0, 10, 11, endpoint=True)
y = np.cos(-x**2/9.)
f = interpolate.interp1d(x, y)
f2 = interpolate.interp1d(x, y, kind="nearest")
f3 = interpolate.interp1d(x, y, kind="cubic")
f4 = interpolate.interp1d(x, y, kind="next")

xnew = np.linspace(0, 10, 71, endpoint=True)

fig,ax=plt.subplots(ncols=4, figsize=(15,4))
ax[0].plot(x, y, 'o', xnew, f(xnew), 'r-', label="linear")
ax[1].plot(x, y, 'o', xnew, f2(xnew), 'g-', label="nearest")
ax[2].plot(x, y, 'o', xnew, f3(xnew), 'b-', label="cubic")
ax[3].plot(x, y, 'o', xnew, f4(xnew), 'k-', label="next")
for a in ax:
    a.legend()
plt.show()
```

### Multivariate data interpolation

Suppose you have multidimensional data, for instance for an underlying function $f(x, y)$ you only know the values at points ($x[i]$, $y[i]$) that do not form a regular grid.

Suppose we want to interpolate the 2-D function:

```python
def func(x, y):
    return x*(1-x)*np.cos(4*np.pi*x) * np.sin(4*np.pi*y**2)**2

grid_x, grid_y = np.mgrid[0:1:100j, 0:1:200j]
```

but we only know its values at 1000 data points:

```python
points = np.random.rand(1000, 2)
values = func(points[:,0], points[:,1])
```

This can be done with `griddata` – below we try out all of the interpolation methods:

```python
grid_z0 = interpolate.griddata(points, values, (grid_x, grid_y), method="nearest")
grid_z1 = interpolate.griddata(points, values, (grid_x, grid_y), method="linear")
grid_z2 = interpolate.griddata(points, values, (grid_x, grid_y), method="cubic")

fig, ax = plt.subplots(ncols=3, figsize=(15,4))

for i,p in enumerate([grid_z0, grid_z1, grid_z2]):
    ax[i].imshow(p)
for i,c in enumerate(["nearest","linear","cubic"]):
    ax[i].set_title(c)
    ax[i].axis("off")
```

### Spline interpolation

Spline interpolation requires two essential steps: (1) a spline representation of the curve is computed, and (2) the spline is evaluated at the desired points. In order to find the spline representation, there are two different ways to represent a curve and obtain (smoothing) spline coefficients: directly and parametrically. The direct method finds the spline representation of a curve in a two- dimensional plane using the function `splrep`:

```python
x = np.arange(0, 2*np.pi+np.pi/4, 2*np.pi/8)
y = np.sin(x)
tck = interpolate.splrep(x, y, s = 0)
tck
```

The keyword argument, s , is used to specify the amount of smoothing to perform during the spline fit. The default value of $s$ is $s=m-\sqrt{2m}$ where $m$ is the number of data points being fit. Thus if no smoothing is desired $s=0$.

Once the spline representation of the data has been determined, functions are available for evaluating the spline (`splev`) and its derivatives (`splev`, `spalde`) at any point and the integral of the spline between any two points ( `splint`):

```python
xnew = np.arange(0, 2*np.pi, np.pi/50)
ynew = interpolate.splev(xnew, tck, der=0)

plt.plot(x, y, 'x', xnew, ynew, xnew, np.sin(xnew), x, y, 'b')
plt.legend(["Linear","Cubic","True"])
```

## Multidimensional image processing

Image processing and analysis are generally seen as operations on two-dimensional arrays of values. There are however a number of fields where images of higher dimensionality must be analyzed. Good examples of these are **medical imaging** and **biological imaging**. `numpy` is suited very well for this type of applications due its inherent multidimensional nature. The `scipy.ndimage` packages provides a number of general image processing and analysis functions that are designed to operate with arrays of arbitrary dimensionality. The packages currently includes functions for linear and non-linear filtering, binary morphology, B-spline interpolation, and object measurements.

To access this functionality, we import the `ndimage` package:

```python
from scipy import ndimage
```

### Importing images from file

Creating a numpy array from an image file:

```python
fig,ax=plt.subplots(ncols=3, figsize=(15,5))

fly = plt.imread("butterfly.jpg")
print(fly.shape, fly.dtype)
ax[0].imshow(fly)
for a in ax:
    a.axis("off")
# different interpolations
ax[1].imshow(fly, interpolation="bilinear")
ax[2].imshow(fly, interpolation="nearest")
plt.show()
```

### Basic Manipulations

Including **masking** and **rotation**:

```python
# create a copy to manipulate
porthole_fly = fly.copy()

lx, ly, lz = fly.shape
X, Y = np.ogrid[0:lx, 0:ly]
mask = (X - lx / 2) **2 + (Y - ly / 2) **2 > lx * ly / 4

porthole_fly[mask,:] = 0

fig,ax=plt.subplots(ncols=2, figsize=(14,5))

ax[0].imshow(porthole_fly)
ax[0].axis("off")

fly_rot = ndimage.rotate(fly, 45, reshape=False)
ax[1].imshow(fly_rot)
ax[1].axis("off")
plt.show()
```

### Blurring/Smoothing

Note that this has selected only on the *gray* channel:

```python
blurred = ndimage.gaussian_filter(fly, sigma=3)
very_blurred = ndimage.gaussian_filter(fly, sigma=5)
unif_fly = ndimage.uniform_filter(fly, size=11)

fig,ax=plt.subplots(ncols=3, figsize=(15,7))
for a in ax:
    a.axis("off")
ax[0].imshow(blurred)
ax[1].imshow(very_blurred)
ax[2].imshow(unif_fly)
plt.show()
```

### Sharpening

To sharpen an image, we apply a blurring filter and then remove the gaussian filter from the image:

```python
filter_blurred = ndimage.gaussian_filter(blurred, 1)
# select an alpha
alpha = 5
sharpened = blurred + alpha * (blurred - filter_blurred)

fig,ax=plt.subplots(ncols=2, figsize=(15,7))
ax[0].imshow(fly)
ax[1].imshow(sharpened)
plt.axis("off")
plt.show()
```

### Edge Detection

We can use a **gradient operator** (Sobel) to find high intensity variations:

```python
sq = np.zeros((256,256))
sq[64:-64, 64:-64] = 1
sq = ndimage.rotate(sq, 30, mode="constant")
sq = ndimage.gaussian_filter(sq, 8)

sx = ndimage.sobel(sq, axis=0, mode="constant")
sy = ndimage.sobel(sq, axis=1, mode="constant")
sob = np.hypot(sx, sy)

fig, ax=plt.subplots(ncols=3, figsize=(15,4))
for a in ax:
    a.axis("off")
for i,p in enumerate([sx, sy, sob]):
    ax[i].imshow(p, cmap="hot")
```

```python

```

There is substantially more that can be found with processing images, however the scope of this session is just to cover some basic operations to show how things can be done.

## Sparse Matrices

Normal matrices are 2-D objects that store numerical values, and every value is stored in memory in a contiguous chunk. This provides benefits such as very fast access to individual items, but what about when most of the data values are null?

We can use `scipy.sparse` for a selection of different strategies for representing **sparse** data, and it even helps when we have cases where memory grows exponentially.

Sparse matrices act to *compress* the data to save memory usage, by not representing zero values. Applications include:

- solution to partial differential equations (finite elements etc.)
- graph theory (nodes and edges)

Sparsity can be visualised with `matplotlib` using `plt.spy`:

```python
X_sp = np.random.choice([0, 1], size=(200,200), p=[.95, .05])
plt.spy(X_sp, cmap="Blues")
plt.show()
```

Sparse matrices offer the data structure to store large, sparse matrices, and allows us to perform complex matrix computations. The ability to do such computations is incredibly powerful in a variety of data science problems. Learning to work with Sparse matrix, a large matrix or 2d-array with a lot elements being zero, can be extremely handy.

Python’s SciPy library has a lot of options for creating, storing, and operating with Sparse matrices. There are 7 different types of sparse matrices available.

1. __csc_matrix__: Compressed Sparse Column format
1. __csr_matrix__: Compressed Sparse Row format
1. __bsr_matrix__: Block Sparse Row format
1. __lil_matrix__: List of Lists format
1. __dok_matrix__: Dictionary of Keys format
1. __coo_matrix__: COOrdinate format
1. __dia_matrix__: DIAgonal format

The default type is the **csr_matrix**, and NumPy converts your sparse matrix to this format before it conducts arithmetic operations on it. The table below highlights the opportunities of each format:

| format | matrix `*` vector | get item | fancy get | set item | fancy set | solvers | note |
| ------ | ---- | ------ | ---- | ------ | ---- | ------ | ---- |
| DIA | sparsetools | . | . | . | . | iterative | has data array, specialized |
| LIL | via CSR | yes | yes | yes | yes | iterative | arithmetics via CSR, incremental construction |
| DOK | python | yes | one axis only | yes | yes | iterative | O(1) item access, incremental construction |
| COO | sparsetools | . | . | . | . | iterative | has data array, facilitates fast conversion |
| CSR | sparsetools | yes | yes | slow | . | any | has data array, fast row-wise operations |
| CSC | sparsetools | yes | yes | slow | . | any | has data array, fast column-wise operations |
| BSR | sparsetools | . | . | . | . | specialized | has data array, specialized |

```python
from scipy import sparse
```

**WARNING**: When multiplying `scipy.sparse` matrices, it acts as *matrix multiplication* (i.e dot product), not element-wise.

### Example

Here we will create a **lil_matrix**, assign some random numbers, convert to CSR and use `sparse.solve`:

```python
dim_size = 10000
subsets = 1000
A = sparse.lil_matrix((dim_size,dim_size))
A[0, :subsets] = np.random.rand(subsets)
A[1, subsets:subsets*2] = np.random.rand(subsets)
A.setdiag(np.random.rand(dim_size))
```

```python
A = A.tocsr()
b = np.random.rand(dim_size)
# solve using scipy.sparse.linalg
x = sparse.linalg.spsolve(A, b)
# solve non-sparse by converting A back to numpy!
x_ = np.linalg.solve(A.toarray(), b)
# error between methods
err = np.linalg.norm(x - x_)
print(err)
```

### CSC/CSR format

These are the best formats, as they allow for fast matrix-vector products and other arithmetics along the appropriate axis, in addition to efficient row/column slicing.

```python
x = np.random.choice([0,1], size=(5,5), p=[.8, .2])
x
```

```python
sparse.csc_matrix(x)
```

```python
sparse.csr_matrix(x)
```

### Diagonal sparse matrices

This is natural, as a diagonal matrix is by definition mostly sparse, only containing non-zero values on the *diagonal* of the matrix.

```python
sparse.dia_matrix(np.ones((10000,10000)))
```

# Tasks

## Task 1

The force $F$ on an area $A$ at a depth $y$ in a liquid of density $w$ is given by:

$$
F=wyA
$$

Imagine this applied to a plate submerged vertically in a liquid.

The **total force** on the plate is given by:

$$
F=w\int_a^b xy \, dy
$$

where $x$ is the length (in m) of the element of area expressed in terms of $y$, $y$ is depth (in m) of the element of area, $w$ is the density of the liquid (in $Nm^{-3}$), $a$ is the depth at the top of the area (in m), and $b$ is the depth at the bottom of the area (in m).

Calculate the force on one side of a cubical container 10.0cm on an edge if the container is filled with water. The weight density of water is $w=9800Nm^{-3}$.

```python
# your codes here
```

### Task 2

Consider the motion of a spring that is subject to a frictional force or a damping force. An example is the damping force supplied by a shock absorber in a car or a bicycle. We assume that the damping force is proportion to the velocity of the mass and acts in the direction opposite to the motion. Thus:

$$
F_d=-c\frac{dx}{dt}
$$

where $c$ is a damping constant. Newton's second law thus gives:

$$
m\frac{d^2x}{dt^2}=F_r + F_d=-kx-c\frac{dx}{dt}
$$

which we re-arrange to:

$$
m\frac{d^2x}{dt^2}+c\frac{dx}{dt}+kx=0
$$

Solve the linear system of equations using `odeint`, with initial conditions $x(0)=0$ and $x'(0)=0.6$. Ensure that $m$, $c$ and $k$ are all positive constants, but initially test with $m=5$, $c=10$ and $k=128$. Create a timespace as $t \in [0, 10]$ with a sensible number of steps.

Once you done one run, try tweaking $m$ and $c$ and see the different plots you find.

Ensure that you plot both $x(t)$ and $x'(t)$.

```python
# your codes here
```

### Task 3

Import the image `bigcat.jpg`. Compute the laplace transformation from the method `laplace()` found in `ndimage`. Plot the image of the big cat and it's laplace transformation. How well does the image capture the big cat from the scenery?

```python
# your codes here
```

### Task 4

Trying different $\sigma$, draw 9 laplace-transformed cats using 9 different values for in the logspace range of $\sigma \in [10^{-1}, 5]$, within a `gaussian_filter()`. Print the sigma at the top of each image.

```python
# your codes here
```

### Task 5

We can try to label groups of pixels in this image using `ndimage.label`, which accepts a boolean matrix. This boolean matrix which it tries to associate groups to can be generated by a number of ways; in this instance we will simply select points that are greater than the pixel mean across all pixels:

$$
B_{ij}=
\begin{cases}
1 & B_{ij} > \bar{B} \\
0 & B_{ij} \le \bar{B}
\end{cases}
$$

Plot 3 images:
1. The labelled unfiltered image
2. The labelled laplacian-gaussian filtered image
3. The labelled sobel-filtered image

Sobel-filters can be generated using `ndimage.sobel`. You may choose to use a gaussian filter and/or laplace transform before using `sobel()`.

You may use any parameters to `gaussian_filter(sigma)` as necessary to get interesting results.

```python
# your codes here
```

## Solutions

**WARNING**: _Please attempt to solve the problems before fetching the solutions!_

See the solutions to all of the problems here:

```python
%load solutions/02_solutions.py
```


### Logistic Regression From Scratch Using Iris Data Set
Logistic Regression is a classification algorithm that aims at predicting the probability that a certain observation $X_i$ belongs to the class $y=1$.

Here we will first give the formulas that we will implement LR with using Gradient descent algorithm (Following the Likelihood approach).

Generally,we want to find a function that gives the probability of a linear combination of the input variable/features $X$ i.e. $$p=\theta_0+\theta_1x_1+ .... +\theta_nx_n$$. We know that a probability a probaility $0\leq p\leq 1$, from the function $p$ it is evident that the probility might be great than $1$ or even less than $0$.

We can therefore, instead of assignig $p$ to the function we assign the odds i.e $\frac{p}{1-p}$. Such that ,
 $$\frac{p}{1-p}=\theta_0+\theta_1x_1+ .... +\theta_nx_n$$

 We also know that $\theta_0+\theta_1x_1+ .... +\theta_nx_n$ might sometimes be negative, to ensure that then negatives are taken care of we apply $\log$ on $\frac{p}{1-p}$.This leads to,
 $$\log\bigg(\frac{p}{1-p}\bigg)=
 \theta_0+\theta_1x_1+ .... +\theta_nx_n=\theta^TX \quad \text{(In vector notation)}$$

 Equation above ensures that the output is within the range of $0$ and $1$. To simplify things further and solve for $p$, we can exponentiate both sides of the equation above to obtain,
 \begin{align}
 &\frac{p}{1-p}=e^{\theta^TX}\\
 &p=e^{\theta^TX}-pe^{\theta^TX}\\
 &p+pe^{\theta^TX}=e^{\theta^TX}\\
 &p\bigg(1+e^{\theta^TX}\bigg)=e^{\theta^TX}\\
 &p=\frac{e^{\theta^TX}}{1+e^{\theta^TX}}\\
 &p=\frac{1}{1+e^{-\theta^TX}}
 \end{align}
The last equation is the famous sigmoid/logistic function.

Since we have already found the probility $p$, which gives the probability that a certain observation $X_i$ belongs to a certain class, in this case $1$, then the probaility that that the obsercation $X_i$ belongs to the class $0$ is $1-p$. This is because we have only two classes $\{1,0\}$. This is represented as:
$$P(y=1|X;\theta)=p\\
P(y=0|X;\theta)=1-p$$
For such an obsertion $X_i=x$, the probability above can be written in a more compact form as,$$P(y|x)=p^{y}(1-p)^{1-y}$$

And since we have many observations in $X$, the to obtain the probality for all the observations, we multiply the individual probability with each other i.e.
$$P(y|x)=p^{y_1}(1-p)^{1-y_1}\times p^{y_2}(1-p)^{1-y_2}\times...\times p^{y_n}(1-p)^{1-y_n
}$$
For $n$ observations. The product above is called the Likelihood (which needs to be minimized/optimized)of a certain outcome given inputs. It written as,
$$L(y;x)=\prod_{i=1}^{n}p^{y_i}(1-p)^{1-y_i}$$

But we know that GradienT Descent (GD) uses derivatives, therefore it is wise not find derivatives of products (Not easy). We therefore apply logarithm on the likelihood to obtain the log-likelihood (l)
$$l=\log(L(y;x))=\log\bigg(\prod_{i=1}^{n}p^{y_i}(1-p)^{1-y_i}\bigg)=\frac{1}{n}\sum_{i=1}^{n}y_i\log(p)+(1-y_i)\log(1-p)$$

Recall that $$p=\frac{1}{1+e^{-\theta^TX}}$$
We can therefore find the partial derivatives of $\theta_j$, since we want to to find the parameters $\theta_j$ that optimizes the log-likelihood function.

After applying chain rule on the log-likelihood function,we obtain
$$\frac{\partial l}{\partial \theta_j}=\frac{1}{n}\sum_{i=1}^{n}X_i(p-y_i)$$

To find the optimum $\theta_j$ we use GD, which iteratively updates the values of \theta_j untill convergence using:
$$\theta_j \text(new)=\theta_j \text(old)-\alpha \frac{\partial l}{\partial \theta_j}$$
where alpha is learning rate choosen experimentaly.

Once the optimum values of $ \theta_j$ is found can use $p=\frac{1}{1+e^{-\theta^TX}}$ to classify unseen data $X$.

Let us now implement Binary Logistic Regression from scratch and also using inbuilt package. We will use data from kaggle Iris data but with two categories instead of three. Kindly download the data here. If you want to use the same data.

```python
## Import the necessary packages
import numpy as np # performing array calculations
import pandas as pd # Load and manipulating the dataset
import seaborn as sns # visualization
import matplotlib.pyplot as plt # visualization
```

```python
#LOad the data which is local in my machine
data=pd.read_csv('Iris.csv')
#check the first 5 rows
data.head()
```

      Id
      SepalLengthCm
      SepalWidthCm
      PetalLengthCm
      PetalWidthCm
      Species

      0
      1
      5.1
      3.5
      1.4
      0.2
      Iris-setosa

      1
      2
      4.9
      3.0
      1.4
      0.2
      Iris-setosa

      2
      3
      4.7
      3.2
      1.3
      0.2
      Iris-setosa

      3
      4
      4.6
      3.1
      1.5
      0.2
      Iris-setosa

      4
      5
      5.0
      3.6
      1.4
      0.2
      Iris-setosa

```python
#Check the shape of our dataset
data.shape
```

    (150, 6)

#### The data has four features $X=\{SepalLengthC,SepalWidthCm,PetalLengthCm PetalWidthCm\}$  and one Target variable $y=Species$,   with $n=150$.

```python
#The species has how namy unique categories
data['Species'].unique()
```

    array(['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'], dtype=object)

#### As seen above,There are three categories ('Iris-setosa', 'Iris-versicolor', 'Iris-virginica'). We remove one so that its binary classification.

```python
#we remove one species using pandas
data1=data[data['Species']!='Iris-virginica']
print(data1.Species.unique() )#Two classes are left.
print(data1.shape)
```

    ['Iris-setosa' 'Iris-versicolor']
    (100, 6)

#### Now $n=100$

```python
#Next we drop the id column and check whether there missing values in the dataset
data2=data1.drop('Id', axis=1)
data2.info()
#No null /or missing values
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 100 entries, 0 to 99
    Data columns (total 5 columns):
     #   Column         Non-Null Count  Dtype
    ---  ------         --------------  -----
     0   SepalLengthCm  100 non-null    float64
     1   SepalWidthCm   100 non-null    float64
     2   PetalLengthCm  100 non-null    float64
     3   PetalWidthCm   100 non-null    float64
     4   Species        100 non-null    object
    dtypes: float64(4), object(1)
    memory usage: 4.7+ KB

#### The next step we convert the species column to 0 and 1  since it is of onject datatype using the Label Encoder from scikit learn

```python
#encode the Species/Target Vaiable
from sklearn.preprocessing import LabelEncoder
le=LabelEncoder()
data2['Species']=le.fit_transform(data2['Species'])
data2.Species.unique()
```

    array([0, 1])

#### Here we see that Iris-setosa was encoded as zero and Iris-versicolor as 1.

#### The data appears to be recorded in a certain order we shuffle to mix the data well.  Uisng the sample method.

```python
data2=data2.sample(frac=1)
```

```python
# we can now assing X and y their respective values.
X=data2.drop('Species',axis=1).values
y=data2.Species.values
```

#### Since the dataset isn't that large we shall only train and check the score on the training data. Remember that it is of paramount to have a testing set in order to have a better model. In this tutorial, we just want to show how Logistic regression works with GD

```python
#hstack a column onf ones in X
ones=np.ones([100,1])
X=np.hstack([ones,X])
```

```python
#initialize the weights
weights=np.zeros(5)
weights
```

    array([0., 0., 0., 0., 0.])

```python
epochs=1000
cost=[]
def sigmoid(p):
    return 1/(1+np.exp(-p))
def gradient(X,y,p):
    return 1/X.shape[0]*(np.dot(X.T,p-y))

```

```python
def logistic(X,y,epochs,alpha):
    weights=np.zeros(5)
    for i in range(epochs):
        product=np.dot(X,weights)
        #print(product)
        p=sigmoid(product)
        l=-1/X.shape[0]*np.sum(y*np.log(p)+(1-y)*np.log(1-p))
        #print(l)
        cost.append(l)
        weights=weights-alpha*gradient(X,y,p)
    return weights
```

```python
w1=logistic(X,y,500,0.02)
diff=np.round(sigmoid(X@w1))-y
(len(y)-np.count_nonzero(diff))/len(y)*100
```

    100.0

```python
np.linalg.pinv?
```

```python

```


# Regression

Regression is the process of approximating noisy data with functions.  The same numerical methods can also be used to approximate a complicated function with a simpler function.  We'll begin by looking at some limited cases in terms of linear algebra.

## Polynomial regression

Long ago (in the Linear Algebra notebook), we solved an over-determined linear system to compute a polynomial approximation of a function.
Sometimes we make approximations because we are interested in the polynomial coefficients.
That is usually only for very low order (like linear or quadratic fits).
Inferring higher coefficients is ill-conditioned (as we saw with Vandermonde matrices) and probably not meaningful.
We now know that Chebyshev bases are good for representing high-degree polynomials, but if the points are arbitrarily spaced, how many do we need for the Chebyshev approximation to be well-conditioned?

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')

def cosspace(a, b, n=50):
    return (a + b)/2 + (b - a)/2 * (
        np.cos(np.linspace(-np.pi, 0, n)))

def vander_chebyshev(x, n=None):
    if n is None:
        n = len(x)
    T = np.ones((len(x), n))
    if n > 1:
        T[:,1] = x
    for k in range(1,n-1):
        T[:,k+1] = 2 * x * T[:,k] - T[:,k-1]
    return T

def runge1(x):
    return 1 / (1 + 10*x**2)
```

```python
def chebyshev_regress_eval(x, xx, n):
    V = vander_chebyshev(x, n)
    Q, R = np.linalg.qr(V)
    return vander_chebyshev(xx, n) @ np.linalg.solve(R, Q.T)

xx = np.linspace(-1, 1, 100)
x = np.linspace(-1, 1, 80)
plt.plot(x, runge1(x), '.k')
plt.plot(xx, chebyshev_regress_eval(x, xx, 40) @ runge1(x), label='p(x)')
plt.plot(xx, runge1(xx), '-.', label='runge1(x)')
plt.legend(loc='upper left');
```

* What is the degree $k$ of the polynomial that is used?
* What distribution of points is used for the data?
* Would this have artifacts if we used a degree $k$ polynomial to interpolate noise-free samples at $k+1$ points using the same distribution?

```python
ns = np.geomspace(5, 1000, dtype=int)
conds = [np.linalg.cond(vander_chebyshev(np.linspace(-1, 1, n),
                                         int(n**.5)))
         for n in ns]
plt.loglog(ns, conds)
plt.xlabel('$n$')
plt.ylabel('cond');
```

* If we have $n$ data points, can we use a polynomial of degree $k = \lfloor n/5 \rfloor$?
* What about $k = n^{3/4}$?
* What expression $k = ?(n)$ appears to be sufficient?

## Noisy data

Regression really comes into its own when the data is noisy.

```python
def runge1_noisy(x, sigma):
    return runge1(x) + np.random.randn(*x.shape)*sigma

x = np.linspace(-1, 1, 500)
y = runge1_noisy(x, 0.5)
plt.plot(x, runge1(x), label='runge1(x)')
plt.plot(x, y, '.')
plt.plot(x, chebyshev_regress_eval(x, x, 7) @ y, label='regress(x)')
plt.legend(loc='upper left');
```

### Probability distributions

In order to interpret real data, we need a model for the noise.  We have chosen the most common and computationally convenient choice when creating the synthetic data above.
The function `randn` draws samples from the "standard normal" or Gaussian distribution.

$$ p(t) = \frac{1}{\sqrt{2\pi}} e^{-t^2/2} . $$

```python
def stdnormal(t):
    return np.exp(-t**2/2) / np.sqrt(2*np.pi)

n = 1000
w = np.random.randn(n)
plt.hist(w, bins=40, range=(-3,3), density=True)
t = np.linspace(-3, 3)
plt.plot(t, stdnormal(t))
plt.xlabel('$t$')
plt.ylabel('$p(t)$');
```

### Regression on noisy data

We can just go run our regression algorithm on the noisy data.

```python
x = np.linspace(-1, 1, 500)
y = runge1_noisy(x, sigma=0.1)
```

```python
plt.plot(x, y, '.')
plt.plot(xx, chebyshev_regress_eval(x, xx, 9) @ y, label='p(x)')
plt.plot(xx, runge1(xx), label='runge1(x)')
plt.legend(loc='upper left');
```

## What problem are we solving?

### Why do we call it a linear model?

We are currently working with algorithms that express the regression as a linear function of the model parameters.  That is, we search for coefficients $c = [c_0, c_1, \dotsc]^T$ such that

$$ V(x) c \approx y $$

where the left hand side is linear in $c$.  In different notation, we are searching for a predictive model

$$ f(x_i, c) \approx y_i \text{ for all $(x_i, y_i)$} $$

that is linear in $c$.

### Assumptions

1. The independent variables $x$ are error-free
1. The prediction (or "response") $f(x,c)$ is linear in $c$
1. The noise in the measurements $y$ is independent (uncorrelated)
1. The noise in the measurements $y$ has constant variance

There are reasons why all of these assumptions may be undesirable in practice, thus leading to more complicated methods.

### Loss functions

The error in a single prediction $f(x_i,c)$ of an observation $(x_i, y_i)$ is often measured as
$$ \frac 1 2 \big( f(x_i, c) - y_i \big)^2, $$
which turns out to have a statistical interpretation when the noise is normally distributed.
It is natural to define the error over the entire data set as
\begin{align} L(c; x, y) &= \sum_i \frac 1 2 \big( f(x_i, c) - y_i \big)^2 \\
&= \frac 1 2 \lVert f(x, c) - y \rVert^2
\end{align}
where I've used the notation $f(x,c)$ to mean the vector resulting from gathering all of the outputs $f(x_i, c)$.
The function $L$ is called the "loss function" and is the key to relaxing the above assumptions.

#### Optimization
Given data $(x,y)$ and loss function $L(c; x,y)$, we wish to find the coefficients $c$ that minimize the loss, thus yielding the "best predictor" (in a sense that can be made statistically precise).  I.e.,
$$ \bar c = \arg\min_c L(c; x,y) . $$

It is usually desirable to design models such that the loss function is differentiable with respect to the coefficients $c$, because this allows the use of more efficient optimization methods.  For our chosen model,
\begin{align} \nabla_c L(c; x,y) = \frac{\partial L(c; x,y)}{\partial c} &= \sum_i \big( f(x_i, c) - y_i \big) \frac{\partial f(x_i, c)}{\partial c} \\
&= \sum_i \big( f(x_i, c) - y_i \big) V(x_i)
\end{align}
where $V(x_i)$ is the $i$th row of $V(x)$.
A more linear algebraic way to write the same expression is
\begin{align} \nabla_c L(c; x,y) &= \big( f(x,c) - y \big)^T V(x) \\
&= \big(V(x) c - y \big)^T V(x) \\
&= V(x)^T \big( V(x) c - y \big)
\end{align}
A necessary condition for the loss function to be minimized is that $\nabla_c L(c; x,y) = 0$.

* Is the condition sufficient for general $f(x, c)$?
* Is the condition sufficient for the linear model $V(x) c$?
* Have we seen this sort of equation before?

##### Uniqueness
We can read the expression $ V(x)^T \big( V(x) c - y \big) = 0$ as saying that the residual $V(x) c - y$ is orthogonal to the range of $V(x)$.  Suppose $c$ satisfies this equation and $c' \ne c$ is some other value of the coefficients.  Then
\begin{align}
V(x)^T \big( V(x) c' - y \big) &= V(x)^T V(x) c' - V(x)^T y \\
&= V(x)^T V(x) c' - V(x)^T V(x) c \\
&= V(x)^T V(x) (c' - c) \ne 0
\end{align}
whenever $V(x)^T V(x)$ is nonsingular, which happens any time $V(x)$ has full column rank.

##### Gradient descent
Instead of solving the least squares problem using linear algebra (QR factorization), we could solve it using gradient descent.  That is, on each iteration, we'll take a step in the direction of the negative gradient.

```python
def grad_descent(loss, grad, c0, gamma=1e-3, tol=1e-5):
    """Minimize loss(c) via gradient descent with initial guess c0
    using learning rate gamma.  Declares convergence when gradient
    is less than tol or after 500 steps.
    """
    c = c0.copy()
    chist = [c.copy()]
    lhist = [loss(c)]
    for it in range(500):
        g = grad(c)
        c -= gamma * g
        chist.append(c.copy())
        lhist.append(loss(c))
        if np.linalg.norm(g) < tol:
            break
    return c, np.array(chist), np.array(lhist)

class quadratic_loss:
    """Test problem to give example of gradient descent."""
    def __init__(self):
        self.A = np.array([[1, 1], [1, 4]])
    def loss(self, c):
        return .5 * c @ self.A @ c
    def grad(self, c):
        return self.A @ c

def test(gamma):
    q = quadratic_loss()
    c, chist, lhist = grad_descent(q.loss, q.grad, .9*np.ones(2), gamma=gamma)
    plt.semilogy(lhist)
    plt.ylabel('loss')
    plt.xlabel('cumulative iterations')

    plt.figure()
    l = np.linspace(-1, 1)
    x, y = np.meshgrid(l, l)
    z = [q.loss(np.array([x[i,j], y[i,j]])) for i in range(50) for j in range(50)]
    plt.contour(x, y, np.reshape(z, x.shape))
    plt.plot(chist[:,0], chist[:,1], 'o-')
    plt.title('gamma={}'.format(gamma))

test(.45)
```

```python
class chebyshev_regress:
    def __init__(self, x, y, n):
        self.V = vander_chebyshev(x, n)
        self.y = y
        self.n = n

    def init(self):
        return np.zeros(self.n)

    def loss(self, c):
        r = self.V @ c - y
        return 0.5 * (r @ r)

    def grad(self, c):
        r = self.V @ c - y
        return self.V.T @ r

reg = chebyshev_regress(x, y, 6)
c, _, lhist = grad_descent(reg.loss, reg.grad, reg.init(), gamma=3e-3)
plt.semilogx(np.arange(1, 1+len(lhist)), lhist)
plt.ylabel('loss')
plt.xlabel('cumulative iterations')
plt.ylim(bottom=0);
```

* How does changing the "learning rate" `gamma` affect convergence?
* How does changing the number of basis functions affect convergence rate?

#### Did we find the same solution?
We intend to solve the same problem as we previously solved using QR, so we hope to find the same solution.

```python
plt.plot(xx, chebyshev_regress_eval(x, xx, 6) @ y, '-k', label='QR')
plt.plot(xx, vander_chebyshev(xx, 6) @ c, '--', label='grad_descent')
plt.plot(x, y, '.')
plt.legend();
```

#### Observations

* This algorithm is kind of finnicky:
 * It takes many iterations to converge
 * The "learning rate" needs to be empirically tuned
 * When not tuned well, the algorithm can diverge
* It could be made more robust using a line search
* The $QR$ algorithm is a more efficient and robust way to solve these problems

## Nonlinear regression

Instead of the linear model
$$ f(x,c) = V(x) c = c_0 + c_1 \underbrace{x}_{T_1(x)} + c_2 T_2(x) + \dotsb $$
let's consider a rational model with only three parameters
$$ f(x,c) = \frac{1}{c_0 + c_1 x + c_2 x^2} = (c_0 + c_1 x + c_2 x^2)^{-1} . $$
We'll use the same loss function
$$ L(c; x,y) = \frac 1 2 \lVert f(x,c) - y \rVert^2 . $$

We will also need the gradient
$$ \nabla_c L(c; x,y) = \big( f(x,c) - y \big)^T \nabla_c f(x,c) $$
where
\begin{align}
\frac{\partial f(x,c)}{\partial c_0} &= - f(x,c)^2 \\
\frac{\partial f(x,c)}{\partial c_1} &= - f(x,c)^2 x \\
\frac{\partial f(x,c)}{\partial c_2} &= - f(x,c)^2 x^2 .
\end{align}

```python
class rational_regress:
    def __init__(self, x, y):
        self.x = x
        self.y = y
        self.n = 3

    def init(self):
        return np.ones(self.n)

    def f(self, c):
        x = self.x
        return 1 / (c[0] + c[1]*x + c[2]*x**2)

    def df(self, c):
        x = self.x
        f2 = self.f(c)**2
        return np.array([-f2, -f2*x, -f2*x**2]).T

    def loss(self, c):
        r = self.f(c) - self.y
        return 0.5 * (r @ r)

    def grad(self, c):
        r = self.f(c) - self.y
        return r @ self.df(c)

reg = rational_regress(x, y)
c, _, lhist = grad_descent(reg.loss, reg.grad, reg.init(), gamma=2e-2)
plt.semilogx(np.arange(1, 1+len(lhist)), lhist)
plt.ylabel('loss')
plt.xlabel('cumulative iterations')
plt.ylim(bottom=0)
plt.title('rational c={}'.format(c));
```

```python
x = np.linspace(-1, 1, 500)
y = runge1_noisy(x, sigma=0.1)

c0 = np.array([1, 0, 1.])
c, _, lhist = grad_descent(reg.loss, reg.grad, c0, gamma=2e-2)
plt.semilogx(np.arange(1, 1+len(lhist)), lhist)
plt.ylabel('loss')
plt.xlabel('cumulative iterations')
plt.ylim(bottom=0)
plt.title('rational c={}'.format(c));
```

```python
plt.plot(x, y, '.')
plt.plot(xx, runge1(xx), label='runge1(x)')
plt.plot(xx, chebyshev_regress_eval(x, xx, 6) @ y, label='Polynomial')
plt.plot(xx, 1/(c[0] + c[1]*xx + c[2]*xx**2), '--k', label='Rational')
plt.legend();
```

#### Observations

* There can be local minima or points that look an awful lot like local minima.
* Convergence is sensitive to learning rate.
* It takes a lot of iterations to converge.
* A well-parametrized model (such as the rational model above) can accurately reconstruct the mean even with very noisy data.

### Outlook

* The optimization problem can be solved using a Newton method.  It can be onerous to implement the needed derivatives.
* The [Gauss-Newton method](https://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm) (see homework) is often more practical than Newton while being faster than gradient descent, though it lacks robustness.
* The [Levenberg-Marquardt method](https://en.wikipedia.org/wiki/Levenberg%E2%80%93Marquardt_algorithm) provides a sort of middle-ground between Gauss-Newton and gradient descent.
* Many globalization techniques are used for models that possess many local minima.
* One pervasive approach is stochastic gradient descent, where small batches (e.g., 1 or 10 or 20) are selected randomly from the corpus of observations (500 in our current example), and a step of gradient descent is applied to that reduced set of observations.  This helps to escape saddle points and weak local minima.
* Among expressive models $f(x,c)$, some may converge much more easily than others.  Having a good optimization algorithm is essential for nonlinear regression with complicated models, especially those with many parameters $c$.
* Classification is a very similar problem to regression, but the observations $y$ are discrete, thus

 * models $f(x,c)$ must have discrete output
 * the least squares loss function is not appropriate.
* [Why momentum really works](https://distill.pub/2017/momentum/)


```python
import sympy
sympy.init_printing()
import numpy
from sympy.plotting import plot  #para plotear 2 variables
import matplotlib.pyplot as plt
%matplotlib inline

t, s = sympy.symbols('t, s')
a = sympy.symbols('a', real=True, positive=True)

f = sympy.exp(-a*t)
f
```

```python
sympy.integrate(f*sympy.exp(-s*t), (t, 0, sympy.oo))
```

$\displaystyle \begin{cases} \frac{1}{s \left(\frac{a}{s} + 1\right)} & \text{for}\: \left|{\arg{\left(s \right)}}\right| \leq \frac{\pi}{2} \\\int\limits_{0}^{\infty} e^{- a t} e^{- s t}\, dt & \text{otherwise} \end{cases}$

```python
sympy.laplace_transform(f, t, s)
```

$\displaystyle \left( \frac{1}{a + s}, \  0, \  \text{True}\right)$

```python
def L(f):
    return sympy.laplace_transform(f, t, s, noconds=True)
def invL(F):
    return sympy.inverse_laplace_transform(F, s, t)
```

```python
F=L(f)
F
```

```python
f_=invL(F)
f_
```

```python
sympy.plot(sympy.Heaviside(t));
```

```python
omega = sympy.Symbol('omega', real=True)
exp = sympy.exp
sin = sympy.sin
cos = sympy.cos
functions = [1,
         t,
         exp(-a*t),
         t*exp(-a*t),
         t**2*exp(-a*t),
         sin(omega*t),
         cos(omega*t),
         1 - exp(-a*t),
         exp(-a*t)*sin(omega*t),
         exp(-a*t)*cos(omega*t),
         ]
functions
```

```python
Fs = [L(f) for f in functions]
Fs
```

```python
F = ((s + 1)*(s + 2)* (s + 3))/((s + 4)*(s + 5)*(s + 6))
display(F)
F.apart(s)
```

```python
display(invL(F))
invL(F.apart(s))
```

```python
s = sympy.Symbol('s')
t = sympy.Symbol('t', real=True)
tau = sympy.Symbol('tau', real=True, positive=True)
G = K/(tau*s + 1)
G
```

```python
u = 1/s
stepresponse = invL(G*u)
stepresponse
```

```python
u = 1/s**2
rampresponse = invL(G*u)
rampresponse
```

```python
g = sympy.inverse_laplace_transform(G.subs({tau: 1,K:10}), s, t)
display(g)
sympy.plot(g, (t, -1, 10), ylim=(0, 1.1))
```

```python
s = sympy.symbols("s")
w = sympy.symbols("omega", real=True)
display(G)
Gw = G.subs({tau : 1,K:10, s : sympy.I*w})
Gw
```

```python
RA = abs(Gw)
phi = sympy.arg(Gw)
sympy.plot(RA, (w, 0.01, 100), xscale="log", yscale="log", ylabel="RA", xlabel="$\omega$")
sympy.plot(phi*180/sympy.pi, (w, 0.01, 100), xscale="log", xlabel="$\omega$", ylabel="$\phi$")
```

```python
from ipywidgets import interact
evalfimpulse = sympy.lambdify((K, tau, t), impulseresponse, 'numpy')
evalfstep = sympy.lambdify((K, tau, t), stepresponse, 'numpy')
evalframp = sympy.lambdify((K, tau, t), rampresponse, 'numpy')
```

```python
ts = numpy.linspace(0, 10)

def firstorder(tau_in, K_in):
    plt.figure(figsize=(12, 6))
    ax_impulse = plt.subplot2grid((2, 2), (0, 0))
    ax_step = plt.subplot2grid((2, 2), (1, 0))
    ax_complex = plt.subplot2grid((2, 2), (0, 1), rowspan=2)

    ax_impulse.plot(ts, evalfimpulse(K_in, tau_in, ts))
    ax_impulse.set_title('Impulse response')
    ax_impulse.set_ylim(0, 10)

    tau_height = 1 - numpy.exp(-1)
    ax_step.set_title('Step response')
    ax_step.plot(ts, evalfstep(K_in, tau_in, ts))
    ax_step.axhline(K_in)
    ax_step.plot([0, tau_in, tau_in], [K_in*tau_height]*2 + [0], alpha=0.4)
    ax_step.text(0, K_in, '$K=${}'.format(K_in))
    ax_step.text(0, K_in*tau_height, '{:.3}$K$'.format(tau_height))
    ax_step.text(tau_in, 0, r'$\tau={:.3}$'.format(tau_in))
    ax_step.set_ylim(0, 10)

    ax_complex.set_title('Poles plot')
    ax_complex.scatter(-1/tau_in, 0, marker='x', s=30)
    ax_complex.axhline(0, color='black')
    ax_complex.axvline(0, color='black')
    ax_complex.axis([-10, 10, -10, 10])
```

```python
firstorder(1., 10.)
```

```python

```


# What is probability?
1. Frequentist: Probabilities represent long run frequencies of events
2. Bayesian:
  - Probability is used to quantify our uncertainty about something
  - It can be used to model our uncertainty about events that do not have long term frequencies

## Discrete random variables

For an event $A \in \mathcal{X}$, like "it will rain tomorrow".
 - **Probability mass function (PMF)**: p(A) means probability that event A is true
 - $0 \leq p(A) \leq 1$
 - $\sum_{a \in \mathcal{X}}p(a) = 1$

### Joint Probability
Given two events $A, B \in \mathcal{X}$, the probability of the joint event A and B:
\begin{align}
p(A, B) = p(A \cap B) = p(A | B)p(B) = p(B | A)p(A)
\end{align}

### Marginal distribution (边缘分布)

如果我们把每一个变量的概率分布称为一个概率分布，那么边缘分布就是若干个变量的概率加和所表现出的分布。

Given a joint distribution, the marginal distribution is defined as:
\begin{equation}
p(A) = \sum_{b \in \mathcal{X}}p(A, B=b) = \sum_{b \in \mathcal{X}}p(A|B=b)p(B=b)
\end{equation}

举个例子，假设$p(B),p(C),p(A|B),p(A|C)$已知，求P(A):
\begin{equation}
p(A) = \sum_{b \in [B, C]}p(A, B=b) = p(A, B) + p(A, C) = p(A|B)p(B) + P(A|C)p(C)
\end{equation}
### Conditional Probability

事件A在另外一个事件B已经发生条件下的发生概率。条件概率表示为P（A|B），读作“在B条件下A的概率”。

We define the conditional probabiltity of an event A, given that event B is true:
\begin{equation}
p(A|B) = \frac{p(A, B)}{p(B)} \; \textrm{if} \; p(B) \; \gt 0
\end{equation}

### Bayes Rule

According to conditional and marginal probabilities, bayes rule is defined by:
\begin{equation}
\underbrace{p(X=\mathcal{x}|Y=\mathcal{y}) = \frac{p(X=\mathcal{x}, Y=\mathcal{y})}{p(Y=\mathcal{y})}}_{\textrm{conditional probability}} \; \textrm{and} \; \underbrace{p(y) = \sum_{x'}p(X=\mathcal{x'})p(Y=\mathcal{y} | X=\mathcal{x'})}_{\textrm{marginal probability}}  \\ \Downarrow \\
p(X=\mathcal{x}|Y=\mathcal{y}) = \frac{p(X=\mathcal{x}, Y=\mathcal{y})}{\sum_{x'}p(X=\mathcal{x'})p(Y=\mathcal{y} | X=\mathcal{x'})}
\end{equation}

### A medical diagnoise problem using Bayes Rule

**Example**: Suppose you are a women in your 40s, and you decide to have medical test ($X = 1$) for breast cancer ($Y \in [0, 1]$), which is called mammogram. If the test is positive, what's probability you have cancer $p(Y=1 | X=1)$?

\begin{proof}
Assume that $p(X=1|Y=1) = 0.8, p(X=1|Y=0) = 0.1, p(Y=1) = 0.004$
\begin{equation}
\begin{aligned}
p(Y=1|X=1) &= \frac{p(X=1, Y=1)}{p(X=1)} \\
&= \frac{p(X=1|Y=1)p(Y=1)}{p(X=1, Y=1) + p(X=1, Y=0)} \\
&= \frac{p(X=1|Y=1)p(Y=1)}{p(X=1|Y=1)p(Y=1) + p(X=1|Y=0)p(Y=0)} \\
&= \frac{0.8 * 0.004}{0.8 * 0.0004 + 0.1*0.0996} = 0.031
\end{aligned}
\end{equation}
\end{proof}

### Conditional Independence

- If random variables $X$ and $Y$ are said to be independent if $p(X, Y) = p(X)p(Y)$, alternatively, $p(X|Y) = p(X), or p(Y|X) = p(Y)$, which is denoted by $X \perp Y$
- Conditional independent: $X \perp Y | Z \Leftrightarrow p(X, Y | Z) = p(X|Z)p(Y|Z)$

## Continuous Random Variable

- Probability density $p(x) (\geq 0)$
- The larger $p(x)$ for a variable $x$, the more likely that a variable around $x$ will be generated by this distribution.
- The probability that $x$ falls into an interval $[a, b]$ can be computed as $\int_{a}^{b} p(x) \,dx$
- The likelihood that any variable drawn from $p(x)$ must fall between postive and negative infinity $\int_{-\infty}^{\infty}p(x)dx = 1$

## Summary stastistic Property

Suppose that we are dealing with a random variables X. The distribution itself can be hard to interpret. It is often useful to be able to summarize the behavior of a random variable concisely. Numbers that help us capture the behavior of a random variable are called summary statistics. The most commonly encountered ones are the mean, the variance, and the standard deviation.

### Mean (a.k.a, expected value)

- Discrete Random Variable: $\mu = \mathbb{E}[X] = \sum_{\mathcal{x} \in X} x p(x)$. then the mean is given by the weighted average: sum the values times the probability that the random variable takes on that value
- Continuous Random Variable: $\mu = \mathbb{E}[X] = \int_{\mathcal{x}}xp(x)dx$

Because they are helpful, let us summarize a few properties.
- $\mu_{aX+b} = a\mu_{X} + b$
- $\mu_{X + Y} = \mu_X + \mu_Y$

Means are useful for understanding the average behavior of a random variable, however the mean is not sufficient to even have a full intuitive understanding.

### Variance (方差, "spread" of a distribution)

This is a quantitative measure of how far a random variable deviates from the mean
\begin{equation}
\begin{aligned}
\textrm{var}[X] &= \mathbb{E}[(X-\mu)^2] = \mathbb{E}[X^2 - 2\mu X + \mu^2] \\
&= \mathbb{E}[X^2] - \mathbb{E}[2\mu X] + \mathbb{E}[\mu^2] \\
&= \mathbb{E}[X^2] - 2\mu \mathbb{E}[X] + \mathbb{E}[\mu^2] \\
&= \mathbb{E}[X^2] - 2\mu^2 + \mu^2 = \mathbb{E}[X^2] - \mu^2
\end{aligned}
\end{equation}

Alternatively, proof as follow:
\begin{proof}
\begin{equation}
\begin{aligned}
\textrm{var}[X] &= \mathbb{E}[(X-\mu)^2] = \int (x-\mu)^2p(x)dx \\
&= \int x^2p(x) dx - \int 2\mu x p(x) dx + \int \mu^2 p(x)dx \\
&= \int x^2p(x) dx - 2\mu \int x p(x) dx + \mu^2 \int p(x)dx \\
&= \mathbb{E}[X^2] - 2\mu \mathbb{E}[X] + \mu^2 * 1 \\
&= \mathbb{E}[X^2] - \mu^2
\end{aligned}
\end{equation}
\end{proof}

We will list a few properties of variance below:
- $Var[X] \geq 0$; If X is constant, then $Var[X] = 0$
- $Var[aX+b] = a^2Var[X]$
- if X is independent Y, then $Var[X+Y] = Var[X] + Var[Y]$

### Standard deviation

- $\delta_X = \textrm{std}[X] = \sqrt{\textrm{var}[X]} $

The properties we had for the variance can be restated for the standard deviation.
- $\delta_X \geq 0$
- $\delta_{aX + b} = |a|\delta_X$
- if X is independent Y, \delta_{X + Y} = \sqrt_{\delta^2_X + \delta^2_Y}

### Covariance (协方差)

For two jointly distributed real-valued random variables X and Y with finite second moments, the covariance is defined as the expected value (or mean) of the product of their deviations from their individual expected values:
\begin{equation}
\begin{aligned}
\delta_{XY} = \textrm{cov}(X, Y) &= \mathbb{E}[(X-\mathbb{E}(X))(Y-\mathbb{E}(Y))] = \sum_{i, j}(x_i - \mu_X)(y_j - \mu_Y)P_{ij}\\
&= \mathbb{E}[(X-\mu_X)(Y- \mu_Y)] \\
&= \mathbb{E}[XY-X*\mu_Y - Y*\mu_X + \mu_X*\mu_Y] \\
&= \mathbb{E}[XY]-\mathbb{E}[X*\mu_Y] - \mathbb{E}[Y*\mu_X] + \mathbb{E}[\mu_X*\mu_Y] \\
&= \mathbb{E}[XY]-\mathbb{E}[X]*\mu_Y - \mathbb{E}[Y]*\mu_X + \mu_X*\mu_Y \\
&= \mathbb{E}[XY]-\mu_X*\mu_Y - \mu_Y*\mu_X + \mu_X*\mu_Y \\
&= \mathbb{E}[XY]-\mu_X*\mu_Y \\
&= \mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]
\end{aligned}\end{equation}

If X is a d-dimnsional random vector, its covariance matrix is:
\begin{equation}
\begin{aligned}
\textrm{cov}[X] &= \mathbb{E}[(X-\mathbb{E}[X])(X-\mathbb{E}[X])^T] \\
&= \begin{bmatrix}
\color{red}{\textrm{cov}[X_1, X_1]}  & \textrm{cov}[X_1, X_2] & \cdots &\textrm{cov}[X_1, X_d]\\
\textrm{cov}[X_2, X_1] & \color{red}{\textrm{cov}[X_2, X_2]} & \cdots &\textrm{cov}[X_2, X_d]\\
\vdots & \vdots & \ddots & \vdots \\
\textrm{cov}[X_d, X_1]& \textrm{cov}[X_d, X_2] & \cdots &\color{red}{\textrm{cov}[X_d, X_d]}\\
\end{bmatrix} = \begin{bmatrix}
\color{red}{\textrm{var}[X_1]}  & \textrm{cov}[X_1, X_2] & \cdots &\textrm{cov}[X_1, X_d]\\
\textrm{cov}[X_2, X_1] & \color{red}{\textrm{var}[X_2]} & \cdots &\textrm{cov}[X_2, X_d]\\
\vdots & \vdots & \ddots & \vdots \\
\textrm{cov}[X_d, X_1]& \textrm{cov}[X_d, X_2] & \cdots &\color{red}{\textrm{var}[X_d]}\\
\end{bmatrix}
\end{aligned}
\end{equation}

Let us see some properties of covariances:

- $Cov[X, X] = Var[X]$
- $Cov[aX + b, Y] = Cov[X, aY + b] = a Cov[X, Y]$
- if X is independent of Y, then $Cov[X, Y] = 0$.
- $Var[X + Y] = Var[X] + 2Cov[X, Y] + Var[Y]$

### Correlation Coefficient

- The (Pearson) correlation coefficient between X and Y measures the linear relationship, which is defined by:
\begin{equation}
-1 \leq \rho_{XY} = \textrm{corr}[X, Y] = \frac{\textrm{cov}[X, Y]}{\sqrt{\textrm{var}[X]\textrm{var}[Y]}} = \frac{Cov[X, Y]}{\delta_X\delta_Y}\leq 1
\end{equation}

consider X as any random variable, and Y = aX +b as any linear deterministic function of X. Then, one can compute that
$$\delta_Y = \delta_{aX + b} = |a|\delta_X$$
$$Cov[X, Y] = Cov[X, aX+b] = a Cov[X, X] = a Var[X]$$
and thus that:
$$\rho_{XY} = \frac{aVar[x]}{|a|\delta^2_X} = \frac{a}{|a} = sign(a)$$
Thus we see that the correlation is +1 for any a > 0, and -1 for any a < 0 illustrating that **correlation measures the degree and directionality the two random variables are related, not the scale that the variation takes.**

Let us list a few properties of the correlation below.
- \$rho_{XX} = 1$
- \forall X, Y, a, b \in \mathcal{R}: \rho[aX+b, Y] = \rho[X, aY+b] = \rho[X, Y]
- If X and Y are independent with non-zero variance then $\rho[X, Y] = 0$
- Indeed if we think of norms as being related to standard deviations, and correlations as being cosines of angles, much of the intuition we have from geometry can be applied to thinking about random variables.

## Common distributions (TODO)

### Bernoulli Distribution (Discrete)
- Toss a coin only once
- Let X be a binary variable $X \in {0, 1}$

- What's the probability of X that a toss shows up as "head":
\begin{equation}
\textrm{Ber}(x|\theta) =  \left\{
	\begin{array}{ll}
		\theta  & \mbox{if } x = 0 \\
		1 - \theta & \mbox{if } x = 0
	\end{array}
\right.
\end{equation}

### Binomial Distribution (Discrete)

### Multinomial Distribution (Discrete)

### Gaussian Distribution (Continuous)

### Laplace Distribution  (Continuous)

### Beta Distribution  (Continuous)

# Bayesian Theory

所谓的贝叶斯方法源于他生前为解决一个“逆概”问题写的一篇文章，而这篇文章是在他死后才由他的一位朋友发表出来的。在贝叶斯写这篇文章之前，人们已经能够计算“正向概率”，如“假设袋子里面有N个白球，M个黑球，你伸手进去摸一把，摸出黑球的概率是多大”。而一个自然而然的问题是反过来：“如果我们事先并不知道袋子里面黑白球的比例，而是闭着眼睛摸出一个（或好几个）球，观察这些取出来的球的颜色之后，那么我们可以就此对袋子里面的黑白球的比例作出什么样的推测”。这个问题，就是所谓的逆概问题。

贝叶斯是机器学习的核心方法之一。这背后的深刻原因在于，现实世界本身就是不确定的，人类的观察能力是有局限性的，我们日常所观察到的只是事物表面上的结果，沿用刚才那个袋子里面取球的比方，我们往往只能知道从里面取出来的球是什么颜色，而并不能直接看到袋子里面实际的情况。这个时候，我们就需要提供一个假设（hypothesis）。所谓假设，当然就是不确定的（可能是有限个，也可能是无限多种），为了确定哪个假设是正确的，我们需要做两件事情：1、算出各种不同猜测的可能性大小。2、算出最靠谱的猜测是什么。第一个就是计算特定猜测的后验概率（Posterior），对于连续的猜测空间则是计算猜测的概率密度函数。第二个则是所谓的模型比较，模型比较如果不考虑先验概率（Prior）的话就是最大似然方法。

## Generative Classifer

对于一个训练数据集X，如何判断他属于哪个类型？

通用的方法就是求解在已经数据集X的情况下，求出每个分类针对于该数据集的概率，概率最大的就是最大的可能性。 其中$\theta$是关于这个模块的参数。

Given a dataset $X$, the probability of (Y=c) for this dataset is defined using the class conditional density ($p(X|Y=c)$) and the class prior $p(Y=c)$:
\begin{equation}
\label{eq:baysian_rule}
\begin{aligned}
p(Y=c|X, \theta) &= \frac{p(X, Y=c, \theta)}{p(X, \theta)} \\
& = \frac{p(X|Y=c, \theta)p(Y=c, \theta)}{\sum_{c'}p(Y=c', \theta)p(X|Y=c', \theta)} \\
&\propto p(X|Y=c, \theta)p(Y=c, \theta)
\end{aligned}
\end{equation}
- $p(Y=c | X, \theta)$: Posterior probability (后验概率)
- $f(\theta) = p(X|Y=c, \theta)$: likelihood probability, we also call this function likelihood function (似然函数)
- $p(Y=c, \theta)$: prior probability (先验概率)
- $p(X, \theta)$: Normalized factor(证据因子)

**Solution:** the posterior equals to the likelihood times the prior, up to a constant.
\begin{equation}
\begin{aligned}
\textrm{Posterior (后验概率)} &= \frac{\textrm{prior (先验概率)} \times \textrm{likelihood (似然函数)}}{\textrm{证据因子}} \\
&\propto \textrm{prior (先验概率)} \times \textrm{likelihood (似然函数)}
\end{aligned}
\end{equation}

## Example: Number Game (from Josh Tenenbaum's PhD thesis)

Inferring abstract patterns from sequence of integers

**Problem:** There is a group of hypotheses $\mathcal{H}$: {'Primer number', 'a number between 1 and 10', $\ldots$}. In this task, a set of positive examples (training datasets) $D = \{x_1, x_2, \ldots, x_n\}$ drawn from a reasonable arithmetical concept $h \in \mathcal{H}$. Finally, I would ask that whether a new test case $x$ belong to $h$ (classifying $x$)
- **Posterior predictive distribution**: what's probability of that $X \in h$ given the dataset $D$: $p(x \in h |D, \theta)$.
\begin{equation}
\begin{aligned}
p(x \in h | D, \theta) & = \frac{p(x \in h, D, \theta)}{p(D, \theta)} \\
&= \frac{p(D|x \in h, \theta)p(x \in h, \theta)}{p(D, \theta)} \\
& = \frac{p(D|x \in h, \theta)p(x \in h, \theta)}{\sum_{h' \in \mathcal{H}}p(D, D \in h', \theta)} \\
&\propto p(D|x \in h, \theta)p(x \in h, \theta)
\end{aligned}
\end{equation}

**Example**
- For simplicity, assume all numbers are integers between 1 and 100
- $D = \{2, 8, 16, 64\}$ which are samples from IID (independent and identically distributed) datasets

**Solution**

1. **Hypothesis space of concepts $\mathcal{H}$**

Usually, the models favors the simplest hypothesis consistent with the data (Occam's razor). In this case, we choose $\mathcal{H} = \{h_1, h_2, h_3\}$
- "powers of two", that's $h_1 = h_{two} = \{2, 4, 8, 16, 32, 64\}$, where $|h_{1}| = 6$
- "even numbers", that's $h_2 = h_{even} = \{2, 4, 6, \ldots, 100\}$, where $|h_{2}|=50$
- "powers of two except 32", that's $h_3 = h_{two-32} = \{2, 4, 8, 16, 64\}$, where $|h_{3}| = 5$

2. **Prior p(h)**

Usually, prior probability captures the background knowledge, domain knowledge, pre-existing biases. In our case, it looks like $h_{two-32}$ is much better fit than $h_{two}$, however, the former seems "conceptually unnatural". Therefore, we can capture such intuition by assigning lower prior probability to unnatural concepts.
\[ p(\mathcal{H}) = \begin{bmatrix} p(h_1) = 0.19, \\ p(h_2) = 0.8, \\ p(h_3) = 0.01 \end{bmatrix}\]

3. **Likelihood p(D|h)**

In this case, we define likelihood function (statistical information in examples) as follows:
\[ p(D | h) = [\frac{1}{size(h)}]^n = [\frac{1}{|h|}]^n \; \textrm{if} x_1, \ldots, x_n \in h\]
where $n$ means the total number of elements in $D$. Smaller hypotheses receive greater likelihood, and exponentially more so as n increases.

Therefore, we can get likelihood probability for $h_1$:
\begin{equation}
\begin{aligned}
p(D|h_1) &= p(x_1|h_1) \times p(x_2|h_1) \times p(x_3|h_1) \times p(x_4|h_1) \;\;\; (\textrm{data from IID}) \\
&= \frac{1}{6} \times \frac{1}{6} \times \frac{1}{6} \times \frac{1}{6} \\
&= [\frac{1}{|h_1|}]^4  \;\;\; (|h_1| = 6)  \\
&= [\frac{1}{6}]^4
\end{aligned}
\end{equation}
Similarly, we can compute likelihood probabilities for $h_2$ and $h_3$:
\[p(D|h_2) = [\frac{1}{|h_2|}]^4 = [\frac{1}{50}]^4\]
\[p(D|h_3) = [\frac{1}{|h_3|}]^4 = [\frac{1}{5}]^4\]

4. **Posterior p(h|D)**

\begin{equation}
\begin{aligned}
p(x \in h_1 | D) & = \frac{p(x \in h_1, D)}{p(D)} \\
&= \frac{p(D|x \in h_1)p(x \in h_1)}{p(D)} \\
& = \frac{p(D|x \in h_1)p(x \in h_1)}{\sum_{h' \in \mathcal{H}}p(D, D \in h')} \\
&\propto p(D|x \in h_1, \theta)p(x \in h_1) \\
&= p(D|h_1)p(h_1)
\end{aligned}
\end{equation}

## Approaches to Estimate Parameters from IID data

Usually, the posterior is simply the liklihood times the prior, and then normalized:
\begin{equation}
\begin{aligned}
p(h|D, \theta) &= \frac{p(D, h, \theta)}{p(D, \theta)} \\
& = \frac{p(D|h, \theta)p(h, \theta)}{\sum_{h'}p(Y=h', \theta)p(D|h', \theta)} \\
&\propto p(D|h, \theta)p(h, \theta)
\end{aligned}
\end{equation}

Maximum Likelihood Estimation (MLE) and Maximum A Posteriori (MAP), are both a method for estimating some variable in the setting of probability distributions or graphical models. They are similar, as they compute a single estimate, instead of a full distribution.

### Maximum Likelihood Estimation (MLE)
MLE, as we, who have already indulge ourselves in Machine Learning, would be familiar with this method. This is the concept that when working with a probabilistic model with unknown parameters, the parameters which make the data have the highest probability are the most likely ones.

Suppose that we have a model with parameters $\theta$ and a collection of data examples X. For concreteness, we can imagine
that $\theta$ is a single value representing the probability that a coin comes up heads when flipped, and X is a sequence of independent coin flips. We will look at this example in depth later.

If we want to find the most likely value for the parameters of our model, that means we want to find:
\[ \text{arg max} P(\theta | X) \]

By Bayesʼ rule, this is the same thing as
\[ \text{arg max} P(\theta | X) = \text{arg max} \frac{P(X|\theta)P(\theta)}{P(X)} \] where
1. Evidence $P(X)$ is a a parameter agnostic probability of generating the data, does not depend on $\theta$ at all. so it can be dropped without changing the best choice of $\theta$
2. Prior $P(\theta)$ does not depend on $\theta$, that's uninformative prior.
3. Likelihood $P(X | \theta)$ is the probability of the data given the parameters.

Given above two assumptions, we see that our application of Bayesʼ rule shows that our best choice of $\theta$ is the maximum likelihood estimate for $\theta$:
\[ \hat{\theta} = \underset{\theta}{\text{arg max}} P(X | \theta)\]

Sometimes, we even use it without knowing it. Take for example, when fitting a Gaussian to our dataset, we immediately take the sample mean and sample variance, and use it as the parameter of our Gaussian. This is MLE, as, if we take the derivative of the Gaussian function with respect to the mean and variance, and maximizing it (i.e. setting the derivative to zero), what we get is functions that are calculating sample mean and sample variance. Another example, most of the optimization in Machine Learning and Deep Learning (neural net, etc), could be interpreted as MLE.

<!-- \begin{equation}
\begin{aligned}
\hat{h}^{MLE} &= \underset{h \in \mathcal{H}}{\textrm{argmax}} [p(h|D, \theta)] \\
&= \underset{h \in \mathcal{H}}{\textrm{argmax}} [\frac{p(D|h, \theta)p(h, \theta)}{\sum_{h'}p(Y=h', \theta)p(D|h', \theta)}] \\
&\propto \underset{h \in \mathcal{H}}{\textrm{argmax}}[\log(p(D|h, \theta)]\;\;\;(\text{If we ignore the prior} )
\end{aligned}
\end{equation} -->

Speaking in more abstract term, let’s say we have a likelihood function $P(X|\theta)$. Then, the MLE for $\theta$, the parameter we want to infer, is:
\begin{equation}
\begin{aligned}
\theta_{MLE} &= \underset{\theta}{\textrm{argmax}} P(X|\theta) \\
&= \underset{\theta}{\textrm{argmax}}\prod_{i} P(x_i|\theta) \;\;\; (x_i \in X \;\text{sampled from IID datasets})
\end{aligned}
\end{equation}

As taking a product of some numbers less than 1 ($ 0 \leq P(x_i|\theta) \leq 1$) would approaching 0 as the number of those numbers goes to infinity, it would be not practical to compute, because of computation underflow. Hence, we will instead work in the **log** space, as logarithm is monotonically increasing, so maximizing a function is equal to maximizing the log of that function.

\begin{equation}
\begin{aligned}
\theta_{MLE} &= \underset{\theta}{\textrm{argmax}} \log P(X|\theta) \\
&= \underset{\theta}{\textrm{argmax}} \log  \prod_{i} P(x_i|\theta) \;\;\; (x_i \in X \;\text{sampled from IID datasets}) \\
&= \underset{\theta}{\textrm{argmax}} \sum_{i} \log P(x_i|\theta)
\end{aligned}
\end{equation}
To use this framework, we just need to derive the log likelihood of our model, then maximizing it with regard of  $\theta$ using our favorite optimization algorithm like Gradient Descent.

### Maximum A Posteriori (MAP)

MAP usually comes up in **Bayesian** setting. Because, as the name suggests, it works on a posterior distribution, not only the likelihood. When we have enough data, the posterior becomes peaked on a single concept.

\begin{equation}
\begin{aligned}
\hat{h}^{MAP} &= \underset{h \in \mathcal{H}}{\textrm{argmax}} [p(h|D, \theta)] \\
&= \underset{h \in \mathcal{H}}{\textrm{argmax}} [\frac{p(D|h, \theta)p(h, \theta)}{\sum_{h'}p(Y=h', \theta)p(D|h', \theta)}] \\
&\propto \underset{h \in \mathcal{H}}{\textrm{argmax}}[\log(p(D|h, \theta)p(h, \theta))] \\
&= \underset{h \in \mathcal{H}}{\textrm{argmax}}[\log p(D|h, \theta)  + \log p(h, \theta)]
\end{aligned}
\end{equation}

在众多的$\mathcal{H}$中，哪个posterior probability是最大的，那么当前这个数据集最属于那个$h$

Recall, with Bayes’ rule, we could get the posterior as a product of likelihood and prior:
\begin{equation}
\begin{aligned}
P(\theta|X) &= \frac{P(X|\theta)P(\theta)}{P(X)} \\
&\propto P(X|\theta)P(\theta)
\end{aligned}
\end{equation}
We are ignoring the **normalizing constant** as we are strictly speaking about optimization here, so proportionality is sufficient. If we replace the likelihood in the MLE formula above with the posterior, we get:

\begin{equation}
\begin{aligned}
\theta_{MAP} &= \underset{\theta}{\textrm{argmax}}  \log P(X|\theta)P(\theta) \\
&= \underset{\theta}{\textrm{argmax}} [\log P(X|\theta) + \log P(\theta)] \\
&= \underset{\theta}{\textrm{argmax}} [\log \prod_{i}{P(x_i | \theta)} + \log P(\theta)]  \;\;\; (x_i \in X \;\text{sampled from IID datasets})\\
&= \underset{\theta}{\textrm{argmax}} \sum_{i} \log P(x_i | \theta) + \underset{\theta}{\textrm{argmax}}  \log P(\theta)
\end{aligned}
\end{equation}

### Comparison between MLE and MAP
Comparing both MLE and MAP equation, the only thing differs is the inclusion of prior $P(\theta)$ in MAP, otherwise they are identical. What it means is that, the likelihood is now weighted with some weight coming from the prior.
\begin{equation}
\begin{aligned}
\theta_{MLE} &= \underset{\theta}{\textrm{argmax}} \sum_{i} \log P(x_i | \theta) \\
\theta_{MAP} &= \underset{\theta}{\textrm{argmax}} \sum_{i} \log P(x_i | \theta) + \underset{\theta}{\textrm{argmax}}  \log P(\theta)
\end{aligned}
\end{equation}

There are several options for prior distribution:

- **Uniform Prior:** This means, we assign equal weights everywhere, on all possible values of the $\theta$. The implication is that the likelihood equivalently weighted by some constants. Being constant, we could be ignored from our MAP equation, as it will not contribute to the maximization. $\theta_{MLE} = \theta_{MAP}$
- **Gaussian Prior:** Depending on the region of the distribution, the probability is high or low, never always the same.
- Others, like Beta distribution etc.

What we could conclude then, **is that MLE is a special case of MAP, where the prior is uniform!**

### Numerical Optimization and the Negative Log-Likelihood
Using $\log$ space in MLE and MAP with following considerations
1. Notice that if we make the assumption that all the data examples are independent, we can no longer practically consider the likelihood itself as it is a product of many probabilities. For example, $p(x) = 0.5$, then product of $(1/2)^1000000000$ is far below machine precision. We cannot work with that directly. We can leverage log-likelihood to overcome this issue.
\[ \log((1/2)^1000000000) = 1000000000*\log(1/2) \approx -301029995.6...\]
This number fits perfectly within even a single precision 32-bit float. Thus, we should consider the log-likelihood, which is
\[ \log P[X|\theta]\]
Since the function $x \rightarrow \log(x)$ is increasing, maximizing the likelihood (e.g. $\underset{\theta}{\text{arg max} P(X|\theta)}$) is the same thing as maximizing the log-likelihood  (e.g. $\underset{\theta}{\text{arg max} \log P(X|\theta)}$) .
We often work with loss functions, where we wish to minimize the loss. We may turn **maximum likelihood** into the **minimization** of a loss by taking $-\log(P(X|\theta))$, which is the **negative log-likelihood**.

To illustrate this, consider the coin flipping problem from before, and pretend that we do not know the closed form solution. We may compute that:
\[ -\log (P(X | \theta)) = -\log(\theta^{n_H}(1-\theta)^{n_T}) = -(n_H\log(\theta)+n_T\log(1-\theta)) \]

```python
# This can be written into code, and freely optimized even for billions of coin flips

import torch
# Set up our data
n_H = 8675309
n_T = 25624
# Initialize our paramteres
theta = torch.tensor(0.5, requires_grad=True)

# Perform gradient descent
lr = 0.00000000001
for iter in range(10):
    loss = -(n_H * torch.log(theta) + n_T * torch.log(1 - theta))
    loss.backward()
    with torch.no_grad():
        theta -= lr * theta.grad
    theta.grad.zero_()
# Check output
theta, n_H / (n_H + n_T)
```

    (tensor(0.5017, requires_grad=True), 0.9970550284664874)

2. The second reason we consider the log-likelihood is the simplified application of calculus rules. As discussed above, due to independence assumptions, most probabilities we encounter in machine learning are products of individual probabilities.
\[ P(X | \theta) = p(x_1|\theta)p(x_2|\theta)\ldotsp(x_n|\theta)\]
This means that if we directly apply the product rule to compute a derivative we get a signficant computations, which require $n(n-1)$ multiplications, along with $n-1$ additions. So it is proportional to **quadratic time** in the inputs! For the negative log-likelihood we have instead:
\[ -\log P(X | \theta) = -(\log(p(x_1|\theta)) + \log(p(x_2|\theta)) + \ldots+ \log(p(x_n|\theta)))\]
which then gives
\[ -\frac{\partial}{\partial \theta} \log(P(X | \theta)) = \frac{1}{P(x_1 | \theta)}\frac{\partial}{\partial \theta} P(x_1 | \theta) + \ldots +  \frac{1}{P(x_n | \theta)}\frac{\partial}{\partial \theta} P(x_n | \theta)  \]
This requires only n divides and n - 1 sums, and thus is **linear time** in the inputs.
3. The third and final reason to consider the negative log-likelihood is the relationship to information theory. This is a rigorous mathematical theory which gives a way to measure the degree of information or randomness in a random variable. The key object of study in that field is the entropy which is: $H(p) = - \sum_{i}p_i\log_2(p_i)$, which measures the randomness of a source. Notice that this is nothing more than the average -log probability, and thus if we take our negative log-likelihood and divide by the number of data examples, we get a relative of entropy known as **cross-entropy**.

### Example 1: Bernoulli model

We perform following steps to estimate the parameters of the model:

1. We observed $N$ IID coin tossing: $D = \{x_1, x_2,  \ldots, x_n\} \;\text{where}\; x_i \in \{0, 1\}$, For example, $D = \{1, 0, 1, \ldots, 0\}$,
2. The model for an instance $x \in D$ is defined as follow:
\begin{equation}
P(x|\theta) = \textrm{Ber}(x|\theta) =  \left\{
	\begin{array}{ll}
		\theta  & \mbox{if } x = 0 \\
		1 - \theta & \mbox{if } x = 0
	\end{array}
\right. \, \Rightarrow \, P(x|\theta) = \theta^x(1-\theta)^{1-x}
\end{equation}
3. **MLE:** We need to maximize the objective function - log likelihood of dataset $D$:
\begin{equation}
\begin{aligned}
L(\theta; D) &= \log P(D|\theta) \\
&= \log \prod_{i}P(x_i|\theta) = \log \prod_{i} \theta^{x_i}(1-\theta)^{1-x_i} \\
&= \log \theta^{\sum_{i}x_i}(1-\theta)^{\sum_i(1-x_i)} \\
&= \log \theta^{n_h}(1-\theta)^{n_t} \\
&= n_h log \theta + n_t \log (1 - \theta) \\
&= n_h log \theta + (N - n_h) \log (1 - \theta)
\end{aligned}
\,\Rightarrow\, s.t. \underset{\theta}{\textrm{argmax}} L(\theta; D)
\end{equation}

4. In order to get max value of this objective function, we take derivatives of $\theta$:
\begin{equation}
\frac{\partial L(\theta; D)}{\partial \theta} = \frac{n_h}{\theta} - \frac{N-n_h}{1 - \theta} = 0 \\ \Downarrow \\ \hat{\theta}_{MLE} = \frac{n_h}{N} = \frac{n_h}{n_h + n_t} \; \text{or} \; \hat{\theta}_{MLE} = \frac{1}{N}\sum_{i} x_i
\end{equation}

**Overfiting Problem:** What if we tossed too few times so that we saw zero head? $\hat{\theta}_{MLE} = 0$. To overcome this problem, we make it more formal:
\begin{equation}
\hat{\theta}_{MLE} = \frac{n_h + n'}{n_h + n_t + n'} \; \text{where} \; n' \text{ is known as the pseudo (imaginary) count.}
\end{equation}

5. ***MAP:*** If we take Beta distribution as the prior of model parameter $\theta$:
\begin{equation}
P(\theta; \alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)}\theta^{\alpha - 1}(1-\theta)^{\beta - 1} = B(\alpha, \beta)\theta^{\alpha - 1}(1-\theta)^{\beta - 1}
\end{equation}
Then, according Bayesian rule:
\begin{equation}
\begin{aligned}
P(\theta|D) &\propto P(D|\theta)P(\theta) \\
&= \underbrace{\theta^{n_h}(1-\theta)^{n_t}}_{P(D|\theta)} \times \underbrace{B(\alpha, \beta)\theta^{\alpha - 1}(1-\theta)^{\beta - 1}}_{P(\theta)} \\
&= B(\alpha, \beta)\theta^{n_h + \alpha - 1}(1-\theta)^{n_t + \beta - 1}
\end{aligned}
\end{equation}
Maximum a posterior (MAP) estimation:
\begin{equation}
\theta_{MAP} = \underset{\theta}{\textrm{argmax}}  \log P(\theta|D) \propto \underset{\theta}{\textrm{argmax}}  \log P(D|\theta)P(\theta)
\end{equation}
According to step 4 to get the corresponding derivatives of $\theta$:
\begin{equation}
\hat{\theta_{MAP}} = \hat{\theta_{Bays}} = \int \theta p(\theta|D)d\theta = C\int \theta \times \theta^{n_h + \alpha - 1}(1-\theta)^{n_t + \beta - 1}d\theta = \frac{n_h + \alpha}{N + \alpha + \beta}
\end{equation}
where $A = \alpha + \beta$ is prior strength, which can be interoperated as the size of an imaginary data set from which we obtain the **pseudo-counts**.

### Example 2: Univariate Normal (Gaussian)
We perform following steps to estimate the parameters of the model:

1. We observed $N$ IID coin tossing: $D = \{x_1, x_2,  \ldots, x_n\}$, For example, $D = \{-0.1, 10, 1, \ldots, 3\}$,
2. The model with parameter $\mu, \delta$ for an instance $x \in D$ is defined as follow:
\begin{equation}
P(x|\mu, \delta) = \frac{1}{\sqrt{2\pi \delta^2}}e^{-\frac{(x-\mu)^2}{2\delta^2}}
\end{equation}
3. **MLE:** We need to maximize the objective function - log likelihood of dataset $D$:
 \begin{equation}
\begin{aligned}
L(\mu, \delta; D) &= log P(D|\mu, \delta) \\
&= \log \prod_{i}P(x_i|\theta) \\
&= \sum_{i} \log P(x_i | \theta) \\
&= \sum_{i} \log \frac{1}{\sqrt{2\pi \delta^2}}e^{-\frac{(x_i-\mu)^2}{2\delta^2}} \\
&= \sum_{i}(-\frac{1}{2} \log (2\pi \delta^2) - \frac{(x_i-\mu)^2}{2\delta^2}) \\
&= -\frac{N}{2} \log (2\pi \delta^2) - \sum_{i}(\frac{(x_i-\mu)^2}{2\delta^2})
\end{aligned}
\,\Rightarrow\, s.t. \underset{\theta}{\textrm{argmax}} L(\theta; D)
\end{equation}

4. In order to get max value of this objective function, we take derivatives of $\mu, \delta^2$:
\begin{equation}
\frac{\partial L}{\partial \mu} = \frac{\sum_{i}(x_i - \mu)}{\delta^2} = 0 \Rightarrow \mu_{MLE} = \frac{\sum_{i}{x_i}}{N} \\
\frac{\partial L}{\partial \delta^2} = -\frac{N}{2\delta^2} + \frac{1}{2\delta^2}\sum_{i}(x_i - \mu)^ = 0 \Rightarrow \delta^2_{MLE} = \frac{1}{N}\sum_{i}(x_i - \mu_{MLE})^2
\end{equation}

5. **MAP:** Similarly, we can assume normal prior for model parameter $\mu$:
\[ P(u) = \frac{1}{\sqrt{2\pi \tau^2}}e^{-\frac{(x-\mu)^2}{2\tau^2}} \]
Maximum a posterior (MAP) estimation:
\begin{equation}
\theta_{MAP} = \underset{\theta}{\textrm{argmax}}  \log P(\mu, \delta |D) \propto \underset{\theta}{\textrm{argmax}}  \log P(D|\mu, \delta)P(\mu)
\end{equation}
According to step 4 to get the corresponding derivatives of $\mu$ and $\delta$:
TODO


# Artificial Neural Networks (ANNs) and Backpropagation

Classify images into ten digits by using an artificial neural network. Let $x \in \Re^{784\times N}$ be a collection of N input images (stacked in columns) and $y \in [0, 1]^{10\times N}$ be the collection of our class predictions made for all images and encoded by so-called one-hot codes meant to approximate class probabilities.

A neural network makes the prediction y from the collection x of images by forward propagation which is defined recursively as follows
$$
\boldsymbol{y}=\boldsymbol{h}_L \quad with \quad \boldsymbol{h}_k=g_k(\boldsymbol{a}_k), \quad \boldsymbol{a}_k=\boldsymbol{W}_k \boldsymbol{h}_{k-1} + \boldsymbol{b}_k \quad and \quad \boldsymbol{h}_0=\boldsymbol{x}
$$
with $L$ the number of layers, $k$ the layer index, $g_k$ a so-called activation function, $\boldsymbol{h}_k$ an array of $N$ hidden feature vectors (also stacked in columns), $\boldsymbol{a}_k$ an array of $N$ vectors of activations (also called potentials), $\boldsymbol{W}_k$ a matrix of synaptic weights and $\boldsymbol{b}_k$ a vector of biases.

We will use a shallow neural network which means $L = 2$. The hidden layer will use ReLU activation function and the output layer will use Softmax.

The matrices $\boldsymbol{W}_k$ and vectors $\boldsymbol{b}_k$ will be learned/estimated from the MNIST training set by using logistic regression. This consists of minimizing the cross-entropy loss, between the collection d of desired one-hot codes ($d_{ij} = 1$ if the j-th image represents the digit i, 0 otherwise) and the prediction $y$, which is defined as follows
$$
E=-\Sigma_{j=1}^{N} \Sigma_{i=1}^{10} d_{ij} \log{y_{ij}}
$$

The optimization will be performed by gradient descent with backpropagation that can be implemented
iteratively, for $t$ = 0, 1, ..., and some initializations $\boldsymbol{W}_k^0$ and $\boldsymbol{b}_k^0$ , as:

```python
import numpy as np
from MNISTtools import load, show
from matplotlib import pyplot as plt
```

## 1. Load dataset

```python
xtrain, ltrain = load(dataset='training', path='dataset/')
```

```python
print(f'xtrain.shape = {xtrain.shape}')
print(f'ltrain.shape = {ltrain.shape}')
print(f'size of training set = {xtrain.shape[1]}')
print(f'feature dimension = {xtrain.shape[0]}')
```

    xtrain.shape = (784, 60000)
    ltrain.shape = (60000,)
    size of training set = 60000
    feature dimension = 784

## 2. Display the image of index 42

```python
print(f'label of index 42 is {ltrain[42]}')
show(xtrain[:,42].reshape((28,28)))
```

## 3. The range and type of `xtrain`

```python
print(f'range of xtrain = from {xtrain.min()} to {xtrain.max()}')
print(f'type of xtrain = {type(xtrain)}')
print(f'dtype of xtrain = {xtrain.dtype}')
```

    range of xtrain = from 0 to 255
    type of xtrain = <class 'numpy.ndarray'>
    dtype of xtrain = uint8

## 4. Normalize `xtrain`
Normalize `xtrain` s.t. it is in the range `[-1,1]` of type `float32`.

```python
def normalize_MNIST_images(x):
    '''
    Inputs:
        x: data
    '''
    x_norm = x.astype(np.float32)
    return x_norm*2/255-1
```

```python
xtrain = normalize_MNIST_images(xtrain)
print(f'min normalized xtrain = {np.min(xtrain)}')
print(f'max normalized xtrain = {np.max(xtrain)}')
```

    min normalized xtrain = -1.0
    max normalized xtrain = 1.0

## 5. One-hot encoding

```python
def label2onehot(lbl):
    '''
    Convert label (n,) to one-hot form (d,n).
    '''
    d = np.zeros((lbl.max() + 1, lbl.size))
    d[lbl, np.arange(lbl.size)] = 1
    return d
```

```python
dtrain = label2onehot(ltrain)
print(f'dtrain shape = {dtrain.shape}')
print(f'{np.where(dtrain[:,42] == 1)}')
print(f'{np.where(dtrain[:,42] == 1) == ltrain[42]}')
```

    dtrain shape = (10, 60000)
    (array([7]),)
    [[ True]]

## 6. One-hot decoding

```python
def onehot2label(d):
    '''
    Inputs:
        d: one-hot encoding labels
    '''
    lbl = d.argmax(axis=0)
    return lbl
```

```python
all(ltrain == onehot2label(dtrain))
```

    True

## 7. The softmax function
$$
y_i=g(\boldsymbol{a})_i=\frac{e^{a_i}}{\Sigma_{j=1}^{10} e^{a_j}} \quad where \quad i\in[1,10]
$$

```python
def softmax(a):
    '''
    Inputs:
        a: data
    '''
    exp_a = np.exp(a - np.max(a, axis=0))
    return exp_a / np.sum(exp_a, axis=0)
```

## 8. $$
\frac{\partial g(\boldsymbol{a})_i}{\partial a_i} = {g(\boldsymbol{a})}_i (1-{g(\boldsymbol{a})_i})
$$

Proof:

$$
\begin{align}
\frac{\partial g(\boldsymbol{a})_i}{\partial a_i} & =
  \frac{\partial}{\partial a_i}\frac{e^{a_i}}{\Sigma_{k=1}^{N} e^{a_k}} \\
& = \frac{e^{a_i}\Sigma_{k=1}^{N}e^{a_k}-e^{a_i}e^{a_i}}{(\Sigma_{k=1}^{N}e^{a_k})^2} \\
& = \frac{e^{a_i}}{\Sigma_{k=1}^{N}e^{a_k}} \frac{\Sigma_{k=1}^{N}e^{a_k}-e^{a_i}}{\Sigma_{k=1}^{N}e^{a_k}} \\
& = {g(\boldsymbol{a})}_i (1-{g(\boldsymbol{a})_i})
\end{align}
$$

## Q9$$
\frac{\partial{g(\boldsymbol{a})_i}}{\partial a_j}=-g(\boldsymbol{a})_i g(\boldsymbol{a})_j \quad for \, j \neq i
$$

Proof:

$$
\begin{align}
\frac{\partial g(\boldsymbol{a})_i}{\partial a_j} & =
  \frac{\partial}{\partial a_j} \frac{e^{a_i}}{\Sigma_{k=1}^{N} e^{a_k}} \\
& = \frac{0-e^{a_i}e^{a_j}}{(\Sigma_{k=1}^{N}e^{a_k})^2} \\
& = -\frac{e^{a_i}}{\Sigma_{k=1}^{N}e^{a_k}} \frac{e^{a_j}}{\Sigma_{k=1}^{N}e^{a_k}} \\
& = -{g(\boldsymbol{a})}_i g(\boldsymbol{a})_j
\end{align}
$$

## 10. Compute the $\delta$
$$
\delta = g(a)\otimes e-<g(a),e>g(a)
$$

```python
def softmaxp(a, e):
    '''
    Compute delta for the backpropagation.

    Inputs:
        a: predictions before softmax
        e: onehot labels

    Returns:
        delta
    '''
    softmax_a = softmax(a)

    # element-wise product
    e_prod = softmax_a*e

    # dot product
    dot_prod = np.sum(e_prod, axis=0)

    return e_prod - dot_prod*softmax_a
```

# 11. Numerical approximations for $\delta\$
$$
\delta=\frac{\partial g(a)}{\partial a}\times e = \lim_{\epsilon \to 0}\frac{g(a+\epsilon e)-g(a)}{\epsilon}
$$

```python
eps = 1e-6 # finite difference step
a = np.random.randn(10, 200) # random inputs
e = np.random.randn(10, 200) # random directions

diff = softmaxp(a, e)
diff_approx = (softmax(a + eps*e) - softmax(a)) / eps
rel_error = np.abs(diff-diff_approx).mean() / np.abs(diff_approx).mean()
print(rel_error, 'should be smaller than 1e-6')
```

    4.888515969765918e-07 should be smaller than 1e-6

## 12. ReLU and its directional derivative

```python
def relu(a):
    return np.maximum(a, 0)

def relup(a, e):
    '''
    The directional derivative of the ReLU function.
    '''
    eps = 1e-6
    return (relu(a + eps * e) - relu(a)) / eps
```

```python
eps = 1e-6 # finite difference step
a = np.random.randn(10, 200) # random inputs
e = np.random.randn(10, 200) # random directions

diff = relup(a, e)
diff_approx = (relu(a + eps*e) - relu(a)) / eps
rel_error = np.abs(diff-diff_approx).mean() / np.abs(diff_approx).mean()
print(rel_error, 'should be smaller than 1e-6')
```

    0.0 should be smaller than 1e-6

## 13. Initialization of the net
Utilize `He and Xavier initializations`.

```python
def init_shallow(Ni, Nh, No):
    '''
    Inputs:
        Ni: dimension of input layer
        Nh: dimension of output layer
        No: number of unit of output layer

    Returns:
        parameters of the net
    '''
    b1 = np.random.randn(Nh, 1) / np.sqrt((Ni+1.)/2.)
    W1 = np.random.randn(Nh, Ni) / np.sqrt((Ni+1.)/2.)
    b2 = np.random.randn(No, 1) / np.sqrt((Nh+1.))
    W2 = np.random.randn(No, Nh) / np.sqrt((Nh+1.))
    return W1, b1, W2, b2
```

```python
Ni = xtrain.shape[0] # 784
Nh = 64
No = dtrain.shape[0] # 10
netinit = init_shallow(Ni, Nh, No)
```

## 14. Foward propagation

```python
def forwardprop_shallow(x, net):
    '''
    Inputs:
        x: data
        net: parameters of the net

    Returns:
        prediction
    '''
    W1 = net[0] # (64, 784)
    b1 = net[1] # (64, 1)
    W2 = net[2] # (10, 64)
    b2 = net[3] # (10, 1)

    a1 = W1.dot(x) + b1 # (64, 60000)
    h1 = relu(a1) # (64, 60000)

    a2 = W2.dot(h1) + b2 # (10, 60000)
    y = softmax(a2) # (10, 60000)
    return y
```

```python
yinit = forwardprop_shallow(xtrain, netinit)
```

## 15. Loss

```python
def eval_loss(y, d):
    '''
    Inputs:
        y (10, 60000): prediction
        d (10, 60000): ground truth

    Returns:
        Average cross-entropy loss (1, 1)
    '''
    return -np.sum(d*np.log(y))/y.shape[0]/y.shape[1]
```

```python
print(eval_loss(yinit, dtrain), 'should be around .26')
```

    0.26297468338637725 should be around .26

## 16. Error rate

```python
def eval_perfs(y, lbl):
    '''
    Compute the percentage of misclassified samples.

    Inputs:
        y (10, 60000): prediction
        lbl (10, 60000): ground truth

    Returns:
        Error rate
    '''
    return np.sum(onehot2label(y)!=lbl)/lbl.size
```

```python
print(eval_perfs(yinit, ltrain))
```

    0.8715

## 17. Update the parameters
Perform one backpropagation update for the network.

$$
E=-\Sigma_{j=1}^{N} \Sigma_{i=1}^{10} d_{ij} \log{y_{ij}} \\
(\nabla_y E)_i=-\frac{d_i}{y_i}
$$

```python
def update_shallow(x, d, net, gamma=.05):
    '''
    Inputs:
        x: data
        d: ground truth
        net: parameters of the net
        gamma: learning rate

    Returns:
        parameters of the net
    '''
    W1 = net[0]
    b1 = net[1]
    W2 = net[2]
    b2 = net[3]
    Ni = W1.shape[1]
    Nh = W1.shape[0]
    No = W2.shape[0]
    gamma = gamma / x.shape[1] # normalized by the training dataset size

    a1 = W1.dot(x) + b1
    h1 = relu(a1)
    a2 = W2.dot(h1) + b2
    y = softmax(a2)

    # derivative of cross-entropy loss
    e = -d/y + (1-d)/(1-y)

    # backprop
    delta2 = softmaxp(a2, e)
    delta1 = relup(a1, W2.T.dot(delta2))

    W2 = W2 - gamma * delta2.dot(h1.T)
    W1 = W1 - gamma * delta1.dot(x.T)
    b2 = b2 - gamma * delta2.sum(axis=1, keepdims=True)
    b1 = b1 - gamma * delta1.sum(axis=1, keepdims=True)

    return W1, b1, W2, b2
```

## 18. Backpropagation

```python
def backprop_shallow(x, d, net, T, gamma=.05):
    '''
    Inputs:
        x: data
        d: ground truth
        net: parameters of the net
        T: number of updates
        gamma: learning rate

    Returns:
        parameters of the net
    '''
    lbl = onehot2label(d)
    for t in range(T):
        # UPDATE NET
        y = forwardprop_shallow(x, net)
        # DISPLAY LOSS AND PERFS
        net = update_shallow(x, d, net, gamma)
        print(f'\niter = {t+1}')
        print(f'loss = {eval_loss(y, d):.4f}')
        print(f'error rate = {eval_perfs(y, lbl):.4f}')
    return net
```

```python
print('\niter_max = 100\n')
nettrain = backprop_shallow(xtrain, dtrain, netinit, 100)
```

    iter_max = 100

    iter = 1
    loss = 0.2630
    error rate = 0.8715

    iter = 2
    loss = 0.2224
    error rate = 0.8073

    iter = 3
    loss = 0.2094
    error rate = 0.7125

    iter = 4
    loss = 0.2003
    error rate = 0.6544

    iter = 5
    loss = 0.1919
    error rate = 0.5986

    iter = 6
    loss = 0.1840
    error rate = 0.5502

    iter = 7
    loss = 0.1765
    error rate = 0.5087

    iter = 8
    loss = 0.1695
    error rate = 0.4803

    iter = 9
    loss = 0.1634
    error rate = 0.4573

    iter = 10
    loss = 0.1589
    error rate = 0.4576

    iter = 11
    loss = 0.1629
    error rate = 0.4871

    iter = 12
    loss = 0.1730
    error rate = 0.5871

    iter = 13
    loss = 0.1893
    error rate = 0.5526

    iter = 14
    loss = 0.1635
    error rate = 0.5360

    iter = 15
    loss = 0.1357
    error rate = 0.3711

    iter = 16
    loss = 0.1290
    error rate = 0.3540

    iter = 17
    loss = 0.1291
    error rate = 0.3766

    iter = 18
    loss = 0.1290
    error rate = 0.3786

    iter = 19
    loss = 0.1385
    error rate = 0.4335

    iter = 20
    loss = 0.1280
    error rate = 0.3814

    iter = 21
    loss = 0.1240
    error rate = 0.3857

    iter = 22
    loss = 0.1174
    error rate = 0.3435

    iter = 23
    loss = 0.1181
    error rate = 0.3681

    iter = 24
    loss = 0.1080
    error rate = 0.3079

    iter = 25
    loss = 0.1064
    error rate = 0.3271

    iter = 26
    loss = 0.1007
    error rate = 0.2932

    iter = 27
    loss = 0.1031
    error rate = 0.3241

    iter = 28
    loss = 0.0951
    error rate = 0.2821

    iter = 29
    loss = 0.0975
    error rate = 0.3120

    iter = 30
    loss = 0.0922
    error rate = 0.2815

    iter = 31
    loss = 0.0948
    error rate = 0.3071

    iter = 32
    loss = 0.0892
    error rate = 0.2742

    iter = 33
    loss = 0.0898
    error rate = 0.2942

    iter = 34
    loss = 0.0861
    error rate = 0.2661

    iter = 35
    loss = 0.0865
    error rate = 0.2833

    iter = 36
    loss = 0.0826
    error rate = 0.2547

    iter = 37
    loss = 0.0824
    error rate = 0.2675

    iter = 38
    loss = 0.0795
    error rate = 0.2461

    iter = 39
    loss = 0.0792
    error rate = 0.2559

    iter = 40
    loss = 0.0768
    error rate = 0.2387

    iter = 41
    loss = 0.0761
    error rate = 0.2450

    iter = 42
    loss = 0.0747
    error rate = 0.2339

    iter = 43
    loss = 0.0735
    error rate = 0.2349

    iter = 44
    loss = 0.0730
    error rate = 0.2298

    iter = 45
    loss = 0.0710
    error rate = 0.2251

    iter = 46
    loss = 0.0714
    error rate = 0.2264

    iter = 47
    loss = 0.0688
    error rate = 0.2159

    iter = 48
    loss = 0.0700
    error rate = 0.2228

    iter = 49
    loss = 0.0670
    error rate = 0.2084

    iter = 50
    loss = 0.0687
    error rate = 0.2192

    iter = 51
    loss = 0.0654
    error rate = 0.2009

    iter = 52
    loss = 0.0673
    error rate = 0.2145

    iter = 53
    loss = 0.0643
    error rate = 0.1947

    iter = 54
    loss = 0.0660
    error rate = 0.2091

    iter = 55
    loss = 0.0633
    error rate = 0.1913

    iter = 56
    loss = 0.0646
    error rate = 0.2040

    iter = 57
    loss = 0.0624
    error rate = 0.1885

    iter = 58
    loss = 0.0632
    error rate = 0.1981

    iter = 59
    loss = 0.0614
    error rate = 0.1844

    iter = 60
    loss = 0.0617
    error rate = 0.1912

    iter = 61
    loss = 0.0602
    error rate = 0.1805

    iter = 62
    loss = 0.0601
    error rate = 0.1847

    iter = 63
    loss = 0.0587
    error rate = 0.1757

    iter = 64
    loss = 0.0585
    error rate = 0.1787

    iter = 65
    loss = 0.0572
    error rate = 0.1693

    iter = 66
    loss = 0.0570
    error rate = 0.1725

    iter = 67
    loss = 0.0557
    error rate = 0.1630

    iter = 68
    loss = 0.0555
    error rate = 0.1662

    iter = 69
    loss = 0.0542
    error rate = 0.1572

    iter = 70
    loss = 0.0541
    error rate = 0.1605

    iter = 71
    loss = 0.0529
    error rate = 0.1530

    iter = 72
    loss = 0.0531
    error rate = 0.1573

    iter = 73
    loss = 0.0519
    error rate = 0.1515

    iter = 74
    loss = 0.0529
    error rate = 0.1585

    iter = 75
    loss = 0.0519
    error rate = 0.1555

    iter = 76
    loss = 0.0541
    error rate = 0.1708

    iter = 77
    loss = 0.0540
    error rate = 0.1713

    iter = 78
    loss = 0.0566
    error rate = 0.1869

    iter = 79
    loss = 0.0578
    error rate = 0.1914

    iter = 80
    loss = 0.0568
    error rate = 0.1855

    iter = 81
    loss = 0.0580
    error rate = 0.1931

    iter = 82
    loss = 0.0551
    error rate = 0.1737

    iter = 83
    loss = 0.0554
    error rate = 0.1807

    iter = 84
    loss = 0.0527
    error rate = 0.1605

    iter = 85
    loss = 0.0525
    error rate = 0.1659

    iter = 86
    loss = 0.0506
    error rate = 0.1497

    iter = 87
    loss = 0.0503
    error rate = 0.1547

    iter = 88
    loss = 0.0491
    error rate = 0.1431

    iter = 89
    loss = 0.0487
    error rate = 0.1477

    iter = 90
    loss = 0.0479
    error rate = 0.1386

    iter = 91
    loss = 0.0476
    error rate = 0.1432

    iter = 92
    loss = 0.0471
    error rate = 0.1365

    iter = 93
    loss = 0.0469
    error rate = 0.1402

    iter = 94
    loss = 0.0465
    error rate = 0.1349

    iter = 95
    loss = 0.0463
    error rate = 0.1384

    iter = 96
    loss = 0.0461
    error rate = 0.1338

    iter = 97
    loss = 0.0459
    error rate = 0.1369

    iter = 98
    loss = 0.0458
    error rate = 0.1333

    iter = 99
    loss = 0.0455
    error rate = 0.1362

    iter = 100
    loss = 0.0455
    error rate = 0.1331

## 19.  Evaluate the performance on the testing dataset

```python
xtest, ltest = load(dataset='testing', path='dataset/')
print(f'xtest.shape = {xtest.shape}')
print(f'ltest.shape = {ltest.shape}')
```

    xtest.shape = (784, 10000)
    ltest.shape = (10000,)

```python
# use trained network to evaluate the performance on testing data
xtest = normalize_MNIST_images(xtest)
pred_test = forwardprop_shallow(xtest, nettrain)
print(f'error rate = {eval_perfs(pred_test, ltest):.4f}')
```

    error rate = 0.1286

## 20. Backpropagation based on minibatch gradient descent

```python
def backprop_minibatch_shallow(x, d, net, T, B=100, gamma=.05):
    '''
    Inputs:
        x: data
        d: ground truth
        net: parameters of net
        T: number of epoch
        B: minibatches
        gamma: learning rate

    Returns:
        parameters of net
    '''
    N = x.shape[1]
    NB = int((N+B-1)/B)
    lbl = onehot2label(d)
    for t in range(T): # epoch
        shuffled_indices = np.random.permutation(range(N))
        for l in range(NB):
            minibatch_indices = shuffled_indices[B*l:min(B*(l+1), N)]
            # UPDATE NET
            net = update_shallow(x[:,minibatch_indices], d[:,minibatch_indices], net, gamma)
        y = forwardprop_shallow(x, net)
        # DISPLAY LOSS AND PERFS
        print(f'\nepoch = {t+1}')
        print(f'loss = {eval_loss(y, d):.4f}')
        print(f'error rate = {eval_perfs(y, lbl):.4f}')
    return net
```

## 21. Evaluate the performance on the testing dataset based on network with minibatch

```python
netminibatch = backprop_minibatch_shallow(xtrain, dtrain, netinit, 5, B=100)
```

    epoch = 1
    loss = 0.0267
    error rate = 0.0790

    epoch = 2
    loss = 0.0205
    error rate = 0.0584

    epoch = 3
    loss = 0.0191
    error rate = 0.0574

    epoch = 4
    loss = 0.0148
    error rate = 0.0432

    epoch = 5
    loss = 0.0124
    error rate = 0.0361

```python
# use trained network to evaluate the performance on testing data
pred_test_mini = forwardprop_shallow(xtest, netminibatch)
print(f'error rate = {eval_perfs(pred_test_mini, ltest):.4f}')
```

    error rate = 0.0399


# The geometry of linear equation

Use matrix form to study linear equations and relative pircture of the matrix

For example:

\begin{align}
2x &-y &=0 \\
-x &+2y &=3
\end{align}

It can be rapresented as a matrix  2 rows X 2 colums:

$$
\begin{bmatrix}
2 & -1 \\
-1 & 2
\end{bmatrix} \circ
\begin{bmatrix}
x \\
y
\end{bmatrix} =
\begin{bmatrix}
0 \\
3
\end{bmatrix}
$$

I can not resist writing the matrix form that can be called at this point a **Linear Equation**

$$ A \cdot X = b $$

Let's write the **Row Picture** of the two equation and the solution in the poiunt *P(1,2)* as the intersection of the the lines.

```python
import numpy as np
import matplotlib.pyplot as plt
plt.xlim(-5, 5)
plt.ylim(-5, 5)
x = np.linspace(-5,5, 10)
plt.quiver([0, -5], [-5, 0], [0, 10], [10, 0], angles='xy', scale_units='xy', scale=1)
plt.plot(x, 2 * x, 'r-', label="2x - y = 0")
plt.plot(x, 0.5 * x + 1.5, 'b-', label = "-x + 2y = 3")
plt.scatter(1,2, c='blue', marker='o')
plt.annotate('P (1,2)', xy=(1, 2), xytext=(1, 1.5))
plt.legend()
plt.show()
```

Now let's weite the column picture:

$$
X
\begin{bmatrix}
2  \\
-1
\end{bmatrix} +
Y
\begin{bmatrix}
-1 \\
2
\end{bmatrix} =
\begin{bmatrix}
0 \\
3
\end{bmatrix}
$$

And this form is called **Lienar combination of columns** Let's write the **Column Picture**

```python
plt.close()
plt.xlim(-3, 3)
plt.ylim(-3, 3)
plt.quiver([0, -3], [-3, 0], [0, 6], [6, 0], angles='xy', scale_units='xy', scale=1)
plt.quiver([0, 0], [0, 0], [2, -1], [-1, 2], color='b',angles='xy', scale_units='xy', scale=1)
plt.show()
```

```python

```