# Kalman Filter

## Introduction

Kalman filtering is an algorithm that provides estimates of some unknown variables given the measurements observed over time. Kalman filters have been demonstrating its usefulness in various applications. Kalman filters have relatively simple form and require small computational power

## Problem definition

Kalman filters are used to estimate states based on linear dynamical systems in
state space format. The Kalman filter represents beliefs by the moments parameterization: At time $t$, the belief is represented by the the mean $\mu_t$ and the covariance $\Sigma_t$. The process model defines the evolution of the state from time $t - 1$ to time $t$. The state transition probability $p(x_t | u_t, x_{t−1})$ must be a linear function in its arguments with added Gaussian noise. This is expressed by the following equation:

$x_t = A_tx_{t−1} + B_tu_t + \varepsilon_t$

Here $x_t$ and $x_{t−1}$ are state vectors, and ut is the control vector at time t.
In our notation, both of these vectors are vertical vectors. They are of the
form

$x_{t}^{n} = \begin{pmatrix} \\ x_{t}^{1}\\ x_{t}^{2}\\ \vdots \\ x_{t}^{n}\\ \end{pmatrix}$ and $u_{t}^{m} = \begin{pmatrix} \\ u_{t}^{1}\\ u_{t}^{2}\\ \vdots \\ u_{t}^{m}\\ \end{pmatrix}$

where $A_t$ is the state transition matrix applied to the previous state vector $x_{t−1}$, $A_t$ is a square matrix of size $n \times n$, where $n$ is
the dimension of the state vector $x_t$. $B_t$ is the control-input matrix applied to the control vector $u_{k}$, $B_t$ have a size $n \times m$, with $m$ being the dimension of the control vector $u_t$.  and $\varepsilon_t$ is the process noise vector that is assumed to be zero-mean Gaussian with the covariance $R_t$, $\varepsilon_t \sim 𝒩(0,R)$.

The measurement probability $p(z_t | x_t)$ must also be linear in its arguments, with added Gaussian noise. The process model is paired with the measurement model that describes the relationship between the state and the measurement at the current time step t as:

$z_t = C_tx_t + \delta_t$

where $z_t$ is the measurement vector, $C_t$ is the measurement matrix, $C_t$ is a matrix of size $k \times n$, where $k$ is the dimension of the measurement vector $z_t$. The $\delta_t$ is the measurement noise vector that is assumed to be zero-mean Gaussian with the covariance $Q_t$ , $\delta_t \sim 𝒩(0,Q)$.

The role of the Kalman filter is to provide estimate of $x_t$ at time $t$, given the initial estimate of $x_0$ , the series of measurement, $z_1,z_2,…,z_t$ , and the information of the system described by $A_t$ , $B_t$ , $C_t$ , $Q$, and $R$. Note that subscripts to these matrices are omitted here by assuming that they are invariant over time as in most applications. Although the covariance matrices are supposed to reflect the statistics of the noises, the true statistics of the noises is not known or not Gaussian in many practical applications. Therefore, Q and R are usually used as tuning parameters that the user can adjust to get desired performance.

## Pseudocode

$1: Algorithm Kalmanfilter(μt−1, Σt−1, ut, zt):$

$2:    \bar{\mu}_t = A_t \mu_{t−1} + B_t u_t$

$3:    \bar{\Sigma}_t = A_t \Sigma_{t−1} A^T_t + R_t$

$4:    K_t = \bar{\Sigma}_t C^T_t (C_t \Sigma_t C^T_t + Q_t)^{−1}$

$5:    \mu_t = \bar{\mu}_t + K_t(z_t − C_t \bar{\mu}_t)$

$6:    \Sigma_t = (I − K_t C_t)\bar{\Sigma}_t$

$7:    return (\mu_t, \Sigma_t)$

## Summary

### Prediction:

| Description                | Representation in the pseudocodigo|
|----------------------------|-------------------------------------------------------|
| Predicted state estimate   | $\bar{\mu} _t = A_t \mu_ {t−1}  + B_t u_t$            |
| Predicted error covariance | $\bar{\Sigma}  _t = A_t \Sigma_  {t−1}   A^T_t + R_t$ |

### Update:

| Description              | Representation in the pseudocodigo |
|--------------------------|----------------------------------------------------------------|
| Measurement residual     | $(z_t − C_t \bar{\mu} _t)$                                     |
| Kalman gain              | $K_t = \bar{\Sigma} _t C^T_t (C_t \Sigma_t C^T_t + Q_t)^{−1} $ |
| Updated state estimate   | $\mu_t = \bar{\mu} _t + K_t(z_t − C_t \bar{\mu} _t)$           |
| Updated error covariance | $\Sigma_t = (I − K_t C_t)\bar{\Sigma} _t$                      |

## Kalman Filter for Sensor Fusion

## The Kalman Filter 1-D

Kalman filters are discrete systems that allows us to define a dependent variable by an independent variable, where by we will solve for the independent variable so that when we are given measurements (the dependent variable),we can infer an estimate     of the independent variable assuming that noise exists from our input measurement and noise also exists in how we’ve modeled the world with our math equations because of inevitably unaccounted for factors in the non-sterile world.Input variables become more valuable when modeled as a system of equations,ora  matrix, in order to make it possible to determine the relationships between those values. Every variables in every dimension will contain noise, and therefore the introduction of related inputs will allow weighted averaging to take place based on the predicted differential at the next step, the noise unaccounted for in the system,and the noise introduced by the sensor inputs.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.mlab as mlab
import seaborn as sb
from scipy import stats
import time
from numpy.linalg import inv
import scipy.stats as scs
```

```python
%matplotlib inline
fw = 10 # figure width
```

#### Plot the Distributions in this range:

```python
x = np.linspace(-100,100,1000)
```

```python
mean0 = 0.0   # e.g. meters or miles
var0  = 20.0
```

```python
plt.figure(figsize=(fw,5))
plt.plot(x, scs.norm.pdf(x, mean0, var0), 'b', label='Normal Distribution')
plt.ylim(0, 0.1);
plt.legend(loc='best');
plt.xlabel('Position');
```

## Now we have something, which estimates the moved distance

#### The Mean is  meters, calculated from velocity*dt or step counter or wheel encoder ...

#### VarMove is the Estimated or determined with static measurements

```python
meanMove = 25.0
varMove  = 10.0
```

```python
plt.figure(figsize=(fw,5))
plt.plot(x,scs.norm.pdf(x, meanMove, varMove), 'r', label='Normal Distribution')
plt.ylim(0, 0.1);
plt.legend(loc='best');
plt.xlabel('Distance moved');
```

Both Distributions have to be merged together
$\mu_\text{new}=\mu_\text{0}+\mu_\text{move}$ is the new mean and $\sigma^2_\text{new}=\sigma^2_\text{0}+\sigma^2_\text{move}$ is the new variance.

```python
def predict(var, mean, varMove, meanMove):
    new_var = var + varMove
    new_mean= mean+ meanMove
    return new_var, new_mean
```

```python
new_var, new_mean = predict(var0, mean0, varMove, meanMove)
```

```python
plt.figure(figsize=(fw,5))
plt.plot(x,scs.norm.pdf(x, mean0, var0), 'b', label='Beginning Normal Distribution')
plt.plot(x,scs.norm.pdf(x, meanMove, varMove), 'r', label='Movement Normal Distribution')
plt.plot(x,scs.norm.pdf(x, new_mean, new_var), 'g', label='Resulting Normal Distribution')
plt.ylim(0, 0.1);
plt.legend(loc='best');
plt.title('Normal Distributions of 1st Kalman Filter Prediction Step');
plt.savefig('Kalman-Filter-1D-Step.png', dpi=150)
```

### What you see: The resulting distribution is flat > uncertain.

The more often you run the predict step, the flatter the distribution get

First Sensor Measurement (Position) is coming in...
#### Sensor Defaults for Position Measurements
(Estimated or determined with static measurements)

```python
meanSensor = 25.0
varSensor  = 12.0
```

```python
plt.figure(figsize=(fw,5))
plt.plot(x,scs.norm.pdf(x, meanSensor, varSensor), 'c')
plt.ylim(0, 0.1);
```

Now both Distributions have to be merged together
$\sigma^2_\text{new}=\cfrac{1}{\cfrac{1}{\sigma^2_\text{old}}+\cfrac{1}{\sigma^2_\text{Sensor}}}$ is the new variance and the new mean value is $\mu_\text{new}=\cfrac{\sigma^2_\text{Sensor} \cdot \mu_\text{old} + \sigma^2_\text{old} \cdot \mu_\text{Sensor}}{\sigma^2_\text{old}+\sigma^2_\text{Sensor}}$

```python
def correct(var, mean, varSensor, meanSensor):
    new_mean=(varSensor*mean + var*meanSensor) / (var+varSensor)
    new_var = 1/(1/var +1/varSensor)
    return new_var, new_mean
```

```python
var, mean = correct(new_var, new_mean, varSensor, meanSensor)
```

```python
plt.figure(figsize=(fw,5))
plt.plot(x,scs.norm.pdf(x, new_mean, new_var), 'g', label='Beginning (after Predict)')
plt.plot(x,scs.norm.pdf(x, meanSensor, varSensor), 'c', label='Position Sensor Normal Distribution')
plt.plot(x,scs.norm.pdf(x, mean, var), 'm', label='New Position Normal Distribution')
plt.ylim(0, 0.1);
plt.legend(loc='best');
plt.title('Normal Distributions of 1st Kalman Filter Update Step');
```

###### This is called the Measurement or Correction step! The Filter get's more serious about the actual state.

#### Let's put everything together: The 1D Kalman Filter
"Kalman-Filter: Predicting the Future since 1960"

Let's say, we have some measurements for position and for distance traveled. Both have to be fused with the 1D-Kalman Filter.

```python
positions = (10, 20, 30, 40, 50)+np.random.randn(5)
distances = (10, 10, 10, 10, 10)+np.random.randn(5)
```

```python
positions
```

    array([10.3266042 , 20.89387408, 29.3947916 , 39.875685  , 48.93473369])

```python
distances
```

    array([ 9.07429798, 10.3234591 , 11.58789189,  9.57728332, 11.52106393])

```python
for m in range(len(positions)):

    # Predict
    var, mean = predict(var, mean, varMove, distances[m])
    #print('mean: %.2f\tvar:%.2f' % (mean, var))
    plt.plot(x,scs.norm.pdf(x, mean, var), label='%i. step (Prediction)' % (m+1))

    # Correct
    var, mean = correct(var, mean, varSensor, positions[m])
    print('After correction:  mean= %.2f\tvar= %.2f' % (mean, var))
    plt.plot(x,scs.norm.pdf(x, mean, var), label='%i. step (Correction)' % (m+1))

plt.ylim(0, 0.1);
plt.xlim(-20, 120)
plt.legend();
```

The sensors are represented as normal distributions with their parameters ($\mu$ and $\sigma^2$) and are calculated together with addition or convolution. The prediction decreases the certainty about the state, the correction increases the certainty.

Prediction: Certainty $\downarrow$
Correction: Certainty $\uparrow$

## Kalman Filter - Multi-Dimensional Measurement

### Kalman Filter Implementation for Constant Velocity Model (CV) in Python

Situation covered: You drive with your car in a tunnel and the GPS signal is lost. Now the car has to determine, where it is in the tunnel. The only information it has, is the velocity in driving direction. The x and y component of the velocity ($\dot x$ and $\dot y$) can be calculated from the absolute velocity (revolutions of the wheels) and the heading of the vehicle (yaw rate sensor).

First, we have to initialize the matrices and vectors. Setting up the math.

## State Vector

Constant Velocity Model for Ego Motion

$$x_t= \left[ \matrix{ x \\ y \\ \dot x \\ \dot y} \right] = \matrix{ \text{Position x} \\ \text{Position y} \\ \text{Velocity in x} \\ \text{Velocity in y}}$$

Formal Definition (Motion of Law):

$$x_{t} = \textbf{$A_t$} \cdot x_{t-1}$$

which is

$$x_{t} = \begin{bmatrix}1 & 0 & \Delta t & 0 \\ 0 & 1 & 0 & \Delta t \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix} \cdot \begin{bmatrix} x \\ y \\ \dot x \\ \dot y \end{bmatrix}_{t-1}$$

Observation Model:

$$z_t = \textbf{$C_t$}\cdot x_t$$

which is

$$z_t = \begin{bmatrix}0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1\end{bmatrix} \cdot x_t$$ means: You observe the velocity directly in the correct unit

### Initial State $x_0$

$$x_{0} = \begin{bmatrix}0 \\ 0 \\ 0 \\ 0\end{bmatrix}$$

```python
x = np.matrix([[0.0, 0.0, 0.0, 0.0]]).T
print(x, x.shape)
plt.scatter(float(x[0]),float(x[1]), s=100)
plt.title('Initial Location')
```

### Covariance Matrix $P_0$ ($\Sigma_0$)

An uncertainty must be given for the initial state $x_0$ . In the 1D case, the $\mu_0$ , now a matrix, defines an initial uncertainty for all states.

This matrix is most likely to be changed during the filter passes. It is changed in both the Predict and Correct steps. If one is quite sure about the states at the beginning, one can use low values here, if one does not know exactly how the values of the state vector are, the covariance matrix should be initialized with very large values (1 million or so) to allow the filter to converge relatively quickly (find the right values based on the measurements).

$$P_{0} = \begin{bmatrix}\sigma^2_x & 0 & 0 & 0 \\ 0 & \sigma^2_y & 0 & 0 \\ 0 & 0 & \sigma^2_{\dot x} & 0 \\ 0 & 0 & 0 & \sigma^2_{\dot y} \end{bmatrix}$$

with $\sigma$ as the standard deviation

```python
P = np.diag([1000.0, 1000.0, 1000.0, 1000.0])
print(P, P.shape)
```

    [[1000.    0.    0.    0.]
     [   0. 1000.    0.    0.]
     [   0.    0. 1000.    0.]
     [   0.    0.    0. 1000.]] (4, 4)

```python
fig = plt.figure(figsize=(6, 6))
im = plt.imshow(P, interpolation="none", cmap=plt.get_cmap('binary'))
plt.title('Initial Covariance Matrix $P$')
ylocs, ylabels = plt.yticks()
# set the locations of the yticks
plt.yticks(np.arange(7))
# set the locations and labels of the yticks
plt.yticks(np.arange(6),('$x$', '$y$', '$\dot x$', '$\dot y$'), fontsize=22)

xlocs, xlabels = plt.xticks()
# set the locations of the yticks
plt.xticks(np.arange(7))
# set the locations and labels of the yticks
plt.xticks(np.arange(6),('$x$', '$y$', '$\dot x$', '$\dot y$'), fontsize=22)

plt.xlim([-0.5,3.5])
plt.ylim([3.5, -0.5])

from mpl_toolkits.axes_grid1 import make_axes_locatable
divider = make_axes_locatable(plt.gca())
cax = divider.append_axes("right", "5%", pad="3%")
plt.colorbar(im, cax=cax);
```

### Dynamic Matrix $A$

It is calculated from the dynamics of the Egomotion.

$$x_{t} = x_{t-1} + \dot x_{t-1} \cdot \Delta t$$
$$y_{t} = y_{t} + \dot y_{t-1} \cdot \Delta t$$
$$\dot x_{t} = \dot x_{t-1}$$
$$\dot y_{t} = \dot y_{t-1}$$

```python
dt = 0.1 # Time Step between Filter Steps

A = np.matrix([[1.0, 0.0, dt, 0.0],
              [0.0, 1.0, 0.0, dt],
              [0.0, 0.0, 1.0, 0.0],
              [0.0, 0.0, 0.0, 1.0]])
print(A, A.shape)
```

    [[1.  0.  0.1 0. ]
     [0.  1.  0.  0.1]
     [0.  0.  1.  0. ]
     [0.  0.  0.  1. ]] (4, 4)

### Measurement Matrix $C_t$

We directly measure the Velocity $\dot x$ and $\dot y$

$$H = \begin{bmatrix}0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1\end{bmatrix}$$

```python
C = np.matrix([[0.0, 0.0, 1.0, 0.0],
              [0.0, 0.0, 0.0, 1.0]])
print(C, C.shape)
```

    [[0. 0. 1. 0.]
     [0. 0. 0. 1.]] (2, 4)

### Measurement Noise Covariance $Q_t$

Tells the Kalman Filter how 'bad' the sensor readings are.

$$Q_t = \begin{bmatrix}\sigma^2_{\dot x} & 0 \\ 0 & \sigma^2_{\dot y} \end{bmatrix}$$

```python
ra = 10.0**2

Q = np.matrix([[ra, 0.0],
              [0.0, ra]])
print(Q, Q.shape)
```

    [[100.   0.]
     [  0. 100.]] (2, 2)

```python
# Plot between -10 and 10 with .001 steps.
xpdf = np.arange(-10, 10, 0.001)
plt.subplot(121)
plt.plot(xpdf, norm.pdf(xpdf,0,Q[0,0]))
plt.title('$\dot x$')

plt.subplot(122)
plt.plot(xpdf, norm.pdf(xpdf,0,Q[1,1]))
plt.title('$\dot y$')
plt.tight_layout()
```

### Process Noise Covariance $R$

The Position of the car can be influenced by a force (e.g. wind), which leads to an acceleration disturbance (noise). This process noise has to be modeled with the process noise covariance matrix R.

$$R = \begin{bmatrix}\sigma_{x}^2 & \sigma_{xy} & \sigma_{x \dot x} & \sigma_{x \dot y} \\ \sigma_{yx} & \sigma_{y}^2 & \sigma_{y \dot x} & \sigma_{y \dot y} \\ \sigma_{\dot x x} & \sigma_{\dot x y} & \sigma_{\dot x}^2 & \sigma_{\dot x \dot y} \\ \sigma_{\dot y x} & \sigma_{\dot y y} & \sigma_{\dot y \dot x} & \sigma_{\dot y}^2 \end{bmatrix}$$

One can calculate R as

$$R = G\cdot G^T \cdot \sigma_v^2$$

with $G = \begin{bmatrix}0.5dt^2 & 0.5dt^2 & dt & dt\end{bmatrix}^T$ and $\sigma_v$ as the acceleration process noise, which can be assumed for a vehicle to be $8.8m/s^2$, according to: Schubert, R., Adam, C., Obst, M., Mattern, N., Leonhardt, V., & Wanielik, G. (2011). [Empirical evaluation of vehicular models for ego motion estimation](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=5940526). 2011 IEEE Intelligent Vehicles Symposium (IV), 534–539. doi:10.1109/IVS.2011.5940526

```python
sv = 8.8

G = np.matrix([[0.5*dt**2],
               [0.5*dt**2],
               [dt],
               [dt]])

R = G*G.T*sv**2
```

```python
from sympy import Symbol, Matrix
from sympy.interactive import printing
printing.init_printing()
dts = Symbol('dt')
Rs = Matrix([[0.5*dts**2],[0.5*dts**2],[dts],[dts]])
Rs*Rs.T
```

```python
fig = plt.figure(figsize=(6, 6))
im = plt.imshow(R, interpolation="none", cmap=plt.get_cmap('binary'))
plt.title('Process Noise Covariance Matrix $P$')
ylocs, ylabels = plt.yticks()
# set the locations of the yticks
plt.yticks(np.arange(7))
# set the locations and labels of the yticks
plt.yticks(np.arange(6),('$x$', '$y$', '$\dot x$', '$\dot y$'), fontsize=22)

xlocs, xlabels = plt.xticks()
# set the locations of the yticks
plt.xticks(np.arange(7))
# set the locations and labels of the yticks
plt.xticks(np.arange(6),('$x$', '$y$', '$\dot x$', '$\dot y$'), fontsize=22)

plt.xlim([-0.5,3.5])
plt.ylim([3.5, -0.5])

from mpl_toolkits.axes_grid1 import make_axes_locatable
divider = make_axes_locatable(plt.gca())
cax = divider.append_axes("right", "5%", pad="3%")
plt.colorbar(im, cax=cax);
```

### Identity Matrix $I$

```python
I = np.eye(4)
print(I, I.shape)
```

    [[1. 0. 0. 0.]
     [0. 1. 0. 0.]
     [0. 0. 1. 0.]
     [0. 0. 0. 1.]] (4, 4)

## Measurements

For example, we are using some random generated measurement values

```python
m = 200 # Measurements
vx= 20 # in X
vy= 10 # in Y

mx = np.array(vx+np.random.randn(m))
my = np.array(vy+np.random.randn(m))

measurements = np.vstack((mx,my))

print(measurements.shape)

print('Standard Deviation of Acceleration Measurements=%.2f' % np.std(mx))
print('You assumed %.2f in Q.' % Q[0,0])
```

    (2, 200)
    Standard Deviation of Acceleration Measurements=0.96
    You assumed 100.00 in Q.

```python
fig = plt.figure(figsize=(16,5))

plt.step(range(m),mx, label='$\dot x$')
plt.step(range(m),my, label='$\dot y$')
plt.ylabel(r'Velocity $m/s$')
plt.title('Measurements')
plt.legend(loc='best',prop={'size':18})
```

```python
# Preallocation for Plotting
xt = []
yt = []
dxt= []
dyt= []
Zx = []
Zy = []
Px = []
Py = []
Pdx= []
Pdy= []
Rdx= []
Rdy= []
Kx = []
Ky = []
Kdx= []
Kdy= []

def savestates(x, Z, P, Q, K):
    xt.append(float(x[0]))
    yt.append(float(x[1]))
    dxt.append(float(x[2]))
    dyt.append(float(x[3]))
    Zx.append(float(Z[0]))
    Zy.append(float(Z[1]))
    Px.append(float(P[0,0]))
    Py.append(float(P[1,1]))
    Pdx.append(float(P[2,2]))
    Pdy.append(float(P[3,3]))
    Rdx.append(float(Q[0,0]))
    Rdy.append(float(Q[1,1]))
    Kx.append(float(K[0,0]))
    Ky.append(float(K[1,0]))
    Kdx.append(float(K[2,0]))
    Kdy.append(float(K[3,0]))
```

# Kalman Filter

```python
for n in range(len(measurements[0])):

    # Time Update (Prediction)
    # ========================
    # Project the state ahead
    x = A*x

    # Project the error covariance ahead
    P = A*P*A.T + R

    # Measurement Update (Correction)
    # ===============================
    # Compute the Kalman Gain
    S = C*P*C.T + Q
    K = (P*C.T) * np.linalg.pinv(S)

    # Update the estimate via z
    Z = measurements[:,n].reshape(2,1)
    y = Z - (C*x)                            # Innovation or Residual
    x = x + (K*y)

    # Update the error covariance
    P = (I - (K*C))*P

    # Save states (for Plotting)
    savestates(x, Z, P, Q, K)
```

# Let's take a look at the filter performance

### Kalman Gains $K$

```python
def plot_K():
    fig = plt.figure(figsize=(16,9))
    plt.plot(range(len(measurements[0])),Kx, label='Kalman Gain for $x$')
    plt.plot(range(len(measurements[0])),Ky, label='Kalman Gain for $y$')
    plt.plot(range(len(measurements[0])),Kdx, label='Kalman Gain for $\dot x$')
    plt.plot(range(len(measurements[0])),Kdy, label='Kalman Gain for $\dot y$')

    plt.xlabel('Filter Step')
    plt.ylabel('')
    plt.title('Kalman Gain (the lower, the more the measurement fullfill the prediction)')
    plt.legend(loc='best',prop={'size':22})
```

```python
plot_K()
```

### Uncertainty Matrix $P$

```python
def plot_P():
    fig = plt.figure(figsize=(16,9))
    plt.plot(range(len(measurements[0])),Px, label='$x$')
    plt.plot(range(len(measurements[0])),Py, label='$y$')
    plt.plot(range(len(measurements[0])),Pdx, label='$\dot x$')
    plt.plot(range(len(measurements[0])),Pdy, label='$\dot y$')

    plt.xlabel('Filter Step')
    plt.ylabel('')
    plt.title('Uncertainty (Elements from Matrix $P$)')
    plt.legend(loc='best',prop={'size':22})
```

```python
plot_P()
```

### State Estimate $x$

```python
def plot_x():
    fig = plt.figure(figsize=(16,9))
    plt.step(range(len(measurements[0])),dxt, label='$\dot x$')
    plt.step(range(len(measurements[0])),dyt, label='$\dot y$')

    plt.axhline(vx, color='#999999', label='$\dot x_{real}$')
    plt.axhline(vy, color='#999999', label='$\dot y_{real}$')

    plt.xlabel('Filter Step')
    plt.title('Estimate (Elements from State Vector $x$)')
    plt.legend(loc='best',prop={'size':22})
    plt.ylim([0, 30])
    plt.ylabel('Velocity')
```

```python
plot_x()
```

## Position x/y

```python
def plot_xy():
    fig = plt.figure(figsize=(16,16))
    plt.scatter(xt,yt, s=20, label='State', c='k')
    plt.scatter(xt[0],yt[0], s=100, label='Start', c='g')
    plt.scatter(xt[-1],yt[-1], s=100, label='Goal', c='r')

    plt.xlabel('X')
    plt.ylabel('Y')
    plt.title('Position')
    plt.legend(loc='best')
    plt.axis('equal')
```

```python
plot_xy()
```

```python

```


Demo - Poisson equation 1D
=======================

Solve Poisson equation in 1D with homogeneous Dirichlet bcs

$$
\begin{align}
\nabla^2 u(x) &= f(x), \quad \forall \, x \in [-1, 1]\\
u(\pm 1) &= 0
\end{align}
$$

Use either Chebyshev basis $P=\{T_k(x)\}_{k=0}^{N-1}$ or Legendre $P=\{L_k(x)\}_{k=0}^{N-1}$ and define Shen's composite basis as

$$
V^N = \{P_k - P_{k+2}\, | \, k=0, 1, \ldots, N-3\}.
$$

We also need the weighted inner product

$$
 (u, v)_w = \int_{-1}^1 u v w \, \text{dx},
$$

where $w(x)$ is a weight associated with the chosen basis. For Legendre the weight is simply $w(x)=1$, whereas for Chebyshev it is $w(x)=1/\sqrt{1-x^2}$. The weight ensures that inner products are orthogonal. We use quadrature to solve the integral

$$
\int_{-1}^1 u v w \, \text{dx} = \sum_{i=0}^{N-1} u(x_i) v(x_i) \omega_i,
$$

where $\{\omega_i\}_{i=0}^{N-1}$ are the quadrature weights associated with the chosen basis (C/L) and quadrature rule. For Chebyshev there are `Chebyshev-Gauss` or `Chebyshev-Gauss-Lobatto`, whereas for Legendre there are `Legendre-Gauss` or `Legendre-Gauss-Lobatto`. The quadrature points are denoted as $\{x_i\}_{i=0}^{N-1}$ and their value depends on the chosen basis and quadrature rule.

With the weighted product in place we can create variational problems from the original PDE by multiplying with a test function and integrating over the domain. For a Legendre basis we get:

Find $u \in V^N$ such that

$$     (\nabla u, \nabla v) = -(f, v), \quad \forall \, v \in V^N.$$

For a Chebyshev basis the integration by parts is complicated due to the weight and the variational problem used is instead:

Find $u \in V^N$ such that

$$     (\nabla^2 u, v)_w = (f, v)_w, \quad \forall \, v \in V^N.$$

We now break the problem down to linear algebra. With any choice of basis or quadrature rule we use $\phi_k(x)$ to represent the test function $v$ and thus

$$
\begin{align}
v(x) &= \phi_k(x) \\
u(x) &= \sum_{j=0}^{N-3} \hat{u}_j \phi_j(x)
\end{align}
$$

Insert into the variational problem for Legendre

$$
\begin{align}
(\nabla \sum_{j=0}^{N-3} \hat{u}_j \phi_j, \nabla \phi_k) &= -(f, \phi_k) \\
\sum_{j=0}^{N-3} \underbrace{(\nabla \phi_j, \nabla \phi_k)}_{a_{kj}} \hat{u}_j  &= -\underbrace{(f, \phi_k)}_{\tilde{f}_k} \\
A \boldsymbol{\hat{u}} &= -\boldsymbol{\tilde{f}}
\end{align}
$$

where $A = (a_{kj})_{0 \ge k, j \ge N-3}$ is the stiffness matrix, $\hat{\boldsymbol{u}} = \{\hat{u}_j\}_{j=0}^{N-3}$ and $\boldsymbol{\tilde{f}} = \{\tilde{f}_k\}_{k=0}^{N-3}$.

```python
from shenfun import *
import matplotlib.pyplot as plt

N = 100
V = Basis(N, 'Legendre', quad='LG', bc=(0, 0))
x, w = V.points_and_weights()
v = TestFunction(V)
u = TrialFunction(V)
A = inner(grad(u), grad(v))
```

Use a manufactured solution to create a right hand side $f(x)$

```python
import sympy
x = sympy.symbols('x')
u = (1-x**2)*(sympy.cos(4*x)*sympy.sin(6*x))
f = u.diff(x, 2)
```

```python
fl = sympy.lambdify(x, f, 'numpy')
fj = Array(V)
```

```python
fj[:] = fl(V.mesh())
```

```python
f_tilde = inner(v, -fj)
```

```python
u_hat = Function(V)
u_hat = A.solve(f_tilde, u=u_hat)
```

```python
uj = u_hat.backward()
```

```python
ul = sympy.lambdify(x, u, 'numpy')
ue = ul(V.mesh())
```

```python
T = Basis(N, 'L', quad='LG')
```

```python
du = project(Dx(u_hat, 0, 1), T)
```


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats
```

```python
n_params = [1, 2, 4]  # Number of trials
p_params = [0.25, 0.5, 0.75]  # Probability of success

x = np.arange(0, max(n_params)+1)
f,ax = plt.subplots(len(n_params), len(p_params), sharex=True,
                    sharey=True,
                    figsize=(8, 7), constrained_layout=True)

for i in range(len(n_params)):
    for j in range(len(p_params)):
        n = n_params[i]
        p = p_params[j]

        y = stats.binom(n=n, p=p).pmf(x)

        ax[i,j].vlines(x, 0, y, colors='C0', lw=5)
        ax[i,j].set_ylim(0, 1)
        ax[i,j].plot(0, 0, label="N = {:3.2f}\nθ = {:3.2f}".format(n,p), alpha=0)
        ax[i,j].legend()

        ax[2,1].set_xlabel('y')
        ax[1,0].set_ylabel('p(y | θ, N)')
        ax[0,0].set_xticks(x)

```

### Choosing the prior

As a prior we will use a beta distribution, which is a very common distribution:

\begin{equation}
p\left(\theta\right) = \frac{\Gamma\left(\alpha+\beta\right)}{\Gamma\left(\alpha\right)\Gamma\left(\beta\right)}\theta^(\alpha - 1)\left(1-\theta\right)^{\beta -1}
\end{equation}

The beta distribution is a common choice of prior in Bayesian statistics:

```python
params = [0.5, 1, 2, 3]
x = np.linspace(0, 1, 100)
f, ax = plt.subplots(len(params), len(params), sharex=True,
                     sharey=True,
                     figsize=(8, 7), constrained_layout=True)
for i in range(4):
    for j in range(4):
        a = params[i]
        b = params[j]
        y = stats.beta(a, b).pdf(x)
        ax[i,j].plot(x, y)
        ax[i,j].plot(0, 0, label="α = {:2.1f}\nβ = {:2.1f}".format(a,
                     b), alpha=0)
        ax[i,j].legend()
ax[1,0].set_yticks([])
ax[1,0].set_xticks([0, 0.5, 1])
f.text(0.5, 0.05, 'θ', ha='center')
f.text(0.07, 0.5, 'p(θ)', va='center', rotation=0)
```

```python

```


# Neural Networks Optimized with Matrix Calculation

## Abstract

I implemented a neural network optimized with matrix calculation. Though neural networks can be implemented with many 'for' loops, which may be easy to read, the traininig time may become longer because we need at leas 3 'for' loops. When we implement such a network with 30 epochs, 50000 mnist training data, and mini_batch size 10, the network has to implement 150000 loops in tota.

This is quite expensive, so I implemented a neural network which uses two for loops and some matrix calculation. I used matrix caculation on update over mini-batch example. The update shares the same weights and biases in the network, so we can conduct the update simultaneously over the examples in the mini-batch.

## Introduction

Usually, update over mini-batch is conducted using one 'for' loop. We iterate over the examples in the mini-batch. In each loop, we calculate the gradient of the cost function with respect to weight matrix nad biases of the neurons.

Suppose that ${C_x}_i$ is the cost calculated over a specific example $x_i$ in the mini_batch. Althoug we want to calculate the gradient of the cost $C_{all}$ over the entier training examples, it would be quite expensive computation. So we estimate the gradient with some randomely chosen examples. Then out estimate of the gradient is:
$$
\widehat{ \nabla_{w} C_{all} } = \frac{1}{m}\sum_{i=0}^{m}\nabla_{w}C_{xi}
$$
$$
\widehat{ \nabla_{b} C_{all} } = \frac{1}{m}\sum_{i=0}^{m}\nabla_{b}C_{xi}
$$
where m is the mini-batch size. Each gradient of $C_x$ with respect to $\mathbf{w}$ and b can be calculated using backpropagation. Suppose that we use the corss entropy as the cost function and that add a L2 regularization term. Then the cost over the entire training examples is representes as:

$$C_{all} = \sum_{x}^{}C_{0x} + \frac{\lambda}{2n}\sum_{\mathbf{w}}^{}\mathbf{w}^2 $$
$$\text{where }C_{0x} = \sum_{j}^{}[ y_j log a_j^L + (1-y_j)log( 1 - a_j^L )] $$

$a_j^L$ is the jth activation output from the output layer. So we update $\mathbf{w}$ as :

$$\mathbf{w} \leftarrow \mathbf{w} - \eta \nabla_{w}C_{all} $$
$$= \mathbf{w} - \eta ( \nabla_{w}C_{0all} + \frac{\lambda}{n}\mathbf{w} )$$
$$= ( 1 - \frac{\eta\lambda}{n} )\mathbf{w} - \eta\nabla_{w}C_{0all}$$

Because we can not clculate $C_{0all}$, we estimate it over the examples in a mini_batch.

$$\mathbf{w} = ( 1 - \frac{\eta\lambda}{n} )\mathbf{w} - \eta\frac{1}{m}\sum_{i=0}^{m}\nabla_{w}C_{0xi} $$

and biases are also updated as:

$$\mathbf{b} \leftarrow \mathbf{b} - \frac{\eta}{m}\sum_{x}^{}\nabla_{b}C_{0xi}$$

But we cannot directly drive these
gradient of the cost function, so we use the back propagation algorithm to get them. Let $\mathbf{z}^l$ and $\mathbf{a}^l$ be the inputs to the neurons in the lth layer and its sigmoid activation respectively. And we also define $\mathbf{\delta}^l$ as the gradient of the const function with respect to $\mathbf{z}^l$
$$\mathbf{z}^{l+1} = \mathbf{w}^{l}\mathbf{z}^{l} + \mathbf{b}^{l}$$
$$\mathbf{a}^{l} = \sigma ( \mathbf{z}^{l} )$$
$$\mathbf{\delta}^{l} = \nabla_{z^l}C_{0xi}$$

## Python code
The python code is composed with three separete file.
The first code $\textbf{network_optimized.py}$ defines a class called 'Network'. This class holds basic parameters that defines the structure of the network. We give the number of neurons in the hidden layer when we instanciate it. The main function for training is 'SGD'. We give the function some hyper parameters specifying how the network learns, such as learing late $\eta$, the number of epochs, mini-bach size, and lambda $\lambda$. $\lambda$ defines how much the regularization term affects the way the networks learns. When it is large, it will be more beneficial, when we minimize the cost function, to make the weights small rather than minimize the non-regularized part of the cost function. And hence, the network will have strong generality. Conversely, when $\lambda = 0$, the network will behave as a non-regularized one and the weights may become larger as the training proceeds.

SGD function calls 'update_mini_batch' function to update $\mathbf{w}$ and $b$ over an mini-batch. We use matrix calculation to derive the estimated gradient of the cost function rather than iterate over the examples in the mini-batch. The update function takes mini-batch examples minibatch_x and minibatch_y, which are matrices. The number of rows is the mini-batch size. minibatch_x and minibatch_y have 784 and 10 columns respectively.

We first feedforward through the network to calculate the $\mathbf{z}$ vectors and its activations. Note that we calculate them simultaneously over all the exmples in the mini-batch using matrix calculation. Afther the feedforward, we conduct the back-propagation. Also in the back-propagation, $\delta$ vector is expanded to a matrixs, where jth column in the matrix represents $\delta$ of the jth example in the mini-batch. $\mathbf{z}$ vecotors and its activations also have the same structure.
$$
\begin{align}
\mathbf{z} &= ( \mathbf{z_1}, \dotsc, \mathbf{z_m} ) \\
\mathbf{a} &= \sigma( \mathbf{z} )

\end{align}
$$

Calculating the gradient of the cost function with respect to $\mathbf{w}$ is a little complicated. Usually, when we do that with 'for' loop, we first calculate the gradients for each example in the mini-batch and then take its average. However, with the matrix compulation we conduct the two operations simultraneously in a single line of code.

```python

# Last Updated: March 6, 2018
# Note:
# the vectorized version of the network2. Only one hidden layer is allowed,
# and we use only cross entropy cost.

import random
import sys
import numpy as np

class Network ( object ) :
    def __init__ ( self, num_hidden_neuron ) :
        # the number of neurons in the hidded layer
        self.num_hidden_neuron = num_hidden_neuron
        # the definistions of the biases vecotors and weights matrices.
        self.biases_hidlayer = np.random.randn ( num_hidden_neuron )
        self.biases_outlayer = np.random.randn ( 10 )
        self.weights_hidlayer = \
          np.random.randn ( num_hidden_neuron, 784 ) / 10.
        self.weights_outlayer = \
          np.random.randn ( 10, num_hidden_neuron ) / num_hidden_neuron

    def SGD ( self,
              # when whole_training_data is the whole mnist training data,
              # whole_training_data [ 0 ] <-shape( 50000, 784 )
              # whole_training_data [ 1 ] <-shape( 50000, 10 )
              whole_training_data,
              epochs,
              mini_batch_size, # 10 in demo.py
              eta, # 0.3 in demp.py
              lmbda = 0.0, # 3.0 in demo.py
              # evaluation data is either validation data or test data.
              evaluation_data = None,
              evaluate_on_training_data = False,
              evaluate_on_evaluation_data = False ) :

        if evaluation_data : n_data = len ( evaluation_data [ 0 ] ) # 10000
        whole_training_data_size = len ( whole_training_data [ 0 ] ) # 50000
        training_cost, training_accuracy = [], []
        evaluation_cost, evaluation_accuracy = [], []
        index = np.arange ( whole_training_data_size )
        for epc in xrange ( epochs ) :
            print "Epoch %d: Start" % epc
            np.random.shuffle ( index )
            whole_training_x = whole_training_data [ 0 ] [ index, : ]
            whole_training_y = whole_training_data [ 1 ] [ index, : ]
            for j in xrange ( 0, whole_training_data_size, mini_batch_size  ) :
                minibatch_x = \
                  whole_training_x [ j : j + mini_batch_size, : ]
                minibatch_y = \
                  whole_training_y [ j : j + mini_batch_size, : ]
                self.update_mini_batch ( minibatch_x, minibatch_y, eta,
                                    lmbda, whole_training_data_size )

            print "Epoch %d: training complete" % epc
            if evaluate_on_training_data :
                accuracy, cost = self.evaluate ( whole_training_data [ 0 ],
                                                 whole_training_data [ 1 ],
                                                 lmbda )
                training_cost.append ( cost )
                training_accuracy.append ( accuracy )
            if evaluate_on_evaluation_data and evaluation_data :
                accuracy, cost = self.evaluate ( evaluation_data [ 0 ],
                                                 evaluation_data [ 1 ],
                                                 lmbda )
                evaluation_cost.append ( cost )
                evaluation_accuracy.append ( accuracy )

        return training_cost, training_accuracy,\
               evaluation_cost, evaluation_accuracy

    def update_mini_batch ( self, minibatch_x, minibatch_y, eta, lmbda,
                            whole_training_data_size ) :
        mini_batch_size = minibatch_x.shape [ 0 ]
        ## feedforward
        a_inlayer = minibatch_x.T
        z_hidlayer = np.dot ( self.weights_hidlayer, a_inlayer ) + \
                     self.biases_hidlayer [ :, None ]
        a_hidlayer = sigmoid ( z_hidlayer )
        z_outlayer = np.dot ( self.weights_outlayer, a_hidlayer ) + \
                     self.biases_outlayer [ :, None ]
        a_outlayer = sigmoid ( z_outlayer )

        ## back propagate
        delta_outlayer = a_outlayer - minibatch_y.T
        biases_outlayer_grad = delta_outlayer.mean ( axis = 1 )
        weights_outlayer_grad = \
          np.dot ( delta_outlayer, a_hidlayer.T ) / mini_batch_size
        delta_hidlayer = \
          sigmoid_prime ( z_hidlayer ) * \
          np.dot ( self.weights_outlayer.T, delta_outlayer )
        biases_hidlayer_grad = delta_hidlayer.mean ( axis = 1 )
        weights_hidlayer_grad = \
          np.dot ( delta_hidlayer, a_inlayer.T ) / mini_batch_size

        ## update biases
        self.biases_outlayer = \
          self.biases_outlayer - eta * biases_outlayer_grad
        self.biaess_hidlayer = \
          self.biases_hidlayer - eta * biases_hidlayer_grad
        # update weights
        self.weights_outlayer = \
          ( 1. - eta * lmbda / whole_training_data_size ) * \
          self.weights_outlayer - eta * weights_outlayer_grad
        self.weights_hidlayer = \
          ( 1. - eta * lmbda / whole_training_data_size ) * \
          self.weights_hidlayer - eta * weights_hidlayer_grad

    def evaluate ( self, data_x, data_y, lmbda ) :
        '''
        calcuate accuracy and cost on the given data. the format of data_x
        and data_y is the same as the training_data. The cost function is
        the coross entropy cost
        '''
        num_data = data_x.shape [ 0 ]

        ## feedforward
        a_inlayer = data_x.T
        z_hidlayer = np.dot ( self.weights_hidlayer, a_inlayer ) + \
                     self.biases_hidlayer [ :, None ]
        a_hidlayer = sigmoid ( z_hidlayer )
        z_outlayer = np.dot ( self.weights_outlayer, a_hidlayer ) + \
                     self.biases_outlayer [ :, None ]
        a_outlayer = sigmoid ( z_outlayer )

        ## calcualte accuracy
        network_output = a_outlayer.argmax ( axis = 0 )
        desired_output = data_y.argmax ( axis = 1 )
        succeeds = ( network_output == desired_output ).sum ()
        accuracy = ( succeeds * 100.0 ) / num_data

        # calculate cost
        cost_C0 = np.nan_to_num \
          ( - data_y.T * np.log ( a_outlayer ) - \
          ( 1 - data_y.T ) * np.log ( 1 - a_outlayer ) ).sum () / num_data
        cost_regularization_term = \
          0.5 * lmbda / num_data *\
          ( np.linalg.norm ( self.weights_hidlayer ) ** 2 +\
            np.linalg.norm ( self.weights_outlayer ) ** 2 )
        cost = cost_C0 + cost_regularization_term

        return accuracy, cost

def sigmoid ( x ) :
    return 1./ ( 1. + np.exp ( -x ) )

def sigmoid_prime ( x ) :
    # return sigmoid ( x ) * ( 1 - sigmoid ( x ) )
    return 1./ ( 2. + np.exp ( x ) + np.exp ( -x ) )

```


# GraviPy - tutorial

## _Coordinates_ and _MetricTensor_

To start working with the gravipy package you must load the package and initialize a pretty-printing mode in Jupyter environment

```python
from gravipy.tensorial import * # import GraviPy package
from sympy import init_printing
import inspect
init_printing()
```

The next step is to choose coordinates and define a metric tensor of a particular space. Let's take, for example, the Schwarzschild metric - vacuum solution to the Einstein's field equations which describes the gravitational field of a spherical mass distribution.

```python
# define some symbolic variables
t, r, theta, phi, M = symbols('t, r, \\theta, \phi, M')
# create a coordinate four-vector object instantiating
# the Coordinates class
x = Coordinates('\chi', [t, r, theta, phi])
# define a matrix of a metric tensor components
Metric = diag(-(1-2*M/r), 1/(1-2*M/r), r**2, r**2*sin(theta)**2)
# create a metric tensor object instantiating the MetricTensor class
g = MetricTensor('g', x, Metric)
```

Each component of any tensor object, can be computed by calling the appropriate instance of the _GeneralTensor_ subclass with indices as arguments. The covariant indices take positive integer values (1, 2, ..., dim). The contravariant indices take negative values (-dim, ..., -2, -1).

```python
x(-1)
```

```python
g(1, 1)
```

```python
x(1)
```

Matrix representation of a tensor can be obtained in the following way

```python
x(-All)
```

```python
g(All, All)
```

```python
g(All, 4)
```

## Predefined _Tensor_ Classes

The GraviPy package contains a number of the _Tensor_ subclasses that can be used to calculate a tensor components. The _Tensor_ subclasses available in the current version of GraviPy package are

```python
print([cls.__name__ for cls in vars()['Tensor'].__subclasses__()])
```

    ['Christoffel', 'Ricci', 'Riemann', 'Einstein', 'Geodesic']

### The _Christoffel_ symbols

The first one is the _Christoffel_ class that represents Christoffel symbols of the first and second kind. (Note that the Christoffel symbols are not tensors) Components of the _Christoffel_ objects are computed from the below formula

$$ \Gamma_{\rho \mu \nu} = g_{\rho \sigma}\Gamma^{\sigma}_{\ \mu \nu} = \frac{1}{2}(g_{\rho \mu, \nu} + g_{\rho \nu, \mu} - g_{\mu \nu, \rho})$$

Let's create an instance of the _Christoffel_ class for the Schwarzschild metric g and compute some components of the object

```python
Ga = Christoffel('Ga', g)
Ga(1, 2, 1)
```

Each component of the _Tensor_ object is computed only once due to memoization procedure implemented in the _Tensor_ class. Computed value of a tensor component is stored in _components_ dictionary (attribute of a _Tensor_ instance) and returned by the next call to the instance.

```python
Ga.components
```

The above dictionary consists of two elements because the symmetry of the Christoffel symbols is implemented in the _Christoffel_ class.

If necessary, you can clear the _components_ dictionary

```python
Ga.components = {}
Ga.components
```

The _Matrix_ representation of the Christoffel symbols is the following

```python
Ga(All, All, All)
```

You can get help on any of classes mentioned before by running the command

```python
help(Christoffel)
```

    Help on class Christoffel in module gravipy.tensorial:

    class Christoffel(Tensor)
     |  Christoffel(symbol, metric, *args, **kwargs)
     |
     |  Christoffel.
     |
     |  Represents a class of Christoffel symbols of the first and second kind.
     |
     |  Parameters
     |  ==========
     |
     |  symbol : python string - name of the Christoffel symbol
     |  metric : GraviPy MtricTensor object
     |
     |  Examples
     |  ========
     |
     |  Define a Christoffel symbols for the Schwarzschild metric:
     |
     |  >>> from gravipy import *
     |  >>> t, r, theta, phi = symbols('t, r, \\theta, \phi')
     |  >>> chi = Coordinates('\chi', [t, r, theta, phi])
     |  >>> M = Symbol('M')
     |  >>> Metric = diag(-(1 - 2 * M / r), 1 / (1 - 2 * M / r), r ** 2,
     |  ...                  r ** 2 * sin(theta) ** 2)
     |  >>> g = MetricTensor('g', chi, Metric)
     |  >>> Ga = Christoffel('Ga', g)
     |  >>> Ga(-1, 2, 1)
     |  -M/(r*(2*M - r))
     |  >>> Ga(2, All, All)
     |  Matrix([
     |  [M/r**2,               0,  0,                 0],
     |  [     0, -M/(2*M - r)**2,  0,                 0],
     |  [     0,               0, -r,                 0],
     |  [     0,               0,  0, -r*sin(\theta)**2]])
     |  >>> Ga(1, -1, 2) # doctest: +IGNORE_EXCEPTION_DETAIL
     |  Traceback (most recent call last):
     |  GraviPyError: "Tensor component Ga(1, -1, 2) doesn't  exist"
     |
     |  Method resolution order:
     |      Christoffel
     |      Tensor
     |      GeneralTensor
     |      builtins.object
     |
     |  Methods defined here:
     |
     |  __init__(self, symbol, metric, *args, **kwargs)
     |      Initialize self.  See help(type(self)) for accurate signature.
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes inherited from Tensor:
     |
     |  TensorObjects = [<gravipy.tensorial.Christoffel object>]
     |
     |  ----------------------------------------------------------------------
     |  Methods inherited from GeneralTensor:
     |
     |  __call__(self, *idxs)
     |      Call self as a function.
     |
     |  covariantD(self, *idxs)
     |
     |  partialD(self, *idxs)
     |
     |  ----------------------------------------------------------------------
     |  Static methods inherited from GeneralTensor:
     |
     |  get_nmatrixel(M, idxs)
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors inherited from GeneralTensor:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes inherited from GeneralTensor:
     |
     |  GeneralTensorObjects = [<gravipy.tensorial.Coordinates object>, <gravi...

Try also "_Christoffel?_" and  "_Christoffel??_"

### The _Ricci_ tensor

$$ R_{\mu \nu} = \frac{\partial \Gamma^{\sigma}_{\ \mu \nu}}{\partial x^{\sigma}} - \frac{\partial \Gamma^{\sigma}_{\ \mu \sigma}}{\partial x^{\nu}} + \Gamma^{\sigma}_{\ \mu \nu}\Gamma^{\rho}_{\ \sigma \rho} - \Gamma^{\rho}_{\ \mu \sigma}\Gamma^{\sigma}_{\ \nu \rho} $$

```python
Ri = Ricci('Ri', g)
Ri(All, All)
```

Contraction of the _Ricci_ tensor $R = R_{\mu}^{\ \mu} = g^{\mu \nu}R_{\mu \nu}$

```python
Ri.scalar()
```

### The _Riemann_ tensor

$$ R_{\mu \nu \rho \sigma} = \frac{\partial \Gamma_{\mu \nu \sigma}}{\partial x^{\rho}} - \frac{\partial \Gamma_{\mu \nu \rho}}{\partial x^{\sigma}} + \Gamma^{\alpha}_{\ \nu \sigma}\Gamma_{\mu \rho \alpha} - \Gamma^{\alpha}_{\ \nu \rho}\Gamma_{\mu \sigma \alpha} - \frac{\partial g_{\mu \alpha}}{\partial x^{\rho}}\Gamma^{\alpha}_{\ \nu \sigma} + \frac{\partial g_{\mu \alpha}}{\partial x^{\sigma}}\Gamma^{\alpha}_{\ \nu \rho} $$

```python
Rm = Riemann('Rm', g)
```

Some nonzero components of the _Riemann_ tensor are

```python
from IPython.display import display, Math
from sympy import latex
for i, j, k, l in list(variations(range(1, 5), 4, True)):
    if Rm(i, j, k, l) != 0 and k<l and i<j:
        display(Math('R_{'+str(i)+str(j)+str(k)+str(l)+'} = '+ latex(Rm(i, j, k, l))))
```

$\displaystyle R_{1212} = - \frac{2 M}{r^{3}}$

$\displaystyle R_{1313} = \frac{M \left(- 2 M + r\right)}{r^{2}}$

$\displaystyle R_{1414} = \frac{M \left(- 2 M + r\right) \sin^{2}{\left(\theta \right)}}{r^{2}}$

$\displaystyle R_{2323} = \frac{M}{2 M - r}$

$\displaystyle R_{2424} = \frac{M \sin^{2}{\left(\theta \right)}}{2 M - r}$

$\displaystyle R_{3434} = 2 M r \sin^{2}{\left(\theta \right)}$

You can also display the matrix representation of the tensor

```python
# Rm(All, All, All, All)
```

Contraction of the _Riemann_ tensor $R_{\mu \nu} = R^{\rho}_{\ \mu \rho \nu} $

```python
ricci = sum([Rm(i, All, k, All)*g(-i, -k)
             for i, k in list(variations(range(1, 5), 2, True))],
            zeros(4))
ricci.simplify()
ricci
```

### The _Einstein_ tensor

$$ G_{\mu \nu} = R_{\mu \nu} - \frac{1}{2}g_{\mu \nu}R $$

```python
G = Einstein('G', Ri)
G(All, All)
```

### _Geodesics_

$$ w_{\mu} = \frac{Du_{\mu}}{d\tau} = \frac{d^2x_{\mu}}{d\tau^2} - \frac{1}{2}g_{\rho \sigma, \mu} \frac{dx^{\rho}}{d\tau}\frac{dx^{\sigma}}{d\tau} $$

```python
tau = Symbol('\\tau')
w = Geodesic('w', g, tau)
w(All).transpose()
```

Please note that instantiation of a _Geodesic_ class for the metric $g$ automatically turns on a _Parametrization_ mode for the metric $g$. Then all coordinates are functions of a world line parameter $\tau$

```python
Parametrization.info()
```

```python
x(-All)
```

```python
g(All, All)
```

_Parametrization_ mode can be deactivated by typing

```python
Parametrization.deactivate(x)
Parametrization.info()
```

    No parametrization activated

```python
x(-All)
```

```python
g(All, All)
```

## Derivatives

### Partial derivative

All instances of a _GeneralTensor_ subclasses inherits _partialD_ method which works exactly the same way as SymPy _diff_ method.

```python
T = Tensor('T', 2, g)
T(1, 2)
```

```python
T.partialD(1, 2, 1, 3) # The first two indices belongs to second rank tensor T
```

```python
T(1, 2).diff(x(-1), x(-3))
```

The only difference is that computed value of _partialD_ is saved in  "_partial_derivative_components_" dictionary an then returned by the next call to the _partialD_ method.

```python
T.partial_derivative_components
```

### Covariant derivative

Covariant derivative components of the tensor ___T___  can be computed by the covariantD method from the formula

$$ \nabla_{\sigma} T_{\mu}^{\ \nu} = T_{\mu \ ;\sigma}^{\ \nu} = \frac{\partial T_{\mu}^{\ \nu}}{\partial x^{\sigma}} - \Gamma^{\rho}_{\ \mu \sigma}T_{\rho}^{\ \nu} + \Gamma^{\nu}_{\ \rho \sigma}T_{\mu}^{\ \rho}$$

Let's compute some covariant derivatives of a scalar field C

```python
C = Tensor('C', 0, g)
C()
```

```python
C.covariantD(1)
```

```python
C.covariantD(2, 3)
```

All _covariantD_ components of every _Tensor_ object are also memoized

```python
for k in C.covariant_derivative_components:
    display(Math(str(k) + ': '
                 + latex(C.covariant_derivative_components[k])))
```

$\displaystyle (1,): \frac{\partial}{\partial t} C{\left(t,r,\theta,\phi \right)}$

$\displaystyle (2,): \frac{\partial}{\partial r} C{\left(t,r,\theta,\phi \right)}$

$\displaystyle (3,): \frac{\partial}{\partial \theta} C{\left(t,r,\theta,\phi \right)}$

$\displaystyle (4,): \frac{\partial}{\partial \phi} C{\left(t,r,\theta,\phi \right)}$

$\displaystyle (2, 3): \frac{r \frac{\partial^{2}}{\partial r\partial \theta} C{\left(t,r,\theta,\phi \right)} - \frac{\partial}{\partial \theta} C{\left(t,r,\theta,\phi \right)}}{r}$

```python
C.covariantD(1, 2, 3)
```

Proof that the covariant derivative of the metric tensor $g$ is zero

```python
not any([g.covariantD(i, j, k).simplify()
         for i, j, k in list(variations(range(1, 5), 3, True))])
```

    True

Bianchi identity in the Schwarzschild spacetime

$$ R_{\mu \nu \sigma \rho ;\gamma} + R_{\mu \nu \gamma \sigma ;\rho} + R_{\mu \nu \rho \gamma ;\sigma} = 0$$

```python
not any([(Rm.covariantD(i, j, k, l, m) + Rm.covariantD(i, j, m, k, l)
          + Rm.covariantD(i, j, l, m, k)).simplify()
         for i, j, k, l, m in list(variations(range(1, 5), 5, True))])
```

    True

## User-defined tensors

To define a new scalar/vector/tensor field in some space you should __extend__ the _Tensor_ class or __create an instance__ of the _Tensor_ class.

### _Tensor_ class instantiation

Let's create a third-rank tensor field living in the Schwarzshild spacetime as an instance of the _Tensor_ class

```python
S = Tensor('S', 3, g)
```

Until you define (override) the _\_compute\_covariant\_component_ method of the __S__ object, all of $4^3$ components are arbitrary functions of coordinates

```python
S(1, 2, 3)
```

```python
inspect.getsourcelines(T._compute_covariant_component)
```

    (['    def _compute_covariant_component(self, idxs):\n',
      '        if len(idxs) == 0:\n',
      '            component = Function(str(self.symbol))(*self.coords.c)\n',
      '        elif len(idxs) == 1:\n',
      '            component = Function(str(self.symbol) +\n',
      "                                 '(' + str(idxs[0]) + ')')(*self.coords.c)\n",
      '        else:\n',
      '            component = Function(str(self.symbol) + str(idxs))(*self.coords.c)\n',
      '        return component\n'],
     157)

Let's assume that tensor __S__ is the commutator of the covariant derivatives of some arbitrary vector field __V__ and create a new _\_compute\_covariant\_component_ method for the object __S__

```python
V = Tensor('V', 1, g)
V(All)
```

```python
def S_new_method(idxs): # definition
    component = (V.covariantD(idxs[0], idxs[1], idxs[2])
                 - V.covariantD(idxs[0], idxs[2], idxs[1])).simplify()
    S.components.update({idxs: component}) # memoization
    return component
S._compute_covariant_component = S_new_method
# _compute_covariant_component method was overriden
```

```python
S(1, 1, 3)
```

One can check that the well known formula is correct

$$ V_{\mu ;\nu \rho} - V_{\mu ;\rho \nu} = R^{\sigma}_{\ \mu \nu \rho}V_{\sigma} $$

```python
zeros = reduce(Matrix.add, [Rm(-i, All, All, All)*V(i)
                            for i  in range(1, 5)]) - S(All, All, All)
zeros.simplify()
zeros
```

Another way of tensor creation is to make an instance of the _Tensor_ class with components option. Tensor components stored in _Matrix_ object are  writen to the _components_ dictionary of the instance by this method.

```python
Z = Tensor('Z', 3, g, components=zeros, components_type=(1, 1, 1))
```

```python
not any(Z.components.values())
```

    True

### _Tensor_ class extension

As an example of the _Tensor_ class extension you can get the source code of any of the predefined _Tensor_ subclasses

```python
print([cls.__name__ for cls in vars()['Tensor'].__subclasses__()])
```

    ['Christoffel', 'Ricci', 'Riemann', 'Einstein', 'Geodesic']

```python
inspect.getsourcelines(Christoffel)
```

    (['class Christoffel(Tensor):\n',
      '    r"""Christoffel.\n',
      '\n',
      '    Represents a class of Christoffel symbols of the first and second kind.\n',
      '\n',
      '    Parameters\n',
      '    ==========\n',
      '\n',
      '    symbol : python string - name of the Christoffel symbol\n',
      '    metric : GraviPy MtricTensor object\n',
      '\n',
      '    Examples\n',
      '    ========\n',
      '\n',
      '    Define a Christoffel symbols for the Schwarzschild metric:\n',
      '\n',
      '    >>> from gravipy import *\n',
      "    >>> t, r, theta, phi = symbols('t, r, \\\\theta, \\phi')\n",
      "    >>> chi = Coordinates('\\chi', [t, r, theta, phi])\n",
      "    >>> M = Symbol('M')\n",
      '    >>> Metric = diag(-(1 - 2 * M / r), 1 / (1 - 2 * M / r), r ** 2,\n',
      '    ...                  r ** 2 * sin(theta) ** 2)\n',
      "    >>> g = MetricTensor('g', chi, Metric)\n",
      "    >>> Ga = Christoffel('Ga', g)\n",
      '    >>> Ga(-1, 2, 1)\n',
      '    -M/(r*(2*M - r))\n',
      '    >>> Ga(2, All, All)\n',
      '    Matrix([\n',
      '    [M/r**2,               0,  0,                 0],\n',
      '    [     0, -M/(2*M - r)**2,  0,                 0],\n',
      '    [     0,               0, -r,                 0],\n',
      '    [     0,               0,  0, -r*sin(\\theta)**2]])\n',
      '    >>> Ga(1, -1, 2) # doctest: +IGNORE_EXCEPTION_DETAIL\n',
      '    Traceback (most recent call last):\n',
      '    GraviPyError: "Tensor component Ga(1, -1, 2) doesn\'t  exist"\n',
      '\n',
      '    """\n',
      '\n',
      '    def __init__(self, symbol, metric, *args, **kwargs):\n',
      '        super(Christoffel, self).__init__(\n',
      '            symbol, 3, metric, index_types=(0, 1, 1), *args, **kwargs)\n',
      '        self.is_connection = True\n',
      '        self.conn = self\n',
      '        self.metric.conn = self\n',
      '\n',
      '    def _compute_covariant_component(self, idxs):\n',
      '        component = Rational(1, 2) * (\n',
      '            self.metric(idxs[0], idxs[1]).diff(self.coords(-idxs[2])) +\n',
      '            self.metric(idxs[0], idxs[2]).diff(self.coords(-idxs[1])) -\n',
      '            self.metric(idxs[1], idxs[2]).diff(self.coords(-idxs[0]))) \\\n',
      '            .together().simplify()\n',
      '        self.components.update({idxs: component})\n',
      '        if self.apply_tensor_symmetry:\n',
      '            self.components.update({(idxs[0], idxs[2], idxs[1]): component})\n',
      '        return component\n'],
     515)


# Introduction to NumPy

## Topics
- Basic Synatx
    - creating vectors matrices
    - special: ones, zeros, identity eye
    - add, product, inverse
- Mechanics: indexing, slicing, concatenating, reshape, zip
- Numpy load, save data files
- Random numbers $\rightarrow$ distributions
- Similarity: Euclidian vs Cosine
- Example Nearest Neighbor search
- Example Linear Regression

# Basics
this section uses content created by Rob Hicks http://rlhick.people.wm.edu/stories/linear-algebra-python-basics.html

## Loading libraries

The python universe has a huge number of libraries that extend the capabilities of python. Nearly all of these are open source, unlike packages like stata or matlab where some key libraries are proprietary (and can cost lots of money).  In lots of my code, you will see this at the top:

```python
%matplotlib inline
import math
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sbn
##from scipy import *
```

This code sets up Ipython Notebook environments (lines beginning with `%`), and loads several libraries and functions.  The core scientific stack in python consists of a number of free libraries.  The ones I have loaded above include:

1. sympy: provides for symbolic computation (solving algebra problems)
2. numpy: provides for linear algebra computations
3. matplotlib.pyplot: provides for the ability to graph functions and draw figures
4. scipy: scientific python provides a plethora of capabilities
5. seaborn: makes matplotlib figures even pretties (another library like this is called bokeh).  This is entirely optional and is purely for eye candy.

## Creating arrays, scalars, and matrices in Python

Scalars can be created easily like this:

```python
x = .5
print x
```

**Vectors and Lists**

The numpy library (we will reference it by np) is the workhorse library for linear algebra in python.  To creat a vector simply surround a python list ($[1,2,3]$) with the np.array function:

```python
x_vector = np.array([1,2,3])
print x_vector
```

We could have done this by defining a python list and converting it to an array:

```python
c_list = [1,2]
print "The list:",c_list
print "Has length:", len(c_list)

c_vector = np.array(c_list)
print "The vector:", c_vector
print "Has shape:",c_vector.shape
```

```python
z = [5,6]
print "This is a list, not an array:",z
print type(z)
```

##  Matrix Addition and Subtraction

### Adding or subtracting a scalar value to a matrix

To learn the basics, consider a small matrix of dimension $2 \times 2$, where $2 \times 2$ denotes the number of rows $\times$ the number of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Consider adding a scalar value (e.g. 3) to the A.
$$
\begin{equation}
	A+3=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}+3
	=\begin{bmatrix}
	  a_{11}+3 & a_{12}+3 \\
	  a_{21}+3 & a_{22}+3
	\end{bmatrix}
\end{equation}
$$
The same basic principle holds true for A-3:
$$
\begin{equation}
	A-3=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}-3
	=\begin{bmatrix}
	  a_{11}-3 & a_{12}-3 \\
	  a_{21}-3 & a_{22}-3
	\end{bmatrix}
\end{equation}
$$
Notice that we add (or subtract) the scalar value to each element in the matrix A.  A can be of any dimension.

This is trivial to implement, now that we have defined our matrix A:

```python
result = A + 3
#or
result = 3 + A
print result
```

### Adding or subtracting two matrices
Consider two small $2 \times 2$ matrices, where $2 \times 2$ denotes the \# of rows $\times$ the \# of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$ and $B$=$\bigl( \begin{smallmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{smallmatrix} \bigr)$.  To find the result of $A-B$, simply subtract each element of A with the corresponding element of B:

$$
\begin{equation}
	A -B =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix} -
	\begin{bmatrix} b_{11} & b_{12} \\
	  b_{21} & b_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  a_{11}-b_{11} & a_{12}-b_{12} \\
	  a_{21}-b_{21} & a_{22}-b_{22}
	\end{bmatrix}
\end{equation}
$$

Addition works exactly the same way:

$$
\begin{equation}
	A + B =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix} +
	\begin{bmatrix} b_{11} & b_{12} \\
	  b_{21} & b_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  a_{11}+b_{11} & a_{12}+b_{12} \\
	  a_{21}+b_{21} & a_{22}+b_{22}
	\end{bmatrix}
\end{equation}
$$

An important point to know about matrix addition and subtraction is that it is only defined when $A$ and $B$ are of the same size.  Here, both are $2 \times 2$.  Since operations are performed element by element, these two matrices must be conformable- and for addition and subtraction that means they must have the same numbers of rows and columns.  I like to be explicit about the dimensions of matrices for checking conformability as I write the equations, so write

$$
A_{2 \times 2} + B_{2 \times 2}= \begin{bmatrix}
  a_{11}+b_{11} & a_{12}+b_{12} \\
  a_{21}+b_{21} & a_{22}+b_{22}
\end{bmatrix}_{2 \times 2}
$$

Notice that the result of a matrix addition or subtraction operation is always of the same dimension as the two operands.

Let's define another matrix, B, that is also $2 \times 2$ and add it to A:

```python
B = np.random.randn(2,2)
print B
```

## Matrix Multiplication

### Multiplying a scalar value times a matrix

As before, let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Suppose we want to multiply A times a scalar value (e.g. $3 \times A$)

$$
\begin{equation}
	3 \times A = 3 \times \begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  3a_{11} & 3a_{12} \\
	  3a_{21} & 3a_{22}
	\end{bmatrix}
\end{equation}
$$

is of dimension (2,2).  Scalar multiplication is commutative, so that $3 \times A$=$A \times 3$.  Notice that the product is defined for a matrix A of any dimension.

Similar to scalar addition and subtration, the code is simple:

```python
A * 3
```

### Multiplying two matricies

Now, consider the $2 \times 1$ vector $C=\bigl( \begin{smallmatrix} c_{11} \\
  c_{21}
\end{smallmatrix} \bigr)$

Consider multiplying matrix $A_{2 \times 2}$ and the vector $C_{2 \times 1}$.  Unlike the addition and subtraction case, this product is defined.  Here, conformability depends not on the row **and** column dimensions, but rather on the column dimensions of the first operand and the row dimensions of the second operand.  We can write this operation as follows

$$
\begin{equation}
	A_{2 \times 2} \times C_{2 \times 1} =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}_{2 \times 2}
    \times
    \begin{bmatrix}
	c_{11} \\
	c_{21}
	\end{bmatrix}_{2 \times 1}
	=
	\begin{bmatrix}
	  a_{11}c_{11} + a_{12}c_{21} \\
	  a_{21}c_{11} + a_{22}c_{21}
	\end{bmatrix}_{2 \times 1}
\end{equation}
$$

Alternatively, consider a matrix C of dimension $2 \times 3$ and a matrix A of dimension $3 \times 2$

$$
\begin{equation}
	A_{3 \times 2}=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
	,
	C_{2 \times 3} =
	\begin{bmatrix}
		  c_{11} & c_{12} & c_{13} \\
		  c_{21} & c_{22} & c_{23} \\
	\end{bmatrix}_{2 \times 3}
	\end{equation}
$$

Here, A $\times$ C is

$$
\begin{align}
	A_{3 \times 2} \times C_{2 \times 3}=&
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
	\times
	\begin{bmatrix}
	  c_{11} & c_{12} & c_{13} \\
	  c_{21} & c_{22} & c_{23}
	\end{bmatrix}_{2 \times 3} \\
	=&
	\begin{bmatrix}
	  a_{11} c_{11}+a_{12} c_{21} & a_{11} c_{12}+a_{12} c_{22} & a_{11} c_{13}+a_{12} c_{23} \\
	  a_{21} c_{11}+a_{22} c_{21} & a_{21} c_{12}+a_{22} c_{22} & a_{21} c_{13}+a_{22} c_{23} \\
	  a_{31} c_{11}+a_{32} c_{21} & a_{31} c_{12}+a_{32} c_{22} & a_{31} c_{13}+a_{32} c_{23}
	\end{bmatrix}_{3 \times 3}
\end{align}
$$

So in general, $X_{r_x \times c_x} \times Y_{r_y \times c_y}$ we have two important things to remember:

* For conformability in matrix multiplication, $c_x=r_y$, or the columns in the first operand must be equal to the rows of the second operand.
* The result will be of dimension $r_x \times c_y$, or of dimensions equal to the rows of the first operand and columns equal to columns of the second operand.

Given these facts, you should convince yourself that matrix multiplication is not generally commutative, that the relationship $X \times Y = Y \times X$ does **not** hold in all cases.
For this reason, we will always be very explicit about whether we are pre multiplying ($X \times Y$) or post multiplying ($Y \times X$) the vectors/matrices $X$ and $Y$.

For more information on this topic, see this
http://en.wikipedia.org/wiki/Matrix_multiplication.

```python
# Let's redefine A and C to demonstrate matrix multiplication:
A = np.arange(6).reshape((3,2))
C = np.random.randn(2,2)

print A.shape
print C.shape
```

We will use the numpy dot operator to perform the these multiplications. You can use it two ways to yield the same result:

```python
print A.dot(C)
print np.dot(A,C)
```

```python
# What would happen to
C.dot(A)
```

## Matrix Division
The term matrix division is actually a misnomer.  To divide in a matrix algebra world we first need to invert the matrix.  It is useful to consider the analog case in a scalar work.  Suppose we want to divide the $f$ by $g$.  We could do this in two different ways:
$$
\begin{equation}
	\frac{f}{g}=f \times g^{-1}.
\end{equation}
$$
In a scalar seeting, these are equivalent ways of solving the division problem.  The second one requires two steps: first we invert g and then we multiply f times g.  In a matrix world, we need to think about this second approach.  First we have to invert the matrix g and then we will need to pre or post multiply depending on the exact situation we encounter (this is intended to be vague for now).

###Inverting a Matrix

As before, consider the square $2 \times 2$ matrix $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22}\end{smallmatrix} \bigr)$.  Let the inverse of matrix A (denoted as $A^{-1}$) be

$$
\begin{equation}
	A^{-1}=\begin{bmatrix}
             a_{11} & a_{12} \\
		     a_{21} & a_{22}
           \end{bmatrix}^{-1}=\frac{1}{a_{11}a_{22}-a_{12}a_{21}}	\begin{bmatrix}
		             a_{22} & -a_{12} \\
				     -a_{21} & a_{11}
		           \end{bmatrix}
\end{equation}
$$

The inverted matrix $A^{-1}$ has a useful property:
$$
\begin{equation}
	A \times A^{-1}=A^{-1} \times A=I
\end{equation}
$$
where I, the identity matrix (the matrix equivalent of the scalar value 1), is
$$
\begin{equation}
	I_{2 \times 2}=\begin{bmatrix}
             1 & 0 \\
		     0 & 1
           \end{bmatrix}
\end{equation}
$$
furthermore, $A \times I = A$ and $I \times A = A$.

An important feature about matrix inversion is that it is undefined if (in the $2 \times 2$ case), $a_{11}a_{22}-a_{12}a_{21}=0$.  If this relationship is equal to zero the inverse of A does not exist.  If this term is very close to zero, an inverse may exist but $A^{-1}$ may be poorly conditioned meaning it is prone to rounding error and is likely not well identified computationally.  The term $a_{11}a_{22}-a_{12}a_{21}$ is the determinant of matrix A, and for square matrices of size greater than $2 \times 2$, if equal to zero indicates that you have a problem with your data matrix (columns are linearly dependent on other columns).  The inverse of matrix A exists if A is square and is of full rank (ie. the columns of A are not linear combinations of other columns of A).

For more information on this topic, see this
http://en.wikipedia.org/wiki/Matrix_inversion, for example, on inverting matrices.

```python
# note, we need a square matrix (# rows = # cols), use C:
C_inverse = np.linalg.inv(C)
print C_inverse
```

Check that $C\times C^{-1} = I$:

```python
print C.dot(C_inverse)
print "Is identical to:"
print C_inverse.dot(C)
```

## Transposing a Matrix

At times it is useful to pivot a matrix for conformability- that is in order to matrix divide or multiply, we need to switch the rows and column dimensions of matrices.  Consider the matrix
$$
\begin{equation}
	A_{3 \times 2}=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
\end{equation}
$$
The transpose of A (denoted as $A^{\prime}$) is
$$
\begin{equation}
   A^{\prime}=\begin{bmatrix}
	  a_{11} & a_{21} & a_{31} \\
	  a_{12} & a_{22} & a_{32} \\
	\end{bmatrix}_{2 \times 3}
\end{equation}
$$

```python
A = np.arange(6).reshape((3,2))
B = np.arange(8).reshape((2,4))
print "A is"
print A

print "The Transpose of A is"
print A.T
```

One important property of transposing a matrix is the transpose of a product of two matrices.  Let matrix A be of dimension $N \times M$ and let B of of dimension $M \times P$.  Then
$$
\begin{equation}
	(AB)^{\prime}=B^{\prime}A^{\prime}
\end{equation}
$$
For more information, see this http://en.wikipedia.org/wiki/Matrix_transposition on matrix transposition.  This is also easy to implement:

```python
print B.T.dot(A.T)
print "Is identical to:"
print (A.dot(B)).T
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

# Mechanics

## Indexing and Slicing
examples from https://www.tutorialspoint.com/numpy/numpy_indexing_and_slicing.htm

```python
a = np.arange(10)
s = slice(2,7,2)
print a[s]
```

    [2 4 6]

```python
a = np.arange(10)
b = a[2:7:2]
print b
```

    [2 4 6]

```python
a = np.arange(10)
b = a[5]
print b
```

    5

```python
a = np.arange(10)
print a[2:]
```

```python
import numpy as np
a = np.arange(10)
print a[2:5]
```

```python
a = np.array([[1,2,3],[3,4,5],[4,5,6]])
print a

# slice items starting from index
print 'Now we will slice the array from the index a[1:]'
print a[1:]
```

```python
# array to begin with
a = np.array([[1,2,3],[3,4,5],[4,5,6]])

print 'Our array is:'
print a
print '\n'

# this returns array of items in the second column
print 'The items in the second column are:'
print a[...,1]
print '\n'

# Now we will slice all items from the second row
print 'The items in the second row are:'
print a[1,...]
print '\n'

# Now we will slice all items from column 1 onwards
print 'The items column 1 onwards are:'
print a[...,1:]
```

## Logic, Comparison

```python
A = np.random.rand(5,5)*10
print A[:,1]>4

A[A[:,1]>4]
```

    [ True False  True False  True]

    array([[ 7.42784212,  9.2688158 ,  7.51932649,  1.45105311,  3.03417883],
           [ 4.98843413,  4.25826129,  7.37323683,  4.84517475,  9.11407095],
           [ 1.91211807,  9.12048643,  2.11283513,  2.18281044,  8.9195815 ]])

```python

```

```python

```

## Concatenate, Reshape

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

# Numpy load, save data files

```python

```

# Similarity: Euclidian vs Cosine

```python

```

# Example Nearest Neighbor search
Nearest Neighbor search is a common technique in Machine Learning

```python
### Pure iterative Python ###
points = [[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]]
qPoint = [4,5,3]
minIdx = -1
minDist = -1
for idx, point in enumerate(points):  # iterate over all points
        dist = sum([(dp-dq)**2 for dp,dq in zip(point,qPoint)])**0.5  # compute the euclidean distance for each point to q
        if dist < minDist or minDist < 0:  # if necessary, update minimum distance and index of the corresponding point
            minDist = dist
            minIdx = idx

print 'Nearest point to q: ', points[minIdx]
```

    Nearest point to q:  [3, 4, 4]

```python
# # # Equivalent NumPy vectorization # # #
import numpy as np
points = np.array([[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]])
qPoint = np.array([4,5,3])
minIdx = np.argmin(np.linalg.norm(points-qPoint,axis=1))  # compute all euclidean distances at once and return the index of the smallest one
print 'Nearest point to q: ', points[minIdx]
```

    Nearest point to q:  [3 4 4]

```python

```

# Example: Linear Regression

**Linear regression** is an approach for modeling the relationship between a scalar dependent variable $y$ and one or more explanatory variables (or independent variables) denoted $X$. The case of one explanatory variable is called *simple linear regression*. For more than one explanatory variable, the process is called *multiple linear regression*.$^1$ (This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.$^2$

We assume that the equation

$y_i = \beta_0 + \beta_1 X_i + \epsilon_i$ where $\epsilon_i \approx N(0, \sigma^2)$

***
$^1$ David A. Freedman (2009). Statistical Models: Theory and Practice. Cambridge University Press. p. 26. A simple regression equation has on the right hand side an intercept and an explanatory variable with a slope coefficient. A multiple regression equation has two or more explanatory variables on the right hand side, each with its own slope coefficient

$^2$ Rencher, Alvin C.; Christensen, William F. (2012), "Chapter 10, Multivariate regression – Section 10.1, Introduction", Methods of Multivariate Analysis, Wiley Series in Probability and Statistics, 709 (3rd ed.), John Wiley & Sons, p. 19, ISBN 9781118391679.

```python
n = 100  # numeber of samples
Xr = np.random.rand(n)*99.0
y = -7.3 + 2.5*Xr + np.random.randn(n)*27.0
plt.plot(Xr, y, "o", alpha=0.5)
```

Let's add the *bias*, i.e. a column of $1$s to the explanatory variables

```python
X = np.vstack((np.ones(n), Xr)).T
print X.shape
X[0:10,:]
```

    (100, 2)

    array([[  1.        ,  23.39898613],
           [  1.        ,   8.89135227],
           [  1.        ,  19.25333926],
           [  1.        ,  60.96497966],
           [  1.        ,  92.27628388],
           [  1.        ,  68.41731103],
           [  1.        ,  55.72185762],
           [  1.        ,  26.11182375],
           [  1.        ,  52.18754121],
           [  1.        ,  79.63390856]])

## Closed-form Linear Regression

And compute the parametes $\beta_0$  and $\beta_1$ according to
$$ \beta = (X^\prime X)^{-1} X^\prime y $$
***
**Note:**
This not only looks elegant but can also be written in Julia code. However, matrix inversion $M^{-1}$ requires $O(d^3)$ iterations for a $d\times d$ matrix.<br />
https://www.coursera.org/learn/ml-regression/lecture/jOVX8/discussing-the-closed-form-solution

```python
beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
```

```python
yhat = X.dot(beta)
yhat.shape
```

    (100,)

```python
plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=1, color="red")
```

## Multiple Linear Regression

```python
n = 100  # numeber of samples
X1 = np.random.rand(n)*99.0
X2 = np.random.rand(n)*51.0 - 26.8
X3 = np.random.rand(n)*5.0 + 6.1
X4 = np.random.rand(n)*1.0 - 0.5
X5 = np.random.rand(n)*300.0

y_m = -7.3 + 2.5*X1 + -7.9*X2 + 1.5*X3 + 10.0*X4 + 0.13*X5 + np.random.randn(n)*7.0
plt.hist(y_m, bins=20)
;
```

```python
X_m = np.vstack((np.ones(n), X1, X2, X3, X4, X5)).T
X_m.shape
```

    (100, 6)

```python
beta_m = np.linalg.inv(X_m.T.dot(X_m)).dot(X_m.T).dot(y_m)
beta_m
```

    array([ -3.89296672,   2.53740144,  -7.97042339,   1.03433261,
            10.95288369,   0.1256619 ])

```python
yhat_m = X.dot(beta_m)
yhat_m.shape
```

    (100,)

## Evaluation: Root-mean-square Deviation
The root-mean-square deviation (RMSD) or root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model or an estimator and the values actually observed. The RMSD represents the sample standard deviation of the differences between predicted values and observed values. These individual differences are called residuals when the calculations are performed over the data sample that was used for estimation, and are called prediction errors when computed out-of-sample. The RMSD serves to aggregate the magnitudes of the errors in predictions for various times into a single measure of predictive power. RMSD is a good measure of accuracy, but only to compare forecasting errors of different models for a particular variable and not between variables, as it is scale-dependent.$^1$

***
$^1$  Hyndman, Rob J. Koehler, Anne B.; Koehler (2006). "Another look at measures of forecast accuracy". International Journal of Forecasting. 22 (4): 679–688. doi:10.1016/j.ijforecast.2006.03.001.

```python
import math
RSMD = math.sqrt(np.square(yhat_m-y_m).sum()/n)
print RSMD
```

    5.42682050509

## Regularization, Ridge-Regression

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting.

In general, a regularization term $R(f)$ is introduced to a general loss function:

for a loss function  $V$ that describes the cost of predicting $f(x)$ when the label is
$y$, such as the square loss or hinge loss, and for the term
$\lambda$  which controls the importance of the regularization term.
$R(f)$ is typically a penalty on the complexity of
$f$, such as restrictions for smoothness or bounds on the vector space norm.$^1$

A theoretical justification for regularization is that it attempts to impose Occam's razor on the solution, as depicted in the figure. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters.

Regularization can be used to learn simpler models, induce models to be sparse, introduce group structure into the learning problem, and more.

We're going to add the L2 term $\lambda||\beta||_2^2$ to the regression equation, which yields to$^2$

$$ \beta = (X^\prime X + \lambda I)^{-1} X^\prime y $$

***
$^1$ Bishop, Christopher M. (2007). Pattern recognition and machine learning (Corr. printing. ed.). New York: Springer. ISBN 978-0387310732.

$^2$ http://stats.stackexchange.com/questions/69205/how-to-derive-the-ridge-regression-solution

```python
p = X.shape[1]  ## get number of parameters
lam = 10.0
p, lam
```

    (2, 10.0)

```python

beta2 = np.linalg.inv(X.T.dot(X) + lam*np.eye(p)).dot(X.T).dot(y)
```

```python
yhat2 = X.dot(beta2)
```

```python
RSMD2 = math.sqrt(np.square(yhat2-y).sum()/n)
print RSMD2
```

    22.5182454253

```python
##n = float(X.shape[0])
print "      RMSE = ", math.sqrt(np.square(yhat-y).sum()/n)
print "Ridge RMSE = ", math.sqrt(np.square(yhat2-y).sum()/n)
plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=0.7, color="red")
plt.plot(X[:,1], yhat2, "-", alpha=0.7, color="green")
```

```python

```

```python

```

```python

```

```python

```

```python

```


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Convolution

The convolution of two signals $s(t)$ and $g(t)$ is defined as

\begin{equation}
\int_{-\infty}^{\infty} s(\tau) \cdot g(t - \tau) \; d\tau = s(t) * g(t)
\end{equation}

where $*$ is a common short-hand notation of the convolution integral. As shown in the previous Section, the convolution is an important operation in the theory of signals and systems. Its properties are therefore of general interest.

### Properties

For the signals $s(t)$, $g(t)$, $h(t) \in \mathbb{C}$ the convolution shows the following properties

1. The Dirac impulse is the [identity element](https://en.wikipedia.org/wiki/Identity_element) of the convolution
    \begin{equation}
    s(t) * \delta(t) = s(t)
    \end{equation}

2. The convolution is [commutative](https://en.wikipedia.org/wiki/Commutative_property)
    \begin{equation}
    s(t) * g(t) = g(t) * s(t)
    \end{equation}

3. The convolution is [associative](https://en.wikipedia.org/wiki/Associative_property)
    \begin{equation}
    \left[ s(t) * g(t) \right] * h(t) = s(t) * \left[ g(t) * h(t) \right]
    \end{equation}

5. The convolution is [distributive](https://en.wikipedia.org/wiki/Distributive_property)
    \begin{equation}
    s(t) * \left[ g(t) + h(t) \right] = s(t) * g(t) + s(t) * h(t)
    \end{equation}

5. Multiplication with a scalar $a \in \mathbb{C}$
    \begin{equation}
    a \cdot \left[ s(t) * g(t) \right] = \left[ a \cdot s(t) \right] * g(t) = s(t) * \left[ a \cdot g(t) \right]
    \end{equation}

6. Derivative of the convolution
    \begin{equation}
    \frac{d}{dt} \left[ s(t) * g(t) \right] =  \frac{d s(t)}{dt} * g(t) = s(t) * \frac{d g(t)}{dt}
    \end{equation}

The first property is a consequence of the sifting property of the Dirac pulse, the second to fifth property can be proven by considering the definition of the convolution integral and the sixth property follows from the properties of the derivative of the Dirac delta function.

### Geometrical Interpretation

The convolution can be [interpreted in a graphical manner](https://en.wikipedia.org/wiki/Convolution#Visual_explanation). This provides valuable insights into its calculation and allows to estimate its result. The calculation of the convolution integral

\begin{equation}
y(t) = \int_{-\infty}^{\infty} x(\tau) \cdot h(t-\tau) \; d\tau
\end{equation}

can be decomposed into four steps:

1. substitute $t$ by $\tau$ in both $x(t)$ and $h(t)$,

2. time-reverse $h(\tau)$ (reflection at vertical axis),

3. shift $h(-\tau)$ by $t$ to the right to yield $h(t - \tau)$,

4. shift $h(t - \tau)$ from $t = -\infty$ to $t = \infty$, check if it overlaps with $x(\tau)$ and calculate the convolution integral for the overlapping sections.

**Example**

Above procedure is illustrated in the following example with

\begin{align}
h(t) &= e^{-t} \\
x(t) &= \text{rect} \left(t - \frac{1}{2}\right)
\end{align}

Before proceeding, helper functions for the rectangular signal and plotting of the signals are defined

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, tau = sym.symbols('t tau')
```

```python
class rect(sym.Function):

    @classmethod
    def eval(cls, arg):
        return sym.Heaviside(arg + sym.S.Half) - sym.Heaviside(arg - sym.S.Half)
```

```python
def plot_signals(x_axis, x, h, ylabel, xlabel):
    p1 = sym.plot(x, (x_axis, -5, 5), show=False, line_color='b', ylabel=ylabel, xlabel=xlabel)
    p2 = sym.plot(h, (x_axis, -5, 5), show=False, line_color='r')
    p1.extend(p2)
    p1.show()
```

Now lets define and plot the signals. In the following, the impulse response $h(t)$ is illustrated by the red graph and the input signal $x(t)$ by the blue graph.

```python
h = sym.exp(-t) * sym.Heaviside(t)
x = rect(t - 1/2)

plot_signals(t, x, h, r'$h(t)$, $x(t)$', r'$t$')
```

The **first step** is to substitute $t$ by $\tau$. The horizontal axis of the plot represents $\tau$

```python
h1 = h.subs(t, tau)
x1 = x.subs(t, tau)

plot_signals(tau, x1, h1, r'$h(\tau)$, $x(\tau)$', r'$\tau$')
```

The **second step** is to time-reverse $h(\tau)$

```python
h2 = h1.subs(tau, -tau)

plot_signals(tau, x1, h2, r'$h(-\tau)$, $x(\tau)$', r'$\tau$')
```

The impulse response $h(-\tau)$ is shifted by $t$ to the right in the **third step** to yield $h(t - \tau)$. This is illustrated for $t = -2$

```python
h3 = h2.subs(tau, tau-t)

plot_signals(tau, x1, h3.subs(t, -2), r'$h(t-\tau)$, $x(\tau)$', r'$\tau$')
```

It now becomes obvious that we have to consider three cases with respect to the overlap of $h(t-\tau)$ and $x(\tau)$

1. $t<0$: no overlap
2. $0 \leq t < 1$: partial overlap
3. $t > 0$: full overlap

The **fourth step**, the evaluation of the convolution integral for the three cases is left open as an exercise.

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# Solution of the Laplace Equation on a Polar grid

In this workbook we are going to look at the solution of the Laplace equations on a polar coordinate grid.  In this case we are going to use the Bi-Conjugate Gradient solver to solve the problem.  It should be noted that there is a periodic boundary condition at $\theta=0$ and $\theta=2\pi$ so the coeficient matrix needs to reflecxt this.

$$
\frac{\partial^2 u}{\partial r^2} + \frac{1}{r}\frac{\partial u}{\partial r} + \frac{1}{r^2}\frac{\partial^2 u}{\partial \theta^2} = 0
$$

The boundary conditions on the inside and outside of the pipe are given by
$$\begin{align}
    u(1, \theta) &= 0, \quad 0\le\theta\le 2\pi \label{eq:inner_BC}\\
    \frac{\partial u}{\partial r}(3, \theta) &= \begin{cases}
        0, \quad & 0\le\theta<\pi \\
        1, \quad & \pi\le\theta\le 2\pi
        \end{cases}
\end{align} $$

```python
import numpy as np
import scipy.sparse.linalg as LA
import scipy.sparse as sps
import matplotlib.pyplot as plt
import time

class Grid:
    '''Class defining a 2D computational grid for
    a polar cartesian grid.  We assume that theta
    goes from 0 to 2\pi and has periodic boundaires
    in the theta direction.'''

    def __init__(self,ni,nj):
        # set up information about the grid
        self.origin = 0.0 # unit circle
        self.extent = 1.0
        self.Ni = ni # grid points in i direction
        self.Nj = nj # grid points in j direction

        # initialse x,y and u arrays
        self.u = np.zeros((nj, ni))
        self.r = np.zeros((nj, ni))
        self.theta = np.zeros((nj, ni))

    def set_origin(self, r0):
        self.origin = r0

    def set_extent(self,r1):
        self.extent = r1

    def generate(self,Quiet=True):
        '''generate a uniformly spaced grid covering the domain from the
        origin to the extent.  We are going to do this using linspace from
        numpy to create lists of x and y ordinates and then the meshgrid
        function to turn these into 2D arrays of grid point ordinates.'''

        theta = np.linspace(0.0, 2*np.pi, self.Ni+1)
        theta_ord = np.delete(theta,[-1])
        r_ord = np.linspace(self.origin, self.extent, self.Nj)
        self.theta,self.r = np.meshgrid(theta_ord, r_ord)
        if not Quiet:
            print(self)

    def Delta_r(self):
        # calculate delta x
        return self.r[1,0]-self.r[0,0]

    def Delta_theta(self):
        # calculate delta y
        return self.theta[0,1]-self.theta[0,0]

    def find(self,point):
        '''find the i and j ordinates of the grid cell which contains
        the point (theta,r).  To do this we calculate the distance from
        the point to the origin in the x and y directions and then
        divide this by delta x and delta y.  The resulting real ordinates
        are converted to indices using the int() function.'''
        grid_x = (point[0])/self.Delta_theta()
        grid_y = (point[1] - self.origin)/self.Delta_r()
        return int(grid_x), int(grid_y)

    def plot(self,title):
        '''produce a contour plot of u(theta,r)'''
        # convert polar to Cartesian coordinates
        x = self.r*np.cos(self.theta)
        y = self.r*np.sin(self.theta)

        # produce the plot
        fig, ax = plt.subplots()
        cmap = plt.get_cmap('jet')
        ax.axis('equal')
        ax.set_aspect('equal', 'box')
        cf = ax.contourf(x,y,self.u,cmap=cmap,levels = 21)
        fig.colorbar(cf, ax=ax)
        ax.set_title(f'{title} ({self.Ni} x {self.Nj} grid)')
        return plt

    def __str__(self):
        # describe the object when asked to print it
        return 'Uniform {}x{} polar grid from r = {} to {}.'.format(self.Ni, self.Nj, self.origin, self.extent)
```

```python
#Set up the coursework
def coursework_one(ni,nj):
    # set up a mesh
    mesh = Grid(ni,nj)
    mesh.set_origin(1.0)
    mesh.set_extent(3.0)
    mesh.generate()
    return mesh
```

Now we can implement the solver.  We can use the solver from the BiCGStab workbook as a template, remembering that in this case we know that the r=1 boundary is a Dirichlet BC with $u=(1,\theta)=0$.  The $r=3$ boundary condition is a Neumann condition where

$$\begin{align}
    u(1, \theta) &= 0, \quad 0\le\theta\le 2\pi \label{eq:inner_BC}\\
    \frac{\partial u}{\partial r}(3, \theta) &= \begin{cases}
        0, \quad & 0\le\theta<\pi \\
        1, \quad & \pi\le\theta\le 2\pi
        \end{cases}
\end{align} $$

We also need to enfroce the periodic boundary condition $u(r,0)=u(r,2\pi)$ and to modify the coefficients in the coefficient matrix to include both the additional derivative and the $1/r$ and $1/r^2$ coefficients.

The cylindrical polar form of the Laplace equation is discretised using the centred, 2nd order, finite difference equation, in compas notation are:

$$
\frac{u_N - 2u_O + u_S}{\Delta r^2} + \frac{1}{r_j} \frac{u_N - u_S}{2\Delta r} + \frac{1}{r_j^2} \frac{u_E - 2u_O + u_W}{\Delta \theta^2} = 0
$$

### Coefficients

Collecting terms

$$\frac{
2\Delta r^2 u_W+2\Delta r^2 u_E
+(2\Delta\theta^2 r^2-\Delta r\Delta\theta^2r) u_s
+(2\Delta\theta^2 r^2+\Delta r\Delta\theta^2r) u_N
-(4\Delta\theta^2r^2-4\Delta r^2) u_O}
{2\Delta r^2\Delta\theta^2r^2}=0$$

This gives coefficents for the $A$ matrix of

| Point | Coefficient
| --- | ---
| uN | $\frac{2r+\Delta r}{2r\Delta r^2}$
| uE | $\frac{1}{r^2\Delta\theta^2}$
| uS | $\frac{2r-\Delta r}{2r\Delta r^2}$
| uW | $\frac{1}{r^2\Delta\theta^2}$
| uO | $-2\frac{r^2\Delta\theta^2+\Delta r^2}{r^2\Delta r^2\Delta\theta^2}$

### Boundary conditions

It is useful to think about a small grid and to workout what points are needed in the stencil to deal with the periodic conditions.  Here is a cartoon of a 5x5 grid and the associated sparisity pattern of the $A$ matrix.  Periodic boundary conditions are in yellow.  Because we are directly assembling the matrix, **no grid cells are being used to hold boundary condition information**, so we won't need 1 cell on each side of the grid to hold ghost cell information.

This means the solver uses the whole grid.

On the left and right boundaries (where $\theta=0$ or $\theta=2\pi$ we have a periodic condition, this means that when $j=0$ the west node is at $j=N_j-1$ and when $j=N_j-1$ the east node is at $j=0$.  We need to do this in the matrix assembly routine.

On the bottom ($r=1$) boundary $u(1,\theta)=0$, this means that $0\times(2r-\Delta r)/(2r\Delta r^2)=0$ must be subtracted from $b_k$, so there is nothing to be done.

The top ($r=3$) boundary has a Neumann boundary condition, when the gradient is zero we have

$$\frac{u_N-u_S}{2\Delta r}=0 \implies u_N = u_s.$$

substituting into the stencil

$$
\frac{2u_S - 2u_O}{\Delta r^2} + \frac{1}{r_j^2} \frac{u_E - 2u_O + u_W}{\Delta \theta^2} = 0
$$

and collecting terms we find

$$\frac{\Delta r^2u_W + 2r^2\Delta\theta^2u_S -2(r^2\Delta\theta^2 + \Delta r^2)u_O+\Delta r^2U_E}
{r^2\Delta r^2\Delta\theta^2 }=0.$$

We can see that the coefficients of $u_W$, $u_E$ and $u_O$ remain unchanged, the coefficient of $u_N$ is zero, but the coefficient of $u_S$ becomes

$$C_s=\frac{2}{\Delta r^2}.$$

When $\theta>\pi$ and the gradient is one we have:

$$\begin{align*}
\frac{u_N-u_S}{2\Delta r}&=1 \\
u_N-u_S&=2\Delta r \\
u_N &= u_S+2\Delta r
\end{align*}$$

substituting into the stencil
$$\frac{u_W-2u_O+u_E}{r^2\Delta \theta^2}+\frac{2u_S-2u_O+2\Delta r}{\Delta r^2}+\frac{1}{r}=0$$

and collecting terms we find that $u_s$ ceofficient becomes

$$C_s=\frac{2}{\Delta r^2}.$$

We also now have a constant term on the LHS of the stencil, $\frac{2r+\Delta r}{r\Delta r}$.
So we also need to set $b_k=-\frac{2r+\Delta r}{r\Delta r}$ at the North boundary when $\theta>\pi$.

```python
def PolarLaplaceSolver(grid,tol=0.5e-7, quiet=True):
    '''Solve the two dimensional laplace equation on a polar coordinates
    using the bi-conjugate gradient stabilised matrix solver (BiCGStab).
    This function assembles the coeficient matrix A and the RHS vector b
    taking account of the boundary conditions specified in the question.
    It should call the  BiCGStab solver from scipy.  The value tol is p
    assed to BiCGStab routine The solution vector x is then unpacked
    into grid.u. It returns the info value from BiCGStab if this is
    zero everything worked.'''

    # Create the A matrix using the lil format and the b vector
    # as numpy vector.
    N = grid.Nj*grid.Ni
    A_mat = sps.lil_matrix((N, N), dtype=np.float64)
    b_vec = np.zeros(N, dtype=np.float64)

    d_r = grid.Delta_r()
    d_theta = grid.Delta_theta()

    for j in range(grid.Nj): # radial direction
            for i in range(grid.Ni): # circumferential direction
                # equation number
                k = i + grid.Ni*j

                # calculate the coefficients
                north = (2 * grid.r[j,i] + d_r) / (2*grid.r[j,i] * d_r**2)
                c_theta = 1 / (grid.r[j,i]**2 * d_theta**2)  # east and west
                south = (2 * grid.r[j,i] - d_r)/(2 * grid.r[j,i] * d_r**2)
                centre = - 2 * (d_theta**2 * grid.r[j,i]**2 + d_r**2) / (grid.r[j,i]**2 * d_r**2 * d_theta**2)

                # now assemble the matrix
                A_mat[k,k] += centre
                # North boundary
                if j < (grid.Nj-1):
                    A_mat[k,k+grid.Ni] = north
                    kn=k+grid.Ni
                else:
                    kn=k-grid.Ni
                    # Neumann boundary (depends on theta)
                    if grid.theta[j,i]<np.pi:
                        A_mat[k,kn] = 2/d_r**2
                    else:
                        A_mat[k,kn] = 2/d_r**2
                        b_vec[k] = - (2*grid.r[j,i]+d_r)/(grid.r[j,i]*d_r)

                # South boundary
                if j>0 and j< (grid.Nj-1):
                    ks=k-grid.Ni
                    A_mat[k,ks] += south
                else:
                    ks=k+grid.Ni
                    b_vec[k] += 0.0

                # East boundary (periodic)
                if i<test.Ni-1:
                    ke=k+1
                else:
                    ke=k-grid.Ni+1
                A_mat[k,ke] += c_theta

                # West boundary (periodic)
                if i>0:
                    kw=k-1
                else:
                    kw=k+grid.Ni-1
                A_mat[k,kw] += c_theta

                if not quiet:
                    print(i,j,':',ks,kw,k,ke,kn)
                    plt.spy(A_mat)
                    plt.savefig('Figures/Matrix-structure.png')

    # call bicgstab with an ILU preconditioner, drop_tol and
    # fill factor control the accuracy of the ILU approximation.
    ilu = LA.spilu(A_mat.tocsc(), drop_tol=1e-5, fill_factor=100)
    M_mat = LA.LinearOperator(A_mat.shape, ilu.solve)
    x_vec, info = LA.bicgstab(A_mat, b_vec, atol=tol, M=M_mat)

    if info==0:
        # unpack x_vec into u
        for j in range(grid.Nj):
            for i in range(grid.Ni):
                k = i + grid.Ni*j
                grid.u[j,i]=x_vec[k]

    return info
```

```python
# lets set up the test problem on a 5x5 grid
test = coursework_one(6,5)
print(test)
```

    Uniform 6x5 polar grid from r = 1.0 to 3.0.

```python
info = PolarLaplaceSolver(test,quiet=False)
if info==0:
    i_c = np.ones_like(test.u)
    j_c = np.ones_like(test.u)
    for j in range(test.Nj):
        for i in range(test.Ni):
            i_c[j,i]=i
            j_c[j,i]=j
    axs = (plt.figure(constrained_layout=True)
       .subplots(1, 2, sharex=True, sharey=True))

    axs[0].contourf(i_c,j_c,test.r)
    axs[0].set(title=r'$r$', xlabel='i', ylabel='j')
    axs[1].contourf(i_c,j_c,test.theta)
    axs[1].set(title=r'$\theta$', xlabel='i', ylabel='j')

    plt.show()

    plt = test.plot('Coursework 1')
    plt.show()
else:
    print('Error code ',info,' returned by BiCGStab')
```

```python
# Run on a proper grid
test = coursework_one(361,101)
print(test)
info = PolarLaplaceSolver(test)
if info==0:
    plt = test.plot('Coursework 1')
    plt.savefig('Figures/Coursework1.pdf')
    plt.show()
else:
    print('Error code ',info,' returned by BiCGStab')
```

### refinement analysis

I'm going to use $$\int_{\frac\pi2}^{\frac{3\pi}2} u(2.0,\theta)\ d\theta$$ as the integrated quantity.  Other choices are reasonable.

```python
import scipy.integrate as integrate
from refinement_analysis import refinement_analysis

def integrate_u_dtheta(mesh,theta0,theta1,r):
    '''Calculate U=\int_\theta_0^\theta_1u(x,r) dx  using the
    u value stored on the grid and simpsons rule'''

    # find the left and right grid points
    i0,j = mesh.find((theta0,r))
    i1,j = mesh.find((theta1,r))

    if i1==mesh.Ni:
        i1=-1

    # integrate
    return integrate.simps(mesh.u[j,i0:i1],mesh.theta[j,i0:i1])

```

```python
import datetime # just seconds may not be enough

# we need some lists u and dx values
U_val = []
dx_val = []
run_time = []
n_pts =[]
for grid_index in range(5,0,-1):
    ni = 30*2**grid_index + 1
    nj = 30*2**grid_index + 1
    n_pts.append(ni*nj)

    # set up the problem
    test = coursework_one(ni,nj)
    print(test)

    # solve it with machine precision
    start = time.process_time()
    info = PolarLaplaceSolver(test)
    stop = time.process_time()
    print("The solver took {}, the return code is {}" \
          .format(datetime.timedelta(seconds=int(stop-start)),info))

    # save dx and the integral
    dx_val.append(test.Delta_theta())
    U_val.append(integrate_u_dtheta(test,np.pi/2,3*np.pi/2,2.0))
    run_time.append(stop-start)

    test.plot('Coursework 1')
    print('Integrated value is ',U_val[-1],'\n')
```

```python
from refinement_analysis import refinement_analysis
# lets to the refinement analysis
analysis = refinement_analysis(dx_val,U_val)
analysis.report(r'$\int_{0}^{2\pi}u(2.0,\theta) d\theta$')
analysis.graph(True,r'$\int_{0}^{2\pi}u(2.0,\theta) d\theta$',filename='Figures/CW1-refinement.pdf')
```

```python

```