```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
import scipy as sp
from mpl_toolkits.mplot3d import Axes3D
import statsmodels.formula.api as smf
import linregfunc as lr
import pandas as pd
import seaborn as sns

plt.style.use('fivethirtyeight')
```

#  Common Distributions in Hypothesis Testing

Here we will have a quick refresh of distributions that are commonly used in hypothesis testing.

##  The Normal Distributions

Normal Distribution is the most important one among all, here we provide a graphic reminder of bivariate normal distribution. Please check out my notebooks of linear algebra, there is a whole chapter devoted for normal distribution.

For your reference, the pdf of multivariate normal distribution is

$$
p(\boldsymbol{x} ; \mu, \Sigma)=\frac{1}{(2 \pi)^{n / 2}|\Sigma|^{1 / 2}} \exp \left(-\frac{1}{2}(x-\mu)^{T} \Sigma^{-1}(x-\mu)\right) \tag{1}\label{1}
$$

```python
x = np.linspace(-10,10,500)
y = np.linspace(-10,10,500)
X,Y = np.meshgrid(x,y)

pos = np.array([X.flatten(),Y.flatten()]).T # two columns matrix

fig = plt.figure(figsize=(14,14))

#########################
ax = fig.add_subplot(221, projection='3d')

mu_x = 0
mu_y = 0

sigma_x = 3
sigma_y = 15
rho = 3

rv = sp.stats.multivariate_normal([mu_x, mu_y], [[sigma_x, rho], [rho, sigma_y]]) # frozen distribution
ax.plot_surface(X, Y, rv.pdf(pos).reshape(500,500),cmap='coolwarm')

ax = fig.add_subplot(223)
ax.contour(rv.pdf(pos).reshape(500,500))
string1 = r'$\sigma_x = %.1f$, $\sigma_y = %.1f$, $\rho = %.1f$'% (sigma_x, sigma_y, rho)
ax.annotate(text = string1, xy=(0.2, 0.05), xycoords='axes fraction', color = 'r')

####

mu_x = 0
mu_y = 0

sigma_x = 10
sigma_y = 4
rho = 0

rv = sp.stats.multivariate_normal([mu_x, mu_y], [[sigma_x, rho], [rho, sigma_y]]) # frozen distribution
ax = fig.add_subplot(222, projection='3d')
ax.plot_surface(X, Y, rv.pdf(pos).reshape(500,500),cmap='coolwarm')

ax = fig.add_subplot(224)
ax.contour(rv.pdf(pos).reshape(500,500))
string2 = r'$\sigma_x = %.1f$, $\sigma_y = %.1f$, $\rho = %.1f$'% (sigma_x, sigma_y, rho)
ax.annotate(text = string2, xy=(0.2, 0.05), xycoords='axes fraction', color = 'r')
#########################

plt.show()
```

Keep in your mind: _any linear combination of normally distributed variables is yet a normal distribution_.

##  The Chi-Squared Distribution

If an $n$-random vector follows $iid$ normal distribution, $\boldsymbol{z}\sim N(\boldsymbol{0}, \mathbf{I})$. Then the random variable

$$
y = \boldsymbol{z}^T\boldsymbol{z} = \sum_{i=i}^n z_i^2
$$

is said to follow the **chi-squared distribution** with $n$ degrees of freedom. Denoted as

$$
y\sim\chi^2(n)
$$

The mean is

$$\mathrm{E}(y)=\sum_{i=1}^{m} \mathrm{E}\left(z_{i}^{2}\right)=\sum_{i=1}^{m} 1=m$$

And the variance is

$$\begin{aligned}
\operatorname{Var}(y) &=\sum_{i=1}^{m} \operatorname{Var}\left(z_{i}^{2}\right)=m \mathrm{E}\left(\left(z_{i}^{2}-1\right)^{2}\right) \\
&=m \mathrm{E}\left(z_{i}^{4}-2 z_{i}^{2}+1\right)=m(3-2+1)=2 m
\end{aligned}$$

As $n$ increases, the probability density function of $\chi^2$ approaches the $N(m, 2m)$. Here is the graphic demonstration shows how $\chi^2$ distribution changes as d.o.f. rises.

```python
fig, ax = plt.subplots(figsize = (9, 9))
x = np.linspace(0, 50, 1000)
for i in range(1, 14):
    chi_pdf = sp.stats.chi2.pdf(x, i)
    ax.plot(x, chi_pdf, lw = 3, label = '$\chi^2 (%.0d)$'%i)
    ax.legend(fontsize = 12)
ax.axis([0, 20, 0, .6])
plt.show()
```

###  Quadratic Form of $\chi^2$ Distribution

If an $n$-random vector $\boldsymbol{y} \sim N(\boldsymbol{\mu}, \Sigma)$ then

$$
(\boldsymbol{y} - \boldsymbol{\mu})^T\Sigma^{-1}(\boldsymbol{y}-\boldsymbol{\mu})\sim \chi^2(n)
$$

If $\boldsymbol{y} \sim N(\boldsymbol{0}, \Sigma)$ simplifies the expression

$$
\boldsymbol{y}^T\Sigma^{-1}\boldsymbol{y}\sim \chi^2(n)
$$

We will show why that holds by using diagonal decomposition. Since the $\Sigma$ is symmetric, it is orthogonally diagonalizable,

$$
\Sigma = QDQ^T
$$

where

$$
D=\left[\begin{array}{ccccc}
\lambda_{1} & 0 & 0 & \ldots & 0 \\
0 & \lambda_{2} & 0 & \ldots & 0 \\
\vdots & \vdots & \vdots & & \vdots \\
0 & 0 & 0 & \ldots & \lambda_{n}
\end{array}\right]
$$

$\lambda$s are eigenvalues. And $Q^{-1} = Q^T$, $Q$ holds all the eigenvectors of $\Sigma$ which are mutually perpendicular.

Denote $D^*$ as

$$
D^* =
\left[\begin{array}{ccccc}
\frac{1}{\sqrt{\lambda_{1}}} & 0 & 0 & \ldots & 0 \\
0 & \frac{1}{\sqrt{\lambda_{2}}} & 0 & \ldots & 0 \\
\vdots & \vdots & \vdots & & \vdots \\
0 & 0 & 0 & \ldots & \frac{1}{\sqrt{\lambda_{n}}}
\end{array}\right]
$$

Let the matrix $H = QD^*Q^T$, since $H$ is also symmetric

$$
HH^T= QD^*Q^TQD^*Q^T= Q^TD^*D^*Q = QD^{-1}Q^T =\Sigma^{-1}
$$

Furthermore

$$
H\Sigma H^T = QD^*Q^T\Sigma QD^*Q^T = QD^*Q^TQDQ^T  QD^*Q^T = QD^*DD^*Q^T = QQ^T = I
$$

Back to the results from above, we set $\boldsymbol{z} = H^T (\boldsymbol{y}-\boldsymbol{\mu})$, which is standard normal distribution since

$$
E(\boldsymbol{z})= H^TE(\boldsymbol{y}-\boldsymbol{\mu})=\boldsymbol{0}\\
\text{Var}(\boldsymbol{z}) = H\text{Var}(\boldsymbol{y}-\boldsymbol{\mu})H^T =H\Sigma H^T = I
$$

Back to where we started

$$
(\boldsymbol{y}-\boldsymbol{\mu})^T\Sigma^{-1}(\boldsymbol{y}-\boldsymbol{\mu}) = (\boldsymbol{y}-\boldsymbol{\mu})^THH^T (\boldsymbol{y}-\boldsymbol{\mu}) = (H^T (\boldsymbol{y}-\boldsymbol{\mu}))^T(H^T (\boldsymbol{y}-\boldsymbol{\mu})) = \boldsymbol{z}^T\boldsymbol{z}
$$

here we proved that $(\boldsymbol{y}-\boldsymbol{\mu})^T\Sigma^{-1}(\boldsymbol{y}-\boldsymbol{\mu}) \sim \chi^2(n)$.

More details of proof in this page.

##  The Student’s $t$ Distribution

If $z\sim N(0, 1)$ and $y\sim \chi^2(m)$, and $z$ and $y$ are independent, then

$$
t = \frac{z}{\sqrt{y/m}}
$$

follows the **Student's t distribution** with $m$ d.o.f.

Here is the plot of $t$-distribution, note that $t(1)$ is called **Cauchy distribution** which has no moments at all, because integral does not converge due to fat tails.

```python
fig, ax = plt.subplots(figsize = (12, 7))
x = np.linspace(-5, 5, 1000)
for i in range(1, 6):
    chi_pdf = sp.stats.t.pdf(x, i)
    if i == 1:
        ax.plot(x, chi_pdf, lw = 3, label = 'Cauchy', ls = '--')
        continue
    else:
        ax.plot(x, chi_pdf, lw = 3, label = '$t (%.0d)$'%i)
        ax.legend(fontsize = 12)
ax.axis([-5, 5, 0, .4])
ax.set_title("Student's $t$ Distribution", size = 18)
plt.show()
```

As $m \rightarrow \infty$, $t(m)\rightarrow N(0, 1)$, in the limit process, the tails of $t$ distribution will diminish, and becoming thinner.

##  The $F$ Distribution

If $y_1$ and $y_2$ are independent random variables distributed as $\chi^2(m_1)$ and $\chi^2(m_2)$, then the random variable

$$
F = \frac{y_1/m_1}{y_2/m_2}
$$

follows the **$F$ distribution**, denoted $ F(m_1, m_2)$.

```python
x = np.linspace(.001, 5, 100)
fig, ax = plt.subplots(figsize = (12, 7))

df1 = 10
df2 = 2
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, label = '$df_1 = %.d, df_2 = %.d$' %(df1, df2))

df1 = 2
df2 = 10
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, label = '$df_1 = %.d, df_2 = %.d$ '%(df1, df2))

df1 = 8
df2 = 15
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, color = 'red', label = '$df_1 = %.d, df_2 = %.d$' %(df1, df2))

df1 = 15
df2 = 8
f_pdf = sp.stats.f.pdf(x, dfn = df1, dfd = df2)
ax.plot(x, f_pdf, lw =3, label = '$df_1 = %.d, df_2 = %.d$ '%(df1, df2))

ax.legend(fontsize = 15)
ax.axis([0, 4, 0, .8])
ax.set_title("$F$ Distribution", size = 18)
plt.show()
```

# Single Restriction

Any linear restriction, such as $\beta_1 = 5$, $\beta_1 =2{\beta_2}$ can be tested, however, no loss of generality if linear restriction of $\beta_2=0$ is demonstrated.

We first take a look at single restriction, see how FWL regression can help to construct statistic tests.

The regression model is

$$
\boldsymbol{y} = \boldsymbol{X}_1\boldsymbol{\beta}_1+\beta_2\boldsymbol{x}_2+\boldsymbol{u}
$$

where $\boldsymbol{x}_2$ is an $n$-vector, whereas $\boldsymbol{X}_1$ is an $n\times (m-1)$ matrix.

Project $\boldsymbol{X}_1$ off $\boldsymbol{x}_2$, the FWL regression is

$$
\boldsymbol{M}_1\boldsymbol{y} = \beta_2\boldsymbol{M}_1\boldsymbol{x}_2 + \boldsymbol{M}_1\boldsymbol{u}
$$

Applying OLS estimate and variance formula,

$$
\hat{\beta}_2 = [(\boldsymbol{M}_1\boldsymbol{x}_2)^T\boldsymbol{M}_1\boldsymbol{x}_2]^{-1}(\boldsymbol{M}_1\boldsymbol{x}_2)^T\boldsymbol{M}_1\boldsymbol{y}=(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{-1}\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}=\frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2}\\
\text{Var}(\hat{\beta}_2) = \sigma^2 [(\boldsymbol{M}_1\boldsymbol{x}_2)^T(\boldsymbol{M}_1\boldsymbol{x}_2)]^{-1} =  \sigma^2(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{-1}
$$

If null hypothesis is $\beta_2^0=0$, also assuming that $\sigma^2$ is known. Construct a $z$-statistic

$$
z_{\beta_2} = \frac{\hat{\beta}_2}
{\sqrt{\text{Var}(\hat{\beta}_2)}}=\frac{\frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2}}{ \sigma(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{-\frac{1}{2}}}
= \frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{\sigma(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{\frac{1}{2}}}
$$

However, $\sigma$ is not likely to be known, hence we replace it with $s$, the least square standard error estimator. Recall that

$$
s^2 =\frac{1}{n-k} \sum_{t=1}^n u_t^2 = \frac{\boldsymbol{u}^T\boldsymbol{u}}{n-k}= \frac{(\boldsymbol{M_X y})^T\boldsymbol{M_X y}}{n-k}=\frac{\boldsymbol{y}^T\boldsymbol{M_X y}}{n-k}
$$

Replace the $\sigma$ in $z_{\beta_2}$, we obtain $t$-statistic

$$
t_{\beta_2} =  \frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{s(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{\frac{1}{2}}} = \left(\frac{\boldsymbol{y}^T\boldsymbol{M_X y}}{n-k}\right)^{-\frac{1}{2}}\frac{\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{y}}{(\boldsymbol{x}_2^T\boldsymbol{M}_1\boldsymbol{x}_2)^{\frac{1}{2}}}
$$

Of course we can also show it indeed follows the $t$ distribution by using the definition that the ratio of standard normal variable to a $\chi^2$ variable. However, this is unnecessary for us.

#  Multiple Restrictions

Multiple restrictions test can be formulated as following

$$
H_0:\quad \boldsymbol{y} = \boldsymbol{X}_1\boldsymbol{\beta_1} + \boldsymbol{u}\\
H_1:\quad \boldsymbol{y} = \boldsymbol{X}_1\boldsymbol{\beta_1} +\boldsymbol{X}_2\boldsymbol{\beta_2}+ \boldsymbol{u}
$$

$H_0$ is restricted model, $H_1$ is unrestricted. And we denote restricted residual sum of squares as $\text{RRSS}$, and unrestricted residual sum of squares as $\text{URSS}$, the test statistic is

$$
F_{\beta_2}= \frac{(\text{RRSS}-\text{URSS})/r}{\text{URSS}/(n-k)}
$$

where $r=k_2$, the number of restrictions on $\beta_2$.

Using FWL regression with $\boldsymbol{M}_1$ and $\text{TSS = ESS + RSS}$,

$$
\boldsymbol{M}_1\boldsymbol{y}=\boldsymbol{M}_1\boldsymbol{X}_2\boldsymbol{\beta}_2+\boldsymbol{u} \tag{FWL regression}
$$

\begin{align}
\text{URSS}& =(\boldsymbol{M_1y})^T\boldsymbol{M_1y}- \underbrace{\left[\boldsymbol{M}_1\boldsymbol{X}_2[(\boldsymbol{M}_1\boldsymbol{X}_2)^T\boldsymbol{M}_1\boldsymbol{X}_2]^{-1}(\boldsymbol{M}_1\boldsymbol{X}_2)^T\boldsymbol{y}\right]^T}_{\text{projection matrix}\ P}\boldsymbol{y}\\
 &= \boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{y} -\boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{X}_2(\boldsymbol{X}_2^T\boldsymbol{M}_1\boldsymbol{X}_2)^{-1}\boldsymbol{X}^T_2\boldsymbol{M}_1 \boldsymbol{y} = \boldsymbol{y}^T\boldsymbol{M_X}\boldsymbol{y}
\end{align}

$$
\text{RRSS} = (\boldsymbol{M}_1y)^T\boldsymbol{M}_1y = \boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{y}
$$

Therefore
$$
\text{RRSS}-\text{URSS}=\boldsymbol{y}^T\boldsymbol{M}_1\boldsymbol{X}_2(\boldsymbol{X}_2^T\boldsymbol{M}_1\boldsymbol{X}_2)^{-1}\boldsymbol{X}^T_2\boldsymbol{M}_1 \boldsymbol{y}
$$

We have all parts of $F$ statistic, combine them

$$
F_{\boldsymbol{\beta}_{2}}=\frac{\boldsymbol{y}^{T} \boldsymbol{M}_{1} \boldsymbol{X}_{2}\left(\boldsymbol{X}_{2}^{T} \boldsymbol{M}_{1} \boldsymbol{X}_{2}\right)^{-1} \boldsymbol{X}_{2}^{T} \boldsymbol{M}_{1} \boldsymbol{y} / r}{\boldsymbol{y}^{T} \boldsymbol{M}_{\boldsymbol{X}} \boldsymbol{y} /(n-k)}
$$

In contrast, $t$ statistic will be
$$
t_{\beta_{2}}=\sqrt{\frac{\boldsymbol{y}^{\top} \boldsymbol{M}_{1} \boldsymbol{x}_{2}\left(\boldsymbol{x}_{2}^{\top} \boldsymbol{M}_{1} \boldsymbol{x}_{2}\right)^{-1} \boldsymbol{x}_{2}^{\top} \boldsymbol{M}_{1} \boldsymbol{y}}{\boldsymbol{y}^{\top} \boldsymbol{M}_{\boldsymbol{X}} \boldsymbol{y} /(n-k)}}
$$

To test the equality of two parameter vectors, we modify the $F$ test as
$$
F_{\gamma}=\frac{\left(\mathrm{RRSS}-\mathrm{RSS}_{1}-\mathrm{RSS}_{2}\right) / k}{\left(\mathrm{RSS}_{1}+\mathrm{RSS}_{2}\right) /(n-2 k)}
$$
for example, the sample being divided into two subsamples, to compare the stability of two subsamples, this so-called **Chow test** is a common practice.

#  Asymptotic Theory

**Asymptotic Theory** is concerned with the distribution of estimators and test statistics as the sample size $n$ tends to infinity.

##  Law of Large Numbers

The widely-known **Law of Large Numbers** ($\text{LLN}$) takes the form
$$
\bar{x} = \frac{1}{n}\sum_{t=1}^nx_t
$$
where $x_t$ are independent variables with its own bounded variance $\sigma_t^2$ and a common mean $\mu$. $\text{LLN}$ tells that as $n\rightarrow \infty$, then $\bar{x} \rightarrow \mu$.

The **Fundamental Theorem of Statistics** can be proved with $\text{LLN}$.

An **empirical distribution function** ($\text{EDF}$) can be expressed as
$$
\hat{F}(x) \equiv \frac{1}{n} \sum_{t=1}^{n} I\left(x_{t} \leq x\right)
$$
where $I(\cdot)$ is an **indicator function**, which takes value $1$ when its argument is true, otherwise $0$. To prove the Fundamental Theorem of Statistics, we invoke the $\text{LLN}$, expand the expectation

\begin{aligned}
\mathrm{E}\left(I\left(x_{t} \leq x\right)\right) &=0 \cdot \operatorname{Pr}\left(I\left(x_{t} \leq x\right)=0\right)+1 \cdot \operatorname{Pr}\left(I\left(x_{t} \leq x\right)=1\right) \\
&=\operatorname{Pr}\left(I\left(x_{t} \leq x\right)=1\right)=\operatorname{Pr}\left(x_{t} \leq x\right)=F(x)
\end{aligned}

It turns out that $\hat{F}(x)$ is a consistent estimator of $F(x)$.

##  Central Limit Theorems

The **Central Limit Theorems** ($\text{CLT}$) tells that $\frac{1}{n}$ times the sum of $n$ centered random variables will approximately follow a normal distribution when $n$ is sufficiently large. The most well $\text{CLT}$ is **Lindeberg-Lévy** $\text{CLT}$, the quantity
$$
z \equiv \frac{1}{\sqrt{n}} \sum_{t=1}^{n} \frac{x_{t}-\mu}{\sigma}
$$
is **asymptotically distributed** as $N(0,1)$.

We won't bother to prove them, but they are the implicit theoretical foundation when we are discussing **simulation-based tests**.

#  Bootstrapping

The bootstrapping is a computational technique for handling small size sample or any sample that doesn't follow any known distribution.

It's rare to see it in classical linear regression model, since the sampling distribution of coefficients follow normal distribution if disturbance terms follow normal distribution.

However, for demonstration purpose, we will show how a bootstrap test is perform. The model is a simple linear regression
$$
Y_t = 5 + 6 X_t + 10u_t
$$
where $u_t\sim N(0, 1)$.

Generate $Y$ and $X$ then pass them to a data frame.

```python
ols_obj = lr.OLS_Simu(100, 2, [5, 6], 10)
```

```python
bootstr_params_const, bootstr_params_1 = [], []
nr_bootstrp = 100
```

```python
df = pd.DataFrame(ols_obj.X)
df['y'] = ols_obj.y
df.columns = ['const', 'x1', 'y']
```

```python
resample_size = 100
```

Repetitively sample from the generated dataframe and estimate parameters.

```python
fig, ax = plt.subplots(figsize=(18, 8), nrows=1, ncols=3)
for i in range(nr_bootstrp):
    df_resample = df.sample(resample_size, replace=True)
    ols_fit = smf.ols(formula = 'y ~ x1', data = df_resample).fit()
    bootstrap_params = ols_fit.params
    bootstr_params_const.append(bootstrap_params[0])
    bootstr_params_1.append(bootstrap_params[1])
    y_pred_temp = ols_fit.predict(df_resample['x1'])
    ax[0].plot(df_resample['x1'], y_pred_temp, alpha=0.2)
ax[0].set_title('Bootstrap Regression Plot')
ax[1].hist(bootstr_params_const, bins=30)
ax[1].set_title(r'$\beta_1$')
ax[2].hist(bootstr_params_1, bins=30)
ax[2].set_title(r'$\beta_2$')
plt.show()
```

#  Confidence Interval

A **confidence interval** for some parameter $\theta$ consists all values of $\theta_0$ that can't be rejected by corresponding hypothesis. If the finite-sample distribution is known, we have an **exact confidence interval** $[\theta_l, \theta_u]$, of which $\theta_l$ and $\theta_u$ represents lower and upper limit.

For instance, we have a test statistic
$$
\tau\left(\boldsymbol{y}, \theta_{0}\right) \equiv\frac{\hat{\theta}-\theta_{0}}{s_{\theta}}
$$

Here $\boldsymbol{y}$ denotes the sample used to compute the particular realization of the statistic, $\hat{\theta}$ is the estimate of $\theta$ and $s_\theta$ is the corresponding standard error. The confidence interval can be found by solving the equation
$$
\tau(\boldsymbol{y}, \theta)=c_{\alpha}
$$

where $c_\alpha$ is the critical value with significance level $\alpha$. Solve
$$
\frac{\hat{\theta}-\theta}{s_{\theta}}=c_{\alpha}
$$

Two solutions are
$$
\theta_{l}=\hat{\theta}-s_{\theta} c_{\alpha} \quad \text { and } \quad \theta_{u}=\hat{\theta}+s_{\theta} c_{\alpha}
$$
The confidence interval is
$$
[\hat{\theta}-s_{\theta} c_{\alpha},\quad \hat{\theta}+s_{\theta} c_{\alpha}]
$$

To obtain critical values, it is convenient to use cumulative distribution function, e.g. the common critical value of $t$-statistic is $\pm 1.96$ assuming a large degree of freedom.

```python
sp.stats.t.ppf([0.025, 0.975], df=1000)
```

    array([-1.96233908,  1.96233908])

If you wish to have the same mass on each side of the distribution, the quantiles are $\frac{\alpha}{2}$ and $1-\frac{\alpha}{2}$.

For a linear regression model
$$
\operatorname{Pr}\left(t_{\alpha / 2} \leq \frac{\hat{\beta}-\beta}{s} \leq t_{1-(\alpha / 2)}\right)=1-\alpha
$$
Solve it, we get
$$
\operatorname{Pr}\left(\hat{\beta}_{2}-s_{2} t_{\alpha / 2} \geq \beta_{20} \geq \hat{\beta}_{2}-s_{2} t_{1-(\alpha / 2)}\right)
$$

Or more intuitively written as
$$
[\hat{\beta}_{2}-s_{2} t_{1-(\alpha / 2)}, \quad \hat{\beta}_{2}+s_{2} t_{1-(\alpha / 2)}]
$$

```python
df = pd.read_excel('Basic_Econometrics_practice_data.xlsx', sheet_name='US_CobbDauglas')
df.columns = ('Area', 'Output', 'Labour', 'Capital')
ols_obj_fit = smf.ols(formula='np.log(Output) ~ np.log(Labour) + np.log(Capital)', data=df).fit()
```

```python
print(ols_obj_fit.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:         np.log(Output)   R-squared:                       0.964
    Model:                            OLS   Adj. R-squared:                  0.963
    Method:                 Least Squares   F-statistic:                     645.9
    Date:                Fri, 18 Mar 2022   Prob (F-statistic):           2.00e-35
    Time:                        14:16:17   Log-Likelihood:                -3.4267
    No. Observations:                  51   AIC:                             12.85
    Df Residuals:                      48   BIC:                             18.65
    Df Model:                           2
    Covariance Type:            nonrobust
    ===================================================================================
                          coef    std err          t      P>|t|      [0.025      0.975]
    -----------------------------------------------------------------------------------
    Intercept           3.8876      0.396      9.812      0.000       3.091       4.684
    np.log(Labour)      0.4683      0.099      4.734      0.000       0.269       0.667
    np.log(Capital)     0.5213      0.097      5.380      0.000       0.326       0.716
    ==============================================================================
    Omnibus:                       45.645   Durbin-Watson:                   1.946
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              196.024
    Skew:                           2.336   Prob(JB):                     2.72e-43
    Kurtosis:                      11.392   Cond. No.                         201.
    ==============================================================================

    Notes:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
ols_obj_fit.conf_int()[0]
```

    Intercept          3.090929
    np.log(Labour)     0.269428
    np.log(Capital)    0.326475
    Name: 0, dtype: float64

```python
ols_obj_fit.conf_int()
```

      0
      1

      Intercept
      3.090929
      4.684270

      np.log(Labour)
      0.269428
      0.667236

      np.log(Capital)
      0.326475
      0.716084

#  Full Session of Bootstrapping

```python
df = pd.read_excel('Basic_Econometrics_practice_data.xlsx', sheet_name='US_CobbDauglas')
df.columns = ('Area', 'Output', 'Labour', 'Capital')
ols_obj_fit = smf.ols(formula='np.log(Output) ~ np.log(Labour) + np.log(Capital)', data=df).fit()
```

```python
print(ols_obj_fit.summary())
```

                                OLS Regression Results
    ==============================================================================
    Dep. Variable:         np.log(Output)   R-squared:                       0.964
    Model:                            OLS   Adj. R-squared:                  0.963
    Method:                 Least Squares   F-statistic:                     645.9
    Date:                Fri, 18 Mar 2022   Prob (F-statistic):           2.00e-35
    Time:                        14:16:17   Log-Likelihood:                -3.4267
    No. Observations:                  51   AIC:                             12.85
    Df Residuals:                      48   BIC:                             18.65
    Df Model:                           2
    Covariance Type:            nonrobust
    ===================================================================================
                          coef    std err          t      P>|t|      [0.025      0.975]
    -----------------------------------------------------------------------------------
    Intercept           3.8876      0.396      9.812      0.000       3.091       4.684
    np.log(Labour)      0.4683      0.099      4.734      0.000       0.269       0.667
    np.log(Capital)     0.5213      0.097      5.380      0.000       0.326       0.716
    ==============================================================================
    Omnibus:                       45.645   Durbin-Watson:                   1.946
    Prob(Omnibus):                  0.000   Jarque-Bera (JB):              196.024
    Skew:                           2.336   Prob(JB):                     2.72e-43
    Kurtosis:                      11.392   Cond. No.                         201.
    ==============================================================================

    Notes:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

```python
len(ols_obj_fit.params)
```

    3

```python
ols_obj_fit.params['Intercept']
```

    3.8875995240416508

```python
ols_obj_fit.conf_int()[0]
```

    Intercept          3.090929
    np.log(Labour)     0.269428
    np.log(Capital)    0.326475
    Name: 0, dtype: float64

```python
ols_obj_fit.conf_int()
```

      0
      1

      Intercept
      3.090929
      4.684270

      np.log(Labour)
      0.269428
      0.667236

      np.log(Capital)
      0.326475
      0.716084

```python
formula='np.log(Output) ~ np.log(Labour) + np.log(Capital)'
# number of bootstrap rounds
B = 2000

# placeholder matrix B rows and k columns
results_bootrp = np.zeros((B, len(ols_obj_fit.params)))

# row id range
row_id = range(0, df.shape[0])

for i in range(B):
    # this samples with replacement from rows
    this_sample = np.random.choice(row_id, size=df.shape[0],
                                   replace=True) # gives sampled row numbers
    # Define data for this replicate:
    df_bootstrp = df.iloc[this_sample]
    # Estimate model
    params_bootstrp = smf.ols(formula, data=df_bootstrp).fit().params
    # Store in row r of results_boot:
    results_bootrp[i, :] = params_bootstrp
```

```python
results_bootrp = pd.DataFrame(results_bootrp, columns = ['Intercept', 'Labour', 'Capital'])
```

```python
results_bootrp
```

      Intercept
      Labour
      Capital

      0
      3.673604
      0.712615
      0.320502

      1
      4.925368
      0.417899
      0.498794

      2
      3.883234
      0.478961
      0.511914

      3
      3.775734
      0.411409
      0.577818

      4
      3.494996
      0.473819
      0.543496

      ...
      ...
      ...
      ...

      1995
      3.778701
      0.671204
      0.350403

      1996
      4.385100
      0.524669
      0.438880

      1997
      4.050759
      0.528841
      0.455097

      1998
      3.618295
      0.541526
      0.473604

      1999
      3.938293
      0.314725
      0.651557

2000 rows × 3 columns

```python
x_range = np.arange(B)
fig, ax = plt.subplots(figsize=(18, 12), nrows=3, ncols=2)
ax[0, 0].plot(x_range, results_bootrp['Intercept'], lw=1)
ax[0, 1].hist( results_bootrp['Intercept'], bins=50 ,orientation='horizontal')

ax[1, 0].plot(x_range, results_bootrp['Labour'], lw=1)
ax[1, 1].hist( results_bootrp['Labour'], bins=50 ,orientation='horizontal')

ax[2, 0].plot(x_range, results_bootrp['Capital'], lw=1)
ax[2, 1].hist( results_bootrp['Capital'], bins=50 ,orientation='horizontal')
plt.show()
```

```python
g = sns.PairGrid(results_bootrp)
g.fig.set_size_inches(12,12)
g.map_diag(sns.kdeplot)
g.map_lower(sns.kdeplot, cmap="Blues_d")
g.map_upper(sns.scatterplot)
plt.show()
```

```python

```

Chapter 3b - Hypothesis Test and Confidence Interval.ipynb


NOTE: Videos and graphs do not display immediately on all devices.  If this is the case for you, click the "Cell" menu option, and select "Run All".  If you still do not see the videos/graphs, try a different browser.  If that also does not work, please let your instructor know by email.

# Measures of Central Tendency and Spread

In this notebook, we continue our study of continuous random variables (such as the height of a randomly chosen MATH 136 student, or the cholesterol level of a randomly chosen female living in Edmonton).  We turn our attention to the determination of summary statistics from probability density functions.  We focus on two types of statistics, namely measures of central tendency (summarizing typical outcomes of an experiment) and measures of spread (summarizing how similar/varied the outcome of an experiment are).

Three common measures of central tendency are mean (which the same as average, also known as expected value), median, and mode.  Two common measures of spread are variance and standard deviation.

### 1. Review: Mean or Average of a Discrete Random Variable

Recall that the average of a set of discrete values is the sum of the values divided by the number of values in the set.  Mathematically, if there are $n$ values in the set, and the values are denoted $a_i$ for $i = 1, 2, \ldots, n$, then the average is $$a_{ave} = \frac{a_1 + a_2 + \ldots + a_n}{n} = \frac{1}{n} \sum_{i=1}^n a_i.$$

In preparation for the next section, it is helpful to consider a specific example with repeated values.  Suppose that in a group of 10 people:
- 5 people are 40 years old,
- 3 people are 12 years old, and
- 2 people are 7 years old.

The average age of the 10 people in the group is
$$A_{ave} = \frac{ 5(40) + 3(12) + 2(7) }{ 10 } = 25.$$
We can rewrite this calculation as follows:
$$A_{ave} = 40 \frac{5}{10} + 12 \frac{3}{10} + 7 \frac{2}{10}.$$
Note that the fraction 5/10 represents the proportion of people in the group with age 40, that is, the probability that a randomly selected individual from the group is 40 years old.  Similarly, the fraction 3/10 represents the probability that a randomly selected individual from the group is 12 years old, etc.

In the last calculation, the expression on the right-hand side is written in the form of the sum of "outcome times probability of that outcome".

This form motivates the definition of average of a continuous random variable.

### 2. Mean or Average of a Continuous Random Variable

For the purposes of developing the formula of the mean or average of a continuous random variable, consider the probability density function $f(x)$ for a continuous random variable $X$ defined on the interval $[a,b]$.

We will build a Riemann sum:
- divide the interval $[a,b]$ into $n$ subintervals of length $\displaystyle \Delta x = \frac{b-a}{n},$ and
- let $x_i = a + i \cdot \Delta x$ for $i = 0, 1, 2, \ldots, n$,

as shown in the figure below.

Focus on determining the probability that $X$ takes on a value between $x_{i-1}$ and $x_i$.  This is the area under the probability density function between $x_{i-1}$ and $x_i$, which we approximate by the area of the rectangle with width $\Delta x$ and height $f(x_i)$, that is,
$$P( x_{i-1} \le X \le x_{i} ) = \int_{x_{i-1}}^{x_i} f(s) \ ds \approx f(x_i) \Delta x.$$
Roughly speaking, $f(x_i) \Delta x$ is the probability that $X$ takes on a value approximately equal to $x_i$.

Motivated by the form for the average value from the previous section, namely that the average is the sum of "outcome times probability of that outcome", we thus have
\begin{align}
\text{Average} & = x_0 \cdot f(x_0) \Delta x + x_1 \cdot f(x_1) \Delta x + \ldots + x_n \cdot f(x_n) \Delta x \\
& = \sum_{i=1}^n x_i f(x_i) \Delta x.
\end{align}

In the limit, as $n \to \infty$, and introducing the notation $\mu$ for mean or average and $E[X]$ for expected value, we obtain
$$\mu = E[X] = \int_a^b x f(x) \ dx.$$

In general, the mean (average) or expected value of the continuous random variable $X$ is
$$\mu = E[X] = \int_{-\infty}^{\infty} x f(x) \ dx.$$

### 3. Two Additional Measures of Central Tendency: Median and Mode

The median, denoted by $\alpha$, is the point at which exactly half the area under the probability density function lies to the left and the other half lies to the right, that is, the median is
$$\alpha \text{ such that} \int_{\alpha}^{\infty} f(x) \ dx = \frac{1}{2}.$$

The mode represents the most frequent value(s) of $X$.  If the probability density function $f(x)$ has a single peak, then the mode is the value of $x$ at which $f(x)$ has the absolute maximum.  In some cases, the probability density function $f(x)$ has multiple peaks, in which we report multiple modes, each located at a local maximum of $f(x)$.

For symmetric probability density functions with a single peak, the mean, median, and mode are the same.  As we have seen, probability density functions often are not symmetric, and it is common for the mean, median, and density to be different.  Each measure of central tendency has its advantages and disadvantages.  One often determines all three measures of central tendency to obtain insight into the characteristics of a given continuous random variable.

> Mean, median, and mode are not the only measures of central tendency.  If you are curious about additional measures of central tendency, you may find it interesting to check out this Wikipedia page on Average: https://en.wikipedia.org/wiki/Average

In the following video, we work through the details of determining the mean, median, and mode for a continuous random variable with a given probability density function.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('GLwkwyGwM-4')
```

### 4. Variance and Standard Deviation of a Continuous Random Variable

Variance is a measure of how much a random variable is likely to deviate from the mean.  The higher the variance, the more spread out the graph of the probability and vice versa.

The variance of a continuous random variable $X$ with probability density function $f(x)$ and mean $\mu$ is defined as the expected value of the square of the deviation of a random variable from the mean, that is,
\begin{align}
\text{Var}[X] & = E[ ( X - \mu )^2 ] \\
& = \int_{-\infty}^{\infty} (x-\mu)^2 f(x) \ dx = \int_{-\infty}^{\infty} x^2 f(x) \ dx - \mu^2.
\end{align}

The latter two forms of the variance are equivalent; the last one generally is the easiest to work with in calculations.

The standard deviation, denoted by $\sigma$ or SD[$X$], simply is the square root of the variance, that is,
$$\sigma = \text{SD}[X] = \sqrt{ \text{Var}[X] }.$$

In the following video, we work through the details of determining the variance and standard deviation for a continuous random variable with a given probability density function.

```python
from IPython.display import YouTubeVideo
YouTubeVideo('xw-Gf3z8f9k')
```

### 5. Summary

Measures of Central Tendency

A continuous random variable $X$ with probability density function $f(x)$ has the following measures of central tendency:

- The mean, also known as the average or expected value, is
$$\mu = E[X] = \int_{-\infty}^{\infty} x f(x) \ dx.$$

- The median is
$$\alpha \text{ such that} \int_{\alpha}^{\infty} f(x) \ dx = \frac{1}{2}.$$

- The modes are all values of $x$ where $f(x)$ has a local maximum.

Measures of Spread

A continuous random variable $X$ with probability density function $f(x)$ has the following measures of spread:

- The variance is
\begin{align}
\text{Var}[X] & = \int_{-\infty}^{\infty} \left( x - E[X] \right)^2 f(x) \ dx = \int_{-\infty}^{\infty} \left( x - \mu \right)^2 f(x) \ dx\\
& = \int_{-\infty}^{\infty} x^2 f(x) \ dx - (E[X])^2 = \int_{-\infty}^{\infty} x^2 f(x) \ dx - \mu^2.
\end{align}

- The standard deviation is
$$\sigma = \text{SD}[X] = \sqrt{ \text{Var}[X] }.$$

### 6. Further Study

Please refer to Section 12.5 in the textbook for additional treatment of this topic.  Note: If you have a hardcopy of the textbook, there is no Chapter 12.  The eBook that you have access to through WebAssign does have Chapter 12.

### 7. Don't Forget

Don't forget to return to eClass to complete the pre-class quiz.


```julia
using JuMP, Cbc, Plots     # The usual packages
using DelimitedFiles       # IO reading/writing files
using LinearAlgebra        # For convenience (includes function norm(x))
using Combinatorics        # To perform permutations
```

# MS-E2121 - Linear optimization
## Exercise session 8

### Demo exercise: Travelling salesman problem (TSP) - MTZ formulation
#### Examining the formulation

Show that the following formulation $P_{MTZ}$ is valid for the TSP defined on a directed graph $G = (N,A)$ with $N = \{1,\dots,n\}$ cities and arcs $A = \{(i,j) : i,j\in N, i\neq j\}$ between cities.
$$
P_{MTZ} = \left\{
   \begin{array}{ll}
   \displaystyle \sum_{j \in N \setminus \{i\}} x_{ij} = 1, & \forall i \in N \\
   \displaystyle \sum_{j \in N \setminus \{i\}} x_{ji} = 1, & \forall i \in N \\
   \displaystyle u_{i} - u_{j} + (n-1) x_{ij} \leq n - 2, & \forall i,j \in N \setminus \{ 1 \} : i \neq j ~~(*)\\
    x_{ij} \in \{0,1\}, & \forall i,j \in N : i\neq j\\
  \end{array}
\right.
$$

where $x_{ij} = 1$ if city $j\in N$ is visited immediately after city $i \in N$, and $x_{ij} = 0$ otherwise. Constraints $(*)$ with the variables $u_i \in \mathbb{R}$ for all $i\in N$ are called *Miller-Tucker-Zemlin* (MTZ) subtour elimination constraints.

We want to show that
1. Constraints $(*)$ prevent subtours in any solution $x \in P_{MTZ}$.
2. Every TSP solution $x$ satisfies the constraints $(*)$.

To prove the first one, we first assume that a solution $x \in P_{MTZ}$ *has* a subtour with $k$ nodes and $k$ arcs between them, not going through node 1. For example, assume nodes 2, 3 and 5 form a subtour when $n=5$. Let's write the constraints $(*)$ corresponding to this subtour:

\begin{align}
    u_2 - u_3 + 4 \le 3 \\
    u_3 - u_5 + 4 \le 3 \\
    u_5 - u_2 + 4 \le 3
\end{align}

We observe that if $x_{ij} = 1$ such that $i,j \in N \setminus 1$, the constraint $(*)$ can be written as $u_i \le u_j - 1$, which for integer variables is the same as $u_i < u_j$. For a general result, we denote the nodes in the subtour by $\{i_1, ..., i_k\}$ and get $u_{i_1} < u_{i_k} < u_{i_1}$, which is a contradiction. This tells us that there can be no subtour ($k<n$) that doesn't contain node 1. A subtour containing node 1 would imply another subtour not containing node 1, so that is also forbidden by $(*)$. This proves the first part.

For the second part, we notice that the $u$-variables seem to imply an ordering for the nodes. Assume that all tours start from node 1 and $u_i, \ i \in N \setminus 1$ is the position of the node on the tour (the first node visited after the starting node 1 has $u$-value 2, the second one 3 and so on). For each arc $i \rightarrow j$, we have either $x_{ij}=0$ or $x_{ij}=1$.

If $x_{ij} = 0$: there is no arc from $i$ to $j$, and the constraint is $u_{i} - u_{j} \leq n - 2$, which holds, since we defined that $i$ and $j$ can't be 1, and also not greater than $n$. The upper bound of the difference between two $u$-values is thus $n-2$, and the constraint always holds if $x_{ij}=0$.

If $x_{ij} = 1$: there is an arc from $i$ to $j$, and the constraint is $u_{i} - u_{j} + n-1 \leq n - 2$ or $u_{i} - u_{j} \leq -1$, which looks familiar from before and actually holds when we do not have subtours since by our definition, $u_i - u_j = -1$. The key is that this time, we have no subtours that do not contain node 1, which is treated as a special case.

To conclude, we proved that this formulation does not allow subtours, and all valid solutions $x$ satisfy the constraints. Therefore, the formulation is valid.

#### MTZ and naive TSP implementation
We first write some helper functions, starting with one that computes the distances between coordinates:

```julia
## Function for getting the distances array
function get_dist(xycoord::Matrix{},n::Int)
    # Compute distance matrix (d[i,j]) from city coordinates

    dist = zeros(n,n)
    for i = 1:n
        for j = i:n
            d = norm(xycoord[i,:] - xycoord[j,:])
            dist[i,j] = d
            dist[j,i] = d
        end
    end
    return dist
end
```

    get_dist (generic function with 1 method)

A function to convert the adjacency matrix $x$ describing a tour to a vector representing the tour, starting from city 1.

```julia
# Get the optimal tour
# Input
#     x: solution matrix
#     n: number of cities
# Returns
#     tour: ordering of cities in the optimal tour
function gettour(x::Matrix{Int}, n::Int)
    tour = zeros(Int,n+1)   # Initialize tour vector (n+1 as city 1 appears twice)
    tour[1] = 1             # Set city 1 as first one in the tour
    k = 2                   # Index of vector tour[k]
    i = 1                   # Index of current city
    while k <= n + 1        # Find all n+1 tour nodes (city 1 is counted twice)
        for j = 1:n
            if x[i,j] == 1  # Find next city j visited immediately after i
                tour[k] = j # Set city j as the k:th city in the tour
                k = k + 1   # Update index k of tour[] vector
                i = j       # Move to next city
                break
            end
        end
    end
    return tour             # Return the optimal tour
end
```

    gettour (generic function with 1 method)

```julia
## Defining the colors to be used
c_blue = palette(:auto)[1]                         # color :1
c_orange = palette(:auto)[2]                       # color :2
c_green = palette(:auto)[3];                       # color :3
```

The data is stored in csv-files, a common external data format. We use the DelimitedFiles package to read a file into a matrix ```data```.

```julia
# "data16a.csv" has 3 columns which are stored in (Nullable) Arrays
# data[1], data[2], data[3] after the function call
# data = CSV.read(...) below. The columns contain:
#
#     data[:,1]: all cities i in V
#     data[:,2]: x-coordinate of each city i in V
#     data[:,3]: y-coordinate of each city i in V
#

data  = readdlm("data16a.csv", ',')
# data  = readdlm("data16b.csv", ',')
n = 16                  # number of cities
# println(data)         # Look at the data in compact form
V = data[2:n+1,1]       # All cities i in V
x = data[2:n+1,2]       # x-coordinates of cities i in V
y = data[2:n+1,3]       # y-coordinates of cities i in V
xycoord = [x y];        # n x 2 coordinate matrix
```

```julia
function tsp_naive(xycoord::Matrix{}, n::Int)

    # Create a model
    m = Model(Cbc.Optimizer)

    # Here the costs c are the distances between cities
    c = get_dist(xycoord,n)

    ## Variables

    # x[i,j] = 1 if we travel from city i to city j, 0 otherwise.
    @variable(m, x[1:n,1:n], Bin)

    ## Objective

    # Minimize length of tour
    @objective(m, Min, dot(c,x))

    ## Constraints

    # Ignore self arcs: set x[i,i] = 0
    @constraint(m, sar[i = 1:n], x[i,i] == 0)

    # We must enter and leave every city exactly once
    @constraint(m, ji[i = 1:n], sum(x[j,i] for j = 1:n if j != i) == 1)
    @constraint(m, ij[i = 1:n], sum(x[i,j] for j = 1:n if j != i) == 1)

    optimize!(m)

    cost = objective_value(m)         # Optimal cost (length)
    sol_x = round.(Int, value.(x))    # Optimal solution vector

    return m, sol_x, cost
end;
```

```julia
## Solve the problem and evaluate time and memory with @time macro
(m_naive, x_naive, cost_naive) = @time tsp_naive(xycoord, n);

# Get the optimal tour
tour_naive = gettour(x_naive,n);
```

     25.801976 seconds (57.71 M allocations: 2.882 GiB, 6.41% gc time)
    Welcome to the CBC MILP Solver
    Version: 2.10.5
    Build Date: Jan  1 1970

    command line - Cbc_C_Interface -solve -quit (default strategy 1)
    Continuous objective value is 14779.5 - 0.00 seconds
    Cgl0002I 16 variables fixed
    Cgl0004I processed model has 32 rows, 240 columns (240 integer (240 of which binary)) and 480 elements
    Cbc0038I Initial state - 0 integers unsatisfied sum - 0
    Cbc0038I Solution found of 14779.5
    Cbc0038I Before mini branch and bound, 240 integers at bound fixed and 0 continuous
    Cbc0038I Mini branch and bound did not improve solution (0.02 seconds)
    Cbc0038I After 0.02 seconds - Feasibility pump exiting with objective of 14779.5 - took 0.01 seconds
    Cbc0012I Integer solution of 14779.476 found by feasibility pump after 0 iterations and 0 nodes (0.03 seconds)
    Cbc0001I Search completed - best objective 14779.47644152248, took 0 iterations and 0 nodes (0.03 seconds)
    Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
    Cuts at root node changed objective from 14779.5 to 14779.5
    Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)

    Result - Optimal solution found

    Objective value:                14779.47644152
    Enumerated nodes:               0
    Total iterations:               0
    Time (CPU seconds):             0.03
    Time (Wallclock seconds):       0.03

    Total time (CPU seconds):       0.03   (Wallclock seconds):       0.03

```julia
plt = scatter(xycoord[:,1],xycoord[:,2],
      markercolor = c_blue,
      markerstrokewidth = 0,
      legend = false
)
for i in 1:length(tour_naive)-1
    annotate!(xycoord[tour_naive[i],1]+50, xycoord[tour_naive[i],2]+50, ("$(tour_naive[i])", 7, :left))
    plot!(([xycoord[tour_naive[i],1],xycoord[tour_naive[i+1],1]] , [xycoord[tour_naive[i],2],xycoord[tour_naive[i+1],2]]), c = c_blue, label = "")
end
plt
```

```julia
plt = scatter(xycoord[:,1],xycoord[:,2],
      markercolor = c_blue,
      markerstrokewidth = 0,
      legend = false
)
for i in 1:n
    annotate!(xycoord[i,1]+50, xycoord[i,2]+50, ("$(i)", 7, :left))
    for j in 1:n
        if x_naive[i,j] == 1
            plot!(([xycoord[i,1],xycoord[j,1]] , [xycoord[i,2],xycoord[j,2]]), c = c_blue, label = "")
            break
        end
    end
end
plt
```

```julia
# Solve a directed, TSP instance (MTZ formulation)
# Input
#    xycoord: coordinates of city locations
#    n: number of cities
# Returns
#    tour: ordering of cities in the optimal tour
#    cost: cost (length) of the optimal tour
function tsp_mtz(xycoord::Matrix{}, n::Int)

    # Create a model
    m = Model(Cbc.Optimizer)

    # Here the costs c are the distances between cities
    c = get_dist(xycoord,n)

    ## Variables

    # x[i,j] = 1 if we travel from city i to city j, 0 otherwise.
    @variable(m, x[1:n,1:n], Bin)
    # Variables u for subtour elimination constraints
    @variable(m, u[2:n])

    ## Objective

    # Minimize length of tour
    @objective(m, Min, dot(c,x))

    ## Constraints

    # Ignore self arcs: set x[i,i] = 0
    @constraint(m, sar[i = 1:n], x[i,i] == 0)

    # We must enter and leave every city exactly once
    @constraint(m, ji[i = 1:n], sum(x[j,i] for j = 1:n if j != i) == 1)
    @constraint(m, ij[i = 1:n], sum(x[i,j] for j = 1:n if j != i) == 1)

    # MTZ subtour elimination constraints
    @constraint(m, sub[i = 2:n, j = 2:n, i != j], u[i] - u[j] + (n-1)*x[i,j] <= (n-2))

    optimize!(m)

    cost = objective_value(m)         # Optimal cost (length)
    sol_x = round.(Int, value.(x))    # Optimal solution vector

    return m, sol_x, cost
end;
```

```julia
## Solve the problem and evaluate time and memory with @time macro
(m_mtz, x_mtz, cost_mtz) = @time tsp_mtz(xycoord, n);

# Get the optimal tour
tour_mtz = gettour(x_mtz,n);
```

    Welcome to the CBC MILP Solver
    Version: 2.10.5
    Build Date: Jan  1 1970

    command line - Cbc_C_Interface -solve -quit (default strategy 1)
    Continuous objective value is 15273.6 - 0.00 seconds
    Cgl0002I 16 variables fixed
    Cgl0004I processed model has 242 rows, 255 columns (240 integer (240 of which binary)) and 1110 elements
    Cbc0038I Initial state - 29 integers unsatisfied sum - 2.4
    Cbc0038I Pass   1: suminf.    1.73333 (19) obj. 16862 iterations 48
    Cbc0038I Pass   2: suminf.    1.73333 (7) obj. 38620.3 iterations 54
    Cbc0038I Pass   3: suminf.    1.73333 (10) obj. 37603.8 iterations 20
    Cbc0038I Pass   4: suminf.    1.73333 (11) obj. 37200.1 iterations 18
    Cbc0038I Pass   5: suminf.    1.73333 (9) obj. 43903.4 iterations 38
    Cbc0038I Pass   6: suminf.    1.73333 (14) obj. 43717.4 iterations 15
    Cbc0038I Pass   7: suminf.    1.73333 (11) obj. 37517.3 iterations 36
    Cbc0038I Pass   8: suminf.    2.80000 (6) obj. 39892.6 iterations 34
    Cbc0038I Pass   9: suminf.    1.73333 (10) obj. 36107 iterations 25
    Cbc0038I Pass  10: suminf.    1.73333 (10) obj. 37514.5 iterations 27
    Cbc0038I Pass  11: suminf.    1.33333 (17) obj. 42954.2 iterations 86
    Cbc0038I Pass  12: suminf.    0.53333 (4) obj. 43714.8 iterations 51
    Cbc0038I Pass  13: suminf.    0.53333 (4) obj. 43402.5 iterations 28
    Cbc0038I Pass  14: suminf.    1.33333 (4) obj. 45766.7 iterations 70
    Cbc0038I Pass  15: suminf.    1.06667 (11) obj. 45589.8 iterations 56
    Cbc0038I Pass  16: suminf.    1.06667 (4) obj. 47925.3 iterations 39
    Cbc0038I Pass  17: suminf.    1.06667 (4) obj. 46778 iterations 23
    Cbc0038I Pass  18: suminf.    1.06667 (9) obj. 46040.1 iterations 26
    Cbc0038I Pass  19: suminf.    1.06667 (4) obj. 48761.5 iterations 51
    Cbc0038I Pass  20: suminf.    1.06667 (4) obj. 48761.5 iterations 2
    Cbc0038I Pass  21: suminf.    1.06667 (9) obj. 46515.9 iterations 16
    Cbc0038I Pass  22: suminf.    1.06667 (4) obj. 46778 iterations 19
    Cbc0038I Pass  23: suminf.    1.06667 (4) obj. 46778 iterations 0
    Cbc0038I Pass  24: suminf.    1.06667 (9) obj. 46040.1 iterations 16
    Cbc0038I Pass  25: suminf.    1.06667 (4) obj. 48761.5 iterations 51
    Cbc0038I Pass  26: suminf.    1.06667 (4) obj. 48761.5 iterations 4
    Cbc0038I Pass  27: suminf.    1.06667 (9) obj. 46515.9 iterations 29
    Cbc0038I Pass  28: suminf.    1.06667 (4) obj. 46778 iterations 21
    Cbc0038I Pass  29: suminf.    1.06667 (4) obj. 46778 iterations 0
    Cbc0038I Pass  30: suminf.    1.06667 (9) obj. 46040.1 iterations 17
    Cbc0038I No solution found this major pass
    Cbc0038I Before mini branch and bound, 160 integers at bound fixed and 0 continuous
    Cbc0038I Full problem 242 rows 255 columns, reduced to 242 rows 95 columns - too large
    Cbc0038I Mini branch and bound did not improve solution (0.03 seconds)
    Cbc0038I Full problem 243 rows 255 columns, reduced to 243 rows 255 columns - too large
    Cbc0038I After 0.04 seconds - Feasibility pump exiting - took 0.03 seconds
    Cbc0031I 11 added rows had average density of 143.81818
    Cbc0013I At root node, 11 cuts changed objective from 15273.642 to 20217.641 in 100 passes
    Cbc0014I Cut generator 0 (Probing) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 0.094 seconds - new frequency is -100
    Cbc0014I Cut generator 1 (Gomory) - 958 row cuts average 191.2 elements, 0 column cuts (0 active)  in 0.053 seconds - new frequency is 1
    Cbc0014I Cut generator 2 (Knapsack) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 0.018 seconds - new frequency is -100
    Cbc0014I Cut generator 3 (Clique) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 0.004 seconds - new frequency is -100
    Cbc0014I Cut generator 4 (MixedIntegerRounding2) - 2 row cuts average 129.0 elements, 0 column cuts (0 active)  in 0.027 seconds - new frequency is -100
    Cbc0014I Cut generator 5 (FlowCover) - 0 row cuts average 0.0 elements, 0 column cuts (0 active)  in 0.014 seconds - new frequency is -100
    Cbc0010I After 0 nodes, 1 on tree, 1e+50 best solution, best possible 20217.641 (0.49 seconds)
    Cbc0004I Integer solution of 21446.314 found after 3463 iterations and 5 nodes (0.54 seconds)
    Cbc0012I Integer solution   2.693751 seconds (4.25 M allocations: 221.267 MiB, 3.84% gc time)
    of 21098.73 found by DiveCoefficient after 3540 iterations and 8 nodes (0.55 seconds)
    Cbc0038I Full problem 242 rows 255 columns, reduced to 228 rows 46 columns - 12 fixed gives 211, 18 - still too large
    Cbc0038I Full problem 242 rows 255 columns, reduced to 211 rows 18 columns - too large
    Cbc0001I Search completed - best objective 21098.72984805134, took 4363 iterations and 18 nodes (0.62 seconds)
    Cbc0032I Strong branching done 394 times (9752 iterations), fathomed 2 nodes and fixed 2 variables
    Cbc0035I Maximum depth 4, 467 variables fixed on reduced cost
    Cuts at root node changed objective from 15273.6 to 20217.6
    Probing was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.094 seconds)
    Gomory was tried 125 times and created 979 cuts of which 0 were active after adding rounds of cuts (0.056 seconds)
    Knapsack was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.018 seconds)
    Clique was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.004 seconds)
    MixedIntegerRounding2 was tried 100 times and created 2 cuts of which 0 were active after adding rounds of cuts (0.027 seconds)
    FlowCover was tried 100 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.014 seconds)
    TwoMirCuts was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    ZeroHalf was tried 1 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.001 seconds)
    ImplicationCuts was tried 25 times and created 6 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)

    Result - Optimal solution found

    Objective value:                21098.72984805
    Enumerated nodes:               18
    Total iterations:               4363
    Time (CPU seconds):             0.63
    Time (Wallclock seconds):       0.63

    Total time (CPU seconds):       0.63   (Wallclock seconds):       0.63

```julia
## Print the optimal tour and its cost
println("\nOptimal tour: $(tour_mtz)\n")
println("Optimal length: ", cost_mtz)
```

    Optimal tour: [1, 8, 9, 7, 6, 15, 12, 11, 13, 14, 5, 10, 4, 2, 3, 16, 1]

    Optimal length: 21098.729848051342

```julia
plt = scatter(xycoord[:,1],xycoord[:,2],
      markercolor = c_blue,
      markerstrokewidth = 0,
      legend = false
)
for i in 1:length(tour_mtz)-1
    annotate!(xycoord[tour_mtz[i],1]+50, xycoord[tour_mtz[i],2]+50, ("$(tour_mtz[i])", 7, :left))
    plot!(([xycoord[tour_mtz[i],1],xycoord[tour_mtz[i+1],1]] , [xycoord[tour_mtz[i],2],xycoord[tour_mtz[i+1],2]]), c = c_blue, label = "")
end
plt
```

Now that we have solved the problem using the MTZ formulation, let's try solving the same problem starting with the naive implementation and using successive cutset or subtour elimination constraints:

```julia
## Initialisation
(m_naive, x_naive, cost_naive) = tsp_naive(xycoord, n);
subnodes = []
count = 0
stop = 0
lim = 100
methods = [:elimination,:cutset]
method = methods[2];

## Perform cuts to break subtours until we got an optimal
@time while stop == 0 && count < lim

    S = collect(permutations(subnodes,2))     # Possible connections present in the naive implementation
    NS = setdiff(V,subnodes)                  # Nodes that are still not included in the tour

    if method == :cutset
        ## Cutset constraints
        if length(S) > 0
            @constraint(m_naive,sum(m_naive[:x][subnodes[i],NS[j]] for i in 1:length(subnodes), j in 1:length(NS)) >= 1)
        end
    else
        ## Subtour elimination constraints
        if length(S) > 0
            @constraint(m_naive,sum(m_naive[:x][S[i][1],S[i][2]] for i in 1:length(S)) <= length(subnodes)-1)
        end
    end

    set_silent(m_naive)
    optimize!(m_naive)

    cost2 = objective_value(m_naive)          # Optimal cost (length)
    sol_x = round.(Int, value.(m_naive[:x]))     # Optimal solution vector

    tour2 = gettour(sol_x,n)            # Get the optimal tour

    if length(unique(tour2)) < n
        count = count + 1
        println("Method used: ",method)
        println("Subtours present in node 1, update subnodes vector to break the subtour: ", tour2')
        println("\nIteration $(count); not optimal.\n")
        subnodes = unique(tour2);
    else
        println("Method used: ",method)
        println("Optimal tour: ", tour2')
        println("\n Took $(count) iterations to find the optimal solution.")
        S = []
        stop = 1
    end;
end;
```

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 9 8 1 9 8 1 9 8 1 9 8 1 9 8 1 9]

    Iteration 1; not optimal.

    Welcome to the CBC MILP Solver
    Version: 2.10.5
    Build Date: Jan  1 1970

    command line - Cbc_C_Interface -solve -quit (default strategy 1)
    Continuous objective value is 14779.5 - 0.00 seconds
    Cgl0002I 16 variables fixed
    Cgl0004I processed model has 32 rows, 240 columns (240 integer (240 of which binary)) and 480 elements
    Cbc0038I Initial state - 0 integers unsatisfied sum - 0
    Cbc0038I Solution found of 14779.5
    Cbc0038I Before mini branch and bound, 240 integers at bound fixed and 0 continuous
    Cbc0038I Mini branch and bound did not improve solution (0.01 seconds)
    Cbc0038I After 0.01 seconds - Feasibility pump exiting with objective of 14779.5 - took 0.01 seconds
    Cbc0012I Integer solution of 14779.476 found by feasibility pump after 0 iterations and 0 nodes (0.01 seconds)
    Cbc0001I Search completed - best objective 14779.47644152248, took 0 iterations and 0 nodes (0.01 seconds)
    Cbc0035I Maximum depth 0, 0 variables fixed on reduced cost
    Cuts at root node changed objective from 14779.5 to 14779.5
    Probing was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Gomory was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Knapsack was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    Clique was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    MixedIntegerRounding2 was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    FlowCover was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    TwoMirCuts was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)
    ZeroHalf was tried 0 times and created 0 cuts of which 0 were active after adding rounds of cuts (0.000 seconds)

    Result - Optimal solution found

    Objective value:                14779.47644152
    Enumerated nodes:               0
    Total iterations:               0
    Time (CPU seconds):             0.02
    Time (Wallclock seconds):       0.02

    Total time (CPU seconds):       0.02   (Wallclock seconds):       0.02

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 1 8 1 8 1 8 1 8 1 8 1 8 1 8 1]

    Iteration 2; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 16 3 11 12 15 9 8 1 16 3 11 12 15 9 8 1]

    Iteration 3; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 16 3 14 13 11 12 15 9 8 1 16 3 14 13 11 12]

    Iteration 4; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 16 3 1 16 3 1 16 3 1 16 3 1 16 3 1 16]

    Iteration 5; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 3 16 8 9 1 3 16 8 9 1 3 16 8 9 1 3]

    Iteration 6; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 9 15 12 11 13 5 14 3 16 1 8 9 15 12 11]

    Iteration 7; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 16 3 2 5 14 13 11 12 15 9 8 1 16 3 2 5]

    Iteration 8; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 16 3 11 12 15 6 7 9 8 1 16 3 11 12 15 6]

    Iteration 9; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 9 7 6 15 12 11 13 14 3 16 1 8 9 7 6]

    Iteration 10; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 9 15 12 11 13 14 5 10 4 2 3 16 1 8 9]

    Iteration 11; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 7 6 9 1 8 7 6 9 1 8 7 6 9 1 8]

    Iteration 12; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 7 6 12 15 9 1 8 7 6 12 15 9 1 8 7]

    Iteration 13; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 9 7 6 15 12 11 13 5 14 3 16 1 8 9 7]

    Iteration 14; not optimal.

    Method used: cutset
    Subtours present in node 1, update subnodes vector to break the subtour: [1 8 9 7 6 15 12 11 13 14 5 2 3 16 1 8 9]

    Iteration 15; not optimal.

    Method used: cutset
    Optimal tour: [1 8 9 7 6 15 12 11 13 14 5 10 4 2 3 16 1]

     Took 15 iterations to find the optimal solution.
      2.362742 seconds (1.97 M allocations: 99.730 MiB)

### Scheduling problem
<!---%--->
One way to formulate the scheduling problem is to use the job start times $t_j \ge 0$ as the decision variables and then define constraints for making sure two jobs are not done simultaneously. From the problem description, it is clear that the objective is to minimize $\sum_{j=1}^{n}w_jt_j$. For the constraints, we add binary variables $x_{ij}$ such that $x_{ij}=1$ if and only if job $i$ is done at some point before job $j$. We want the following two implications to hold:
\begin{align}
    x_{ij} = 1 \Rightarrow t_j \ge t_i + p_i \\
    x_{ij} = 0 \Rightarrow t_i \ge t_j + p_j
\end{align}

We will then model these implications using a disjunctive big-M formulation (see the ULS problem in Lecture 8).

\begin{align}
    t_i - t_j + p_i &\le M(1-x_{ij}), \ &\forall i,j \in \{1,...,n\}: i \neq j \\
    t_j - t_i + p_j &\le Mx_{ij}, \ &\forall i,j \in \{1,...,n\}: i \neq j
\end{align}

If we then define a large enough big-M ($M \ge t_i - t_j + p_j, \ \forall i,j \in \{1,...,n\}: i \neq j$), these constraints correspond to what we wanted and we have our formulation.

\begin{align}
    \text{min. } &\sum_{j=1}^{n}w_jt_j \\
    \text{s.t. }&t_i - t_j + p_i \le M(1-x_{ij}), \ \forall i,j \in \{1,...,n\}: i \neq j \\
    &t_j - t_i + p_j \le Mx_{ij}, \ \forall i,j \in \{1,...,n\}: i \neq j \\
    &t_j \ge 0, \ \forall j \in \{1,...,n\} \\
    &x_{ij} \in \{0,1\}, \ \forall i,j \in \{1,...,n\}: i \neq j
\end{align}
<!---%--->


# 15 Partial Differential Equations — 2

## Solving Laplace's or Poisson's equation

Still solving **Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

General solution by iteration:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

```python
def Laplace_Jacobi_slow(Phi):
    # Don't use, very slow AND inefficient
    Phi_new = Phi.copy()
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-2):
        for yj in range(1, Ny-2):
            Phi_new[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                  + Phi[xi, yj+1] + Phi[xi, yj-1])
    Phi[:, :] = Phi_new
    return Phi
```

Fast implementation using numpy array operations (vectorized, run at C speed, not Python speed) (see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/13_PDEs/13_PDEs-2-LectureNotes.pdf) for an explanation for how to set up the numpy array operations that give you a 100-fold speed up over `Laplace_Jacobi_slow()`):

```python
def Laplace_Jacobi(Phi):
    """One update in the Jacobi algorithm"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2])
    return Phi
```

#### Gauss-Seidel method
Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

```python
def Laplace_Gauss_Seidel(Phi):
    """One update in the Gauss-Seidel algorithm"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-2):
        for yj in range(1, Ny-2):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
    return Phi
```

#### Fast Gauss-Seidel-like

Divide the lattice into a checkerboard of black and white cells. Update the odd cells first (like Jacobi) but then use the odd cells to update the even ones (Gauss-Seidel-like; see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/15_PDEs/15_PDEs-2-LectureNotes.pdf)). This leads to faster convergence *and* can be done with numpy array operations.

```python
def Laplace_Gauss_Seidel_odd_even(Phi):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2] + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2])
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2] + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2])

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2] + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2])
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2] + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2])
    return Phi
```

### *Converged* solution of the wire-in-a-box problem

Solve the box-wire problem and **make sure that the solution is converged to  `tol = 1e-3`.**

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
#%matplotlib inline
%matplotlib notebook
```

Only execute the next line if you *don't* want interactive plotting (e.g., when exporting to LaTeX/PDF or html):

```python
%matplotlib inline
```

#### Convience plotting functions.
If you provide a filename then output is only written to a file and figures are close to conserve memory. This allows you to plot files in loops and later assemble them into movies using other programs such as ffmpeg, ImageMagick, mencoder, QuickTime 7, ...

```python
def plot_contour(Phi, filename=None):
    """Plot Phi as a contour plot.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    """
    fig = plt.figure(figsize=(5,4))
    ax = fig.add_subplot(111)

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]
    cset = ax.contourf(X, Y, Z, 20, cmap=plt.cm.coolwarm)
    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_aspect(1)

    cb = fig.colorbar(cset, shrink=0.5, aspect=5)
    cb.set_label(r"potential $\Phi$ (V)")

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax

def plot_surf(Phi, filename=None, offset=-20, zlabel=r'potential $\Phi$ (V)',
             elevation=40, azimuth=20):
    """Plot Phi as a 3D plot with contour plot underneath.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    """

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    surf = ax.plot_surface(X, Y, Z, cmap=plt.cm.coolwarm, rstride=2, cstride=2, alpha=0.3)
    cset = ax.contourf(X, Y, Z, 20, zdir='z', offset=offset+Z.min(), cmap=plt.cm.coolwarm)

    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Z.min(), Z.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

The following solution can also write out intermediate solution steps as PNG graphics (commented out for speed). Filenames and directory names are hard-coded.(Important: `plt.ioff()` to disable interactive plotting in the notebook frontend, otherwise it is very slow.)

Put all images into a directory `wire`:

```python
import os, shutil

def mk_new_dir(dirname):
    """Create new directory (remove existing one)"""
    if os.path.exists(dirname):
        shutil.rmtree(dirname)
        print("Removed directory {}".format(dirname))
    os.mkdir(dirname)
    print("Created empty directory {}".format(dirname))

mk_new_dir("./wire")
```

    Removed directory ./wire
    Created empty directory ./wire

Make a movie by saving an image of the potential every 100 steps.

```python
plt.ioff()  # suppress interactive plotting

nsave = 100

Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
        break
    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090299163

Wireframe plot as usual:

```python
# plot Phi(x,y)
x = np.arange(Nmax)
y = np.arange(Nmax)
X, Y = np.meshgrid(x, y)

Z = Phi[X, Y]

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')

ax.view_init(elev=40, azim=20)
```

    <IPython.core.display.Javascript object>

```python
ax = plot_surf(Phi, elevation=40, azimuth=20)
```

    <IPython.core.display.Javascript object>

## Successive Over Relaxation (SOR)
Accelerate convergence with the scheme

\begin{align}
r_{i, j} &= \Phi_{i,j}^\text{new} - \Phi_{i, j}^\text{old}\\
\Phi_{i,j}^\text{new} &= \Phi_{i,j}^\text{old} + \omega r_{i,j}
\end{align}

where the new solution is computed with the Gauss-Seidel scheme.

Values of $1 \leq \omega \leq 2$ may work well, $\omega > 2$ can lead to numerical instabilities. Experiment!

Create images in directory `./wire_SOR`

```python
mk_new_dir("./wire_SOR")
mk_new_dir("./wire_SOR_surf")
```

    Removed directory ./wire_SOR
    Created empty directory ./wire_SOR
    Created empty directory ./wire_SOR_surf

```python
# %%timeit
plt.ioff()

nsave = 10

Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        #plot_contour(Phi, "wire_SOR/phi_{0:08d}.png".format(n_iter))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, "./wire_SOR/phi_{0:08d}.png".format(n_iter))
        #plot_surf(Phi, "./wire_SOR_surf/phi_{0:08d}.png".format(n_iter),
        #          elevation=40, azimuth=20)

else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    SOR converged in 3797 iterations to 0.0009987117944037717

Run to convergence (tol = 1e-03). Record the number of steps required (and visually check the solution).
* Start with $\omega = 1$. What should you get?
* Change $\omega$ and try to get faster convergence? Who can get it to converge in the fewest number of steps?

#### Tuning the code
Alternative implementation that uses more in-place array operations. (However, only marginally faster (~5%) than the previous solution.)

```python
%%timeit
plt.ioff()

Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        #plot_contour(Phi, "wire_SOR/phi_{0:08d}.png".format(n_iter))
        break
    # SOR
    # Phi[:, :] = Phi_old + omega*residual  = omega*Phi + (1-omega)*Phi_old
    Phi *= omega
    Phi += (1-omega)*Phi_old

    if n_iter % 100 == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        #plot_contour(Phi, "wire_SOR/phi_{0:08d}.png".format(n_iter))
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    SOR converged in 3797 iterations to 0.0009987117944294588
    SOR converged in 3797 iterations to 0.0009987117944294588
    SOR converged in 3797 iterations to 0.0009987117944294588
    SOR converged in 3797 iterations to 0.0009987117944294588
    1 loop, best of 3: 499 ms per loop

## Simple example of Poisson's equation: Wire and charge
Add a positive charge and a negative charge in the box.

Now we need to solve **Poisson's equation**.

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

Finite difference solution:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

### Poisson Solver
Modify the Laplace solvers to now solve Poisson's equation:

```python
import numpy as np

def Poisson_Jacobi(Phi, rho, Delta=1.):
    """One update in the Jacobi algorithm for Poisson's equation"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2]) \
                      + np.pi * Delta**2 * rho[1:-1, 1:-1]
    return Phi

def Poisson_Gauss_Seidel(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm for Poisson's equation"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-2):
        for yj in range(1, Ny-2):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1]) \
                            + np.pi * Delta**2 * rho[xi, yj]
    return Phi

def Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    a = np.pi * Delta**2
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2]
                                + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2]) + a * rho[1:-2:2, 1:-2:2]
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2]
                                + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2]) + a * rho[2:-1:2, 2:-1:2]

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2]
                                + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2]) + a * rho[1:-2:2, 2:-1:2]
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2]
                                + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2]) + a * rho[2:-1:2, 1:-2:2]
    return Phi
```

The following code can write partial solutions as images. To make it a bit more comfortable, we generate a directory for the images from Python:

```python
import os

def makedir(dirname):
    try:
        os.mkdir(dirname)
    except OSError:
        pass
    return dirname
```

Solve the Poisson problem with SOR (as above):

```python
plt.ioff()

nsave = 10

Nmax = 100
Max_iter = 10000
omega = 1.99

dirname = makedir("dipole_wire")
filename = os.path.join(dirname, "phi_{0:08d}.png")

dirname3 = makedir("dipole_wire_3d")
filename3 = os.path.join(dirname3, "phi_{0:08d}.png")

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename=filename.format(n_iter))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename=filename.format(n_iter))
        plot_surf(Phi, filename=filename3.format(n_iter),
                  elevation=40, azimuth=20)
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

print("Images in {0} and {1}".format(dirname, dirname3))
plt.ion()

```

    SOR converged in 3823 iterations to 0.0009991605868815044
    Images in dipole_wire and dipole_wire_3d

```python
plot_surf(Phi, elevation=40, azimuth=20)
```

### Charge density
Given a potential $\Phi$, Poisson's equation gives the charge density:
$$
\rho(\mathbf{x}) = -\frac{1}{4\pi} \nabla^2\Phi(\mathbf{x})
$$

Discretized Laplacian
$$
\nabla^2 \Phi \approx \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) +\, \Phi(x,y-\Delta y) - 4\Phi(x, y)}{\Delta^2}
$$

```python
def laplacian2d(f, Delta=1):
    """Finite difference approximation of Del^2 f.

    Arguments
    ---------
    f : M x N matrix
    Delta : float

    Returns
    -------
    M x N matrix, boundaries set to 0
    """

    L = np.zeros_like(f, dtype=np.float64)
    L[1:-1, 1:-1] = f[1:-1, 2:] + f[1:-1, 0:-2] + f[2:, 1:-1] + f[0:-2, 1:-1] - 4*f[1:-1, 1:-1]
    return L/Delta**2

def laplacian2dsimple(f, Delta=1):
    L = np.zeros_like(f, dtype=np.float64)
    for i in range(1, L.shape[0]-1):
        for j in range(1, L.shape[1]-1):
            L[i, j] = f[i, j+1] + f[i, j-1] + f[i+1, j] + f[i-1, j] - 4*f[i, j]
    return L/Delta**2
```

```python
rhox = - laplacian2d(Phi)/(4*np.pi)
```

Does `rhox` show the charges of $+5$ and $-5$ that we introduced with the charge density $\rho$?

```python
print(rhox.min())
print(rhox.max())
```

    -5.00000418672
    5.0

Plot the charge density:

```python
plot_surf(rhox, zlabel=r"charge density $\rho$", elevation=20, azimuth=20, offset=-3)
```

    <IPython.core.display.Javascript object>

    <matplotlib.axes._subplots.Axes3DSubplot at 0x10e76ff98>

The point charges are found correctly but the charge on the wire appear to be absent. The problem is that the Laplacian is not defined on the boundary.

(Also, if the charges are only defined on a single grid cell then the visualization is sometimes not able to show it correctlty.)

#### Testing the implementation of Laplacian

```python
import numpy as np

def test_laplacian2d():
    ftest = np.random.random((200, 200))
    assert np.allclose(laplacian2d(ftest), laplacian2dsimple(ftest))

test_laplacian2d()
```

```python

```


```python
import open3d as o3d
import numpy as np
import sympy as sp
import modern_robotics as mr
```

    Jupyter environment detected. Enabling Open3D WebVisualizer.
    [Open3D INFO] WebRTC GUI backend enabled.
    [Open3D INFO] WebRTCWindowSystem: HTTP handshake server disabled.
    [Open3D INFO] Resetting default logger to print to terminal.

```python
def Ry_sym(theta):
    ct = sp.cos(theta)
    st = sp.sin(theta)
    R = sp.Matrix([[ct, 0.0, st], [0.0, 1.0, 0.0], [-st, 0, ct]])
    return R

def Rx_sym(theta):
    ct = sp.cos(theta)
    st = sp.sin(theta)
    R = sp.Matrix([[1.0, 0.0, 0.0], [0.0, ct, -st], [0.0, st, ct]])
    return R

def skew(v):
    return sp.Matrix([[0, -v[2], v[1]],
                    [v[2], 0, -v[0]],
                    [-v[1], v[0], 0]])

def exp3(omega, theta):
    omega = skew(omega)
    R = sp.eye(3) + sp.sin(theta) * omega + (1 - sp.cos(theta)) * omega * omega
    return R

def exp6(twist, theta):
    omega = skew(twist[:3])
    v = sp.Matrix(twist[3:])
    T = sp.eye(4)
    T[:3,:3] = exp3(twist[:3], theta)
    T[:3,3] = (sp.eye(3) * theta + (1 - sp.cos(theta)) * omega +
              (theta-sp.sin(theta)) * omega * omega) * v
    return T
```

```python
def calc_v(omega_mat, q_mat):
    #omega_mat and q_mat of type matrix with q_i & omega_i as columns
    #Returns v_mat in same type/format
    assert len(omega_mat) == len(q_mat)

    n_joints = omega_mat.shape[1]
    v_mat = sp.zeros(3, n_joints)

    for i in range(n_joints):
        v_mat[:,i] = (-skew(omega_mat.col(i)) * q_mat.col(i))
    return v_mat

def Slist_maker(omega_mat, q_mat): #omega_mat and q_mat of type matrix with q_i & omega_i as columns
    #Returns v_mat in same type/format
    v_mat = calc_v(omega_mat, q_mat)
    n_joints = omega_mat.shape[1]
    Slist = sp.zeros(6, n_joints)

    for i in range(n_joints):
        Slist[:3,i] = omega_mat[:,i]
        Slist[3:,i] = v_mat[:,i]
    return Slist
```

```python
M1=sp.Matrix([[0, 1, 0, 0],
             [1, 0, 0, 0],
             [0, 0, -1, 200],
             [0, 0, 0, 1]])

M2=sp.Matrix([[0, 1, 0, 25],
             [0, 0, 1, 0],
             [1, 0, 0, 400],
             [0, 0, 0, 1]])

M3=sp.Matrix([[1, 0, 0, 25],
             [0, 0, 1, 0],
             [0, -1, 0, 855],
             [0, 0, 0, 1]])

M4=sp.Matrix([[0, 0, -1, 25+420],
             [0, 1, 0, 0],
             [1, 0, 0, 400+455+35],
             [0, 0, 0, 1]])
M5=sp.Matrix([[1, 0, 0, 25+420],
             [0, 0, 1, 0],
             [0, -1, 0, 400+455+35],
             [0, 0, 0, 1]])
M6=sp.Matrix([[0, 0, -1, 525],
             [0, 1, 0, 0],
             [1, 0, 0, 400+455+35],
             [0, 0, 0, 1]])
Mlist = np.array([M1,M2,M3,M4,M5,M6], dtype=float)

om = sp.zeros(3,6)
om1 = om[:, 0] = M1[:3, 2]
om2 = om[:, 1] = M2[:3, 2]
om3 = om[:, 2] = M3[:3, 2]
om4 = om[:, 3] = M4[:3, 2]
om5 = om[:, 4] = M5[:3, 2]
om6 = om[:, 5] = M6[:3, 2]
q = sp.zeros(3,6)
q1 = q[:,0] = M1[:3, 3]
q2 = q[:,1] = M2[:3, 3]
q3 = q[:,2] = M3[:3, 3]
q4 = q[:,3] = M4[:3, 3]
q5 = q[:,4] = M5[:3, 3]
q6 = q[:,5] = M6[:3, 3]

S_list = Slist_maker(om,q)
S_list
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & -1 & 0 & -1\\0 & 1 & 1 & 0 & 1 & 0\\-1 & 0 & 0 & 0 & 0 & 0\\0 & -400 & -855 & 0 & -890 & 0\\0 & 0 & 0 & -890 & 0 & -890\\0 & 25 & 25 & 0 & 445 & 0\end{matrix}\right]$

```python
# class Robot:
#     #Parameters:
#     # Mlist: Pose of all joints in zero-config as homogenous transformation
#     # link_orient: orientation of link in next joints frame (including ground to Link1) ex: ['z', '-z', 'x', 'x', 'z','x']
#     def __init__(self, Mlist, link_orient='x'):
#         self.robotObjects = [o3d.geometry.TriangleMesh.create_coordinate_frame(size=75)]
#         self.current_config = Mlist
#         self.Mlist = Mlist
#         self.num_joints = len(Mlist)
#         self.num_links = len(Mlist)
#         self.length_links = self.findLinkLengths(Mlist)
#         self.link_orient = ['x']*(self.num_links+1) if link_orient == 'x' else link_orient #link attached to preceeding joints x-axis by default

#         self.joints = []  # Of class Joint
#         self.links = []  # Of class Link
#         self.make_robot_objects()  # create all objects of robot (links, frames, joints)
#         self.update_mesh_list()
#         self.__transform(Mlist) # Transforms all objects from {s} to zero-config

# # Calculates link lengths based on M
#     def findLinkLengths(self, Mlist):
#         linkLengths = np.zeros(self.num_links)

#         for i in range(self.num_links):
#             p = Mlist[i][:3, 3]
#             p_pre = np.array([0, 0, 0]) if i == 0 else Mlist[i-1][:3, 3] #1. link from ground ([0,0,0]) to 1. joint
#             linkLengths[i] = np.linalg.norm(p_pre-p)
#             linkLengths[linkLengths <= 0] = 0.1 # cant have zero-length links because of transformation logic
#         return linkLengths

# # creates all o3d-objects of the robot in {s}
#     def make_robot_objects(self):
#         for i in range(self.num_joints):
#             self.joints.append(Joint())
#         for i in range(self.num_links):
#             self.links.append(Link(self.length_links[i], self.link_orient[i]))

#     def update_mesh_list(self):
#         for Joint in self.joints:
#             self.robotObjects.append(Joint.joint)
#             self.robotObjects.append(Joint.coord)
#         for Link in self.links:
#             self.robotObjects.append(Link.link)

#     def allToOrigin(self):  # Sends all objects to Origin
#         T_origin = []
#         for T in self.current_config:
#             T_origin.append(mr.TransInv(T))
#         self.__transform(T_origin)
#         return

#     def transform(self, Slist, thetas):
#         self.allToOrigin()
#         T_list = []  # List to fill with T01,T02,T03...
#         T = np.eye(4)
#         for i in range(len(thetas)):
#             T = T @ exp6(Slist[:, i], thetas[i])
#             T_list.append(T*self.Mlist[i])
#         self.__transform(T_list)
#         self.current_config = T_list
#         return

#     # Moves all objects from {s} to config given by T_list
#     def __transform(self, T_list):  # Private member function
#         for i, J in enumerate(self.joints):
#             J.transform(T_list[i])
#         for i, L in enumerate(self.links):
#             T_links = np.concatenate(([np.eye(4)], T_list[:-1])) #Transform links after joint
#             L.transform(T_links[i])

#     def draw_robot(self):  # Draws all o3d objects in robotObjects list
#         draw(self.robotObjects)

# #______________Joint Class_______________#
# class Joint(Robot):
#     def __init__(self):
#         self.joint = o3d.geometry.TriangleMesh.create_cylinder(
#             radius=10, height=30)
#         self.coord = o3d.geometry.TriangleMesh.create_coordinate_frame(size=25)
#         self.set_colour()

#     def set_colour(self, colour=[0, 1, 0]):
#         self.joint.paint_uniform_color(colour)

#     def transform(self, T):
#         self.joint = self.joint.transform(T)
#         self.coord = self.coord.transform(T)

# #_______________Link Class_________________#
# class Link(Robot):
#     def __init__(self, lenght, orient):
#         self.lenght = lenght

#         if (orient == 'x'):  # Defines link direction from preceeding joint
#             self.link = o3d.geometry.TriangleMesh.create_cylinder(radius=1, height=self.lenght).rotate(
#                 Ry_sym(np.pi/2)).translate(np.array([self.lenght/2, 0, 0]))
#         elif (orient == 'y'):
#             self.link = o3d.geometry.TriangleMesh.create_cylinder(radius=1, height=self.lenght).rotate(
#                 Rx_sym(-np.pi/2)).translate(np.array([0, self.lenght/2, 0]))
#         elif (orient == 'z'):
#             self.link = o3d.geometry.TriangleMesh.create_cylinder(
#                 radius=1, height=self.lenght).translate(np.array([0, 0, self.lenght/2]))
#         elif (orient == '-z'):
#             self.link = o3d.geometry.TriangleMesh.create_cylinder(
#                 radius=1, height=self.lenght).translate(np.array([0, 0, -self.lenght/2]))
#         self.set_colour()

#     def set_colour(self, colour=[0, 0, 1]):
#         self.link.paint_uniform_color(colour)

#     def transform(self, T):
#         self.link = self.link.transform(T)

```

```python

```

```python
KUKzero = Robot(Mlist, ['z', '-z', 'x', 'x', 'z','x'] )
KUKup = Robot( Mlist, ['z', '-z', 'x', 'x', 'z','x'])
KUKdown = Robot( Mlist, ['z', '-z', 'x', 'x', 'z','x'])
```

```python
KUKdown.transform(S_list, [-1.1416,  1.038 , -4.2855, -1.5708,  1.5708, -0.2475])
KUKup.transform(S_list, [-1.1416, -1.3258,  1.3102, -1.5708,  1.5708,  2.9844])

KUKup.draw_robot()
KUKdown.draw_robot()
KUKzero.draw_robot()

```

    WebVisualizer(window_uid='window_0')

    WebVisualizer(window_uid='window_1')

    WebVisualizer(window_uid='window_2')

```python

```


# 線形代数

## ベクトル

以下のように括弧の中に数を縦に配置したものを **ベクトル** と呼ぶ

$$
\left(\begin{array}{c} 5 \\ 7 \\ 9 \end{array}\right)
$$

ベクトルは $\vec{x}$ や $\vec{x}$ (太字斜体) などの表記で表される

ベクトルは数字を縦に並べたものだが、$ (5, 7, 9)^T $ のように **転置演算子** ($^T$) をつけた行ベクトルで表すこともある（大抵は紙面の都合でスペースを使いたくない時）

$$
\left(\begin{array}{c} 1 \\ 2 \\ 3 \end{array}\right) = (1, 2, 3)^T
$$

ベクトル内部に配置された各数を **成分** と呼び、それぞれの成分を上から順に第1成分、第2成分、... と呼ぶ

また、特別なベクトルとしてすべての成分が 0 である **ゼロベクトル** が以下のように定義されている

$$
\vec{0} = \left(\begin{array}{c} 0 \\ 0 \\ \vdots \\ 0 \end{array}\right)
$$

### ベクトルの次元
- **次元**:
    - ベクトルの成分の個数のこと
    - $d$ 次元ベクトルの実数集合を $\mathbb{R}^d$ と表現する

例えば、

$$
\vec{u} = \left(\begin{array}{c} 2 \\ 3 \end{array}\right), \vec{v} = \left(\begin{array}{c} 4 \\ 5 \\ 6 \\ 7 \end{array}\right)
$$

のとき、$\vec{u}$ は2次元ベクトルであり、$\vec{v}$ は4次元ベクトルである

### ベクトルの計算
ベクトルの和は、要素ごとの加算で定義される

$$
\vec{u} = \left(\begin{array}{c} u_1 \\ u_2 \\ \vdots \\ u_d \end{array}\right)
\quad
\vec{v} = \left(\begin{array}{c} v_1 \\ v_2 \\ \vdots \\ v_d \end{array}\right) \\
\vec{u} + \vec{v} = \left(\begin{array}{c} u_1 + v_1 \\ u_2 + v_2 \\ \vdots \\ u_d + v_d \end{array}\right)
$$

ベクトルの和が定義されるのは、和を取ろうとしているベクトルの次元が等しい場合に限られる

ベクトルと対比して、通常の数（実数など）を **スカラー** と呼ぶことがある

ここで、ベクトルのスカラー倍は、各成分にスカラーを掛けたものと定義される

$$
k\vec{v} = \left(\begin{array}{c} kv_1 \\ kv_2 \\ \vdots \\ kv_d \end{array}\right) \quad (k\in{\mathbb{R}})
$$

特に $k = 0$ のときは、任意ベクトル $\vec{v}$ について、次のようになる

$$
0\vec{v} = \vec{0}
$$

次に $k = -1$ の場合を考えると、

$$
(-1)\cdot{\vec{v}} = -\vec{v} = \left(\begin{array}{c} -v_1 \\ -v_2 \\ \vdots \\ -v_d \end{array}\right)
$$

となり、これはベクトルの符号反転となる

これにより、ベクトルの差は以下のように定義される

$$
\vec{u} - \vec{v} = \vec{u} + (-\vec{v}) = \left(\begin{array}{c} u_1 - v_1 \\ u_2 - v_2 \\ \vdots \\ u_d - v_d \end{array}\right)
$$

あるベクトルに対してそのスカラー倍であるベクトルを考えた時、それらのベクトルは **平行** であると呼ぶ

つまり $\vec{v} = k\vec{u}$ のとき、$\vec{u}$ と $\vec{v}$ は平行である

### 交換法則・結合法則・分配法則
ベクトルにも次のような交換法則・結合法則・分配法則が成り立つ

$$
\begin{align}
\vec{u} + \vec{v} &= \vec{v} + \vec{u} \\
(\vec{u} + \vec{v}) + \vec{w} &= \vec{u} + (\vec{v} + \vec{w}) \\
k(\vec{u} + \vec{v}) &= k\vec{u} + k\vec{v} \\
(k + l)\vec{v} &= k\vec{v} + l\vec{v}
\end{align}
$$

また、ゼロベクトルの性質について以下が成り立つ

$$
\begin{align}
\vec{0} + \vec{v} &= \vec{v} \\
\vec{0} - \vec{v} &= -\vec{v} \\
0\vec{v} &= \vec{0}
\end{align}
$$

### ノルム
- **ノルム**:
    - ベクトルの大きさを表す量
    - ノルムの定義はいくつか考えられるが、ここではL1ノルムとL2ノルムのみ考える
        - L1ノルム: $|\vec{v}|_1 = |v_1| + |v_2| + \dots + |v_d| \quad (|x|:\ xの絶対値)$
        - L2ノルム: $\|\vec{v}\| = \sqrt{v_1^2 + v_2^2 + \dots + v_d^2}$

本稿では、特に断りなくベクトルのノルムというときはL2ノルムを指すこととする

また、ベクトルの大きさ（長さ）というときもL2ノルムを指すこととする

ベクトルのスカラー倍のノルムについては以下の式が成り立つ

$$
\begin{align}
|k\vec{v}|_1 &= |k||\vec{v}|_1 \\
\|k\vec{v}\| &= |k|\|\vec{v}\|
\end{align}
$$

ノルムが1であるベクトルを特に **単位ベクトル** と呼び、$\hat{a}$ で表す

#### 演習
次のような3つのベクトルについて、

$$
\vec{u} = \begin{pmatrix}1\\2\\-2\end{pmatrix},\ \vec{v} = \begin{pmatrix}3\\5\\7\end{pmatrix},\ \vec{w} = \begin{pmatrix}1\\-1\\2\end{pmatrix}
$$

以下を求めよ

1. $\vec{u} + \vec{v}$
2. $\vec{u} - \vec{v}$
3. $3\vec{u}$
4. $2\vec{v} + 3\vec{w}$
5. $|\vec{u}|_1$
6. $\|\vec{u}\|$

##### 解
$$
\begin{align}
&1. \quad \vec{u} + \vec{v} = \begin{pmatrix}1+3\\2+5\\-2+7\end{pmatrix} = \begin{pmatrix}4\\7\\5\end{pmatrix} \\
&2. \quad \vec{u} - \vec{v} = \begin{pmatrix}1-3\\2-5\\-2-7\end{pmatrix} = \begin{pmatrix}-2\\-3\\-9\end{pmatrix} \\
&3. \quad 3\vec{u} = \begin{pmatrix}3\times{1}\\3\times{2}\\3\times(-2)\end{pmatrix} = \begin{pmatrix}3\\6\\-6\end{pmatrix} \\
&4. \quad 2\vec{v} + 3\vec{w} = \begin{pmatrix}2\times{3} + 3\times{1} \\ 2\times{5} + 3\times(-1) \\ 2\times{7} + 3\times{2}\end{pmatrix} = \begin{pmatrix}9\\7\\20\end{pmatrix} \\
&5. \quad |\vec{u}|_1 = |1| + |2| + |-2| = 5 \\
&6. \quad \|\vec{u}\| = \sqrt{1^2 + 2^2 + (-2)^2} = 3 \\
\end{align}
$$

```julia
# ベクトル u, v, w
u = [1, 2, -2]; display(u)
v = [3, 5, 7]; display(v)
w = [1, -1, 2]; display(w)
```

    3-element Vector{Int64}:
      1
      2
     -2

    3-element Vector{Int64}:
     3
     5
     7

    3-element Vector{Int64}:
      1
     -1
      2

```julia
u + v # => [4, 7, 5]
```

    3-element Vector{Int64}:
     4
     7
     5

```julia
u - v # => [-2, -3, -9]
```

    3-element Vector{Int64}:
     -2
     -3
     -9

```julia
3u # => [3, 6, -6]
```

    3-element Vector{Int64}:
      3
      6
     -6

```julia
2v + 3w # => [6, 10, 14] + [3, -3, 6] = [9, 7, 20]
```

    3-element Vector{Int64}:
      9
      7
     20

```julia
# L1ノルム: 各成分の絶対値を計算し、合計を算出
## Juliaでベクトルの各成分に対して関数適用する際は dot 演算子を用いる
abs.(u) |> sum
```

    5

```julia
# L2ノルム: 各成分の2乗を計算し、合計の平方根を算出
u.^2 |> sum |> sqrt
```

    3.0

```julia
# L2ノルム: LinearAlgebra.norm 関数でも算出可能
using LinearAlgebra

norm(u)
```

    3.0

### ベクトルの内積
ベクトル $\vec{u} = (u_1, \dots, u_d)^T$, $\vec{v} = (v_1, \dots, v_d)^T$ の **内積** は以下のように定義される

$$
\vec{u}\cdot{\vec{v}} = \vec{u}^T \vec{v} = u_1 v_1 + u_2 v_2 + \dots + u_d v_d
$$

ここで、内積はベクトルではなくスカラー値になることに注意

内積には次の性質がある

$$
\begin{align}
\vec{0}^T\vec{u} &= 0 \\
\vec{u}^T\vec{v} &= \vec{v}^T\vec{u} \\
(\vec{u} + \vec{v})^T\vec{w} &= \vec{u}^T\vec{w} + \vec{v}^T\vec{w} \\
(k\vec{u})^T\vec{v} &= k(\vec{u}^T\vec{v}) \\
\vec{v}^T\vec{v} &= \|\vec{v}\|^2
\end{align}
$$

#### 演習
$$
\begin{align}
&1. \quad \vec{u} = \begin{pmatrix}1\\3\\5\end{pmatrix},\ \vec{v} = \begin{pmatrix}2\\-1\\1\end{pmatrix}\ のとき、\vec{u}^T\vec{v}を求めよ \\
\\
&2. \quad \vec{u}, \vec{v} \in \mathbb{R}^d について、次の式が成り立つことを証明せよ \\
& \quad\quad \|\vec{u} + \vec{v}\|^2 + \|\vec{u} - \vec{v}\|^2 = 2(\|\vec{u}\|^2 + \|\vec{v}\|^2)
\end{align}
$$

##### 解

```julia
u = [1, 3, 5]
v = [2, -1, 1]

# 内積 u^T*v
## 内積演算子は \cdot <tabキー> で記述可能（REPL環境の場合）
u ⋅ v
```

    4

```julia
# 内積は転置ベクトルとベクトルの積でも表される
u' * v
```

    4

```julia
# 記号処理パッケージ
using SymPy

# シンボル U, V を定義
@vars U
@vars V

# ベクトルの和同士の内積 + 差同士の内積
(U + V)^2 + (U - V)^2 |> simplify
```

$\begin{equation*}2 U^{2} + 2 V^{2}\end{equation*}$

```julia
# 実数ベクトル u, v で確認
## (u + v)^2 + (u - v)^2
(u + v) ⋅ (u + v) + (u - v) ⋅ (u - v) |> println # => 82

## 2(|u|^2 + |v|^2)
2(u⋅u + v⋅v) |> println # => 82
```

    82
    82

```julia

```


```python
from sympy import *
from mpmath import atan2, degrees
```

```python
def get_hypotenuse(a, b):
  return sqrt(a*a + b*b)
```

```python
def get_cosine_law_angle(a, b, c):

  cos_gamma = (a*a + b*b - c*c) / (2*a*b)
  sin_gamma = sqrt(1 - cos_gamma * cos_gamma)
  gamma = atan2(sin_gamma, cos_gamma)

  return gamma
```

```python
# Test cosine function

g = get_cosine_law_angle(3, 4, 6)
print(g) # g = 2.05

g = get_cosine_law_angle(3, 4, 5)
print(g) # g = 1.57
```

    2.04691538770669
    1.5707963267949

```python
# Test hypotenuse function

c = get_hypotenuse(3, 4)
print(c) # 5

c = get_hypotenuse(3, 5)
print(c) # sqrt(34) = 0.53

l = get_hypotenuse( 1.5, 0.054)
print(l)

gamma = atan2(1.5, 0.054)
print(gamma)
```

    5
    sqrt(34)
    1.50097168527591
    1.53481186671284

```python
def get_first_three_angles(wrist_center):
  # given the wrist center which a tuple of 3 numbers x, y, z
  # (x, y, z) is the wrist center point wrt base frame
  # return the angles q1, q2, q3 for each respective joint
  # given geometry of the kuka kr210

  x, y, z  = wrist_center

  a1, a2, a3 = 0.35, 1.25, -0.054
  d1, d4 = 0.75, 1.5
  l = 1.50097168527591 #get_hypotenuse(d4, -a3)
  phi = 1.53481186671284 # atan2(d4, -a3)

  x_prime = get_hypotenuse(x, y)
  mx = x_prime -  a1
  my = z - d1
  m = get_hypotenuse(mx, my)
  alpha = atan2(my, mx)

  gamma = get_cosine_law_angle(l, a2, m)
  beta = get_cosine_law_angle(m, a2, l)

  q1 = atan2(y, x)
  q2 = pi/2 - beta - alpha
  q3 = -(gamma - phi)

  return q1, q2, q3
```

```python
# Test wrist center inverse kinematics
wrist_center = 1.79505, 1.84825, 0.3094

q1, q2, q3 = get_first_three_angles(wrist_center)
print("\n q1:", q1, "\n q2:", q2, "\n q3:", q3)# q1 = 0.8, q2 = 1.1, q3 = -0.4

print(q2.evalf())
```

    gamma 1.93479351084403
    phi 1.53481186671284

     q1: 0.799999272099671
     q2: -0.470820255901082 + pi/2
     q3: -0.399981644131188
    1.09997607089381

```python
print("q2:", -1.8470352502004 + 3.14 / 2 )
print("q3:", -3.25681192184063 + 3.14)

wrist_center = 0.750499, 1.201, 2.475

q1, q2, q3 = get_first_three_angles(wrist_center)
print("\n q1:", q1, "\n q2:", q2, "\n q3:", q3)
# q1 = 1.012, q2 = -0.2758003637377226 , q3 = -0.11568

q2.evalf()
```

    q2: -0.2770352502003999
    q3: -0.11681192184062983
    gamma 1.65003113496368
    phi 1.53481186671284

     q1: 1.01227245775763
     q2: -1.8470352502004 + pi/2
     q3: -0.115219268250835

    -0.276238923405500

```python

```

```python

```