# SOLVING LINEARY SYSTEMS OF DIFFERENTIAL EQUATIONS

# Preliminaries

```python
import control as c
from controlSBML.control_sbml import ControlSBML
import controlSBML.constants as cn
try:
    from symmathSBML.symmath_sbml import SymmathSBML
except:
    pass

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import tellurium as te

import os
```

# Terminology and Theory

Let ${\bf x}$ be a vector of $N$ of the variables $[x_1, \cdots, x_N]^T$ such that $\frac{d x_n}{d t} = a_{n, 1} x_1 + \cdots + a_{n, N} x_N$. Or,
$\frac{{d \bf x}}{dt} = {\bf A} {\bf x}$, where the $n,m$ entry in ${\bf A}$ is $a_{n,m}$.

We call this a **linear, homogeneous differential equation**.
If we are given initial values for the $x_n$, then this is an **initial value problem**.

To solve a homogeneous linear system, consider the eigenvalues $\lambda_i$ and associated eigenvectors $e_i$ of ${\bf A}$.
By definition. ${\bf A} {\bf e}_i = \lambda_i {\bf e}_i$.

Now, consider a candidate solution ${\bf x} = c_1 {\bf e}_1 e^{\lambda_1 t} + \cdots +  c_1  {\bf e}_N  e^{\lambda_N t}$.
Note that

$$
\begin{align}
\frac{{d \bf x}}{dt} && =&& c_1 \lambda_1 {\bf e}_1 e^{\lambda_1 t} + \cdots +  c_N \lambda_N {\bf e}_N  e^{\lambda_N t} \\
\end{align}
$$

On the other hand,
$$
\begin{align}
{\bf A} {\bf x} && =&& {\bf A} \left( c_1 {\bf e}_1 e^{\lambda_1 t} + \cdots +  c_N  {\bf e}_N  e^{\lambda_N t} \right) \\
&& = && c_1  \lambda_1 {\bf e}_1 e^{\lambda_1 t} + \cdots + c_1  \lambda_N {\bf e}_N e^{\lambda_N t}  \\
\end{align}
$$

So, the candidate solution results in equality.

Another representation of the solution to a linear homogeneous differential equation is

${\bf x}(t) = e^{ \bf A t}$, which follows from the fact that

\begin{align}
\frac{d {\bf x } (t)}{dt} &&=&& A e^{ \bf A t} \\
&& = && A {\bf x}(t) \\
\end{align}

We we elaborate later on how to calculate $e^{A t}$.

# Solving Initial Value Problems (IVP)

How do we find the $c_n$ if we are given ${\bf x} (0)$?

Let ${\bf E} = \left[ {\bf e_1} \cdots {\bf e_N} \right]^T$. Then, we have the linear system

${\bf x}(0) = {\bf E} \left[ {c_1} \cdots {c_N} \right]^T$.
If ${\bf E}$ is non-singular, then there is a unique solution.

# Systems with Forced Inputs

What is a forced input?

A forced input, or just **input**, is a vector of settings for actuators.
It can affect state (${\bf x}(t)$) directly or indirectly.
We take this into account by inclding another matrix ${\bf B}$ that translates the
input into its impact on state.
That is, the net effect on state is ${\bf B} {\bf u}(t)$.

**Examples**

${\bf x} (t) = e^{ {\bf A} t} {\bf x} (0) + \int_0^t e^{ {\bf A} (t - \tau)} {\bf B} {\bf u}(\tau) d \tau$

This is easier to interpret if the integral is approximated by a sum.

That is,
${\bf x} (t) \approx e^{ {\bf A} t} {\bf x} (0) + \sum_{m=1}^{M \Delta \tau} e^{ {\bf A} (t - m \Delta \tau)} {\bf B} {\bf u}(m \Delta \tau) \Delta \tau$,
where $\Delta \tau = \frac{t}{M}$.

Let's look at each term in the sum: $e^{ {\bf A} (t - m \Delta \tau)} {\bf B} {\bf u} (m \Delta \tau) $.
Another view of this term is to do the following:
* $t_m^{\prime} = t - m \Delta \tau$;
* ${\bf x}_m^{\prime} (0) = {\bf B} {\bf u}(m \Delta \tau)$.

So, $e^{ {\bf A} (t - m \Delta \tau)} {\bf B} {\bf u}(m \Delta \tau) = e^{ {\bf A} t_m^{\prime}} {\bf x}_m^{\prime}(0)$.
Or,
${\bf x} (t) \approx e^{ {\bf A} t} {\bf x} (0) + \sum_{m=1}^{M \Delta \tau}  e^{ {\bf A} t_m^{\prime}} {\bf x}_m^{\prime}(0)$,
That is, the effect of a forced input is just to add up the effects of running the system in different starting states based
on the value of the forced input.

# Application to Reaction Networks

Here, we show how to obtain time domain solutions for reaction networks.
We start the following sequential pathway.

$S_0 \rightarrow S_1 \rightarrow S_2 \rightarrow S_3$

$S_0$ and $S_3$ are fixed species in that there concentrations do not change (e.g., controlled externally).

```python
LINEAR_MDL = """
J0: $S0 -> S1; k0*$S0
J1: S1 -> S2; k2*S1
J2: S2 -> S3; k3*S2
J3: S3 -> $S4;k4*S3

k0 = 1
k1 = 1
k2 = 1
k3 = 1
k4 = 1
S0 = 10
S1 = 0
S2 = 0
S3 = 0
$S4 = $S0
"""
LINEAR_RR = te.loada(LINEAR_MDL)
LINEAR_RR.plot(LINEAR_RR.simulate(0, 10))
```

## The Sequential Pathway as a Linear System

We obtain the differential equations in the usual way.

Let ${\bf N}$ be the stoichiometry matrix, where the rows are species and the columns are reactions.


# Non-trivial Band Topology and the Chern Number
### Christina Lee
### Category:  Graduate
### Topological Physics Series
* [Quantum Anomolous Hall Effect and the Chern Number](../Graduate/Chern-Number.ipynb)
* [SSH Model and the Winding Number](../Graduate/Winding-Number.ipynb)

## Overview

A Chern number tells us whether something non-trivial is going on in the wavefunction and lets us distinguish between different topological phases.

Now let me clarify what I mean when I say "non-trivial".

Normally, when we are studying materials, we move from a spatial dependence for the wavefunction to a momentum dependence across a Brillouin Zone.

When we are in a non-trivial phase, we can't define a wavefunction across the entire Brillouin Zone at the same time. We can rewrite the wavefunction to cover the area that didn't work before, but then some other section isn't well-defined.

To be clear, the physics is well defined everywhere, and every way we write the wavefunction gives the same physics. The problem lies in our inability to write down a single "chart" for the whole Zone. This conundrum is similar to the problem with plotting a globe in 2 dimensions. We always have to make cuts, but the entire globe can be covered by "charts" that make up an "atlas".

## Our Model

A page of a Review of Modern Physics behemoth, "Classification of topological quantum matter with symmetries" [1], presented this model below. It inspired me to dig deeper and fill in the details.   They attributed this model as describing the Quantum Anomolous Hall Effect, QAHE, but this form describes a wide variety topological phenomena.

\begin{equation}
H(k) = R_0(k) \sigma_0 + \vec{R}(k) \cdot \vec{\sigma}
\end{equation}
\begin{equation}
= R_0(k) \sigma_0 + R_1(k) \sigma_1 + R_2(k) \sigma_2 + R_3(k) \sigma_3
\end{equation}
where $\sigma_0$ is the identity matrix and $\sigma_i$ are simply the Pauli matrices.
Combining the terms, $H(k)$ is a 2x2 matrix.
\begin{equation}=
\begin{pmatrix}
R_0 + R_3 & R_1 -i R_2 \\
R_1 + i R_2 & R_0 - R_3
\end{pmatrix}
\end{equation}
The wavefunction will then be 2-valued.  This could denote two sublattices, or two different types of particles.

While we could use any values for $\vec{R}(k)$, we will use
\begin{equation}
\vec{R}(k) = \begin{pmatrix}
-2 \sin k_x \\
-2 \sin k_y \\
\mu +2 \sum_{x,y} \cos k_i \\
\end{pmatrix}
\end{equation}
as it's a fairly simple form that gives us the physics we want and exhibits phase transitions depending on the value of $\mu$.  I set $\mu$ as $1$ early on.  Go through the notebook with that value, then go through the notebook with $\mu = -5, -1, 1, 5$.

```julia
using Plots
pyplot()
```

    Plots.PyPlotBackend()

```julia
μ=1

labels=["-π","-π/2","0","π/2","π"]
ticks=[-π,-π/2,0,π/2,π];

ks=range(-π,stop=π,length=314)
l=length(ks)

ka=Array{Array{Float64},2}(undef,l,l)
for ii in 1:l
    x=ks[ii]
    for jj in 1:l
        ka[ii,jj]=[x,ks[jj]]
    end
end
```

```julia
function R0(k::Array)
    return 0
end

function R1(k::Array)
    return -2*sin(k[1])
end

function R2(k::Array)
    return -2*sin(k[2])
end

function R3(k::Array)
    return μ+2*cos(k[1])+2*cos(k[2])
end
```

    R3 (generic function with 1 method)

```julia
function R(k::Array)
    return sqrt(R1(k)^2+R2(k)^2+R3(k)^2)
end
```

    R (generic function with 1 method)

## Band Diagram

First, let's just take a look at the energy spectrum.  For each $k$, we calculate the eigenvalues of the 2x2 matrix Hamiltonian.

To make the calculation simpler, I denote the function $R$ as
\begin{equation}
R=\sqrt{R_1^2+R_2^2+R_3^2}
\end{equation}
With that, energy can simply be written as
\begin{equation}
\lambda= R_0 \pm R
\end{equation}
$R_0$ moves the entire spectrum up and down but doesn't effect the gap, and won't affect the physics.  That term won't even factor into the eigenvectors.

```julia
function λp(k::Array{Float64})
    return R0(k)+R(k)
end

function λm(k::Array{Float64})
    return R0(k)-R(k)
end
```

    λm (generic function with 1 method)

Notation Note:  I denote the Array evaluation of a function as the function name followed by 'a'.

```julia
λpa=zeros(Float64,l,l)
λma=zeros(Float64,l,l)

for ii in 1:l
    for jj in 1:l
        λpa[ii,jj]=λp(ka[ii,jj])
        λma[ii,jj]=λm(ka[ii,jj])
    end
end
```

```julia
surface(ks,ks,λpa)
surface!(ks,ks,λma)
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx",
    ylabel="ky",
    zlabel="Energy",
    title="Band Diagram")
```

## Eigenvectors

To find the eigenvectors, we find the nullspace of the following matrix,
\begin{equation}
\begin{vmatrix}
R_0 +R_3 - R_0 \mp R & R_1 - i R_2 \\
R_1 + i R_2 & R_0 - R_3 -R_0 \mp R \\
\end{vmatrix}
\end{equation}

Take the bottom row. Second column value multiples first column and first column value multiples second column.
\begin{equation}
\begin{pmatrix}
\pm R + R_3 \\
R_1 + i R_2
\end{pmatrix}
\end{equation}

And now we need to normalize the state
\begin{equation}
\frac{1}{\sqrt{2 R \left( R \pm R_3 \right) }}
\begin{pmatrix}
\pm R + R_3\\
R_1 + i R_2
\end{pmatrix}
\end{equation}
In low energy, just the minus state will be occupied (lower energy), but it has a singularity at $\vec{R} = ( 0 ,0, R) $.

We have two complex numbers in this vector, and thus four things to plot $r_1, \theta_1, r_2, \theta_2$,
\begin{equation}
|\psi \rangle =
\begin{pmatrix}
r_1 e^{i \theta_1} \\
r_2 e^{i \theta_2}
\end{pmatrix}
\end{equation}

If we chose to create our vector from the top row instead,
\begin{equation}
\begin{pmatrix}
R_1 -i R_2 \\
- R_3 \mp R
\end{pmatrix}
\end{equation}
and normalized
\begin{equation}
\frac{1}{\sqrt{2 R \left( R \mp R_3 \right) }}
\begin{pmatrix}
R_1 - i R_2 \\
\pm R - R_3
\end{pmatrix}
\end{equation}
But the minus state for this one has a singularity at $\vec{R} = (0,0, -R )$.

We can move where the singularity is, but we can't get rid of it.  The problem-point doesn't show up in the physics, only in the wavefunction representation of it.  We cannot well represent the state across the entire Brillouin Zone at the same time.  This problem occurs because we have a topologically non-trivial state and a non-zero Chern number.

```julia
function up1(k::Array)
    front=1/sqrt(2*R(k)*(R(k)+R3(k)))
    return front*(R(k)+R3(k))
end

function up2(k::Array)
    front=1/sqrt(2*R(k)*(R(k)+R3(k)))
    return front*(R1(k)+im*R2(k))
end

up=Function[]
push!(up,up1)
push!(up,up2)
```

    2-element Array{Function,1}:
     up1
     up2

```julia
function um1(k::Array)
    front=1/sqrt(2*R(k)*(R(k)-R3(k)))
    return front*(-R(k)+R3(k))
end

function um2(k::Array)
    front=1/sqrt(2*R(k)*(R(k)-R3(k)))
    return front*(R1(k)+im*R2(k))
end

um=Function[]
push!(um,um1)
push!(um,um2)
```

    2-element Array{Function,1}:
     um1
     um2

```julia
upa=zeros(Complex{Float64},2,l,l)
uma=zeros(Complex{Float64},2,l,l)

for ii in 1:l
    for jj in 1:l
        upa[1,ii,jj]=up[1](ka[ii,jj])
        upa[2,ii,jj]=up[2](ka[ii,jj])
        uma[1,ii,jj]=um[1](ka[ii,jj])
        uma[2,ii,jj]=um[2](ka[ii,jj])
    end
end
```

```julia
# Plotting θ2
heatmap(ks,ks,angle.(uma[2,:,:])-angle.(uma[1,:,:]))
plot!(xticks= (ticks,labels),
yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="difference in phase, lower energy")
```

```julia
surface(ks,ks,abs2.(uma[2,:,:]))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx", ylabel="ky",
    title="magnitude,second component, lower energy")
```

```julia
surface(ks,ks,abs2.(uma[1,:,:]))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
xlabel="kx",
ylabel="ky",
title="magnitude,first component, lower energy")
```

## Calculating the Connection

The first step in calculating the Chern number is evaluating the Berry Connection.

\begin{equation}
\mathcal{A}^{i} = \langle u (k,r) | d_{i} u (k,r) \rangle
\end{equation}

Though $\mathcal{A}^i$ looks like a vector, it is not invariant under gauge tranformation.  If a wavefunction transforms as
\begin{equation}
| u(k,r) \rangle \rightarrow e^{-i \phi} | u(k,r) \rangle
\end{equation}
then the connection transforms as
\begin{equation}
\mathcal{A}^i \rightarrow \mathcal{A}^i -i \partial_i \phi
\end{equation}

Near the Dirac points in the Brillouin Zone, the wavefunction has quite a high curvature, which makes the numerical computation of the derivative prone to errors.  I tried numerical derivation but was unable to arrive at a correct, stable answer.  Therefore, I'm using the 'ForwardDiff' Package to take derivatives analytically.

```julia
using ForwardDiff
```

Before dealing with the physics, let's just look at the syntax for the package.

Here's just an example of taking the gradient of $x^2$.

```julia
ex=x->ForwardDiff.gradient(t->t[1]^2,x)
ex([1])
```

    1-element Array{Int64,1}:
     2

Now let's apply that syntax to our wavefunctions.

The 'ForwardDiff' package seems to only work on purely real functions, so we have to take the derivative of the real and imaginary parts separately.

```julia
dum1=kt->ForwardDiff.gradient(um1,kt)

Rdum2=kt->ForwardDiff.gradient(t->real(um2(t)),kt)
Idum2=kt->ForwardDiff.gradient(t->imag(um2(t)),kt)
dum2(k)=Rdum2(k)+im*Idum2(k)
```

    dum2 (generic function with 1 method)

With the derivatives, we can now calculate the connection.

```julia
Amkx(k)=conj(um1(k))*dum1(k)[1]+conj(um2(k))*dum2(k)[1]
Amky(k)=conj(um1(k))*dum1(k)[2]+conj(um2(k))*dum2(k)[2]
```

    Amky (generic function with 1 method)

```julia
Akxa=Array{Complex{Float64}}(undef,l,l)
Akya=Array{Complex{Float64}}(undef,l,l)
for ii in 1:l
    for jj in 1:l
        Akxa[ii,jj]=Amkx(ka[ii,jj])
        Akya[ii,jj]=Amky(ka[ii,jj])
    end
end
```

```julia
heatmap(ks,ks,log.(abs.(Akxa)))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx",ylabel="ky",
    title="A_kx Magnitude Berry Connection")
```

```julia
heatmap(ks,ks,log.(abs.(Akya)))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx",ylabel="ky",
    title="A_ky Magnitude  Berry Connection")
```

```julia
heatmap(ks,ks,angle.(Akxa))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx",ylabel="ky",
    title="A_kx Berry Connection angle")
```

```julia
heatmap(ks,ks,angle.(Akya))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx",ylabel="ky",
    title="A_ky Berry Connection angle")
```

## Curvature

I previously had a section here about Berry phases, but I need to fix it.  Until then, please believe the mass of physics literature that from the guage-dependent Berry connection, we can get a gauge independent quantity called the <i>Berry Curvature</i>.  Hopefully I can clarify this mathematical magic when I understand it myself.

Of course it's called Berry.  Every thing in this calculation is named after Berry.  At least its not Euler.  Done with side note, back to equations.
Here's our version of Gauss's Theorem:
\begin{equation}
\oint \mathcal{A} \cdot ds  = \iint F_{n}^{xy} d^2 k
\end{equation}

And $F_n^{xy}$ will take on this form:
\begin{equation}
F_{n}^{xy} = \nabla \times \mathcal{A} = \partial_{k_x} \mathcal{A}^y_{n} - \partial_{k_y} \mathcal{A}^x_{n}
\end{equation}
\begin{equation}
= \partial_{k_x} \langle u | \partial_{k_y} u \rangle - \partial_{k_y} \langle u | \partial_{k_x} u \rangle
\end{equation}

TIME FOR MORE DERIVATIVES!

```julia
DRAmkx=kt->ForwardDiff.gradient(t->(real(Amkx(t))),kt )
DImAmkx=kt->ForwardDiff.gradient(t->(imag(Amkx(t))),kt )

DRAmky=kt->ForwardDiff.gradient(t->(real(Amky(t))),kt )
DImAmky=kt->ForwardDiff.gradient(t->(imag(Amky(t))),kt )
```

    #29 (generic function with 1 method)

```julia
F(k)=DRAmky(k)[1]+im*DImAmky(k)[1]-DRAmkx(k)[2]-im*DImAmkx(k)[2]
```

    F (generic function with 1 method)

```julia
Fa=Array{Complex{Float64}}(undef,l,l)
for ii in 1:l
    for jj in 1:l
        Fa[ii,jj]=F(ka[ii,jj])
    end
end
```

```julia
heatmap(ks,ks,abs.(Fa))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx",ylabel="ky",
    title="F Berry curvature norm")
```

```julia
heatmap(ks,ks,angle.(Fa))
plot!(xticks= (ticks,labels),
    yticks=(ticks,labels),
    xlabel="kx",ylabel="ky",
    title="F Berry Curvature phase")
```

## Chern Number

Last Step!

Easy calculation now in terms of code, but this number has a great deal of significance.  I'm still trying to wrap my head around it.  The Chern number seems to pop up in a variety of obscure mathematical stuff over this physicist's head, but hopefully none of that is necessary to grasp its incredible mind-blowing usefullness.

This single integer not only seperates out topological phases from topologically trivial phases, but seperates different topological phases from each other.  And always evaluates to an integer.

\begin{equation}
Ch = \frac{1}{2 \pi i} \iint_{BZ} F^{xy}_n d^2 k
\end{equation}

If you don't get approximately an integer, try a finer mesh.

```julia
sum(Fa)*(ks[2]-ks[1])^2/(2π*im)
```

    1.0250189329637458 + 1.0118627829401277e-34im

## Victory !!!!!!!!!!

You made it!

Even if you didn't understand everything, or really understand much at all, pat yourself on the back.  This is the very frontier of science.

[1] Ching Kai Chiu, Jeffrey C.Y. Teo, Andreas P. Schnyder, and Shinsei Ryu, Classification of topological quantum matter with symmetries, Reviews of Modern Physics 88 2016, no. 3, 1–70.

@article{RevelsLubinPapamarkou2016,
    title = {Forward-Mode Automatic Differentiation in Julia},
   author = {{Revels}, J. and {Lubin}, M. and {Papamarkou}, T.},
  journal = {arXiv:1607.07892 [cs.MS]},
     year = {2016},
     url = {https://arxiv.org/abs/1607.07892}
}

```julia

```


Fill in any place that says `# YOUR CODE HERE` or YOUR ANSWER HERE, as well as your name and collaborators below.
Grading for pre-lecture assignments is all or nothing. Partial credit is available for in-class assignments and checkpoints, but **only when code is commented**.

```python
NAME = ""
COLLABORATORS = ""
```

---

```python
import grading_helper as _test
```

```python
# Space for imports, utility functions, etc.

# YOUR CODE HERE
```

# Time-Dependent Schrödinger Equation

(Adapted from textbook exercise 9.8)

Perhaps the most important partial differential equation, at least for physicists, is the Schrödinger equation. In one dimension the Schrödinger equation for a particle of mass $m$ with no potential energy reads

$$-\frac{\hbar^2}{2m} \frac{\partial^2\psi}{\partial x^2} = \mathrm{i}\hbar \frac{\partial\psi}{\partial t}\,.$$

In units where $\hbar/2m=1$, the equation simplifies to

$$-\frac{\partial^2\psi}{\partial x^2} = \mathrm{i}\frac{\partial\psi}{\partial t}\,.$$

For simplicity, let's put our particle in a box with impenetrable walls, so that we only have to solve the equation in a finite-sized space. The box forces the wavefunction $\psi$ to be zero at the walls, which we'll put at $x=0$ and $x=L$.

Replacing the second derivative in the Schrödinger equation with a finite difference and applying the forward Euler method, we get the FTCS equation

$$\psi(x,t+h) = \psi(x,t) + h \frac{\mathrm{i}}{a^2} \bigl[ \psi(x+a,t) + \psi(x-a,t) - 2\psi(x,t) \bigr]\,,$$

where $a$ is the spacing of the spatial grid points and $h$ is the size of the time-step. (Be careful not to confuse the time-step $h$ with Planck's constant $\hbar$.) Performing a similar step in reverse, we get the
backward Euler method

$$\psi(x,t+h) - h \frac{\mathrm{i}}{a^2} \bigl[ \psi(x+a,t+h) + \psi(x-a,t+h) - 2\psi(x,t+h) \bigr] = \psi(x,t)\,.$$

And taking the average of these two, we get the Crank-Nicolson equation for the Schrödinger equation:

$$\begin{align}
\psi(x,t+h) - h \frac{\mathrm{i}}{2a^2} \bigl[ &\psi(x+a,t+h) + \psi(x-a,t+h) - 2\psi(x,t+h) \bigr]\\
  &= \psi(x,t) + h \frac{\mathrm{i}}{2a^2} \bigl[ \psi(x+a,t) + \psi(x-a,t) - 2\psi(x,t) \bigr]\,.
\end{align}$$

This gives us a set of simultaneous equations, one for each grid point.

The boundary conditions on our problem tell us that $\psi=0$ at $x=0$ and $x=L$ for all $t$. In between these points we have grid points at $a$, $2a$, $3a$, and so forth. Let us arrange the values of $\psi$ at these interior points into a vector

$$ \boldsymbol{\psi}(t) = \begin{pmatrix} \psi(a,t) \\ \psi(2a,t) \\ \psi(3a,t) \\ \vdots \end{pmatrix}\,.$$

Then the Crank-Nicolson equations can be written in the form

$$\mathbf{A}\,\boldsymbol{\psi}(t+h) = \mathbf{B}\,\boldsymbol{\psi}(t)\,,$$

where the matrices $\mathbf{A}$ and $\mathbf{B}$ are both symmetric and tridiagonal:

$$\mathbf{A} = \begin{pmatrix} a_1 & a_2 \\
                         a_2 & a_1 & a_2 \\
                             & a_2 & a_1 & a_2 \\
                             &     & a_2 & a_1 \\
                             &     &     &     & \ddots
\end{pmatrix},\qquad\qquad
\mathbf{B} = \begin{pmatrix} b_1 & b_2 \\
                             b_2 & b_1 & b_2 \\
                                 & b_2 & b_1 & b_2 \\
                                 &     & b_2 & b_1 \\
                                 &     &     &     & \ddots
\end{pmatrix}\,,$$

with

$$ a_1 = 1 + \mathrm{i}\frac{h}{a^2},\qquad a_2 = - \mathrm{i}\frac{h}{2a^2},\qquad b_1 = 1 - \mathrm{i}\frac{h}{a^2},\qquad b_2 = \mathrm{i}\frac{h}{2a^2}\,.$$

The equation $\mathbf{A}\,\boldsymbol{\psi}(t+h) = \mathbf{B}\,\boldsymbol{\psi}(t)$ has precisely the form $\mathbf{A}\mathbf{x} = \mathbf{v}$ of the simultaneous equation problems we studied in Chapter 6 and can be solved using the same methods. Specifically, since the matrix $\mathbf{A}$ is tridiagonal in this case, we can use the fast tridiagonal version of Gaussian elimination.

The last piece we need are the initial conditions. Suppose that at time $t=0$ the wavefunction of the electron has the form

$$\psi(x,0) = \exp \biggl[ -\frac{(x-x_0)^2}{2\sigma^2} \biggr]e^{\mathrm{i}\kappa x}\,,$$

where

$$x_0 = \frac{L}{2},\qquad \sigma = \frac{L}{10},\qquad \kappa = \frac{50}{L}\,,$$

and $\psi=0$ on the walls at $x=0$ and $x=L$. (This expression for $\psi(x,0)$ is not normalized—there should really be an overall multiplying coefficient to make sure that the probability density for the electron integrates to unity. It's safe to drop the constant, however, because the Schrödinger equation is linear, so the constant cancels out on both sides of the equation and plays no part in the solution. If desired, it's easy to normalize the final result.)

Write a program to perform the Crank-Nicolson method for this electron, calculating the vector $\boldsymbol{\psi}(t)$ of values of the wavefunction for each time step between $t=0$ and $t=0.1$. Use the initial wavefunction above and $N=1001$ spatial grid points, so that $a=L/1000$. Since everything scales with $L$, the easiest approach is to set $L=1$.

Your program will have to perform the following steps:

- First, given the vector $\boldsymbol{\psi}(0)$ at $t=0$, you will have to multiply by the matrix $\mathbf{B}$ to get a vector $\mathbf{v} = \mathbf{B}\,\boldsymbol{\psi}$. Because of the tridiagonal form of $\mathbf{B}$, this is fairly simple. The $i$th component of $\mathbf{v}$ is given by

 $$v_i = b_1\psi_i + b_2(\psi_{i+1}+\psi_{i-1})\,.$$

 You will also have to choose a value for the time-step $h$. A reasonable choice is $h=10^{-4}$ (giving us 1000 time steps).

- Second, you will have to solve the linear system $\mathbf{A}\,\mathbf{x}=\mathbf{v}$ for $\mathbf{x}$, which gives you the new value of $\boldsymbol{\psi}$. You could do this using a standard linear equation solver like the function `solve` in `scipy.linalg`, but since the matrix $\mathbf{A}$ is tridiagonal, a better approach would be to use `solve_banded`.

- Third, once you have the code in place to perform a single step of the calculation, extend your program to perform repeated steps and hence solve for $\psi$ at a sequence of times a separation $h$ apart. Note that the
matrix $\mathbf{A}$ is independent of time, so it doesn't change from one step to another. You can set up the matrix just once and then keep on reusing it for every step.

- Finally, make an animation of the time evolution of the probability density $|\psi(x,t)|^2$ for the particle.

```python
%%graded # 20 points

# YOUR CODE HERE
```

Assignments/11.5 Checkpoint - Time Dependent Schrodinger Equation.ipynb


# ЛР №5

# Решение систем нелинейных уравнений

```python
from scipy.optimize import fsolve
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.patches as mpatches
import sympy as sp
import numpy as np
import math
```

### 1
Отделить корни заданного уравнения графически

Заданное уравнение

```python
f = lambda x: np.cos(x) - np.sqrt(x + 2) + 1
df = lambda x: -1 * math.sin(x) - 0.5 * math.pow(x + 2, -0.5)
```

Отделяем корни

```python
%matplotlib inline
```

```python
roots = fsolve(f, 2)
x = np.array(range(-1, 30))
y = f(x)
plt.plot(x, y)
plt.axvline(x=roots, color='r', linestyle='-')
red_patch = mpatches.Patch(color='red', label=f'x={roots[0]:.3f}')
plt.legend(handles=[red_patch])
plt.show()
```

### 2
Решить уравнения методом хорд

```python
x0 = 0
x1 = 1
if f(x0) * f(x1) < 0:
    while abs(x1 - x0) > 1e-5:
        x = x0 - f(x0) * (x1 - x0) / (f(x1) - f(x0))
        x0, x1 = x1, x
    print(f"\nКорень уравнения: {x:.3}")
```

    Корень уравнения: 0.823

### 3
Решить уравнение методом касательных

```python
x0 = 100
x1 = x0 - f(x0) / df(x0)
while abs(x1 - x0) > 1e-5:
    x = x1 - f(x1) / df(x1)
    x0, x1 = x1, x
print(f"\nКорень уравнения: {x:.3}")
```

    Корень уравнения: 0.823

### 4
Отделить корни заданной системы нелинейных уравнений графически

```python
sys_f1 = lambda x, y: sp.tan(x * y) - x**2
sys_f2 = lambda x, y: 0.7 * x**2 + 2 * y**2 - 1
```

Отделяем корни

```python
x, y = sp.symbols('x y')
p0 = sp.plot_implicit(sys_f2(x, y), (x, -1.5, 1.5), (y, -1.5, 1.5), show=False, line_color='b')
p1 = sp.plot_implicit(sys_f1(x, y), (x, -1.5, 1.5), (y, -1.5, 1.5), show=False, line_color='r')
p0.extend(p1)
p0.show()
```

### 5
Решить систему методом простых итераций

```python
F = lambda xy: [np.tan(xy[0] * xy[1]) - xy[0]**2, 0.7 * xy[0]**2 + 2 * xy[1]**2 - 1]
fi = lambda xy: [math.sqrt(math.tan(xy[0] * xy[1])), math.sqrt((1 - 0.7 * xy[0]**2) / 2)]
minus_fi = lambda xy: [-math.sqrt(math.tan(xy[0] * xy[1])), -math.sqrt((1 - 0.7 * xy[0]**2) / 2)]
def J(xy):
    res = [[0.0, 0.0], [0.0, 0.0]]
    res[0][0] = -2 * xy[0] + xy[1] * (np.tan(xy[0]*xy[1])**2 + 1)
    res[0][1] = xy[0] * (np.tan(xy[0]*xy[1])**2 + 1)
    res[1][0] = 1.4 * xy[0]
    res[1][1] = 4 * xy[1]
    return np.array(res)
```

```python
def iterr(x0, itrr_func):
    x1 = itrr_func(x0)
    while abs(x1[0] - x0[0]) > 1e-5 or abs(x1[1] - x0[1]) > 1e-5:
        x = itrr_func(x1)
        x0, x1 = x1, x
    print("Корень системы:", ", ".join(map(str, x)))
```

```python
iterr([0.83, 1], fi)
iterr([0, 0.8], fi)
iterr([-0.83, -0.7], minus_fi)
iterr([0, -0.8], minus_fi)
```

    Корень системы: 0.6310229619451228, 0.6005262403306086
    Корень системы: 0.0, 0.7071067811865476
    Корень системы: -0.6310288456446511, -0.6005230551716196
    Корень системы: 0.0, -0.7071067811865476

### 6
Решить систему методом Ньютона

```python
def newton(x0):
    i = 0
    x1 = [k - m for (k, m) in zip(x0, np.linalg.inv(J(x0)) @ F(x0))]
    while abs(x1[0] - x0[0]) > 1e-5 or abs(x1[1] - x0[1]) > 1e-5:
        i += 1
        x = [k - m for k, m in zip(x1, np.linalg.inv(J(x1)) @ F(x1))]
        x0, x1 = x1, x
    print("Корень системы:", ", ".join(map(str, x)))
    print(f"Количество итераций: {i}\n")
```

```python
newton([0.8, 0.9])
newton([0, 0.5])
newton([-0.6, -0.6])
newton([-0, -0.9])
```

    Корень системы: 0.6310253503923949, 0.6005268124953965
    Количество итераций: 7

    Корень системы: 0.0, 0.7071067811873449
    Количество итераций: 3

    Корень системы: -0.6310253503228611, -0.6005268125070298
    Количество итераций: 2

    Корень системы: 0.0, -0.7071067811865502
    Количество итераций: 3

### 7
Решить систему модифицированным методом Ньютона. Сравнить скорости сходимости методов

```python
def modified_newton(x0):
    i = 0
    J0 = np.linalg.inv(J(x0))
    x1 = [k - m for (k, m) in zip(x0, J0 @ F(x0))]
    while abs(x1[0] - x0[0]) > 1e-5 or abs(x1[1] - x0[1]) > 1e-5:
        i += 1
        x = [k - m for k, m in zip(x1, J0 @ F(x1))]
        x0, x1 = x1, x
    print("Корень системы:", ", ".join(map(str, x)))
    print(f"Количество итераций: {i}\n")
```

```python
modified_newton([0.8, 0.9])
modified_newton([0, 0.5])
modified_newton([-0.6, -0.6])
modified_newton([0, -0.9])
```

    Корень системы: 0.6310296933645543, 0.6005255939269765
    Количество итераций: 94

    Корень системы: 0.0, 0.7071039681017769
    Количество итераций: 11

    Корень системы: -0.6310254516477837, -0.6005267964046216
    Количество итераций: 4

    Корень системы: 0.0, -0.7071086513254751
    Количество итераций: 6


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import scipy as sp
import scipy.stats
import statsmodels.api as sm
import statsmodels.formula.api as smf

import seaborn as sns
from scipy.optimize import minimize
```

#  Simple Linear Regression

This is a simple linear regression model as in every textbooks
$$
Y_i=\beta_1+\beta_2X_i+u_i
$$
where $Y$ is **dependent variable**, $X$ is **independent variable** and $u$ is **disturbance term**. $\beta_1$ and $\beta_2$ are unknown parameters that we are aiming to estimate by feeding the data in the model. Without disturbance term, the model is simple a function of a straight line in $\mathbb{R}^2$, such as
$$
Y = 2 + 3X
$$

```python
X = np.linspace(1, 10, 10)
Y = 2 + 3*X
fig, ax = plt.subplots(figsize = (7, 7))
ax.plot(X, Y)
ax.scatter(X, Y, c ='r')
ax.grid()
ax.set_title('$Y=2+3x$')
ax.set_xlim(0, 10)
ax.set_ylim(0, 40)
plt.show()
```

There are five reasons justified that we need a disturbance term:

1. omission of independent variables

2. aggregation of variables

3. model misspecification

4. function misspecification, eg. should be nonlinear rather than linear

5. measurement error

The second one means that if we intend to aggregate the variable to a macro level, for instance every family has a consumption function, but aggregation on a national level causes discrepancies which contribute to the disturbance term.

The third and forth one will be discussed in details in later chapter.

The fifth one includes all types of error, man-made or natural.

#  Odinary Least Squares

**Odinary Least Squares** is the most common estimation technique used in basic econometrics, it is popular due to its _simplicity_ and _transparency_. You'll be able to derive the whole estimation process by hand-calculation, all steps will have _closed-form expression_.

Let's dive in our first plot.

```python
beta1 = 2
beta2 = 3

X = np.linspace(1, 10, 10)
u = 5 * np.random.randn(10)

Y = beta1 + beta2*X + u

Y_hat = 2 + 3*X
fig, ax = plt.subplots(figsize = (7, 7))

for i in range(len(Y)):
    dot_fit_values = [X[i], X[i]]
    dot_org_values = [Y[i], Y_hat[i]]
    ax.plot(dot_fit_values, dot_org_values, linestyle = '--', color = 'red', label = 'residual')

ax.plot(X, Y_hat)
ax.scatter(X, Y_hat, c = 'k')
ax.scatter(X, Y, c ='r')
ax.grid()
ax.set_title('$\hat Y =2+3X$')
plt.show()
```

We have plotted a fitted line onto $10$ observations (red dots) which was generated by $Y_i = 2+3X_i+5u_i$, where $u_i \sim N(0, 1)$. For easy demonstration, say we have a 'perfect' estimator that provides
$$
b_1 = 2\\
b_2 = 3
$$
where $b_1$ and $b_2$ are estimates, in contrast $\beta_1$ and $\beta_2$ are model parameters.

Therefore we can plot a fitted line (blue line) $\hat{Y} = 2+3X$. The red dashed line is the difference of $Y_i$ and $\hat{Y}_i$, we officially call it **residual**, denoted as $e_i$.
$$
e_i = Y_i - \hat{Y}_i = Y_i - b_1-b_2X_i
$$
The OLS algorithm is aiming find the estimates of $\beta_1$ and $\beta_2$ such that
$$
\text{min}RSS=\sum_{i=1}^n e^2_i
$$
where $RSS$ is the **residual sum of squares**.

To minimise the $RSS$, we simply take _partial derivatives_ w.r.t. $b_2$ and $b_1$ respectively. The results are the OLS estimators of simple linear regression, which are
\begin{align}
b_2 &=\frac{\sum_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})}{\sum^n_{i=1}(X_i-\bar{X})^2}=\frac{\text{Cov}(X, Y)}{\text{Var}(X)}\\
b_1 &= \bar{Y}-b_2\bar{X}
\end{align}

With these formulae in mind, let's perform a serious OLS estimation. Considering possible repetitive use of OLS in this tutorial, we will write a class for OLS.

```python
class S_OLS:
    '''Create instances with S_OLS(X, Y), where X and Y are data array.'''
    def __init__(self, X, Y):
        self.X = X
        self.Y = Y

    def ols(self):
        '''Estimate the data with OLS method, and return b1 and b2.'''
        cov_mat = np.cov(self.X, self.Y)
        self.b2 = cov_mat[0, 1]/cov_mat[0, 0]
        self.b1 = np.mean(self.Y) - self.b2*np.mean(self.X)
        self.Y_hat = self.b1 + self.b2*self.X
        print('b1 estimate: {:.4f}'.format(self.b1))
        print('b2 estimate: {:.4f}'.format(self.b2))
        return self.Y_hat, self.b2, self.b1

    def simul_plot(self, beta1, beta2):
        '''Plot scatter plot and fitted line with ols_plot(self, beta1, beta2),
        beta1 and beta2 are parameters of data generation process.'''
        fig, ax = plt.subplots(figsize = (7, 7))
        for i in range(len(Y)):
            dot_fit_values = [self.X[i], self.X[i]]
            dot_org_values = [self.Y[i], self.Y_hat[i]]
            ax.plot(dot_fit_values, dot_org_values, linestyle = '--', color = 'red')
        ax.scatter(self.X, self.Y_hat, c = 'k')
        ax.scatter(self.X, self.Y, c ='r')
        ax.plot(self.X, self.Y_hat, label = '$b_1$= {:.2f}, $b_2$={:.2f}'.format(b1, b2))
        ax.grid()
        ax.set_title('$\hat Y ={:.2f}+{:.2f}X$'.format(b1, b2))
        Y_hat_perfect = beta1 + beta2*X
        ax.plot(X, Y_hat_perfect, label = r'$\beta_1=2, \beta_2=3$')
        ax.legend()
        plt.show()

    def ols_plot(self, xlabel, ylabel):
        self.xlabel = xlabel
        self.ylabel = ylabel
        fig, ax = plt.subplots(figsize = (7, 7))
        ax.scatter(self.X, self.Y_hat, c = 'k')
        ax.scatter(self.X, self.Y, c ='r')
        ax.plot(self.X, self.Y_hat, label = '$b_1$= {:.2f}, $b_2$={:.2f}'.format(self.b1, self.b2))
        ax.grid()
        ax.set_title('$\hat Y ={:.2f}+{:.2f}X$'.format(self.b1, self.b2))
        ax.set_xlabel(self.xlabel)
        ax.set_ylabel(self.ylabel)

    def r_sq(self):
        '''Calculate coefficient of determination and correlation of Y and Yhat'''
        self.ESS = np.var(self.Y_hat)
        self.RSS = np.var(self.Y-self.Y_hat)
        self.R_sq = self.ESS/self.RSS
        return self.ESS, self.RSS, self.R_sq

```

```python
X = np.random.randn(10)
Y = np.random.randn(10)
np.corrcoef(X, Y)
```

    array([[ 1.        , -0.12628739],
           [-0.12628739,  1.        ]])

```python
s_ols = S_OLS(X, Y)
Y_hat, b2, b1 = s_ols.ols()
```

    b1 estimate: -0.1137
    b2 estimate: -0.1228

A convenient function ```np.polyfit``` of curve fitting could verify our results.

```python
np.polyfit(X, Y, 1)
```

    array([-0.12282449, -0.11372037])

The plot the fitted line $b_1+b_2X$, original line $\beta_1+\beta_2X$ and observations.

```python
s_ols.ols_plot('X', 'Y')
```

From the estimation result and graph above, we can notice $b_1$ and $b_2$ are close to true paramters $\beta_1$ and $\beta_2$,  nevertheless minor discrepancies present, this is absolute normal in estimation, no estimation techniques can guarantee a perfect estimation result.

#  Interpretation of Estimates

We will check on a real dataset to understand some basic princicples of interpreting estimates. The dataset has $25$ Chinese cities with average house ¥/$m^2$ and average annual disposable income in 2020.

Load the data with Pandas.

```python
df = pd.read_excel('Basic_Econometrics_practice_data.xlsx', sheet_name = 'CN_Cities_house_price'); df.head()
```

      cities
      house_price
      salary

      0
      Shenzhen
      87957
      64878

      1
      Beijing
      64721
      69434

      2
      Shanghai
      59072
      72232

      3
      Xiamen
      49803
      58140

      4
      Guangzhou
      39851
      68304

Estimate parameters with OLS algorithm.

```python
s_ols_house_income = S_OLS(df['salary'], df['house_price'])
Y_hat, b2, b1 = s_ols_house_income.ols()
```

    b1 estimate: -29181.1698
    b2 estimate: 1.1010

Plot the observation and regression line.

```python
s_ols_house_income.ols_plot('Disposable Income', 'House Price')
```

$b_2$ can be interpreted literally as the graph shows, as disposable income increases $1$ yuan (Chinese currency unit), the house price increases $1.1$ yuan.

As for $b_1$ we can't interpret that if the disposable income is zero, the house price is $-29181$ yuan, it doesn't make any sense. The basic principle of interpreting $b_1$ is to check if it has a plausible meaning when independent variable equals zero. If no sensible meaning, you don't need to interpret it.

#  Important Results of OLS

Some features of OLS could provide us some insight of internal mechanism of the algorithm.

The **first** one is
$$
\bar{e}=0
$$
It is true because
$$
\bar{e}=\bar{Y}-b_1-b_2\bar{X}=\bar{Y}-(\bar{Y}-b_2\bar{X})-b_2\bar{X}=0
$$
holds. We can demonstrate numerically with the variables that we have defined in house price example.

```python
e = df['house_price'] - Y_hat
np.mean(e)
```

    7.275957614183426e-13

It is not theoretically zero due to some numerical round-off errors, but we treat it as zero.

The **second** feature is
$$
\bar{\hat{Y}}=\bar{Y}
$$

```python
print('Mean of Y hat: {}'.format(np.mean(Y_hat)))
print('Mean of Y: {}'.format(np.mean(df['house_price'])))
```

    Mean of Y hat: 30218.879999999994
    Mean of Y: 30218.88

The **third** and **forth** features are
$$
\sum_i^n X_ie_i=0\\
\sum_i^n \hat{Y}_ie_i=0
$$

This can be shown by using a _dot product_ function.

```python
print(np.dot(df['salary'],e))
print(np.dot(Y_hat,e))
```

    5.364418029785156e-07
    -3.0174851417541504e-07

Actually, lots of econometric theory can be conveniently derived by linear algebra, if you are curious about details and have solid linear algebra skills, you can study my advanced econometric theory lecture notes. For instance, in linear algebra, covariance has a geometric interpretation
$$
\text{Cov}(X, Y)=x\cdot y= ||x||||y||\cos{\theta}
$$
where $x$ and $y$ are vectors in $\mathbb{R}^n$. If dot product equals zero, geometrically these two vectors are perpendicular, denote as $x\perp y$. Therefore the third and forth features are equivalent to
$$
\text{Cov}(X, e)=0\\
\text{Cov}(\hat{Y}, e)=0
$$
i.e. $x\perp e$ and $\hat{y} \perp e$. Traditionally, the vectors are denoted as lower case letters.

#  Variance of Decomposition

**Variance of decomposition** is based on analysis of variance (ANOVA), if you don't know what ANOVA is meant for, check here. We know that any observation can be decomposed as a fitted value and a residual
$$
Y_i = \hat{Y}_i+e_i
$$
Take variance on both sides
$$
\text{Var}(Y)=\text{Var}(\hat{Y}+e)=\operatorname{Var}(\hat{Y})+\operatorname{Var}(e)+ \underbrace{2 \operatorname{Cov}(\hat{Y}, e)}_{=0}
$$
Or in the explicit form
$$
\frac{1}{n} \sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}=\frac{1}{n} \sum_{i=1}^{n}\left(\hat{Y}_{i}-\overline{\hat{Y}}\right)^{2}+\frac{1}{n} \sum_{i=1}^{n}\left(e_{i}-\bar{e}\right)^{2}
$$

Use the OLS features, i.e. $\bar{\hat{Y}}=\bar{Y}$ and $\bar{e}=0$, the equation simplifies into
$$
\underbrace{\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}}_{TSS}=\underbrace{\sum_{i=1}^{n}\left(\hat{Y}_{i}-\bar{Y}\right)^{2}}_{ESS}+\underbrace{\sum_{i=1}^{n} e_{i}^{2}}_{RSS}
$$
where $TSS$ means **total sum of squares**, $ESS$ means **explained sum of squares**.

#  Coefficient of Determination

Though $ESS$ is called 'explained' part, it might be entirely wrong due to misspecification of model. That being said, we still need a quantitative indicator that tells us how much the model is able to 'explain' the behaviour of the dependent variables.

The **coefficient of determination** is most intuitive indicator
$$
R^2 = \frac{ESS}{TSS}=\frac{\sum_{i=1}^{n}\left(\hat{Y}_{i}-\bar{Y}\right)^{2}}{\sum_{i=1}^{n}\left(Y_{i}-\bar{Y}\right)^{2}}
$$

We have written a ```r_sq()``` method in the ```S_OLS``` class.

```python
ess, rss, r_sq = s_ols_house_income.r_sq()
print(r_sq)
```

    0.5744603125471682

It means the disposable income can explain $57\%$ of house price variation. Furthermore,
$$
R^2 = \frac{TSS - RSS}{TSS}=1-\frac{RSS}{TSS}
$$
it is clear that minimise $RSS$ is equivalent to maximise $R^2$.

Alternatively, the $R^2$ can be shown its relationship with correlation coefficient $r_{Y, \hat{Y}}$.

$$
\begin{aligned}
r_{Y, \hat{Y}} &=\frac{\operatorname{Cov}(Y, \hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\frac{\operatorname{Cov}([\hat{Y}+e], \hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\frac{\operatorname{Cov}(\hat{Y}, \hat{Y})+\operatorname{Cov}(e, \hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\frac{\operatorname{Var}(\hat{Y})}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}} \\
&=\frac{\sqrt{\operatorname{Var}(\hat{Y}) \operatorname{Var}(\hat{Y})}}{\sqrt{\operatorname{Var}(Y) \operatorname{Var}(\hat{Y})}}=\sqrt{\frac{\operatorname{Var}(\hat{Y})}{\operatorname{Var}(Y)}}=\sqrt{R^{2}}
\end{aligned}
$$

#  Gauss-Markov Conditions

In order to achieve the best estimation by OLS, the disturbance term ideally has to satisfy four conditions which are called **Gauss-Markov Conditions**. Provided that all G-M conditions satisfied, OLS is the preferred over all other estimators, because mathematically it is proved to be the **Best Linear Unbiased Estimator** (BLUE). This conclusion is called **Gauss-Markov Theorem**.

1. $E(u_i|X_i)=0$
2. $E(u_i^2|X_i)= \sigma^2$ for all $i$, **homoscedasticity**
3. $\text{Cov}(u_i, u_j)=0, \quad i\neq j$, no **autocorrelation**.
4. $\text{Cov}(X_i, u_i)=0$, assuming $X_i$ is non-stochastic.

In addition to G-M conditions, we also assume normality of disturbance term, i.e. $u_i\sim N(0, \sigma^2_u)$, which is guaranteed by _Central Limit Theorem_.

In practice, almost impossible to have all conditions satisfied simultaneously or even one perfectly, but we must be aware of severity of violations, because any violation of G-M condition will compromise the quality of estimation results. And identifying which condition is violated could also lead us to corresponding remedies.

#  Random Components of Regression Coefficients

According to the OLS formula of $b_2$
$$
b_{2}=\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}
$$

Plug in $Y=\beta_1+\beta_2X+u$:
$$
b_{2}=\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}=\frac{\operatorname{Cov}(X, \beta_1+\beta_2X+u)}{\operatorname{Var}(X)}
$$

The covariance operation rules come in handy
$$
\operatorname{Cov}(X, \beta_1+\beta_2X+u)=\operatorname{Cov}\left(X, \beta_{1}\right)+\operatorname{Cov}\left(X, \beta_{2} X\right)+\operatorname{Cov}(X, u)
$$

where $\operatorname{Cov}\left(X, \beta_{1}\right)=0$, and $\operatorname{Cov}\left(X, \beta_{2} X\right)=\beta_2 \operatorname{Var}(X)$, therefore
$$
b_{2}=\frac{\operatorname{Cov}(X, Y)}{\operatorname{Var}(X)}=\beta_{2}+\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)}
$$

If $u$ perfectly uncorrelated with $X$ as in G-M condition, the second term should be $0$. However that rarely happens, so the $b_2$ and $\beta_2$ will always have certain level of discrepancy. And note that we can't decompose $b_2$ in practice, because we don't know the true value of $\beta_2$.

And also note that there are two ways to improve the accuracy of $b_2$, either lower the correlation of $X$ and $u$ or increase the variance of $X$.

##  A Simulated Illustration

We want to perform a Monte Carlo simulation to show the sampling distribution of $b_1$ and $b_2$. First, we write a simple class for OLS Monte Carlo experiment.

We can set $\beta_1$, $\beta_2$, $N$ and $a_u$ for initialisation. The model is simply
$$
Y_i=\beta_1+\beta_2X_i +a_uu_i, \qquad i\in(1, n),\qquad u_i\sim N(0,1)
$$

```python
class OLS_Monte_Carlo:
    def __init__(self, beta1, beta2, N, u_scaler):
        '''Input beta1, beta2, sample size, scaler of disturbance'''
        self.beta1 = beta1
        self.beta2 = beta2
        self.u_scaler = u_scaler
        self.N = N
        self.X = self.N*np.random.rand(self.N) # generate N random X's in the range of (0, N)

    def ols(self):
        '''Estimate the data with OLS method, and return b1 and b2.'''
        self.u = self.u_scaler*np.random.randn(self.N)
        self.Y = self.beta1 + self.beta2*self.X + self.u
        cov_mat = np.cov(self.X, self.Y)
        self.b2 = cov_mat[0, 1]/cov_mat[0, 0]
        self.b1 = np.mean(self.Y) - self.b2*np.mean(self.X)
        return self.b2, self.b1
```

Instantiate the OLS Monte Carlo object with $\beta_1=2$, $\beta_2=3$, $n=10$ and $a_u=1$, then run $10000$ times of simulations, each time with a different set of $u$. All estimated $b_1$ and $b_2$ are collected in their arrays.

```python
ols_mt = OLS_Monte_Carlo(beta1=2, beta2=3, N=10, u_scaler=1)
b2_array, b1_array = [], []
for i in range(10000):
    b2, b1 = ols_mt.ols()
    b2_array.append(b2)
    b1_array.append(b1)
b2_mean = np.mean(b2)
b1_mean = np.mean(b1)
```

Plot the histogram and the mean of estimates. Not difficult to notice that the mean of the $b_1$ and $b_2$ are very close to 'true values'.

```python
fig, ax = plt.subplots(nrows = 2, ncols = 1,figsize=(9, 9))
ax[0].hist(b1_array, bins = 50)
ax[0].axvline(b1_mean, color = 'salmon', label = r'$\bar{b}_1$=%f'% b1_mean)
ax[0].legend()

ax[1].hist(b2_array, bins = 50)
ax[1].axvline(b2_mean, color = 'salmon', label = r'$\bar{b}_2$=%f'% b2_mean)
ax[1].legend()
plt.show()
```

Now we try again with a larger disturbance scaler $a_u = 2$ and keep rest parameters unchanged.

```python
ols_mt = OLS_Monte_Carlo(beta1=2, beta2=3, N=10, u_scaler=2)
b2_array, b1_array = [], []
for i in range(10000):
    b2, b1 = ols_mt.ols()
    b2_array.append(b2)
    b1_array.append(b1)
b2_mean = np.mean(b2)
b1_mean = np.mean(b1)

fig, ax = plt.subplots(nrows = 2, ncols = 1,figsize=(9, 9))
ax[0].hist(b1_array, bins = 50)
ax[0].axvline(b1_mean, color = 'salmon', label = r'$\bar{b}_1$=%f'% b1_mean)
ax[0].legend()

ax[1].hist(b2_array, bins = 50)
ax[1].axvline(b2_mean, color = 'salmon', label = r'$\bar{b}_2$=%f'% b2_mean)
ax[1].legend()
plt.show()
```

Though the histogram has been scaled as seemingly identical as above, but pay attention to the $x$-axis and vertical line. Apparently the variance of $u$ affects the accuracy of estimates, i.e. the variance of sampling distribution of $b_1$ and $b_2$.

It's straightforward to see why this happens from the formula
$$
b_{2}=\beta_{2}+\frac{\operatorname{Cov}(X, a_uu)}{\operatorname{Var}(X)}=\beta_{2}+a_u\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)}
$$

We know from statistics course the increase sample is always doing good for quality of estimates, thus we dial it up to $N=100$, rest are unchanged.

```python
ols_mt = OLS_Monte_Carlo(beta1=2, beta2=3, N=100, u_scaler=2)
b2_array, b1_array = [], []
for i in range(10000):
    b2, b1 = ols_mt.ols()
    b2_array.append(b2)
    b1_array.append(b1)
b2_mean = np.mean(b2)
b1_mean = np.mean(b1)

fig, ax = plt.subplots(nrows = 2, ncols = 1,figsize=(9, 9))
ax[0].hist(b1_array, bins = 50)
ax[0].axvline(b1_mean, color = 'salmon', label = r'$\bar{b}_1$=%f'% b1_mean)
ax[0].legend()

ax[1].hist(b2_array, bins = 50)
ax[1].axvline(b2_mean, color = 'salmon', label = r'$\bar{b}_2$=%f'% b2_mean)
ax[1].legend()
plt.show()
```
Notice how we have improved the accuracy of estimates, pay attention to $x$-axis that the sample distributions are tremendously concentrated than previous runs.
#  Statistical Features of Estimates

After a visial exam of sample distributions, now we can formally discuss the statistical features of estimator.

##  Unbiased of Estimator

If an estimator is biased, we rarely perform any estimation with it, unless you have no choices which is also rare. But how to prove unbiasness of an estimator?

The _rule of thumb_ is to take expectation on both sides of estimator. Here is the OLS example.
$$
E\left(b_{2}\right)=E\left[\beta_{2}+\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)}\right]=\beta_{2}+E\left[\frac{\operatorname{Cov}(X, u)}{\operatorname{Var}(X)}\right]=\beta_2+\frac{E[\operatorname{Cov}(X, u)]}{E[\operatorname{Var}(X)]}
$$
To show $E[\operatorname{Cov}(X, u)]=0$, rewrite covariance in explicit form
$$
E[\operatorname{Cov}(X, u)]=\frac{1}{n} \sum_{i=1}^{n}\left(X_{i}-\bar{X}\right) E\left(u_{i}-\bar{u}\right)=0
$$
Therefore
$$
E\left(b_{2}\right)=\beta_{2}
$$
Again take expectation on $b_1$, immediately we get
$$
E(b_1) = E(\bar{Y})-E(b_2)E(\bar{X})= E(\bar{Y})-\beta_2\bar{X}=\beta_1
$$
where we used $E(\bar{Y})=\beta_{1}+\beta_{2} \bar{X}$.

##  Precision of Estimator

If we know the variance of disturbance term, the population variance of $b_1$ and $b_2$ can be derived
$$
\sigma_{b_{1}}^{2}=\frac{\sigma_{u}^{2}}{n}\left[1+\frac{\bar{X}^{2}}{\operatorname{Var}(X)}\right]\\\sigma_{b_{2}}^{2}=\frac{\sigma_{u}^{2}}{n \operatorname{Var}(X)}
$$
Though we will never really know $\sigma_{u}$, the formulae provide the intuition how the variance of coefficients are determined.

In the visual example of last section, we have seen that the larger $\sigma_{u}$ causes larger $\sigma_{b_{1}}^{2}$ and $\sigma_{b_{2}}^{2}$, here the formulae also present the relation.

And there are two ways to contract the variance of $b_1$ and $b_2$, one is increasing sample size $n$, the other is increasing $\operatorname{Var}(X)$.

In practice, we substitute $\sigma_{u}^{2}$ by its unbiased estimator
$$
s_{u}^{2}=\frac{n}{n-2} \operatorname{Var}(e)
$$

where $\operatorname{Var}(e)$ is the sample variance of residuals. The term $\frac{n}{n-2}$ is for upscaling variance, because generally residuals are smaller than disturbance term, this is determined by its mathematical nature. You will get a very clear linear algebraic view in advanced econometric theory.

After plug-in we get the _standard error_ of $b_1$ and $b_2$
$$
\text { s.e. }\left(b_{1}\right)=\sqrt{\frac{s_{u}^{2}}{n}\left[1+\frac{\bar{X}^{2}}{\operatorname{Var}(X)}\right]}\\
\text { s.e. }\left(b_{2}\right)=\sqrt{\frac{s_{u}^{2}}{n \operatorname{Var}(X)}}
$$
The stardard error is used when we are referring to the _standard deviation of sampling distribution of estimates_, specifically in econometric term, regression coefficients.

#  Hypothesis Testing and $t$-Test

Essentially, hypothesis testing in econometrics is the same as statistics. We propose a theory and test against data collected by experiment or sampling.

That being said, in econometrics, we mainly investigate if the linear relation between independent and dependent variables is plausible, so we rarely test a specific null hypothesis such that $\beta_2 = 2$, but rather $\beta_2 =0$. Therefore
$$
H_0: \beta_1 = 0, \beta_2 =0\\
H_1: \beta_1 \neq 0, \beta_2 \neq 0
$$

Let's reproduce the house price example. We have the estimates, but we would like to investigate how reliable the results are. Reliability hinges on relativity, that means even if the absolute value of estimates are small, such as $.1$, as long as the standard error are smaller enough, such as $.01$, we can safely conclude a rejection of null hyphothesis without hesitation.

$t$-statisc of $b_1$ and $b_2$ are
$$
\frac{b_1-\beta_1^0}{s.e.(b_1)}\qquad\frac{b_2-\beta_2^0}{s.e.(b_2)}
$$
where $\beta^0$ is null hypothesis.

Generally, statistics are intuitive to interpret, it measures how many standard deviations (with known $\sigma$) or standard errors (with unknown $\sigma$) away from the null hypothesis. The further way, the stronger evidence supporting the rejection of null hypothesis.

```python
df = pd.read_excel('Basic_Econometrics_practice_data.xlsx', sheet_name = 'CN_Cities_house_price')
s_ols_house_income = S_OLS(df['salary'], df['house_price'])
Y_hat, b2, b1 = s_ols_house_income.ols()
s_ols_house_income.ols_plot('Disposable Income', 'House Price')
resid = df['house_price'] - Y_hat
```

Compute $t$-statistic as in formulae

```python
n = len(df['house_price'])
s_u_sqr = n/(n-2)*np.var(resid)

std_err_b1 = np.sqrt(s_u_sqr/n*(1+np.mean(df['salary'])**2/np.var(df['salary'])))
std_err_b2 = np.sqrt(s_u_sqr/(n*np.var(df['salary'])))

t_b1 = b1/std_err_b1
t_b2 = b2/std_err_b2
print('t_b1:{:.4f}'.format(t_b1))
print('t_b2:{:.4f}'.format(t_b2))
```

    t_b1:-1.7580
    t_b2:3.6349

Seems both $t$'s are far away from null hypothesis $\beta^0=0$, but how far is real far? Unless we have a quantitative criterion, the interpretations won't sound objective.

That's why we have **significant level** (denoted as $\alpha$) as the decisive criterion. The common levels is $5\%$ and $1\%$. If $t$ falls on the right of **critical value** of $t$ at $5\%$, we can conclude a rejection of null hypothesis.
$$
t > t_{.05}
$$
However in econometrics two-side test is more common, then rejection rules of $\alpha=.05$ are
$$
t<t_{-.025} \text{ and }t > t_{.025}
$$
Critical value of $t$-distribution is obtained by ```sp.stats.t.ppf```, and the shape of $t$-distribution approximate to normal distribution as degree of freedom raise.

The two-side test $t$-statistic is

```python
t_crit = sp.stats.t.ppf(.975, df = n-1); t_crit
```

    2.0638985616280205

Therefore we conclude a rejections of $\beta_2=0$ ($t_{b_2}>t_{crit}$) but fail to reject $\beta_1=0$ ($t_{b_1}<t_{crit}$).

The $b_1=1.1010$ is interpreted that every $1$ yuan increment of disposable income is accompanied by raising $1.1010$ yuan on house price.

In contrast, $b_0= -29181.16$ can't be interpreted that if every family has $0$ disposable income, the house price will be $-29181.16$, because it doesn't have any logical or economic sense. So the rule of thumb for interpretation of constant term: _it's unnecessary to interpret constant coefficient if the it has no logical sense_.

Side note of $t$-distribution

Here below is a figure for refreshing $t$-statistics, we set $\alpha=.05$. The first axes demonstrates how $t$-statistic changes as degree of freedom raises, the second shows that $t$-distribution approximates normal distribution indefinitely with rising degree of freedom.
<\div>

```python
degree_frd = np.arange(1, 50, 1)
t_array = sp.stats.t.ppf(.975, df=degree_frd)
fig, ax = plt.subplots(nrows=2, ncols=1,figsize=(14,12))
ax[0].plot(degree_frd, t_array)
ax[0].set_xlim([0, 20])
ax[0].set_xlabel('degree of freedom')
ax[0].set_ylabel('critical value of $t$-statistic')
ax[0].set_title('The Change of $t$-statistic As d.o.f. Increases')
x = np.linspace(-3, 3, len(degree_frd))
for i in range(len(degree_frd)):
    t_pdf = sp.stats.t.pdf(x, degree_frd[i], loc=0, scale=1) #pdf(x, df, loc=0, scale=1)
    if i % 8 == 0:
        ax[1].plot(x, t_pdf, label = 't-Distribution with d.o.f. = {}'.format(i))
    else:
        ax[1].plot(x, t_pdf)
ax[1].set_title('The Shape of $t$-Distribution Approximate Normal Distribution As d.o.f. Increases')
ax[1].legend()
plt.show()
```

#  $p$-Values

Personally, I myself prefer to $p$-values. It is more informative than $t$-statistic, $p$-value gives the probability of obtaining corresponding $t$-statistic if null hypothesis is true, which is exactly the probability of **type I error**.

With proper use of ```sp.stats.t.cdf```, we can access $p$-value conveniently.

```python
1 - sp.stats.t.cdf(t_b2, df=n) # because t_b2 is positive, so p-value should be deducted from 1, if negative then without
```

    0.0006284669089313688

$p$-value tells that if null hypothesis is true, the probability of obtaining $t_{b_2}=3.6349$ or even higher is merely $0.0006$. That means, very unlikely the null hypothesis is true, we can safely reject null hypothesis with a tiny probability of Type I error.

Medical researches conventionally uses $p$-value, econometrics tend to use estimates with standard error bracketed below. But they just different ways of expressing the same ideas, pick based on your preference unless you don't are writing an economic paper.

Side note of Type I and II Error

The blue shaded area are genuinely generated by null distribution, however they are too distant (i.e. $2\sigma$ away) from the mean ($0$ in this example), and they are mistakenly rejected, this is what we call Type I Error.

The orange shaded area are actually generated by alternative distribution, however they are in the adjacent area of mean of null hypothesis, so we failed to reject they, but wrongly. And this is called Type II Error.

As you can see from the chart, if null distribution and alternative are far away from each other, the probability of both type of errors diminish to trivial.

```python
from plot_material import type12_error
type12_error()
```

#  Confidence Interval

Why bother with confidence interval if we have hypothesis testing?

If you have a theory that house price has a certain linear relationship with disposable income, you test theory with model, this is called _hypothesis testing_. But what if you don't have a theory yet, and runs the regression, you are wondering how confident these estimates can represent the true parameters, the range that you feel confident is called **confidence interval**.

These two procedures complementing each other, that's why we see them often reported together.

Recall the rejection rules are
$$
\frac{b-\beta}{\text { s.e. }\left(b\right)}>t_{\text {crit }} \quad \text { or } \quad \frac{b-\beta}{\text { s.e. }\left(b\right)}<-t_{\text {crit }}
$$
If we slight rearrange and join them, we get the confidence interval
$$
b-\text { s.e. }\left(b\right) \times t_{\text {crit }} \leq \beta\leq b+\text { s.e. }\left(b\right) \times t_{\text {crit }}
$$
The higher significance level, the smaller $\alpha$ is, the larger confidence interval (because of larger $t_{crit}$). For example, if the significance level is $\alpha=.05$, then confidence level is $0.95$.

Here's the confidence interval ($95\%$) of our house price example.

```python
t_crit = sp.stats.t.ppf(.975, df = n-1)
print('C.I. of b1: [{}, {}]'.format(b1-t_crit*std_err_b1, b1+t_crit*std_err_b1))
print('C.I. of b2: [{}, {}]'.format(b2-t_crit*std_err_b2, b2+t_crit*std_err_b2))
```

    C.I. of b1: [-63439.54567466995, 5077.206045321876]
    C.I. of b2: [0.4758472405690619, 1.7261233292171854]

There are $95\%$ chances the true parameter $\beta$ will 'fall' in this confidence interval.

#  One-Tailed vs Two-Tailed Test

So far we have been discussing about two-tailed test, but there are scenarios that one-tailed test make more sense. In our house price example, some practitioners would prefer to test the theory or common sense: _disposable income would not have negative effects on house price_. The alternative would be that _disposable income would have either no effect or positive effects on house price_.

Thus the one-tailed test hypotheses are
$$
H_0: \beta_2<0\\
H_1: \beta_2\geq 0
$$
In one-tailed test, we don't split $\alpha$ anymore since there is only one side, that means the critical value will be smaller, easier to reject null hypothesis.

Here is $t_{crit}$ of $\alpha=5\%$. However, these are conventional rules, if you still prefer $2.5\%$ on one-side, feel free to do so. Especially you have a very significant $t$-statistic, such as $10$, one-tailed or two-tailed won't really matter.

```python
t_crit_oneside = sp.stats.t.ppf(.95, df = n-1); t_crit_oneside
```

    1.7108820799094275

So here the rule of thumb for one-tailed test.

Rules of Thumb for One-Tailed Test

1. If the theory or common sense supports one side test, e.g. household consumption increases as disposable incomes increase.

2. If two-tailed test failed to reject, but one-tailed reject, you can report one-tailed test results if the first rule satisfied too.

#  $F$-test

$F$-test is based on _Analysis of Variance_ (ANOVA), the goal is to test **multiple restrictions** on the regression model. In simple linear regression model, the **joint hypothesis** is usually
\begin{align}
H_0&: \beta_1 = 0,\qquad\beta_2=0\\
H_1&: \text{One or more restrictions does not hold}
\end{align}

once you have ANOVA done, $F$-statistic is an natural byproduct.
$$
F=\frac{E S S /(k-1)}{R S S /(n-k)}
$$
where $k$ is the number of number of parameters in the regression model, here in simple regression model $k=2$ and $n$ is the sample size.

You might have doubt now: why aren't we using same old $t$-tests such that
$$
H_0: b_1=0 \qquad H_0: b_2=0 \qquad \\
H_1: b_1\neq0 \qquad H_1: b_2\neq0\qquad\\
$$

Apparently, the number of $t$-tests will be as large as ${k \choose 2} $ where $k$ is the number of parameters. If there are $5$ parameters, then we have to test ${5 \choose 2}=10$ pairs. With $95\%$ confidence level, $10$ $t$-tests would cut back confidence level dramatically to $95\%^{10}=59.8\%$, which also means the probability of _type I_ error would be around $40\%$.

We have user-defined functions written in the OLS class, so $F$-statistic is

```python
f_stat = (ess/(2-1))/(rss/(len(df['salary'])-2))
print('F-statistic is {:.4f}.'.format(f_stat))
```

    F-statistic is 13.2126.

```python
p_value = 1 - sp.stats.f.cdf(f_stat, 1, len(df['salary'])-2) # sp.stats.f.cdf(df on nominator, df on denom)
print('p-value is {:.4f}.'.format(p_value))
```

    p-value is 0.0014.

To explore further, we can even prove that in simple linear regression $F$ are the just the square of $t$.

$F$-statistic and $t$-statistic

Here's the proof that $F$ and $t$ are connected
$$
F=\frac{R^{2}}{\left(1-R^{2}\right) /(n-2)}=\frac{\frac{\operatorname{Var}(\hat{Y})}{\operatorname{Var}(Y)}}{\left\{1-\frac{\operatorname{Var}(\hat{Y})}{\operatorname{Var}(Y)}\right\} /(n-2)}\\
=\frac{\frac{\operatorname{Var}(\hat{Y})}{\operatorname{Var}(Y)}}{\left\{\frac{\operatorname{Var}(Y)-\operatorname{Var}(\hat{Y})}{\operatorname{Var}(Y)}\right\} /(n-2)}=\frac{\operatorname{Var}(\hat{Y})}{\operatorname{Var}(e) /(n-2)}\\
=\frac{\operatorname{Var}\left(b_{1}+b_{2} X\right)}{\left\{\frac{1}{n} \sum_{i=1}^{n} e_{i}^{2}\right\} /(n-2)}=\frac{b_{2}^{2} \operatorname{Var}(X)}{\frac{1}{n} s_{u}^{2}}=\frac{b_{2}^{2}}{\frac{s_{u}^{2}}{n \operatorname{Var}(X)}}=\frac{b_{2}^{2}}{\left[\operatorname{s.e.} \left(b_{2}\right)\right]^{2}}=t^{2}
$$

$F$-statistic and $R^2$

$F$-statistic is a different angle of evaluating the goodness of fit, it can be shown that $F$ and $R^2$ are closely connected, divide both nominator and denominator by $TSS$:
$$
F=\frac{(E S S / T S S) /(k-1)}{(R S S / T S S) /(n-k)}=\frac{R^{2} /(k-1)}{\left(1-R^{2}\right) /(n-k)}
$$
We prefer to $F$ for hypothesis test, it's because critical value of $F$ is straightforward, and critical value of $R^2$ has to be calculated based on $F_{crit}$:
$$
R_{\mathrm{crit}}^{2}=\frac{(k-1) F_{\mathrm{crit}}}{(k-1) F_{\text {crit }}+(n-k)}
$$

# Regression vs Correlation

Here is the message for all beginner that misinterpret regression relationship as causation.

Does Regression Imply Causation

It's tempting to interpret regression result as causality, but it's not. Regression only implies a statistical relationship, the independent variables may or may not be the cause of dependent variables, sometimes we know thanks to theories, sometimes we don't.

For instance, researches found that parents with higher education tend to have healthier children, but this is hardly a causality. Perhaps higher education parents are in general wealthier, they can afford decent medical packages. Or they spend time with their kids on sports and dining. We can form some hypothesis, but not a definite causality based on one regression.

But regressions do resemble the correlation to some extent

Does Regression Imply Correlation

From formula of $b_2$
$$
b_2 =\frac{\text{Cov}(X, Y)}{\text{Var}(X)}
$$
We can see the regression indeed has a component of correlation (covariance in the formula), but it's normalised by variance (i.e. $\sigma_X\sigma_X$) rather than $\sigma_X\sigma_Y$. To compare with correlation coefficient of $X$ and $Y$
$$
\rho_{XY}=\frac{\text{Cov}(X, Y)}{\sigma_X\sigma_Y}
$$
We can see one important difference is that regression coefficient does not treat both variables symmetrically, but correlation coefficient does. Joining two formulae, we have a different view of the coefficient.

$$
b_2=\frac{\rho_{XY}\sigma_X\sigma_Y}{\text{Var}(X)}= \frac{\rho_{XY}\sigma_X\sigma_Y}{\sigma_X\sigma_X}=\rho_{XY}\frac{\sigma_Y}{\sigma_X}
$$

Besides that, the purpose of these two techniques are different, regression are mainly predicting dependent variables behaviours, but correlation are mainly summarising the direction and strength among two or more variables.

Maybe a chart can share insight of their relationship. All data are simulated by $u\sim N(0, 1)$. It's easy to notice the smaller correlation implies a smaller slope coefficient in terms of absolute value. And larger disturbance term also implies lower correlation coefficient.

```python
from plot_material import reg_corr_plot
reg_corr_plot()
```

#  Joint Confidence Region

**Joint confidence region** are the joint distribution of regression coefficients, it is theoretically an ellipse. It shows the distributed location of the coefficient pair.

Here is a Monte Carlo simulation, we set $\beta_1 = 3$, $\beta_2 = 4$ and $u\sim N(0, 10)$, run $1000$ times then plot the estimates.

```python
beta1, beta2 = 3, 4
beta1_array, beta2_array = [], []
for i in range(1000):
    u = 10*np.random.randn(30)
    X2 = np.linspace(10, 100, 30)
    Y = beta1 + beta2*X2 + u

    df = pd.DataFrame([Y, X2]).transpose()
    df.columns = ['Y', 'X2']

    X_inde = df['X2']
    Y = df['Y']

    X_inde = sm.add_constant(X_inde)

    model = sm.OLS(Y, X_inde).fit()
    beta1_array.append(model.params[0])
    beta2_array.append(model.params[1])

fig, ax = plt.subplots(figsize = (10, 10))
ax.grid()
for i in range(1000):
    ax.scatter(beta1_array[i], beta2_array[i]) # no need for a loop, i just want different colors
ax.set_xlabel(r'$\beta_1$')
ax.set_ylabel(r'$\beta_2$')
plt.show()
```

But why the joint distribution of coefficient has an elliptic shape? If you take a look at any linear regression plot, it wouldn't be difficult to notice that the high slope coefficient $\beta_2$ would cause low the intercept coefficient $\beta_1$, this is a geometric feature of linear regression model.

And from the plot, we can see the range of $\beta_1$ is much larger than $\beta_2$ and even include $0$, especial the data points are far away from $0$, $\beta_1$ can have erratic results, that's also the reason we don't expect to interpret $\beta_1$ most of time.

#  Stochastic Regressors

One of G-M condition is $\text{Cov}(X_i, u_i)=0$, assuming $X_i$ is non-stochastic. But we commonly encounters that $X_i$ is stochastic, for instance you sample $10$ family's annual income, you have no clue on how much they are earning eventually, therefore assuming they are stochastic would be more appropriate.

If $X_i$ is stochastic and distributed independently of $u_i$, it guarantees that $\text{Cov}(X_i, u_i)=0$, but not vice versa.

#  Maximum Likelihood Estimation

In more advanced level of econometric research, **maximum likelihood estimation**(MLE) is used more often than OLS. The reason is that MLE is more flexible on assumption of disturbance term, i.e. not assuming the normality of the disturbance term. But it requires you to have an assumption of certain distribution, it can be exponential or gamma distribution or whatever.

We will provide two examples to illustrate the philosophy of MLE, one is to estimate simple linear regression with MLE and the other is to estimate a mean of an exponential distribution.

##  MLE for Simple Linear Regression

Once you have a dataset, you should know the data are just observations of random variables. For instance, you are about to collect $100$ family's annual income data for year 2021, each family's income is essentially a random variable, hence it follows some distribution, it can be normal distribution or skewed gamma distribution.

Once the data is collected, the observances are done. Each data point, i.e. $Y_1, Y_2, ..., Y_n$ is just single realization of its distribution, the joint distribution of all random variables is
$$
f\left(Y_{1}, Y_{2}, \ldots, Y_{n}\right)
$$
We assume a simple linear regression $Y_{i}=\beta_{1}+\beta_{2} X_{i}+u_{i}$ model to explain $Y$, then the joint distribution is conditional
$$
f\left(Y_{1}, Y_{2}, \ldots, Y_{n} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right)
$$
We also assume each family's income is independent of rest, then joint distribution equals the product of all distributions.
$$
\begin{aligned}
&f\left(Y_{1}, Y_{2}, \ldots, Y_{n} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right)=f\left(Y_{1} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right) f\left(Y_{2} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right) \cdots f\left(Y_{n} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right) =\prod_{i=0}^nf\left(Y_{i} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right)
\end{aligned}
$$

Now we need to ask ourselves, what distribution does $Y$ follow? It's the same as asking what's the distribution of $u$? It's reasonable to assume $u$ follow normal distribution, so are $Y$'s.
$$
f(Y_i)= \frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{1}{2}\frac{[Y_i-E(Y_i)]^2}{\sigma^2}}=\frac{1}{\sigma \sqrt{2\pi}}e^{-\frac{1}{2}\frac{[Y_i-\beta_1-\beta_2X_i]^2}{\sigma^2}}
$$

Then the joint distribution is the product of this PDF function
$$
\prod_{i=0}^nf\left(Y_{i} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right)=\frac{1}{\sigma^{n}(\sqrt{2 \pi})^{n}} e^{-\frac{1}{2} \sum \frac{\left(Y_{i}-\beta_{1}-\beta_{2} X_{i}\right)^{2}}{\sigma^{2}}}
$$

Once we have joint distribution function, in a frequentist view, the observed data is generated by this distribution with certain parameters, in this case $\beta_1, \beta_2$ and $\sigma^2$. So the fundamental the questions are what those parameters are? How to find them?

We give a name to the join distribution function above - **likelihood function**, which means how likely a set of parameters can generate a set of data. We denote likelihood function as $LF(\beta_1, \beta_2, \sigma^2)$.

The MLE, as its name indicates, is to estimate the parameters that in such manner that the probability of generating $Y$'s is the highest.

To derive a analytical solution, usually we use log form likelihood function
$$
\ln{LF}=\ln{\prod_{i=0}^nf\left(Y_{i} \mid \beta_{1}+\beta_{2} X_{i}, \sigma^{2}\right)} =-\frac{n}{2} \ln \sigma^{2}-\frac{n}{2} \ln (2 \pi)-\frac{1}{2} \sum \frac{\left(Y_{i}-\beta_{1}-\beta_{2} X_{i}\right)^{2}}{\sigma^{2}}
$$

Take derivative with respect to $\beta_1$, $\beta_2$ and $\sigma^2$ and equal them to $0$ will yield the **maximum likelihood estimators** for simple linear regression with assumption of normally distributed disturbance term
\begin{align}
b_2 &= \frac{\sum_{i=1}^n (X_i - \bar{X})(Y_i-\bar{Y})}{\sum_{i=1}^n(X_i-\bar{X})^2}=\frac{\text{Cov}(X, Y)}{\text{Var}(X)}\\
b_1 &= \bar{Y}-\beta_2\bar{X}\\
s^2 &= \frac{1}{n} \sum_{i=1}^{n}\left(Y_{i}-\left(b_{0}+b_{1} X_{i}\right)\right)^{2} = \frac{1}{n}\sum_{i=1}^n e_i^2
\end{align}

MLE $b_1$ and $b_2$ are exactly the same as OLS estimators, only the $s^2$ differs from OLS estimator $s^2 = \frac{\sum e^2_i}{n-2}$. The MLE $s^2$ is biased in small sample, but _consistent_ as sample size increases.

We'll experiment with MLE with simulated data. Generate a data for OLS estimation and print the estimated coefficients.

```python
N = 100
X = np.linspace(0, 20, N)
u = np.random.normal(loc = 0, scale = 10, size = N)

beta1, beta2 = 2, 3
Y = beta1 + beta2*X + u
df = pd.DataFrame({'Y':Y, 'X':X})

X = df['X']
Y = df['Y']

X = sm.add_constant(X) # adding a constant

model = sm.OLS(Y, X).fit()

print_model = model.summary()
model.params
```

    const    1.616445
    X        3.077071
    dtype: float64

Estimated standard deviation of disturbance term.

```python
np.sqrt(np.sum(model.resid**2)/(N-2))
```

    10.680968046974376

```statsmodel``` library doesn't have a direct api for maximum likelihood estimation, but we can construct it with ```Scipy```'s ```minimize``` function, so we define a negative log likelihood. The reason of negative function is because ```Scipy``` has only ```minimize``` function.

```python
def neg_log_likelihood(params):
    intercept, beta, std_dev = params[0], params[1], params[2]
    Yhat = intercept + beta*X
    negLL = -np.sum(sp.stats.norm.logpdf(Y, loc=Yhat, scale=std_dev)) # negative log-likelihood
    return(negLL)
```

```python
init_value = np.array([1,1,1])
results = minimize(neg_log_likelihood, init_value, method = 'Nelder-Mead', options={'disp': True}) # options={'disp': True} is not a must
```

Print the relevant results.

```python
resultsdf = pd.DataFrame({'coef':results['x']})
resultsdf.index=['Intercept','Slope','Sigma']
resultsdf
```

As you can see the intercept and slope coefficients are the same as OLS, but $\hat{\sigma}$ is underestimated as the theory tells.

##  MLE for Exponential Distribution

MLE is more general than OLS, it can estimate any parameters as long as you know or assume the distribution, this example we will give an explicit function form for an exponential function.

Consider
$$
f(Y_i; \theta) = \theta e^{-\theta Y_i}
$$
The joint distribution (likelihood function) is
$$
\prod_{i=1}^n f(Y_i; \theta) = \theta^ n  e^{-\theta \sum Y_i}
$$
Log likelihood function is
$$
\ln \prod_{i=1}^n f(Y_i; \theta) = n\ln{\theta} - \theta \sum_{i=1}^n Y_i
$$

Take derivative with respect to $\theta$
$$
\frac{\partial LF}{\partial \theta} = \frac{n}{\theta}-\sum_{i=1}^n Y_i = 0
$$

The result is
$$
\hat{\theta} = \frac{n}{n \bar{Y}} = \frac{1}{\bar{Y}}
$$

The estimated parameter $\hat{\theta}$ is the reciprocal of mean of $Y_i$.

# Statsmodels Library

Fortunately, we don't need to program our own toolbox for estimations, ```statsmodels``` library will be our main tool from now on, all the codes are self-explanatory unless it requires further clarification.

The estimation results are reported in as in Stata or Eview, there is a test run below.

```python
X = df['salary']
Y = df['house_price']

X = sm.add_constant(X) # adding a constant

model = sm.OLS(Y, X).fit()

print_model = model.summary()
print(print_model)
```

However, you might see some people using $R$-style formula such as
```Y ~ X + Z```, which produces the same results.

```python
df = pd.read_excel('Basic_Econometrics_practice_data.xlsx', sheet_name = 'CN_Cities_house_price')
df.head()

model = smf.ols(formula='house_price ~ salary', data=df)
results = model.fit()
print(results.summary())
```

```python

```


# Gyro integrating model

This model is an evolution of the simple model, where we don't assume that we have access to a Digital Motion Processor (DMP) unit inside of the IMU. Instead, we will have to do the integral of the movement ourselves.

Here are our assumptions:

1. The robot moves in the 2D plane.
2. The gyroscope is in the yaw axis.
3. The gyroscope's output has a constant bias ($b_{\omega}$) and some white gaussian noise.
4. The robot starts its integration with a heading of $\theta = 0$.

The state of the robot is the following:

\begin{equation}
\mathbf{x} = \begin{pmatrix}
    x\\
    y\\
    \dot{x}\\
    \dot{y}\\
    \theta\\
    b_{\omega}
    \end{pmatrix}
\end{equation}

We include the bias in the state even if we cannot measure it.
This allows the model to take into account the variance of the bias in the prediction step.

## Future work
* How can we measure "real" yaw from the UWB output?
* Can we use the magnetometer output as a "real" yaw measurement?

```python
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
sp.init_printing()

# defines useful variables
x, y, xdot, ydot, theta, b_omega = sp.symbols('x y v_x v_y theta b_{\omega}')
state = sp.Matrix([x, y, xdot, ydot, theta, b_omega])
dt = sp.symbols('Delta_t')
```

## Prediction step

The prediction step uses outputs of the accelerometer (in body frame) and the angular rate as control inputs:

\begin{equation}
\mathbf{u} = \begin{pmatrix}
    a_x\\
    a_y\\
    \omega
\end{pmatrix}
\end{equation}

The differential equation governing the state evolution is therefore:

\begin{equation}
\mathbf{\dot{x}} = \begin{pmatrix}
\dot{x}\\
\dot{y}\\
\cos(\theta) a_x - \sin(\theta) a_y \\
\sin(\theta) a_x + \cos(\theta) a_y \\
\omega - b_{\omega}\\
0
\end{pmatrix}
\end{equation}

Using forward Euler integration we get:

\begin{equation}
\mathbf{x}_{k+1} = \mathbf{x}_{k} + \Delta_t \mathbf{\dot{x}} = \mathbf{g}\left(\mathbf{x}_{k}, \mathbf{u}_k\right)
\end{equation}

```python
a_x, a_y, omega = sp.symbols('a_x a_y omega') # in body frame
u = [a_x, a_y, omega]
dx = sp.Matrix([xdot,
                ydot,
                sp.cos(theta) * a_x - sp.sin(theta) * a_y,
                sp.sin(theta) * a_x + sp.cos(theta) * a_y,
                omega - b_omega,
                0])
g = state + dt * dx
```

## Measurement step

For the measurement, the UWB system gives us the distance $d$ to a beacon.
The beacon's position $\mathbf{b}$ is known and assumed to be fixed.
Therefore the measurement model becomes:

\begin{equation}
h(\mathbf{x}, \mathbf{b}) = \sqrt{(x - b_x)^2 + (y - b_y)^2}
\end{equation}

```python
b_x, b_y = sp.symbols('b_x b_y')
h = sp.Matrix([sp.sqrt((x - b_x)**2 + (y - b_y)**2)])
```

We also define a function that measures the heading directly (using some type of compass maybe) in order to see how it impacts the performances of the system:

\begin{equation}
h_{\theta}(\mathbf{x}) = \theta
\end{equation}

```python
h_theta = sp.Matrix([theta])
```

## Extended Kalman Filter

For this model we will use an Extended Kalman Filter to fuse the sensor's data.
The Extended Kalman filter requires us to compute the Jacobian of the update and measurement functions:

```python
g.jacobian(state)
```

```python
h.jacobian(state)
```

```python
h_theta.jacobian(state)
```

## Simulation

We can now implement the EKF using the framework we developped in `ekf.py`.
The functions we generated as Sympy expression will need to be slightly wrapped to fit the expected format.

```python
f = 200 # Hz, sample rate of the filter
g_num = sp.lambdify([state, u], g.replace(dt, 1/f), 'numpy')
G_num = sp.lambdify([state, u], g.jacobian(state).replace(dt, 1/f), 'numpy')
h_num = sp.lambdify([[b_x, b_y], state], h, 'numpy')
H_num = sp.lambdify([[b_x, b_y], state], h.jacobian(state), 'numpy')
h_theta_num = sp.lambdify([state], h_theta, 'numpy')
H_theta_num = sp.lambdify([state], h_theta.jacobian(state), 'numpy')
```

```python
# Quick check that the forward equation looks correct
mu = g_num([0,0,1,2,0,0.3], [1,0,1])
assert mu[0] == 0.005
assert mu[2] == 1.005
assert mu[4] == 0.0035
assert mu[5] == 0.3
```

```python
# Defines the kalman filter
import functools
import ekf

# Create a dummy variance model
R = np.diag([0.1, # pos
             0.1,
             1e-4, # speed
             1e-4,
             1e-4, # angle
             1e-4  # bias
            ])
Q = np.diag([10]) # distance measurement

predictor = ekf.Predictor(g_num, G_num, R)

BEACON_POS = [
    (-1.5, 0),
    (1.5, 1),
    (1.5, -1),
]

correctors = [
    ekf.Corrector(functools.partial(h_num, pos),
                  functools.partial(H_num, pos),
                  Q
                 )
for pos in BEACON_POS]

Q_theta = np.diag([10])
angle_corrector = ekf.Corrector(h_theta_num, H_theta_num, Q_theta)
```

```python
# Runs the kalman filter on a circular trajectory
import trajectories

f_uwb = 50 # Update frequency for the UWB beacons

bias = np.deg2rad(5) # rad / s
bias_var = np.deg2rad(3.3)

mu = np.array([0, 0, 0, 0.1, 0, bias]) # initial state
sigma = np.diag([1e-1, 1e-1, # pos
                 1e-3, 1e-3, # speed
                 1e-1, # angle
                 bias_var,
                ])

angle, angle_estimated = [], []
x, xhat = [], []
y, yhat = [], []
acc = []
ts = []

for i, p in zip(range(10000), trajectories.generate_circular_traj(1, np.deg2rad(10), 1/f)):
    # add noise & bias to the measurements
    acc = [p.acc[0], p.acc[1]] + np.random.normal(0, 0.2, 2)
    omega = p.omega + np.random.normal(bias, np.sqrt(bias_var))

    # feeds the input into Kalman
    filter_input = np.array([acc[0], acc[1], omega])
    mu, sigma = predictor(mu, sigma, filter_input)

    # Correct for angle
    #z = np.array([p.theta]) + np.random.normal(0, np.sqrt(Q_theta[0]))
    #mu, sigma = angle_corrector(mu, sigma, z)

    # If we received a radio update, feed it into the EKF
    if i % (f // f_uwb) == 0:
        for corrector, (bx, by) in zip(correctors, BEACON_POS):
            z = corrector.h(np.array([p.pos[0], p.pos[1], 0, 0, 0, 0]))
            z += np.random.normal(0, 0.03)
            mu, sigma = corrector(mu, sigma, z)

    # Saves the data
    ts.append(p.timestamp)
    angle.append(p.theta)
    angle_estimated.append(mu[4])
    x.append(p.pos[0])
    xhat.append(mu[0])
    y.append(p.pos[1])
    yhat.append(mu[1])
```

```python
# Plot various results
plt.plot(ts, angle)
plt.plot(ts, angle_estimated)
plt.xlabel('time')
plt.ylabel('rad')
plt.legend(['truth', 'estimated'])
plt.title('Theta')
plt.show()

plt.plot(ts, x)
plt.plot(ts, xhat)
plt.plot(ts, y)
plt.plot(ts, yhat)
plt.xlabel('time')
plt.ylabel('meters')
plt.title('position')
plt.legend(['x', 'x (estimated)', 'y', 'y (estimated)'])
plt.show()

plt.plot(x, y)
plt.plot(xhat, yhat)
plt.plot([x for x, y in BEACON_POS],[y for x, y in BEACON_POS], 'x')
plt.legend(('reality', 'estimation', 'beacons'))
plt.title('trajectory')
plt.xlabel('x')
plt.ylabel('y')
plt.show()

plt.plot(ts, [np.sqrt((x-xh)**2+(y-yh)**2) for x,xh,y,yh in zip(x, xhat,y,yhat)])
plt.xlabel('timestamp')
plt.ylabel('error [m]')
plt.ylim(0, 0.3)
plt.title('Position error')
plt.show()
```

## Conclusion

It appears this model could very well work. When enabling the angle correction, it manages to track the angular position pretty well. I also found that this model is very sensitive to noise and biases on the distance measurement: a standard deviation of 3 cm on the distance measurement yields reasonable performance, while anything higher results in large noise spikes. Bias, on the other hand, can prevent the model from converging to the correct state. This implies that the implementation of the ranging protocol, and especially the calibration of the antenna delay will have to be done carefully.


```python
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
import numpy as np
import sympy as sp
import scipy.linalg as sl
from sympy.interactive import printing
printing.init_printing(use_latex=True)
```

```python
#Defining the symbols.
S,H,P,A,T,R= sp.symbols('S,H,P,A,T,R')
N,beta_1,beta_2,d,mu_1,p,k_1,k_2,mu_2,mu_3,rho=sp.symbols('N,beta_1,beta_2,d,mu_1,p,k_1,k_2,mu_2,mu_3,rho')
```

```python
#Equations representing our system
dot_S=N-d*beta_1*S*H/N-d*beta_2*p*S/N-d*S-mu_1*S
display(dot_S)
dot_H=d*beta_1*S*H/N+d*beta_2*p*S/N-k_1*H-d*H
display(dot_H)
dot_P=k_1*H-k_2*p-mu_2*p-d*p
display(dot_P)
dot_A=k_2*p-mu_3*A-d*A-rho*A
display(dot_A)
dot_T=mu_2*p+mu_3*A-d*T
display(dot_T)
dot_R=mu_1*S-d*R
display(dot_R)
```

```python
#Creating equations from the expressions above.
SEq = sp.Eq(dot_S,0)
display(SEq)
HEq = sp.Eq(dot_H,0)
display(HEq)
PEq = sp.Eq(dot_P,0)
display(PEq)
AEq = sp.Eq(dot_A,0)
display(AEq)
TEq = sp.Eq(dot_T,0)
display(TEq)
#REq = sp.Eq(dot_R,0)
#display(REq)
```

```python
#Determining the equilibrium points.
EqPnts = sp.solve( (SEq,HEq,PEq,AEq,TEq), S,H,p,A,T )
display(EqPnts)
```

```python
#Linearization of the system.
F_i=sp.Matrix([d*beta_1*S*H/N+d*beta_2*p*S/N,0,0,0])
F_i
```

```python
V_i=sp.Matrix([k_1*H-d*H,-dot_P,-dot_A,-dot_T])
V_i
```

```python
Mat=sp.Matrix([H,p,A,T])
Mat
```

```python
F=F_i.jacobian(Mat)
F
```

```python
V=V_i.jacobian(Mat)
V
```

```python
VInv=V.inv()
VInv
```

```python
G=F*VInv
G
```

```python
p=EqPnts[1][0]
q=EqPnts[1][1]
r=EqPnts[1][2]
s=EqPnts[1][3]
t=EqPnts[1][4]
G1=G.subs([(S,p),(H,q),(p,r),(A,s),(T,t)])
G1
```

```python
EV=G1.eigenvals()
EV
```

```python

```

```python

```


# Introduction

This example gives a simple demostration of chaotic behavior in a simple two body system. The system is made up of a slender rod that is connected to the ceiling at one end with a revolute joint that rotates about the $\hat{\mathbf{n}}_y$ unit vector. At the other end of the rod a flat plate is attached via a second revolute joint allowing the plate to rotate about the rod's axis with aligns with the $\hat{\mathbf{a}_z}$ unit vector.

# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sm
import sympy.physics.mechanics as me
from pydy.system import System
from pydy.viz import Cylinder, Plane, VisualizationFrame, Scene
```

```python
%matplotlib nbagg
```

```python
me.init_vprinting(use_latex='mathjax')
```

# Define Variables

First define the system constants:

- $m_A$: Mass of the slender rod.
- $m_B$: Mass of the plate.
- $l_B$: Distance from $N_o$ to $B_o$ along the slender rod's axis.
- $w$: The width of the plate.
- $h$: The height of the plate.
- $g$: The acceleratoin due to gravity.

```python
mA, mB, lB, w, h, g = sm.symbols('m_A, m_B, L_B, w, h, g')
```

There are two time varying generalized coordinates:

- $\theta(t)$: The angle of the slender rod with respect to the ceiling.
- $\phi(t)$: The angle of the plate with respect to the slender rod.

The two generalized speeds will then be defined as:

- $\omega(t)=\dot{\theta}$: The angular rate of the slender rod with respect to the ceiling.
- $\alpha(t)=\dot{\phi}$: The angluer rate of the plate with respect to the slender rod.

```python
theta, phi = me.dynamicsymbols('theta, phi')
omega, alpha = me.dynamicsymbols('omega, alpha')
```

The kinematical differential equations are defined in this fashion for the `KanesMethod` class:

$$0 = \omega - \dot{\theta}\\
0 = \alpha - \dot{\phi}$$

```python
kin_diff = (omega - theta.diff(), alpha - phi.diff())
kin_diff
```

$$\left ( \omega - \dot{\theta}, \quad \alpha - \dot{\phi}\right )$$

# Define Orientations

There are three reference frames. These are defined as such:

```python
N = me.ReferenceFrame('N')
A = me.ReferenceFrame('A')
B = me.ReferenceFrame('B')
```

The frames are oriented with respect to each other by simple revolute rotations. The following lines set the orientations:

```python
A.orient(N, 'Axis', (theta, N.y))
B.orient(A, 'Axis', (phi, A.z))
```

# Define Positions

Three points are necessary to define the problem:

- $N_o$: The fixed point which the slender rod rotates about.
- $A_o$: The center of mass of the slender rod.
- $B_o$: The center of mass of the plate.

```python
No = me.Point('No')
Ao = me.Point('Ao')
Bo = me.Point('Bo')
```

The two centers of mass positions can be set relative to the fixed point, $N_o$.

```python
lA = (lB - h / 2) / 2
Ao.set_pos(No, lA * A.z)
Bo.set_pos(No, lB * A.z)
```

# Specify the Velocities

The generalized speeds should be used in the definition of the linear and angular velocities when using Kane's method. For simple rotations and the defined kinematical differential equations the angular rates are:

```python
A.set_ang_vel(N, omega * N.y)
B.set_ang_vel(A, alpha * A.z)
```

Once the angular velocities are specified the linear velocities can be computed using the two point velocity thereom, starting with the origin point having a velocity of zero.

```python
No.set_vel(N, 0)
```

```python
Ao.v2pt_theory(No, N, A)
```

$$\left(\frac{L_{B}}{2} - \frac{h}{4}\right) \omega\mathbf{\hat{a}_x}$$

```python
Bo.v2pt_theory(No, N, A)
```

$$L_{B} \omega\mathbf{\hat{a}_x}$$

# Inertia

The central inertia of the symmetric slender rod with respect to its reference frame is a function of its length and its mass.

```python
IAxx = sm.S(1) / 12 * mA * (2 * lA)**2
IAyy = IAxx
IAzz = 0

IA = (me.inertia(A, IAxx, IAyy, IAzz), Ao)
```

This gives the inertia tensor:

```python
IA[0].to_matrix(A)
```

$$\left[\begin{matrix}\frac{m_{A}}{12} \left(L_{B} - \frac{h}{2}\right)^{2} & 0 & 0\\0 & \frac{m_{A}}{12} \left(L_{B} - \frac{h}{2}\right)^{2} & 0\\0 & 0 & 0\end{matrix}\right]$$

The central inerita of the symmetric plate with respect to its reference frame is a function of its width and height.

```python
IBxx = sm.S(1)/12 * mB * h**2
IByy = sm.S(1)/12 * mB * (w**2 + h**2)
IBzz = sm.S(1)/12 * mB * w**2

IB = (me.inertia(B, IBxx, IByy, IBzz), Bo)
```

```python
IB[0].to_matrix(B)
```

$$\left[\begin{matrix}\frac{h^{2} m_{B}}{12} & 0 & 0\\0 & \frac{m_{B}}{12} \left(h^{2} + w^{2}\right) & 0\\0 & 0 & \frac{m_{B} w^{2}}{12}\end{matrix}\right]$$

All of the information to define the two rigid bodies are now available. This information is used to create an object for the rod and the plate.

```python
rod = me.RigidBody('rod', Ao, A, mA, IA)
```

```python
plate = me.RigidBody('plate', Bo, B, mB, IB)
```

# Loads

The only loads in this problem is the force due to gravity that acts on the center of mass of each body. These forces are specified with a tuple containing the point of application and the force vector.

```python
rod_gravity = (Ao, mA * g * N.z)
plate_gravity = (Bo, mB * g * N.z)
```

# Equations of motion

Now that the kinematics, kinetics, and inertia have all been defined the `KanesMethod` class can be used to generate the equations of motion of the system. In this case the independent generalized speeds, independent generalized speeds, the kinematical differential equations, and the inertial reference frame are used to initialize the class.

```python
kane = me.KanesMethod(N, q_ind=(theta, phi), u_ind=(omega, alpha), kd_eqs=kin_diff)
```

The equations of motion are then generated by passing in all of the loads and bodies to the `kanes_equations` method. This produces $f_r$ and $f_r^*$.

```python
bodies = (rod, plate)
loads = (rod_gravity, plate_gravity)

fr, frstar = kane.kanes_equations(loads, bodies)
```

```python
sm.trigsimp(fr)
```

$$\left[\begin{matrix}g \left(- \frac{L_{B} m_{A}}{2} - L_{B} m_{B} + \frac{h m_{A}}{4}\right) \operatorname{sin}\left(\theta\right)\\0\end{matrix}\right]$$

```python
sm.trigsimp(frstar)
```

$$\left[\begin{matrix}\frac{m_{B} w^{2}}{12} \alpha \omega \operatorname{sin}\left(2 \phi\right) - \left(\frac{L_{B}^{2} m_{A}}{3} + L_{B}^{2} m_{B} - \frac{L_{B} h}{3} m_{A} + \frac{h^{2} m_{A}}{12} + \frac{h^{2} m_{B}}{12} + \frac{m_{B} w^{2}}{12} \operatorname{cos}^{2}\left(\phi\right)\right) \dot{\omega}\\- \frac{m_{B} w^{2}}{24} \left(\omega^{2} \operatorname{sin}\left(2 \phi\right) + 2 \dot{\alpha}\right)\end{matrix}\right]$$

# Simulation

The equations of motion can now be simulated numerically. Values for the constants, initial conditions, and time are provided to the `System` class along with the symbolic `KanesMethod` object.

```python
sys = System(kane)
```

```python
sys.constants = {lB: 0.2, # meters
                 h: 0.1, # meters
                 w: 0.2, # meters
                 mA: 0.01, # kilograms
                 mB: 0.1, # kilograms
                 g: 9.81} # meters per second squared
```

```python
sys.initial_conditions = {theta: np.deg2rad(45),
                          phi: np.deg2rad(0.5),
                          omega: 0,
                          alpha: 0}
```

```python
sys.times = np.linspace(0, 10, 500)
```

The trajectories of the states are found with the `integrate` method.

```python
x = sys.integrate()
```

The angles can be plotted to see how they change with respect to time given the initial conditions.

```python
def plot():
    plt.figure()
    plt.plot(sys.times, np.rad2deg(x[:, :2]))
    plt.legend([sm.latex(s, mode='inline') for s in sys.coordinates])
plot()
```

    <IPython.core.display.Javascript object>

# Chaotic Behavior

Now change the intial condition of the plat angle just slighty to see if the behvior of the system is similar.

```python
sys.initial_conditions[phi] = np.deg2rad(1.0)
x = sys.integrate()
plot()
```

    <IPython.core.display.Javascript object>

Seems all good, very similar behavior. But now set the rod angle to $90^\circ$ and try the same slight change in plate angle.

```python
sys.initial_conditions[theta] = np.deg2rad(90)
sys.initial_conditions[phi] = np.deg2rad(0.5)
x = sys.integrate()
plot()
```

    <IPython.core.display.Javascript object>

First note that the plate behaves wildly. What happens when the initial plate angle is altered slightly.

```python
sys.initial_conditions[phi] = np.deg2rad(1.0)
x = sys.integrate()
plot()
```

    <IPython.core.display.Javascript object>

The behavior does not look similar to the previous simulation. This is an example of chaotic behavior. The plate angle can not be reliably predicted because slight changes in the initial conditions cause the behavior of the system to vary widely.

# Visualization

Finally, the system can be animated by attached a cylinder and a plane shape to the rigid bodies. To properly align the coordinate axes of the shapes with the bodies, simple rotations are used.

```python
rod_shape = Cylinder(2 * lA, 0.005, color='red')
plate_shape = Plane(h, w, color='blue')

v1 = VisualizationFrame('rod',
                        A.orientnew('rod', 'Axis', (sm.pi / 2, A.x)),
                        Ao,
                        rod_shape)

v2 = VisualizationFrame('plate',
                        B.orientnew('plate', 'Body', (sm.pi / 2, sm.pi / 2, 0), 'XZX'),
                        Bo,
                        plate_shape)

scene = Scene(N, No, v1, v2, system=sys)
```

The following method opens up a simple gui that shows a 3D animatoin of the system.

```python
scene.display_ipython()
```

```python
%load_ext version_information
%version_information numpy, sympy, scipy, matplotlib, pydy
```

Software	Version

Python	3.5.3 64bit [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

IPython	4.2.0

OS	Linux 4.10.0 21 generic x86_64 with debian stretch sid

numpy	1.12.1

sympy	1.0

scipy	0.19.0

matplotlib	2.0.2

pydy	0.4.0dev

Mon May 29 13:08:51 2017 PDT