# A Biological Example

Suppose we are observing exponential growth $\dot{y} = \theta y$ but we don't know $\theta$ and wish to estimate it. We could assume $\theta \sim {\cal{N}}(\mu, \sigma^2)$ and use least squares or better something like Markov Chain Monte Carlo or Hamiltonian Monte Carlo and any observations to infer $\mu$ and $\sigma$. However, we might want to model that the further we go into the future, the less we know about $\theta$. We can write our system as as

$$
\begin{aligned}
\mathrm{d}y & = \theta y\mathrm{d}t \\
\mathrm{d}\theta & = \sigma\mathrm{d}W_t
\end{aligned}
$$

where $W_t$ is Brownian Motion.

# Fokker-Planck

We can use the [Fokker-Planck equation](https://en.wikipedia.org/wiki/Fokker%E2%80%93Planck_equation) to convert a stochastic differential equation into a partial differential equation.

$$
d \mathbf{X}_{t}=\boldsymbol{\mu}\left(\mathbf{X}_{t}, t\right) d t+\boldsymbol{\sigma}\left(\mathbf{X}_{t}, t\right) d \mathbf{W}_{t}
$$

$$
\frac{\partial}{\partial t} p(t, \mathbf{x})+\sum_{k=1}^{n} \frac{\partial}{\partial x_{k}}\left({\mu}_{k}(t, \mathbf{x}) p(t, \mathbf{x})\right)=\frac{1}{2} \sum_{j=1, k=1}^{n} \frac{\partial^{2}}{\partial x_{j} \partial x_{k}}\left[\left(\sigma(t, \mathbf{x}) \sigma^{T}(t, \mathbf{x})\right)_{j k} p(t, \mathbf{x})\right]
$$

For our particular system we have

$$
\frac{\partial}{\partial t} p(t, y, \theta)+\frac{\partial}{\partial y}\left({\mu}_{1}(t, y, \theta) p(t, y, \theta)\right)+\frac{\partial}{\partial \theta}\left({\mu}_{2}(t, y, \theta) p(t, y, \theta)\right)=\frac{1}{2}\left[\sigma_{y}^{2} \frac{\partial^{2}}{\partial y^{2}} p(t, y, \theta)+\sigma_{\theta}^{2} \frac{\partial^{2}}{\partial \theta^{2}} p(t, y, \theta)\right]
$$

And since $\mu_1 = \theta y$, $\mu_2 = 0$ and $\sigma_y = 0$ this further simplifies to

$$
\frac{\partial}{\partial t} p(t, y, \theta)+\frac{\partial}{\partial y}(\theta y p(t, y, \theta))=\sigma_{\theta}^{2} \frac{\partial^{2}}{\partial \theta^{2}} p(t, y, \theta)
$$

We can note two things:

* This is an advection / diffusion equation with two spatial variables ($y$ and $\theta$).
* If $\sigma_\theta = 0$ then this is a transport (advection?) equation.

$$
\frac{\partial}{\partial t} p(t, y, \theta)+\frac{\partial}{\partial y}(\theta y p(t, y, \theta))=0
$$

Notice that there is nothing stochastic about the biology but we
express our uncertainty about the parameter by making it a
time-varying stochastic variable which says the further we go into the
future the less certain we are about it.

We are going to turn this into a Fokker-Planck equation which we can
then solve using e.g. the method of lines. But before turning to
Fokker-Planck, let's show that we can indeed solve a diffusion
equation using the method of lines.

# Warming Up

Let us solve the heat equation

$$
\frac{\partial u}{\partial t}=k_{x} \frac{\partial^{2} u}{\partial x^{2}}+k_{y} \frac{\partial^{2} u}{\partial y^{2}}+h
$$

with initial condition $u(0, x, y) = 0$ and stationary boundary conditions

$$
\frac{\partial u}{\partial t}(t, 0, y)=\frac{\partial u}{\partial t}(t, 1, y)=\frac{\partial u}{\partial t}(t, x, 0)=\frac{\partial u}{\partial t}(t, x, 1)=0
$$

and a periodic heat source

$$
h(x, y)=\sin (\pi x) \sin (2 \pi y)
$$

This has analytic solution

$$
u(t, x, y)=\frac{1-e^{-\left(k_{x}+4 k_{y}\right) \pi^{2} t}}{\left(k_{x}+4 k_{y}\right) \pi^{2}} \sin (\pi x) \sin (2 \pi y)
$$

The spatial derivatives are computed using second-order centered differences, with the data distributed over $n_x \times n_y$ points on a uniform spatial grid.

$$
u_{i\,j}(t) \triangleq u\left(t, x_{i}, y_{j}\right), \quad x_{i} \triangleq i \Delta x, \quad 0 \leq i \leq n_x-1, \quad  y_{j} \triangleq j \Delta y, \quad 0 \leq j \leq n_y-1
$$

$$
\begin{align}
u_{x x} &= \frac{u_{i+1\,j}-2 u_{i\,j}+u_{i-1\,j}}{\Delta x^{2}} \\
u_{y y} &= \frac{u_{i\,j+1}-2 u_{i\,j}+u_{i\,j-1}}{\Delta y^{2}}
\end{align}
$$

$$
\dot{u}_{i\, j} = \frac{k_x}{(\Delta x)^2}({u_{i+1\,j}-2 u_{i\,j}+u_{i-1\,j}})
                + \frac{k_y}{(\Delta y)^2}({u_{i\,j+1}-2 u_{i\,j}+u_{i\,j-1}})
                + h_{i\, j}
$$

We could try using [Naperian functors and APL-like programming in Haskell](https://www.cs.ox.ac.uk/people/jeremy.gibbons/publications/aplicative.pdf) via this [library](http://hackage.haskell.org/package/Naperian). But the performance is terrible (or it could be that the author's implementation was terrible). Moreover, applied mathematicans tend to think of everything as matrices and vectors. But flattening the above tensor operation into a matrix operation is not entirely trivial. Although the Haskell Ecosystem's support for symbolic mathematics is very rudimentary, we can use what there is to convince ourselves that we haven't made too many errors in the transcription.

```haskell
{-# LANGUAGE DataKinds           #-}
{-# LANGUAGE OverloadedLists     #-}
{-# LANGUAGE ScopedTypeVariables #-}
{-# LANGUAGE FlexibleContexts    #-}
{-# LANGUAGE FlexibleInstances   #-}
{-# LANGUAGE MultiParamTypeClasses #-}
{-# LANGUAGE GADTs               #-}
{-# LANGUAGE TypeApplications    #-}
{-# LANGUAGE TypeOperators       #-}

import Data.Number.Symbolic
import qualified Data.Number.Symbolic as Sym
import Data.Proxy

import qualified Naperian as N
import qualified GHC.TypeLits as M
import           Data.Functor

import           Numeric.Sundials.ARKode.ODE
import           Numeric.LinearAlgebra

import           System.IO
```

We can re-write the semi-discretized equations in a tensor form from which we can derive an implementation.

$$
\dot{u}_{i\, j} = \frac{k_x}{(\Delta x)^2}\sum_{k=0}^{n_x-1}\sum_{l=0}^{n_y-1}A_{i\,j\,k\,l} u_{k\,l}
                + \frac{k_y}{(\Delta y)^2}\sum_{k=0}^{n_x-1}\sum_{l=0}^{n_y-1}B_{i\,j\,k\,l} u_{k\,l}
                + h_{i\, j}
$$

$$
A_{i\, j\, l\, k} =
\begin{cases}
0,& \text{if } i = 0 \\
0,& \text{if } j = 0 \\
1,& \text{if } k = i-1 \text{ and } l = j \\
-2,& \text{if } k = i \text{ and } l = j \\
1,& \text{if } k = i+1 \text{ and } l = j \\
0,& \text{if } i = n_x - 1 \\
0,& \text{if } j = n_y - 1 \\
0,& \text{otherwise}
\end{cases}
$$

$$
B_{i\, j\, l\, k} =
\begin{cases}
0,& \text{if } i = 0 \\
0,& \text{if } j = 0 \\
1,& \text{if } k = i \text{ and } l = j - 1 \\
-2,& \text{if } k = i \text{ and } l = j \\
1,& \text{if } k = i \text{ and } l = j + 1 \\
0,& \text{if } i = n_x - 1 \\
0,& \text{if } i = n_y - 1 \\
0,& \text{otherwise}
\end{cases}
$$

Let's write down the tensor $A$ in Haskell

```haskell
preA :: forall b m n . (M.KnownNat m, M.KnownNat n, Num b) =>
        N.Hyper '[N.Vector n, N.Vector m, N.Vector n, N.Vector m] b
preA = N.Prism $ N.Prism $ N.Prism $ N.Prism $ N.Scalar $
      N.viota @m <&> (\(N.Fin x) ->
      N.viota @n <&> (\(N.Fin w) ->
      N.viota @m <&> (\(N.Fin v) ->
      N.viota @n <&> (\(N.Fin u) ->
      (f m n x w v u)))))
        where
          m = fromIntegral $ M.natVal (undefined :: Proxy m)
          n = fromIntegral $ M.natVal (undefined :: Proxy n)
          f p q i j k l | i == 0               = 0
                        | j == 0               = 0
                        | i == p - 1           = 0
                        | j == q - 1           = 0
                        | k == i - 1 && l == j = 1
                        | k == i     && l == j = -2
                        | k == i + 1 && l == j = 1
                        | otherwise            = 0
```

We can concretize this to symbolic numbers

```haskell
a :: forall a m n . (M.KnownNat m, M.KnownNat n, Floating a, Eq a) =>
      N.Hyper '[N.Vector n, N.Vector m, N.Vector n, N.Vector m] (Sym a)
a = N.binary (*) (N.Scalar $ var "a") preA
```

And do the same for the tensor $B$

```haskell
preB :: forall b m n . (M.KnownNat m, M.KnownNat n, Num b) =>
        N.Hyper '[N.Vector n, N.Vector m, N.Vector n, N.Vector m] b
preB = N.Prism $ N.Prism $ N.Prism $ N.Prism $ N.Scalar $
      N.viota @m <&> (\(N.Fin x) ->
      N.viota @n <&> (\(N.Fin w) ->
      N.viota @m <&> (\(N.Fin v) ->
      N.viota @n <&> (\(N.Fin u) ->
      (f m n x w v u)))))
        where
          m = fromIntegral $ M.natVal (undefined :: Proxy m)
          n = fromIntegral $ M.natVal (undefined :: Proxy n)
          f :: Int -> Int -> Int -> Int -> Int -> Int -> b
          f p q i j k l | i == 0                   = 0
                        | j == 0                   = 0
                        | i == p - 1               = 0
                        | j == q - 1               = 0
                        | k == i     && l == j - 1 = 1
                        | k == i     && l == j     = -2
                        | k == i     && l == j + 1 = 1
                        | otherwise                = 0
```

```haskell
b :: forall a m n . (M.KnownNat m, M.KnownNat n, Floating a, Eq a) =>
           N.Hyper '[N.Vector n, N.Vector m, N.Vector n, N.Vector m] (Sym a)
b = N.binary (*) (N.Scalar $ var "b") preB
```

We can check that our implementation matches the mathematical formula
by rendering it as a $\LaTeX$.

```haskell
ps :: forall m n . (M.KnownNat m, M.KnownNat n) =>
       [N.Vector n (N.Vector m ((Int, Int), Sym Double))]
ps = N.elements $ N.crystal $ N.crystal $ N.hzipWith (,) ss rhs
  where
    h = N.Prism $ N.Prism $ N.Scalar $
      N.viota @n <&> (\(N.Fin x) ->
      N.viota @m <&> (\(N.Fin w) ->
      var ("u_{" ++ show x ++ "," ++ show w ++ "}")))
    rhs = N.foldrH (+) 0 $ N.foldrH (+) 0 $ N.binary (*) preFoo h
    preFoo = N.binary (+) (a @Double @n @m) (b @Double @n @m)
    ss = N.Prism $ N.Prism $ N.Scalar $
         N.viota @n <&> (\(N.Fin x) ->
         N.viota @m <&> (\(N.Fin w) -> (x,w)))
```

```haskell
eqns = mapM_ putStrLn $ zipWith (++) aaa bbb
  where
    aaa = concatMap (N.elements . N.Prism . N.Prism . N.Scalar) $
          fmap (fmap (fmap ((\(i, j)-> "u_{" ++ show i ++ show j ++ "} &= ") . fst))) x
    bbb = concatMap (N.elements . N.Prism . N.Prism . N.Scalar) $
          fmap (fmap (fmap ((++ " \\\\") . show . snd))) x
    x = ps @4 @3
```

```haskell
eqns
```

    u_{00} &= 0.0 \\
    u_{01} &= 0.0 \\
    u_{02} &= 0.0 \\
    u_{03} &= 0.0 \\
    u_{10} &= 0.0 \\
    u_{11} &= a*u_{0,1}+b*u_{1,0}+((-2.0)*a+(-2.0)*b)*u_{1,1}+b*u_{1,2}+a*u_{2,1} \\
    u_{12} &= a*u_{0,2}+b*u_{1,1}+((-2.0)*a+(-2.0)*b)*u_{1,2}+b*u_{1,3}+a*u_{2,2} \\
    u_{13} &= 0.0 \\
    u_{20} &= 0.0 \\
    u_{21} &= 0.0 \\
    u_{22} &= 0.0 \\
    u_{23} &= 0.0 \\

And then getting our notebook to render the $\LaTeX$.

$$
\begin{aligned}
u_{00} &= 0.0 \\
u_{01} &= 0.0 \\
u_{02} &= 0.0 \\
u_{03} &= 0.0 \\
u_{10} &= 0.0 \\
u_{11} &= a*u_{0,1}+b*u_{1,0}+((-2.0)*a+(-2.0)*b)*u_{1,1}+b*u_{1,2}+a*u_{2,1} \\
u_{12} &= a*u_{0,2}+b*u_{1,1}+((-2.0)*a+(-2.0)*b)*u_{1,2}+b*u_{1,3}+a*u_{2,2} \\
u_{13} &= 0.0 \\
u_{20} &= 0.0 \\
u_{21} &= 0.0 \\
u_{22} &= 0.0 \\
u_{23} &= 0.0 \\
\end{aligned}
$$

Now we have checked that our tensors look correct (at least for a particular and small tensor) we can try solving the system numerically

Spatial mesh size:

```haskell
nx, ny :: Int
nx = 3
ny = 4
```

Heat conductivity coefficients:

```haskell
kx, ky :: Floating a => a
kx = 0.5
ky = 0.75
```

x and y mesh spacing:

```haskell
dx :: Floating a => a
dx = 1 / (fromIntegral nx - 1)
dy :: Floating a => a
dy = 1 / (fromIntegral ny - 1)
```

```haskell
c1, c2 :: Floating a => a
c1 = kx/dx/dx
c2 = ky/dy/dy
```

Now we make the tensors more concrete by ensuring their elements come from `Floating`

```haskell
bNum :: forall a m n . (M.KnownNat m, M.KnownNat n, Floating a) =>
        N.Hyper '[N.Vector n, N.Vector m, N.Vector n, N.Vector m] a
bNum = N.binary (*) (N.Scalar c1) preB
```

```haskell
aNum :: forall a m n . (M.KnownNat m, M.KnownNat n, Floating a) =>
        N.Hyper '[N.Vector n, N.Vector m, N.Vector n, N.Vector m] a
aNum = N.binary (*) (N.Scalar c2) preA
```

Again we flatten the system into a matrix form so we can check everything looks as it should

```haskell
bigA :: Matrix Double
bigA = fromLists $
       fmap (N.elements . N.Prism . N.Prism . N.Scalar) $
       N.elements $ N.crystal $ N.crystal $ N.binary (+)
       (aNum @Double @4 @3) (bNum @Double @4 @3)
```

```haskell
bigA
```

    (12><12)
     [ 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0, 6.75, 0.0, 2.0, -17.5, 2.0, 0.0,  6.75, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,  6.75, 0.0, 2.0, -17.5, 2.0, 0.0, 6.75, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0
     , 0.0,  0.0, 0.0, 0.0,   0.0, 0.0, 0.0,   0.0, 0.0, 0.0,  0.0, 0.0 ]

```haskell
h :: forall m n a . (M.KnownNat m, M.KnownNat n, Floating a) =>
                     N.Hyper '[N.Vector m, N.Vector n] a
h = N.Prism $ N.Prism $ N.Scalar $
     N.viota @n <&> (\(N.Fin x) ->
     N.viota @m <&> (\(N.Fin w) ->
     sin (pi * (fromIntegral w) * dx)
             * sin (2 * pi * (fromIntegral x) * dy)))
```

```haskell
c :: Vector Double
c = fromList $ N.elements (h @3 @4 @Double)
```

```haskell
t0, tf :: Double
t0 = 0.0
tf = 0.3

bigNt :: Int
bigNt = 20

dTout :: Double
dTout = (tf - t0) / (fromIntegral bigNt)

ts :: [Double]
ts = map (dTout *) $ map fromIntegral [1..bigNt]

sol :: Matrix Double
sol = odeSolveV SDIRK_5_3_4' Nothing 1.0e-5 1.0e-10 (const bigU') (assoc (nx * ny) 0.0 [] :: Vector Double) (fromList $ ts)
  where
    bigU' bigU = bigA #> bigU + c
```

```haskell
main :: IO ()
main = do
  h1 <- openFile "Haskell.txt" WriteMode
  mapM_ (hPutStrLn h1) $ map (concatMap (' ':)) $ map (map show) $ toLists sol
  hClose h1
  mapM_ (\i -> putStrLn $ show $ sqrt $ (sol!i) <.> (sol!i) / (fromIntegral nx) / (fromIntegral ny)) ([0 .. length ts - 1] :: [Int])
```

```haskell
main
```

    0.0
    4.445792552162428e-3
    7.536496589199581e-3
    9.685474590386838e-3
    1.1179180479250954e-2
    1.2215783380962438e-2
    1.2936469361667507e-2
    1.3437435527874617e-2
    1.3785614451622913e-2
    1.4027670958010267e-2
    1.4196099215739022e-2
    1.4312878038957881e-2
    1.4394259941155774e-2
    1.4450706139605892e-2
    1.4490277045502337e-2
    1.451741051111616e-2
    1.4536171194645382e-2
    1.4549482462569737e-2
    1.45586417237852e-2
    1.4564936158380733e-2

The grid is unrealistically coarse. Let's check the Haskell implementation against a [reference C implmentation](https://github.com/LLNL/sundials/blob/master/examples/arkode/CXX_parallel/README)  with a better grid size.

```
nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/19.09.tar.gz
mpicxx ark_heat2D.cpp -lm -lsundials_arkode -lsundials_nvecparallel -o ark_heat2D
mpiexec -n 1 ./ark_heat2D
python plot_heat2D.py
```

We could run with more processors but it's easier to modify the python to work on the Haskell output if we don't.

```
nix-shell -I nixpkgs=https://github.com/NixOS/nixpkgs/archive/19.09.tar.gz
ghc -O2 -fforce-recomp Heat2D.hs -main-is Heat2D -o Heat2D
./Heat2D
python plot_heat2D.py
```


```python
from sympy.vector import curl, divergence, gradient, CoordSys3D, Del
```

```python
transformation = 'spherical'
var_names = ['r', r'\theta', r'\phi']
vec_names = ['e_' + v for v in var_names]
R = CoordSys3D('', transformation=transformation, variable_names=var_names, vector_names=vec_names)
```

```python
%run -i define_symbols_beta.py
```

    1.1.2.dev

# Coriolis term for horizontal velocity ($u_r \to 0 $)

Vertical component of Earth's rotation vector

```python
# f = 2 * omega1 * sin(R.y)  # When $\theta$ represents the latitude (geographics convention)
f = 2 * omega1 * cos(R.y)  # When $\theta$ represents the co-latitude (physics convention)
f
```

```python
beta = gradient(f).dot(R.j)
beta
```

```python
vort = curl(u)
vort_r = vort.dot(R.i)
div = divergence(u_h)
```

```python
corio = (-f * R.i).cross(u)
```

```python
corio
```

## Coriolis term in vorticity equation

$\nabla \times (-f \hat{e}_r \times \bf{u})$ is given by

```python
curl(corio).dot(R.i).expand()
```

Divergence in spherical coordinates is given by:

```python
div.expand()
```

$u_1 = 0$ in the above equation because it is the 2D divergence --- which can be separated from the expression for $\nabla \times (-f \hat{e}_r \times \bf{u})$

```python
(- beta * u2 - f * div).expand()
```

Therefore linearized vorticity equation can be written as:
$$\partial_t \zeta = -\beta u_\theta - f\nabla . \mathbf{u}$$

## Coriolis term in divergence equation

$\nabla . (-f \hat{e}_r \times \bf{u})$ is given by

```python
divergence(corio).expand()
```

Vertical component of vorticity (in the $\hat{e}_r$ direction), $\zeta$ in spherical coordinates is given by:

```python
vort_r.expand()
```

```python
(beta * u3 + f * vort_r).expand()
```

Therefore linearized divergence equation can be written as:
$$\partial_t \nabla.\mathbf{u} = \beta u_\phi + f\zeta - c^2 \nabla^2 \eta $$

## Linearized equations in vorticity-divergence formulation $(\zeta, \delta, \eta)$

$$
\begin{align}
\partial_t \zeta &= -\beta u_\theta - f\delta \\
\partial_t \delta &= \beta u_\phi + f\zeta - c^2 \nabla^2 \eta \\
\partial_t \eta &= -\delta
\end{align}
$$

Projecting the spherical velocities onto the beta plane, $u_\theta \to v$ (y component) and $u_\phi \to u$ (x component)

$$
\begin{align}
\partial_t \zeta &= -\beta v - f\delta \\
\partial_t \delta &= \beta u + f\zeta - c^2 \nabla^2 \eta \\
\partial_t \eta &= -\delta
\end{align}
$$

## Linearized equations in stream function-potential formulation $(\Psi, \Phi, \eta)$

Let stream function and velocity potential be defined as:

$$ \mathbf u = - \nabla \times (\Psi \hat{e}_z) + \nabla \Phi $$

which implies the velocities are given by:

$$u = -\Psi_y + \Phi_x$$
$$v = \Psi_x + \Phi_y$$

and also implies vorticity and divergence to be represented as:
$$\zeta = \nabla^2 \Psi$$
$$\delta = \nabla^2 \Phi$$

$$
\begin{align}
\partial_t \nabla^2 \Psi &= -\beta(\Psi_x + \Phi_y)  - f \nabla^2 \Phi \\
\partial_t \nabla^2 \Phi &= \beta(-\Psi_y + \Phi_x) + f\nabla^2 \Psi  - c^2 \nabla^2 \eta \\
\partial_t \eta &= -\nabla^2 \Phi
\end{align}
$$

**The eigenvalue problem and normal modes are solved in `normalmode_nb-beta.ipynb`**


```python
import numpy as np
from scipy import ndimage
from scipy import spatial
from scipy import io
from scipy import sparse
from scipy.sparse import csgraph
from scipy import linalg
from matplotlib import pyplot as plt
import seaborn as sns
from skimage import data
from skimage import color
from skimage import img_as_float

import graph3d

%matplotlib inline
```

# Load data

```python
image = img_as_float(data.camera()[::2, ::2])
```

```python
fig, ax = plt.subplots()
plt.imshow(image, cmap='gray')
plt.grid('off')
ax.xaxis.set_ticks([])
ax.yaxis.set_ticks([])
ax.set_title('Original image')
plt.savefig('../img/tikhonov_regularization_0.pdf', bbox_inches='tight')
```

# Crop and add noise

```python
image = image[40:80, 100:140]
noisy_image = image + 0.05*np.random.randn(*image.shape)
```

```python
fig, ax = plt.subplots(1, 2, figsize=(8, 4))

ax[0].imshow(image, cmap='gray')
ax[1].imshow(noisy_image, cmap='gray')
ax[0].grid('off')
ax[1].grid('off')
ax[0].xaxis.set_ticks([])
ax[0].yaxis.set_ticks([])
ax[1].xaxis.set_ticks([])
ax[1].yaxis.set_ticks([])
ax[0].set_title('Cropped image')
ax[1].set_title('Noisy image')
plt.savefig('../img/tikhonov_regularization_1.pdf', bbox_inches='tight')
```

# Perform graph filtering

#### Given a signal $f_0$ corrupted by Gaussian noise $\eta$

\begin{equation}
    \mathbf{y} = \mathbf{f_0} + \mathbf{\eta}
\end{equation}

#### Solve the regularization problem

\begin{equation}
    \underset{f}{\text{argmin}} \{ ||f - y||_2^2 + \gamma f^T L f\}
\end{equation}

#### Solution is given by

\begin{equation}
    f_{*}(i) = \sum_{l=0}^{N-1} \bigg[ \frac{1}{1 + \gamma \lambda_l} \bigg] \hat{y}
(\lambda_l) u_l(i)
\end{equation}

#### Or equivalently

\begin{equation}
    \mathbf{f} = \hat{h}(L) \mathbf{y}
\end{equation}

#### Where L is the laplacian of the adjacency matrix defined by:

\begin{equation}
    W_{i,j} =
    \begin{cases}
    \exp \bigg( - \frac{[dist(i, j)]^2}{2 \theta^2} \bigg) & \text{if $dist(i,j)$ < $\kappa$} \\
    0 & \text{otherwise}
    \end{cases}
\end{equation}

```python
# Parameters
kappa = np.sqrt(2)
theta = 20
gamma = 10
```

```python
# Query neighboring pixels for each pixel
yx = np.vstack(np.dstack(np.indices(noisy_image.shape)))
tree = spatial.cKDTree(yx)
q = tree.query_ball_point(yx, kappa)

# Get pixels I, and neighbors J
I = np.concatenate([np.repeat(k, len(q[k])) for k in range(len(q))])
J = np.concatenate(q)

# Distance metric is difference between neighboring pixels
dist_ij = np.sqrt(((noisy_image.flat[I] - noisy_image.flat[J])**2))

# Thresholded Gaussian kernel weighting function
W = np.exp(- ((dist_ij)**2 / 2*(theta**2)) )
```

```python
# Construct sparse adjacency matrix
A = sparse.lil_matrix((noisy_image.size, noisy_image.size))

for i, j, w in zip(I, J, W):
    A[i, j] = w
    A[j, i] = w

A = A.todense()
```

```python
# Compute Laplacian
L = csgraph.laplacian(A)

# Compute eigenvalues and eigenvectors of laplacian
l, u = linalg.eigh(L)

# Compute filtering kernel
h = u @ np.diag(1 / (1 + gamma*l)) @ u.T

# Filter the image using the kernel
graph_filtered_image = (h @ noisy_image.ravel()).reshape(noisy_image.shape)
```

```python
# Filter the image using traditional gaussian filtering
traditional_filtered_image = ndimage.gaussian_filter(noisy_image, 0.8)
```

```python
# Plot the result
fig, ax = plt.subplots(2, 2, figsize=(6, 6))

ax.flat[0].imshow(image, cmap='gray')
ax.flat[1].imshow(noisy_image, cmap='gray')
ax.flat[2].imshow(graph_filtered_image, cmap='gray')
ax.flat[3].imshow(traditional_filtered_image, cmap='gray')
ax.flat[0].grid('off')
ax.flat[1].grid('off')
ax.flat[2].grid('off')
ax.flat[3].grid('off')
ax.flat[0].xaxis.set_ticks([])
ax.flat[0].yaxis.set_ticks([])
ax.flat[1].xaxis.set_ticks([])
ax.flat[1].yaxis.set_ticks([])
ax.flat[2].xaxis.set_ticks([])
ax.flat[2].yaxis.set_ticks([])
ax.flat[3].xaxis.set_ticks([])
ax.flat[3].yaxis.set_ticks([])
ax.flat[0].set_title('Cropped Image')
ax.flat[1].set_title('Noisy Image')
ax.flat[2].set_title('Graph Filtered')
ax.flat[3].set_title('Gaussian Filtered')
plt.tight_layout()
plt.savefig('../img/tikhonov_regularization_2.pdf', bbox_inches='tight')
```

```python

```


# 3M1 - Search methods for univariate continuous functions

Luca Magri, lm547@cam.ac.uk

(With many thanks to Professor Gábor Csányi.)

Univariate function = function of one variable

## Lecture 2: List of contents
- General algorithm
- Convergence criteria
- Rate of convergence
- Line search
 - Interval reduction
 - Golden section method
- Fitting with quadratic functions
 - Fitting with a polynomial at three points
 - Fitting with a polynomial at one point: Newton's method and quasi-Newton method

## Search methods for multivariate functions

1. Start with an initial guess, $x_0$, for the minimum of $f(x)$
1. Propose a __search direction__, $d_k$
1. Propose a step size $\alpha_k$ along $d_k$, typically, by an inner __line search__ loop to find the lowest value of $f(x)$ along the direction $d_k$
1. Update the estimate of the minimum location, $x_{k+1} = x_k+\alpha_k d_k$
1. Back to 2 until convergence

- In 1D, problem of convergence is "easy", robustness is "hard"
 - Robustness is  the ability of the algorithm to operate at unexpected conditions
- In >1D, step 3 itself is a 1D optimisation problem
- In >1D, most of the effort is in the design of an algorithm for step 2

## Convergence criteria

- It can be assessed in different ways

\begin{align}
&\textrm{Norm of the residual}\;\;\;&\left\| f(x_{k+1}) - f(x_k)\right\| &\lt \varepsilon_f\\
\\
&\textrm{Norm of the error}\;\;\;&\left\| x_{k+1} - x_k\right\| &\lt \varepsilon_x\\
\\
&\textrm{Norm of the gradient}\;\;\;&\left\| \nabla f(x_k)\right\| &\lt \varepsilon_g\\
\end{align}

- $\epsilon_{\bullet}$ are user-defined tolerances
- The test on the norm of the gradient is often used
 - When the curvature is small, it should be combined with a test on the norm of the error
- Search methods find a _local_ minimum
 - Convex functions have only one minimum. Therefore, the local minimum is also a global minimum

## Rate of convergence

- The __rate of convergence__ of an algorithm quantifies how fast the approximate solution approaches the true minimum $x^*$, close to the minimum, iteration after iteration
$$
\lim_{k\rightarrow\infty} \frac{\left\| x_{k+1}-x^*\right\|}{\left\| x_k-x^* \right\|^p} = \beta
$$
The operation $\lim_{k\rightarrow\infty}$ means that the rate of convergence is an _asymptotic_ quantity.
  - For the keen reader, the rate of convergence is the speed at which a sequence converges to its limit, if it converges
- $\beta$ is the __convergence ratio__: the smaller the better

- $p$ is the __order of convergence__: the larger the better
 - $p=1$ $\rightarrow$ linear convergence
 - $p=2$ $\rightarrow$ quadratic convergence. The number of correct digits _roughly_ doubles at each  iteration

- Example
$$ x_k = 2 + 3^{-k}$$
$x_k\rightarrow 2$ for $k\rightarrow \infty$ . The sequence converges linearly to its limit $2$ because
\begin{align}
\lim_{k\rightarrow\infty} \frac{\left\| x_{k+1}-x^*\right\|}{\left\| x_k-x^* \right\|^p} = \frac{2 + 3^{-(k+1)}-2}{2 + 3^{-k}-2}=\frac{ 3^{-(k+1)}}{3^{-k}} = 3^{-1-k+k}=\frac{1}{3}
\end{align}
 Therefore, $\beta=1/3$ and $p=1$.

## Line search

- The search of the minimum for a univariate function is called _line search_
- This is the goal, for example, of the inner loop of multi-variable search
-  __Interval reduction__ is about the best you can do if the gradient is not available

- Example with the nonlinear function
\begin{align}f(x) = x^2 + \frac{345000}{\pi x}\;\;\;\;\;\;\;\;\;\;\;\; 22 < x < 48
\end{align}
The minimum is
$$ x^* = \left(\frac{345000}{2\pi}\right)^{\frac 1 3} \approx 38$$

- Now we seek the minimum by line search

```python
from pylab import *
import numpy as np
# function we will optimize
def R(x):
    return x**2+(345.0*1000.0)/(np.pi*x)
```

```python
figure(figsize=(12,8))
x = np.linspace(23,47,50)
plot(x, R(x))
ymin=4200
ymax=5400
axis((22, 48, ymin, ymax))
plot((25,25),(ymin,ymax), 'k:' ) ; text(25, 4220, "$x_1$", fontsize=24)
plot((45,45),(ymin,ymax), 'k:' ) ; text(45, 4220, "$x_2$", fontsize=24)
plot((40,40),(ymin,ymax), 'k:' ) ; text(40, 4220, "$x_3$", fontsize=24)
plot((33,33),(ymin,ymax), 'r:' ) ; text(33, 4220, "$x_4$", fontsize=24, color='r')
annotate(s='', xy=(25,5200), xytext=(45,5200), arrowprops=dict(arrowstyle='<->')); text(26,5220, "$I_1$", fontsize=24)
annotate(s='', xy=(33,5000), xytext=(45,5000), arrowprops=dict(arrowstyle='<->')); text(35,5020, "$I_2$", fontsize=24)
show()
```

1. Start with three points, $x_1,x_2,x_3$ such that
$$f(x_1) \gt f(x_3)\;\;\;\textrm{and}\;\;\; f(x_3) \lt f(x_2)$$
   $x_1$ and $x_2$ are the bounds, $x_3$ is the interior point
1. As a consequence of [Bolzano theorem](http://mathworld.wolfram.com/BolzanosTheorem.html), there exists a minimum in the interval $I_1 = [x_1,x_2]$.
   This is called __bracketing__
1. Compute $f(x_4)$ at a fourth point $x_4 \in I_1$
1. If $f(x_4) \gt f(x_3)$, the minimum must be in $I_2 = [x_4,x_2]$, otherwise the minimum is in $I'_2=[x_1,x_3]$
1. Continue bracketing until the interval is smaller than a tolerance

## Golden section search

Where to place the interior points?

- We impose a constant reduction factor $\beta$ for the interval length (convergence is linear, $p=1$)
$$\frac{I_2}{I_1} = \frac{\Delta x}{I_2} \text{ with }\Delta x = I_1 - I_2$$
set $\beta=I_2/I_1$, and solve quadratic:

$$\beta = \frac{\sqrt{5}-1}{2}\approx 0.618$$

- This is the inverse of the [Golden ratio](https://en.wikipedia.org/wiki/Golden_ratio), also known as golden section or sectio aurea, hence the name of this search method

```python
figure(figsize=(12,8))
x = np.linspace(23,47,50)
plot(x, R(x))
ymin=4200
ymax=5400
axis((22, 48, ymin, ymax))
plot((25,25),(ymin,ymax), 'k:' ) ; text(25, 4220, "$x_1$", fontsize=24)
plot((45,45),(ymin,ymax), 'k:' ) ; text(45, 4220, "$x_2$", fontsize=24)
plot((37,37),(ymin,ymax), 'k:' ) ; text(37, 4220, "$x_3$", fontsize=24)
plot((33,33),(ymin,ymax), 'r:' ) ; text(33, 4220, "$x_4$", fontsize=24, color='r')
annotate(s='', xy=(25,5200), xytext=(45,5200), arrowprops=dict(arrowstyle='<->')); text(26,5220, "$I_1$", fontsize=24)
annotate(s='', xy=(33,5000), xytext=(45,5000), arrowprops=dict(arrowstyle='<->')); text(35,5020, "$I_2$", fontsize=24)
annotate(s='', xy=(25,4800), xytext=(37,4800), arrowprops=dict(arrowstyle='<->')); text(35,4820, "$I'_2$", fontsize=24)
annotate(s='', xy=(37,4500), xytext=(45,4500), arrowprops=dict(arrowstyle='<->')); text(41,4520, "$\Delta x$", fontsize=24)
annotate(s='', xy=(25,4500), xytext=(33,4500), arrowprops=dict(arrowstyle='<->')); text(26,4520, "$\Delta x$", fontsize=24)
show()
```

```python
def golden_section(f, x1, x2, tol):
    # initial points
    f1 = f(x1) # x1 = left bound
    f2 = f(x2) # x2 = right bound
    # now set up golden section ratios
    r = (np.sqrt(5)-1)/2.0
    # third point
    x3 = x1*(1-r)+x2*r; f3=f(x3) #x3 = interior point

    # now loop until convergence
    traj = []
    it  = 0
    while abs(x1-x2) > tol:
        x4 = x1*r+x2*(1-r); f4=f(x4) # x4 = test point
        traj.append((x1, x2, x3, x4))
        if f4 < f3:
            x2=x3; f2=f3
            x3=x4; f3=f4
        else:
            x1=x2; f1=f2
            x2=x4; f2=f4
        print("x1", "x2",)
        print(round(x1,1), round(x2,1))
        print("\n")
    return x3,traj

x, t = golden_section(R, 25, 45, 0.5)
```

    ('x1', 'x2')
    (45.0, 32.6)

    ('x1', 'x2')
    (32.6, 40.3)

    ('x1', 'x2')
    (40.3, 35.6)

    ('x1', 'x2')
    (40.3, 37.4)

    ('x1', 'x2')
    (37.4, 39.2)

    ('x1', 'x2')
    (37.4, 38.5)

    ('x1', 'x2')
    (38.5, 37.8)

    ('x1', 'x2')
    (37.8, 38.2)

```python

fig = figure(figsize=(12,8))
x = np.linspace(23,47,50)
plot(x, R(x))
ymin=4200
ymax=5400
axis((22, 48, ymin, ymax))
text(25, 4220, "$x_1$", fontsize=24)
text(45, 4220, "$x_2$", fontsize=24)
text(38, 4220, "$x_3$", fontsize=24)
text(33, 4220, "$x_4$", fontsize=24)

def animate(i):
    ax=gca()
    for j in range(1,len(ax.lines)):
        ax.lines[j].set_color('k')
    plot((t[i][0],t[i][0]),(ymin,ymax), 'g' )
    plot((t[i][1],t[i][1]),(ymin,ymax), 'g' )
    plot((t[i][2],t[i][2]),(ymin,ymax), 'k' )
    plot((t[i][3],t[i][3]),(ymin,ymax), "r")

import matplotlib.animation
ani = matplotlib.animation.FuncAnimation(fig, animate, frames=len(t), interval=1000)

from IPython.display import HTML
HTML(ani.to_jshtml())

```

## Robustness

How could the previous program go wrong?
- The initial interval does not contain a minimum

Much of practical optimisation is actually about __robustness__, creating software that copes with unexpected situations

## Fitting with a quadratic polynomial at three points

We approximate the problem with a simpler one, which we can easily solve.

- We can _fit a quadratic function_
$$
q(x) = a_0 + a_1 x + a_2 x^2
$$
to match $f(x)$ at three points
- Find the minimum of the quadratic function
- Test this minimum

- To obtain the coefficients, we impose
$$
\begin{array}
\\
f(x_1) &= a_0 + a_1 x_1 + a_2 x_1^2\\
f(x_2) &= a_0 + a_1 x_2 + a_2 x_2^2\\
f(x_3) &= a_0 + a_1 x_3 + a_2 x_3^2\\
\end{array}
$$
which in matrix form is cast as
$$
\begin{bmatrix}
f(x_1)\\
f(x_2)\\
f(x_3)\\
\end{bmatrix}
=
\begin{bmatrix}
1&x_1&x_1^2\\
1&x_2&x_2^2\\
1&x_3&x_3^2\\
\end{bmatrix}
\begin{bmatrix}
a_0\\
a_1\\
a_2\\
\end{bmatrix}
$$

- The inversion of the matrix provides the solution

```python
figure(figsize=(12,8))
ymin=4200
ymax=5400
axis((22, 48, ymin, ymax))
x = np.linspace(23,47,50)
# function
plot(x, R(x)); text(24, 5200, "$f(x) = x^2 + 345000/(\pi x)}$", color='b', fontsize=24)

# true minimum
plot([38,38], [ymin,ymax], "b-"); text(31, 4250, "true minimum", color="b", fontsize=24)

# quadratic approx
x1=25; x2=37.5; x3=37
m = np.matrix([[1, x1, x1**2],[1, x2, x2**2], [1, x3, x3**2]])
a = np.dot(np.linalg.inv(m),np.matrix([[R(x1)],[R(x2)],[R(x3)]]))
plot(x, a[0,0]+a[1,0]*x+a[2,0]*x**2, 'r-')
plot([x1,x2,x3],[R(x1),R(x2),R(x3)], "ko", markersize=10)
text(27.5, 4800, "$q(x) = a_0 + a_1 x + a_2 x^2$", color='r', fontsize=24);

# minimum of quadratic approx
x4 = -a[1,0]/(2*a[2,0])
plot([x4, x4], [ymin,ymax], "r-") ; text(39,4250, "min at $x=-a_1/(2a_2)$", color="r", fontsize=24);

```

## Fitting with a quadratic polynomial at one point: Newton's method

- Instead of fitting the function values at three points, we can fit the function value, first and second derivatives evaluated at a single point $x_0$

$$
f(x) = f(x_0) + (x-x_0) f'(x_0) + \frac12 (x-x_0)^2 f''(x_0) + h.o.t.
$$

- We neglect the higher order terms ($h.o.t.$) and approximate $f(x)$ as
$$
q(x) = f(x_0) + (x-x_0) f'(x_0) + \frac12 (x-x_0)^2 f''(x_0)
$$

- To calculate the minimum of $q(x)$, we set its derivative to zero
\begin{align}
q'(x) & = f'(x_0) + (x-x_0) f''(x_0) \\
& = 0
\end{align}
- The solution is
$$
x = x_0 - \frac{f'(x_0)}{f''(x_0)}
$$

```python
figure(figsize=(12,8))
ymin=4200
ymax=5400
axis((22, 48, ymin, ymax))
x = np.linspace(23,47,50)
# function
plot(x, R(x)); text(24, 5200, "$f(x) = x^2 + 345000/(\pi x)$", color='b', fontsize=24)

# true minimum
plot([38,38], [ymin,R(38)], "b-"); text(39, 4250, "true minimum", color="b", fontsize=24)

x0=25
plot([x0],[R(x0)], "ko", markersize=10)
text(24,4950, "$x_0$", fontsize=24)

# quadratic approx
def Rp(x):
    return 2*x-(345*1000)/(np.pi*x**2)
def Rpp(x):
    return 2*(345*1000)/(np.pi*x**3)

def q(x):
    return R(x0)+(x-x0)*Rp(x0)+0.5*(x-x0)**2*Rpp(x0)
plot(x[:35], q(x[:35]), 'r-')
text(26.5, 4900, "$q(x) = f(x_0) + (x-x_0)f'(x_0) + 0.5(x-x_0)^2 f''(x_0)$", color='r', fontsize=24)
x1=x0-Rp(x0)/Rpp(x0)
plot([x1, x1], [ymin,q(x1)], "r");
```

### Remarks on Newton's method

The Newton's method
- might not be as good as the previous 3-point method
- is sensitive to the choice of the initial point
- can make the estimate worse
- has problems when $f''(x_0)$ is zero or negative

- Both polynomial fitting methods can be iterated to refine esimate
- Newton's method, when it works, has a quadratic ($p=2$) convergence
- Newton's method _generalises_ to multidimensional searches

## Newton's method can converge to the wrong extremum

- Example with the nonlinear function
$$ f(x) = -10000e^{-\frac{x^2}{350}}+\frac{345000}{\pi x}\;\;\;\;\;\;\;\;\;\;\;\; 9 < x < 40$$
We focus on the positive $x$-axis. The minimum is
$$ x^* \approx 15.7$$
The maximum is
$$ x^*_{max} \approx 30.9$$

<!--- - $f''(x)=0$ when $x=20.8$ --->

- Now we plot the approximation made by Newton's method

```python
figure(figsize=(12,8))
ymin=1900
ymax=3600
axis((9, 40, ymin, ymax))
x = np.linspace(10,40,50)
# function
def f(x):
    return -10000*exp(-x**2.0/350.0)+(345*1000)/(np.pi*x)
plot(x, f(x)); text(12, 3200, "$f(x) = -10000*\exp(-x^2/350) + 345000/(\pi x)$", color='b', fontsize=24)

# true minimum
m = 15.7
plot([m,m], [ymin,f(m)], "b-"); text(m+1, 1950, "true minimum", color="b", fontsize=24)

x0=14.#  # 12 14 17 19 21
plot([x0],[f(x0)], "ko", markersize=10)
text(25,2550, "$x_0$", fontsize=24)

# quadratic approx
def fp(x):
    return 10000*exp(-x**2.0/350.0)*2*x/350-(345*1000)/(np.pi*x**2)
def fpp(x):
    return -10000*exp(-x**2.0/350.0)*4*x**2/350**2 + 10000*exp(-x**2/350)*2/350 + 2*(345*1000)/(np.pi*x**3)

def qq(x):
    return f(x0)+(x-x0)*fp(x0)+0.5*(x-x0)**2*fpp(x0)
plot(x, qq(x), 'r-')
text(15, 3000, "$q(x) = f(x_0) + (x-x_0)f'(x_0) + 0.5(x-x_0)^2 f''(x_0)$", color='r', fontsize=24)
x1=x0-fp(x0)/fpp(x0)
plot([x1, x1], [ymin,qq(x1)], "r");
```

## Quasi-Newton methods

- In anticipation of multivariate functions, we stress that the computation of the second derivatives in the Hessian $H=\nabla^2 f(x)$ can be computationally prohibitive
- A large number of methods are designed to approximate the second derivative
- The simplest approximation is the __secant method__, which takes the current and previous points to estimate the second derivative at the current point

$$
f''(x_1) \approx \frac{f'(x_1)-f'(x_0)}{x_1-x_0}
$$

- The Taylor expansion around $x_1$ is approximated as
$$ s(x) = f(x_1) + (x-x_1)f'(x_1) + \frac{1}{2}(x-x_1)^2 \left(\frac{f'(x_1)-f'(x_0)}{x_1-x_0}\right)$$

- Setting $s'(x)=0$ yields the estimate for the minimum

$$
x = x_1 - f'(x_1) \frac{x_1-x_0}{f'(x_1)-f'(x_0)}
$$

- It can be shown that convergence is *superlinear*, $p = (1+\sqrt{5})/2 \approx 1.618$
- It has higher dimensional generalisations:
 - Family of Broyden's methods, including the most widely used BFGS method
 - The Barzilai-Borwein method (two-point steepest descent), which is often more robust
  - We will see the Barzilai-Borwein method in action in the next lecture

```python
figure(figsize=(12,8))
ymin=4200
ymax=5400
axis((22, 48, ymin, ymax))
x = np.linspace(23,47,50)
# function
plot(x, R(x)); text(24, 5200, "$f(x) = x^2 + 345000/(\pi x)$", color='b', fontsize=24)

# true minimum
plot([38,38], [ymin,R(38)], "b-"); text(39, 4250, "true minimum", color="b", fontsize=24)

x0=25.0
x1=27.0
plot([x0,x1],[R(x0), R(x1)], "ko", markersize=10)
text(24,4950, "$x_0$", fontsize=24)
text(26,4700, "$x_1$", fontsize=24)

def q(x):
    return R(x0)+(x-x0)*Rp(x0)+0.5*(x-x0)**2*Rpp(x0)
plot(x[:35], q(x[:35]), 'r-')
text(25, 5100, "$q(x) = f(x_0) + (x-x_0)f'(x_0) + 0.5(x-x_0)^2 f''(x_0)$ Newton", color='r', fontsize=18)
x1N=x0-Rp(x0)/Rpp(x0)
plot([x1N, x1N], [ymin,q(x1N)], "r");

def s(x):
    return R(x1)+(x-x1)*Rp(x1)+0.5*(x-x1)**2*(Rp(x1)-Rp(x0))/(x1-x0)
plot(x[:35], s(x[:35]), 'g-')
text(26, 5000, "$s(x) = f(x_1) + (x-x_1)f'(x_1) + 0.5(x-x_1)^2 (f'(x_1)-f'(x_0))/(x_1-x_0)$ Secant", color='g', fontsize=18)
x2=x1-Rp(x1) * (x1-x0)/(Rp(x1)-Rp(x0))
plot([x2, x2], [ymin,s(x2)], "g-");
```


# Part 1: Introduction to Python
## 1.1 [About Python](https://lectures.quantecon.org/py/about_py.html#id3)
### 1.1.1 Overview
### 1.1.2 What's Python?
### 1.1.3 Scientific Programming
- [Machine learing and data science](http://scikit-learn.org/stable/)
- [Astronomy(天文学)](http://www.astropy.org/)
- [Artificial intelligence](https://wiki.python.org/moin/PythonForArtificialIntelligence)
- [Chemistry](http://chemlab.github.io/chemlab/)
- [Computational biology](http://biopython.org/)
- [Meteorology(气象学)](https://pypi.python.org/pypi/metrology)

### 1.1.4 Learn More
#### Numerical Programming

```python
import numpy as np                      # Load the library

a = np.linspace(-np.pi,np.pi,100)        # Create even grid from -pi to pi
b = np.cos(a)                            # Apply cosine to each element of a
c = np.sin(a)                            # Apply sin to each element of a
```

```python
np.dot(b,c)       # matrix multiplication
```

    -1.8041124150158794e-16

```python
b @ c            # matrix multiplication
# element-wise product: np.multiply(), 或 *
```

    -1.8041124150158794e-16

#### SciPy
The SciPy library is built on top of NumPy and provides additional functionality For example, let’s calculate $\int^2_{-2}{\phi(z)dz}$ where $\phi$ is the standard normal density

```python
from scipy.stats import norm              # Load norm (normal distribution)
from scipy.integrate import quad          # Load quad (caculating intergration)
theta = norm()                            # Theta follows normal distribution
value, error = quad(theta.pdf, -2, 2)     # Integrate using Gaussian quadrature  .pdf 概率密度函数 .cdf 累计密度函数 -2,2 积分上下限
value                                     # 返回value的值
```

    0.9544997361036417

```python
# 使用cdf手动方法计算
import scipy.stats                       # Load library
value=1-2*scipy.stats.norm.cdf(-2,loc=0,scale=1)              #http://blog.csdn.net/claroja/article/details/72830515
value
```

    0.95449973610364158

SciPy includes many of the standard routines used in
- [linear algebra(线性代数)](https://docs.scipy.org/doc/scipy/reference/linalg.html)
- [integration(整合)](https://docs.scipy.org/doc/scipy/reference/integrate.html)
- [interpolation(插补)](https://docs.scipy.org/doc/scipy/reference/interpolate.html)
- [optimization](https://docs.scipy.org/doc/scipy/reference/optimize.html)
- [distributions and random number generation](https://docs.scipy.org/doc/scipy/reference/stats.html)
- [signal processing](https://docs.scipy.org/doc/scipy/reference/signal.html)
- [etc.](https://docs.scipy.org/doc/scipy/reference/index.html)

#### Graphics
[Library Matplotlib](https://matplotlib.org/gallery.html)
Other graphics libraries include
- [Plotly](https://plot.ly/python/)
- [Bokeh](http://bokeh.pydata.org/en/latest/)
- [VPython](http://www.vpython.org/)

#### Symbolic Algebra(符号代数)

```python
from sympy import Symbol
```

Symbolic Computing (符号计算)

```python
x, y = Symbol('x'), Symbol('y')    # Treat 'x' and 'y' as algebraic symbols（代数符号）
x + x + x + y
```

    3*x + y

Factor Analysis (因子分析)

```python
expression = (x + y)**2        # **  乘方
expression.expand()            # .expand 展开式
```

    x**2 + 2*x*y + y**2

Solving Polynomials

```python
from sympy import solve
solve(x**2 + x + 2)           # 解方程
```

    [-1/2 - sqrt(7)*I/2, -1/2 + sqrt(7)*I/2]

```python
solve(x**2 + 2*x + 1)
```

    [-1]

Calculating Limits, Derivatives and Integrals(极限，导数，积分)

```python
from sympy import limit, sin, diff
limit(1 / x, x, 0)                          #  求1/x的极限，其中x趋向于0
```

    oo

```python
limit(sin(x) / x, x, 0)
```

    1

```python
diff(sin(x), x)                         #  求sin（x）对x的导数
```

    cos(x)

#### Statistics
Python’s data manipulation and statistics libraries have improved rapidly over the last few years
#### Pandas
How to generate random number
如何生成随机数？
- numpy.random.uniform(low=0.0, high=1.0, size=None)  [0,1)均匀分布
- numpy.random.random(size=None)
  - numpy.random.random((2, 3))   元组形式指定
- numpy.random.normal(size,loc,scale): 给出均值为loc，标准差为scale的高斯随机数
- numpy.random.rand(d0, d1, ..., dn)   [0,1)    标准正态分布
  numpy.random.randn(d0, d1, ..., dn)  没有限制   标准正态分布
  - 当函数括号内没有参数时，则返回一个浮点数
  - 当函数括号内有一个参数时，则返回秩为1的数组，不能表示向量和矩阵
  - 当函数括号内有两个及以上参数时，则返回对应维度的数组，能表示向量或矩阵
  - numpy.random.standard_normal() 函数与np.random.randn()类似，但是np.random.standard_normal() 的输入参数为元组（tuple）
- numpy.random.randint(low, high=None, size=None, dtype=’l’) 整数型

```python
np.random.uniform()
```

    0.8474738087470629

```python
np.random.uniform(2,3,6)              # [2,3)中均匀分布中的6个随机数
```

    array([ 2.9860889 ,  2.19519615,  2.00777363,  2.23653008,  2.05505467,
            2.89117726])

```python
np.random.uniform(2,3,(2,2))
```

    array([[ 2.53819935,  2.85976506],
           [ 2.91869204,  2.64991819]])

```python
np.random.random()
```

    0.7304829610087114

```python
np.random.random(10)
```

    array([ 0.86273288,  0.5279635 ,  0.33135421,  0.72708269,  0.86928968,
            0.47071769,  0.05077265,  0.69756921,  0.16713672,  0.54353137])

```python
np.random.random((3,4))
```

    array([[ 0.96642415,  0.37624851,  0.32901655,  0.58515682],
           [ 0.62271202,  0.56993156,  0.47964188,  0.42931829],
           [ 0.60408005,  0.77272266,  0.00162158,  0.52869527]])

```python
np.random.normal(1,0.5,10)
```

    array([ 1.20349355,  1.17745893,  0.98227701,  1.58086968,  1.54173165,
            1.01738413,  0.91850826,  1.16709306,  1.03949584,  1.00972302])

```python
np.mean(np.random.normal(1,0.5,1000))
```

    0.97477448694503321

```python
np.random.normal(5,0.8,(2,4))
```

    array([[ 5.37148902,  5.52165235,  3.10762657,  3.21051536],
           [ 6.40331996,  5.35974761,  5.53317151,  4.81759955]])

```python
np.random.rand()
```

    0.6991420637529272

```python
np.random.rand(3)
```

    array([ 0.59372934,  0.2606707 ,  0.85677761])

```python
np.random.rand(2,2)
```

    array([[ 0.12969677,  0.26830264],
           [ 0.26575202,  0.15031921]])

```python
np.random.rand(3,2,2)           # 3组 2*2
```

    array([[[ 0.04667944,  0.56022617],
            [ 0.99076839,  0.2932098 ]],

           [[ 0.70883542,  0.72276065],
            [ 0.0241512 ,  0.66399433]],

           [[ 0.56175472,  0.93324534],
            [ 0.47822311,  0.29943678]]])

```python
np.random.rand(2,4,3,3)                    #  4 组 3*3 ， 这样的4组有2个
```

    array([[[[ 0.91655043,  0.12037943,  0.85462637],
             [ 0.31528861,  0.59644266,  0.24527812],
             [ 0.41429085,  0.80717022,  0.21825278]],

            [[ 0.89791812,  0.19810456,  0.75128835],
             [ 0.34516507,  0.8161068 ,  0.6074346 ],
             [ 0.19217673,  0.46841586,  0.22780444]],

            [[ 0.0995358 ,  0.38507414,  0.13525457],
             [ 0.71547676,  0.58165184,  0.54236188],
             [ 0.94241399,  0.42352624,  0.52700985]],

            [[ 0.55401586,  0.97291657,  0.26568722],
             [ 0.94813474,  0.69113245,  0.87098301],
             [ 0.58442572,  0.24803309,  0.18808223]]],

           [[[ 0.82405861,  0.06174618,  0.44058703],
             [ 0.04691451,  0.01917401,  0.97859124],
             [ 0.95750291,  0.38926466,  0.18569959]],

            [[ 0.19203849,  0.44958773,  0.61960534],
             [ 0.43102461,  0.79000366,  0.43515503],
             [ 0.16416595,  0.6032064 ,  0.1556198 ]],

            [[ 0.62336311,  0.26819523,  0.01729941],
             [ 0.01069535,  0.69215579,  0.03171449],
             [ 0.6912115 ,  0.17182265,  0.24050492]],

            [[ 0.59704035,  0.48744065,  0.74663878],
             [ 0.85071639,  0.89530915,  0.23641053],
             [ 0.3558962 ,  0.63559743,  0.79752732]]]])

```python
np.random.randn(2,2,3)
```

    array([[[ 0.41860832,  0.20000057,  1.61377788],
            [ 0.34181873, -1.41733621,  0.31367332]],

           [[-2.18577938,  0.47208238,  1.08411372],
            [ 0.56983538, -1.25560168, -1.44829982]]])

```python
np.random.standard_normal((2,2,3))
```

    array([[[ 0.33496069,  0.15313107,  0.68528172],
            [ 0.20353637, -0.06418014, -0.2242228 ]],

           [[ 0.01099221,  0.10426834, -0.51487265],
            [ 1.17121491, -0.55703709, -0.22933557]]])

```python
np.random.randint(5,7,(2,2,3))            # [5,7）  结构 （2,2,3）
```

    array([[[6, 6, 6],
            [5, 5, 5]],

           [[5, 6, 6],
            [6, 5, 6]]])

```python
import pandas as pd
np.random.seed(1234)                                                 #  在随机数中从1234这个位置开始抽取随机数

data = np.random.randn(5,2)                                          #  5x2 matrix of N(0, 1) random draws
dates = pd.date_range('01/03/2018', periods = 5)                     #  生成5期data，从2018/03/01开始

df = pd.DataFrame(data, columns = ('price', 'weight'), index = dates)  # 数据框格式，（数据，列名，编号为dates）
print(df)
```

                   price    weight
    2018-01-03  0.471435 -1.190976
    2018-01-04  1.432707 -0.312652
    2018-01-05 -0.720589  0.887163
    2018-01-06  0.859588 -0.636524
    2018-01-07  0.015696 -2.242685

```python
df.mean()                # 均值
```

    price     0.411768
    weight   -0.699135
    dtype: float64

#### Other Useful Statistics Libraries
- [statsmodels](http://www.statsmodels.org/stable/index.html)— various statistical routines
- [scikit-learn](http://scikit-learn.org/stable/) —(machine learning in Python (sponsored by Google, among others)
- [pyMC](http://pymc-devs.github.io/pymc/)—for Bayesian data analysis
- [pystan](https://pystan.readthedocs.io/en/latest/) Bayesian analysis based on [stan](http://mc-stan.org/)

#### Networks and Graphs
Library [NetworkX](http://networkx.github.io/)
A simple netword sample

```python
import networkx as nx                              #  network library
import matplotlib.pyplot as plt                    #  plot library
import numpy as np
np.random.seed(1234)                               # seef of random number

# Generate random graph
# function: numpy.random.uniform(low,high,size)
p = dict((i,(np.random.uniform(0,1),np.random.uniform(0,1))) for i in range(200))
# dict: dictionary
# genenrate {0:(0.5,0.8), 1:(0.6,0.7), 2:(0.2,0.9),.....,199:(0.3,0.8)}
G = nx.random_geometric_graph(200, 0.12, pos = p)
# position is stored as node attribute data for random_geometric_graph
# <networkx.classes.graph.Graph at 0xc03c588>
pos = nx.get_node_attributes(G, 'pos')
#  Get node attributes from graph
```

`nx.random_geometric_graph(n, radius, dim=2, pos=None)`
-   `n` : int;              Number of nodes
-   `radius`: float ;         Distance threshold value
-   `dim` : int, optional;      Dimension of graph
-   `pos` : dict, optional ;    A dictionary keyed by node with node positions as values.

` nx.get_node_attributes(G, name)`
-   `G` :            NetworkX Graph
-   `name` : string ;    Attribute name

```python
# find node nearest the center point (0.5, 0.5)
# 离中心越远，颜色变化的设定？
dists = [(x - 0.5)**2 + (y - 0.5)**2 for x,y in list(pos.values())]
# dists = [(x - 0.2)**2 + (y - 0.8)**2 for x,y in list(pos.values())]
ncenter = np.argmin(dists)               # argmin()  求出离中心最近的那个点，命名为C点

```

```python
# Plot graph, coloring by path length from central node
p = nx.single_source_shortest_path_length(G, ncenter)    # 计算C点离其他点的距离
plt.figure()                                             # 设置一副空图
nx.draw_networkx_edges(G, pos, alpha=0.5)                # 画出线条  alpha：边缘透明度
nx.draw_networkx_nodes(G,                                # 画出点
                      pos,                               # 点的属性
                      nodelist=list(p.keys()),           # Draw only specified nodes (default G.nodes())
                      node_size=120,alpha=0.5,
                      node_color=list(p.values()),
                      cmap=plt.cm.jet_r)                # Colormap for mapping intensities of nodes (default=None)
plt.show()
```

#### Cloud Computing
Library: [Wakar](https://www.wakari.io/)
See also
- [Amazon Elastic Compute Cloud](http://aws.amazon.com/ec2/)
- [The Google App Engine (Python, Java, PHP or Go)](https://cloud.google.com/appengine/)
- [Pythonanywhere](https://www.pythonanywhere.com/)
- [Sagemath Cloud](https://cloud.sagemath.com/)

#### Parallel Processing
- [Parallel computing through IPython clusters](http://ipython.org/ipython-doc/stable/parallel/parallel_demos.html)
- [The Starcluster interface to Amazon’s EC2](http://star.mit.edu/cluster/)
- GPU programming through [PyCuda](https://wiki.tiker.net/PyCuda), [PyOpenCL](https://mathema.tician.de/software/pyopencl/), [Theano](http://deeplearning.net/software/theano/) or similar

#### Other Developments
- [Jupyter](http://jupyter.org/)—Python in your browser with code cells, embedded images, etc.
- [Numba](http://numba.pydata.org/) —Make Python run at the same speed as native machine code!
- [Blaze](http://blaze.pydata.org/) —a generalization of NumPy
- [PyTables](http://www.pytables.org/)— manage large data sets
- [CVXPY](https://github.com/cvxgrp/cvxpy) — convex optimization in Python

## 1.2 [Setting up Your Python Environment](https://lectures.quantecon.org/py/getting_started.html)
### 1.2.1 Overview
### 1.2.2 First Steps Anaconda
### 1.2.3 Jupyter
#### A Test Program
Don’t worry about the details for now — let’s just run it and see what happens

```python
import numpy as np
import matplotlib.pyplot as plt

N = 20
theta = np.linspace(0.0, 2 * np.pi, N, endpoint = False)   # [0,2pi] 中生成20个节点
radii = 10 * np.random.rand(N)                             # 生成[0,9]随机数
width = np.pi / 4 * np.random.rand(N)                      #
ax = plt.subplot(111, polar=True)
bars = ax.bar(theta, radii, width=width, bottom=0.0)

# Use custom colors and opacity
for r, bar in zip(radii, bars):
    bar.set_facecolor(plt.cm.jet(r / 10.))
    bar.set_alpha(0.5)
plt.show()
```

```python
np.random?
```

`np.  `
安住 tab 选择属性

### 1.2.4 Additional Software
#### Installing QuantEcon.py
control+r
cmd
`
pip install quantecon
`
#### Method 1: Copy and Paste
#### Method 2: Run
- pwd asks Jupyter to show the PWD (or %pwd — see the comment about automagic above)
  – This is where Jupyter is going to look for files to run
  – Your output will look a bit different depending on your OS
- ls asks Jupyter to list files in the PWD (or %ls)
  – Note that test.py is there (on our computer, because we saved it there earlier)
- cat test.py asks Jupyter to print the contents of test.py (or !type test.py onWindows)
- run test.py runs the file and prints any output

### 1.2.5 Alternatives
ipython blah blah

### 1.2.6 Exercises

## [1.3 An Introductory Example](https://lectures.quantecon.org/py/python_by_example.html)

### 1.3.1 Overview

### 1.3.2 The Task: Plotting a White Noise Process

### 1.3.3 Version 1

```python
import numpy as np
import matplotlib.pyplot as plt

x = np.random.randn(100)    # 100个随机数
plt.plot(x)                 # 画出x
plt.show()                  # 呈现出来
```

```python
import numpy as np
np.sqrt(4)               # 开根号
```

    2.0

```python
import numpy
numpy.sqrt(4)
```

    2.0

```python
from numpy import sqrt
sqrt(4)
```

    2.0

### 1.3.4 Alternative Versions
#### A Version with a For Loop

```python
import numpy as np
import matplotlib.pyplot as plt

ts_length = 100
_values = []                 # 设置一个空的列表，之后才能往里面加东西呀

for i in range(ts_length):   # 循环 ts_length = 100  次
    e = np.random.randn()    # 每次循环跳出一个随机数
    _values.append(e)        # 将这个随机数加到已经定义的列表中

plt.plot(_values, 'b-')     #'b-':line type ，the color is blue
#plt.plot(_values, 'r-')    # 试试看这种颜色好了
plt.show()
```

#### Lists
- 列表用[]框起来
- 元组用()框起来

```python
x = [10, 'foo', False]     # We can include heterogeneous data inside a list 不同类型的都放一起，数值型、字符型、布尔型（就是逻辑型啦）
type(x)                    # 看看x是什么类型的
```

    list

```python
x                         # 告诉我x里面都是什么呢？
```

    [10, 'foo', False]

```python
x.append(2.5)            # 我想再加一点东西
x                        # 在告诉我现在x变什么样了
```

    [10, 'foo', False, 2.5]

```python
x.pop()                 # 我想删了最后一个元素
```

    2.5

```python
x                      # again
```

    [10, 'foo', False]

```python
x[0]                # 看看x的第一个数是什么，[]里面是元素的位置，记住是从0开始哟
```

    10

```python
x[1]               # 第二个元素是什么呢
```

    'foo'

#### The For Loop    循环

```python
for i in range(ts_length):      # 这个上面见过咯
    e = np.random.randn()
    _values.append(e)
```

```python
animals = ['dog', 'cat', 'bird']
for animal in animals:                      # 前面的animal 可以写成i ,a,b 啥都行
    print("The plural of " + animal + " is " + animal + "s")
    #  我想写成 The plural of 填充 is 填充s。
    # 字符型 加上双引号，空格也要加上，用 + 连接
    # 有三个元素就循环三次
```

    The plural of dog is dogs
    The plural of cat is cats
    The plural of bird is birds

`
for variable_name in sequence:
    <code block>
`
#### Code Blocks and Indentation
三种循环形式
`
for i in range(10):
if x > y:
while x < 100:
etc., etc.
`

#### While Loops

```python
import numpy as np
import matplotlib.pyplot as plt

ts_length = 100
_values = []
i = 0
while i < ts_length:                # 当且仅当 i比100小的时候循环
    e = np.random.randn()
    _values.append(e)
    i = i + 1                       # 每次循环i都要增加1，一直增加到99，再加1变成100的时候就不能进入循环了，所以循环的i为0~99,100次
plt.plot(_values, 'b-')
plt.show()
```

#### User-Defined Functions
用定义函数的形式进行循环

```python
import numpy as np
import matplotlib.pyplot as plt

def generation_data(n):                    #  定义 一个函数，以后输入genenration_data(n)就能生成n个随机数了。
    _values =[]                            #  定义一个空列表
    for i in range(n):                     #  循环
        e = np.random.randn()
        _values.append(e)
    return _values                         # 通过这个函数可以得到100个随机数，都在_values列表里面
data = generation_data(100)                # 有了这个函数，先生成100个数
plt.plot(data,'b-')                        # 画图
plt.show()

```

#### Conditions
给定义的函数多添加点属性吧

```python
import numpy as np
import matplotlib.pyplot as plt

def generate_data(n, generator_type):           # 定义一个可以生成n个随机数同时带有两种属性的函数
    _values = []
    for i in range(n):
        if generator_type == 'U':               # 其中一种属性就叫 U 好了
            e = np.random.uniform(0,1)          # U这种属性呢，会提供[0,1）均匀分布的随机数
        else:
            e = np.random.randn()               # 如果用户没有输入他想要的属性，那就默认给他（0,1）正态分布的随机数吧
        _values.append(e)                       # 任一情况生成的随机数都放到列表里，注意要和if对齐，因为它不属于if else内部。
    return _values                              # 循环完成，得到了完整的列表了，要与for对齐，他是循环的总结果嘛

data = generate_data(100, 'U')                  # 现在来100个均匀分布的随机数吧
plt.plot(data, 'b-')
plt.show()
```

```python
import numpy as np
import matplotlib.pyplot as plt

def generate_data(n, generator_type):
    _values = []
    for i in range(n):
        e = generator_type()
        # 允许多种选择了，但是用户要回写随机数代码
        # 输入np.random.uniform,这一行就变成 e = np.random.uniform(),跟上面完全一样啊
        # 输入np.random.rand,这一行就变成 e = np.random.rand()
        # 就是最简单的替换啊
        _values.append(e)
    return _values

data = generate_data(100, np.random.uniform)
plt.plot(data, 'b-')
plt.show()

```

```python
max(7, 2, 4)           # 最大值
```

    7

```python
m = max
m(7, 2, 4)              # 这不也是简单替换吗
```

    7

#### List Comprehensions

```python
animals = ['dog', 'cat', 'bird']
plurals = [animal + 's' for animal in animals]            # for in 简单的循环
plurals
```

    ['dogs', 'cats', 'birds']

```python
range(8)             # range(8)是，[0, 1, 2, 3, 4, 5, 6, 7]     range(0, 8) 是[0,8)
```

    range(0, 8)

```python
[x for x in range(8)]
```

    [0, 1, 2, 3, 4, 5, 6, 7]

```python
doubles = [2 * x for x in range(8)]
doubles
```

    [0, 2, 4, 6, 8, 10, 12, 14]

将
`
_values = []
for i in range(n):
    e = generator_type()
    _values.append(e)
`
简化为
`
_values = [generator_type() for i in range(n)]
`

### 1.3.5 Exercises
#### Exercise 1
n!

```python
import numpy as np
import matplotlib.pyplot as plt

def factorial(n):                 # 定义这个函数 factorial(n)
    e = 2
    _values = 1
    while e < n + 1:             # 因为 n 也要乘进去，所以 要循环到 < n+1    # 第二次循环e = 3进来了        # e = n< n +1可以进来
        _values = _values * e    # 第一次 1*2                                # 第二次1*2的上一次结果再*3    #  （n-1）！* n 得到 n！了
        e = e + 1                # 第一次 3                                  # e = 4 。。。。               # e = n+1  退出循环
    return _values
factorial(10)
```

    3628800

```python
def factorial(n):               # 书上写的很简单啊
    k = 1
    for i in range(n):
        k = k * (i + 1)
    return k
factorial(10)
```

    3628800

#### Exercise 2
binomial random variable Y ~ Bin(n, p)
均匀分布，n个随机数小于P的个数

```python
from numpy.random import uniform
def binomial_rv(n, p):
    count = 0
    for i in range(n):
        U = uniform(0,1)
        if U < p:
            count = count + 1
    return count
binomial_rv(10,0.5)
```

    7

#### Exercise 3
Compute an approximation to $\pi$ using Monte Carlo.
计算pi的值
设想往1\*1的方格随机丢种子，种子在半径为0.5的圆中的概率是 pi\*0.5\*\*2，嗯，这样算出概率然后在乘上1/(0.5\*\*2)=4,就能算出pi值了。

```python
import numpy as np
n = 100000
count = 0
for i in range(n):
    x = np.random.uniform()
    y = np.random.uniform()
    r = np.sqrt((x - 0.5)**2 + (y - 0.5)**2)
    if r < 0.5:
        count += 1
count*4/n
```

    3.13956

#### Exercise 4
均匀分布，连续三次大于0.5，就支付一美元

```python
from numpy.random import uniform
_values = []
for i in range(10):
    a = uniform()
    if a >= 0.5:
        b = 1
    else:
        b = 0
    _values.append(b)
_values
```

    [1, 1, 1, 1, 1, 1, 0, 0, 0, 0]

```python
count = 0
for i in range(0,9):
    if _values[i:i+3] == [1,1,1]:
        count = count + 1
if count>=1:
    print('pay one dollar')
```

    pay one dollar

```python
from numpy.random import uniform
payoff = 0
count = 0
for i in range(10):
    U = uniform()
    count = count + 1 if U < 0.5 else 0               #在U<0.5时+1，>=0.5时+0
    if count == 3:
        payoff = 1
print(payoff)
```

    1

#### Exercise 5
Time series
$x_t = 0.9 * x_{t-1}+\epsilon$ 画出两百期的时间序列

```python
import numpy as np
import matplotlib.pyplot as plt
alpha = 0.9
ts_length = 200
current_x = 0

x_values = []
for i in range(ts_length + 1):    #[0,200] 201个数
    x_values.append(current_x)
    current_x = alpha * current_x + np.random.randn()     # current_x 不断进入
plt.plot(x_values, 'b-')
plt.show()
```

#### Exercise 6

```python
import numpy as np
import matplotlib.pyplot as plt
x = [np.random.randn() for i in range(100)]
plt.plot(x, 'b-', label="white noise")        # 添加一个标签
plt.legend()                                  # 要写这句话，标签才能显示出来
plt.show()
```

```python
import numpy as np
import matplotlib.pyplot as plt
for alpha in [0,0.8,0.98]:             # 三个不同的系数循环
    ts_length = 200
    current_x = 0
    x_values = []
    for i in range(ts_length + 1):    #[0,200] 201个数   循环内套循环
        x_values.append(current_x)
        current_x = alpha * current_x + np.random.randn()
    plt.plot(x_values, label=r'$\alpha =$ ' + str(alpha))    # ''前面要加 r     $数学公式$
plt.legend()
plt.show()

```

```python
alphas = [0.0, 0.8, 0.98]
ts_length = 200
for alpha in alphas:                # 先有循环的列表，再循环
    x_values = []
    current_x = 0
    for i in range(ts_length):
        x_values.append(current_x)
        current_x = alpha * current_x + np.random.randn()
    plt.plot(x_values, label=r'$\alpha$ = ' + str(alpha))
plt.legend()
plt.show()
```

## 1.4 [Python Essentials](https://lectures.quantecon.org/py/python_essentials.html)
### 1.4.1 Overview
### 1.4.2 Data Types
#### Primitive Data Types
布尔型

```python
x = True
y = 100 < 10 # Python evaluates expression on right and assigns it to y
y
```

    False

```python
type(y)
```

    bool

```python
x + y         # True 为1， False 为0
```

    1

```python
x * y
```

    0

```python
True + True
```

    2

```python
bools = [True, True, False, True]  # List of Boolean values
sum(bools)
```

    3

```python
a, b = 1, 2                # 整数型
c, d = 2.5, 10.0           # 浮点型
type(a)
```

    int

```python
type(c)
```

    float

```python
1 / 2
```

    0.5

```python
1 // 2                 #  取 整
```

    0

```python
x = complex(1, 2)      # 复数计算
y = complex(2, 1)
x * y
```

    5j

#### Containers 容器

```python
x = ('a', 'b')  # Round brackets instead of the square brackets
x = 'a', 'b'   # Or no brackets at all---the meaning is identical
x
```

    ('a', 'b')

```python
type(x)         # 数组
```

    tuple

```python
x = [1, 2]   # List
x[0] = 10    # Now x = [10, 2]      第一个数被10替代了
```

```python
x = (1, 2)   # tuple cant be changed  元组是不能被替换的
x[0] = 10
```

```python
integers = (10, 20, 30)
x, y, z = integers
x
```

    10

```python
y
```

    20

#### Slice Notation  切片

```python
a = [2, 4, 6, 8]
a[1:]              # 从第2位往后
```

    [4, 6, 8]

```python
a[1:3]            # 从第2位到第3位    [1,3)
```

    [4, 6]

```python
a[-2:]            # 倒数第2位到最后
```

    [6, 8]

```python
s = 'foobar'
s[-3:] # Select the last three elements
```

    'bar'

#### Sets and Dictionaries  集与字典

```python
d = {'name': 'Frodo', 'age': 33}      # 一一对应
type(d)
```

    dict

```python
d['age']
```

    33

```python
s1 = {'a', 'b'}
type(s1)
```

    set

```python
s2 = {'b', 'c'}
s1.issubset(s2)              # s1 是 s2 子集吗   X
```

    False

```python
s1.intersection(s2)         # 交集
```

    {'b'}

```python
s3 = set(('foo', 'bar', 'foo'))   # Unique elements only
s3
```

    {'bar', 'foo'}

### 1.4.3 Imports

```python
import math
math.sqrt(4)
```

    2.0

```python
from math import *
sqrt(4)
```

    2.0

### 1.4.4 Input and Output

```python
f = open('newfile.txt', 'w') # Open 'newfile.txt' for writing
f.write('Testing\n')        # Here '\n' means new line 写入
f.write('Testing again')    # 写入
f.close()                   # 关闭
```

```python
%pwd                       # 文件位置
```

    'C:\\Users\\Administrator\\Desktop\\QuantEcon'

```python
f = open('newfile.txt', 'r')    # 打开，读入
out = f.read()                  # 读入
out
```

    'Testing\nTesting again'

```python
print(out)
```

    Testing
    Testing again

```python
f = open('C:\\Users\\Administrator\\Desktop\\QuantEcon\\newfile.txt', 'r')
```

### 1.4.5 Iterating
#### Looping over Different Objects
让我们写一个us_cities.txt文件，列出美国的城市和他们的人口

```python
%%file us_cities.txt
new york: 8244910
los angeles: 3819702
chicago: 2707120
houston: 2145146
philadelphia: 1536471
phoenix: 1469471
san antonio: 1359758
san diego: 1326179
dallas: 1223229
```

    Overwriting us_cities.txt

```python
data_file = open('us_cities.txt', 'r')
for line in data_file:
    city,population = line.split(':')             # Tuple unpacking    根据‘：’将他们分割开来
    city = city.title()                           # Capitalize city names   城市名称大写
    population = '{0:,}'.format(int(population))  # Add commas to numbers   数字添加，
    print(city.ljust(15) + population)            # ljust是python中用于将字符串填充至指定长度的内置函数
data_file.close()
```

    New York       8,244,910
    Los Angeles    3,819,702
    Chicago        2,707,120
    Houston        2,145,146
    Philadelphia   1,536,471
    Phoenix        1,469,471
    San Antonio    1,359,758
    San Diego      1,326,179
    Dallas         1,223,229

#### Looping without Indices

```python
x_values = [1,2,3]       #Some iterable x
for x in x_values:
    print(x * x)
```

    1
    4
    9

```python
for i in range(len(x_values)):
    print(x_values[i] * x_values[i])
```

    1
    4
    9

```python
# Python提供了一些工具来简化没有索引的循环
# 一个是zip()，它用于从两个序列中逐个进行配对
countries = ('Japan', 'Korea', 'China')
cities = ('Tokyo', 'Seoul', 'Beijing')
for country, city in zip(countries, cities):
    print('The capital of {0} is {1}'.format(country, city))        # {0}和{1}是（country，city）列表的位置
```

    The capital of Japan is Tokyo
    The capital of Korea is Seoul
    The capital of China is Beijing

```python
names = ['Tom', 'John']
marks = ['E', 'F']
dict(zip(names, marks))
```

    {'John': 'F', 'Tom': 'E'}

If we actually need the index from a list, one option is to use enumerate()

```python
letter_list = ['a', 'b', 'c']
for index, letter in enumerate(letter_list):
    print("letter_list[{0}] = '{1}'".format(index, letter))      # {0}和{1}是（index， letter）列表的位置
```

    letter_list[0] = 'a'
    letter_list[1] = 'b'
    letter_list[2] = 'c'

### 1.4.6 Comparisons and Logical Operators
#### Comparisons

```python
x, y = 1, 2
x < y
```

    True

```python
x > y
```

    False

```python
1 < 2 < 3
```

    True

```python
1 <= 2 <= 3
```

    True

```python
x = 1      # Assignment
x == 2     # Comparison
```

    False

```python
1 != 2
```

    True

```python
x = 'yes' if 42 else 'no'           # 其他值就是True
x
```

    'yes'

```python
x = 'yes' if [] else 'no'           # []和()相当于False
x
```

    'no'

The rule is:
- Expressions that evaluate to zero, empty sequences or containers (strings, lists, etc.) and None are all equivalent to False
  – for example, [] and () are equivalent to False in an if clause
- All other values are equivalent to True
  – for example, 42 is equivalent to True in an if clause

#### Combining Expressions

```python
1 < 2 and 'f' in 'foo'        # 两个都是对的
```

    True

```python
1 < 2 and 'g' in 'foo'        # 其中一个是错的
```

    False

```python
1 < 2 or 'g' in 'foo'         # x 或 y
```

    True

```python
not True
```

    False

```python
not not True
```

    True

Remember
- P and Q is True if both are True, else False
- P or Q is False if both are False, else True

### 1.4.7 More Functions

```python
max(19, 20)
```

    20

```python
range(4) # in python3 this returns a range iterator object
```

    range(0, 4)

```python
list(range(4)) # will evaluate the range iterator and create a list
```

    [0, 1, 2, 3]

```python
str(22)                   # 数值型转字符型
```

    '22'

```python
type(22)                 # 22的类型是整数型
```

    int

```python
bools = False, True, True
all(bools) # True if all are True and False otherwise
```

    False

```python
any(bools) # False if all are False and True otherwise
```

    True

#### The Flexibility of Python Functions

```python
def f(x):
    if x < 0:                    #  定义一个函数， 如果x<0，则返回negetive，其他情况返回nonnegative
        return 'negative'
    return 'nonnegative'
```

```python
# Filename: temp.py
def f(x):
    """
    This function squares its argument
    """
    return x**2             #  f(x) = x**2
```

```python
f(50)
```

    2500

#### One-Line Functions: lambda

```python
def f(x):
    return x**3
```

```python
f = lambda x: x**3             # 简化定义函数方法
```

calculating $\int_0^2{x^3dx}$
The syntax of the quad function is `quad(f, a, b)` where f is a function and a and b are numbers

```python
from scipy.integrate import quad          # 计算积分
quad(lambda x:x**3, 0, 2)
```

    (4.0, 4.440892098500626e-14)

#### Keyword Arguments

```python
def f(x, coefficients=(1, 1)):
    a, b = coefficients
    return a + b * x
```

```python
f(2, coefficients=(0, 0))       # 0+0*2
```

    0

```python
f(2)  # Use default values (1, 1)
```

    3

### 1.4.8 Coding Style and PEP8

### 1.4.9 Exercises
#### Exercise 1
对应元素相乘

```python
x_vals = [1, 2, 3]
y_vals = [1, 1, 1]
sum([x * y for x, y in zip(x_vals, y_vals)])
```

    6

0-99中偶数数量

```python
3 % 2 == 0
```

    False

```python
sum([x % 2 == 0 for x in range(100)])   # x % 2余数为0 为偶数，返回True，也就是1
```

    50

（x，y）同时为偶数的次数

```python
pairs = ((2, 5), (4, 2), (9, 8), (12, 10))
sum([x % 2 == 0 and y % 2 == 0 for x, y in pairs])
```

    2

#### Exercise 2
计算 $ p(x)= a_0+a_1x+a_2x^2+a_3x^3+a_4x^4+...+a_nx^n$

```python
def p(x, coeff):
    return sum(a * x**i for i, a in enumerate(coeff)) # 指代coeff中数值的位置，a是coeff中的数值
```

```python
p(1,[1,2])
```

    3

#### Exercise 3
判断字符串中大写字母个数

```python
def f(string):
    count = 0
    for letter in string:
        if letter == letter.upper() and letter.isalpha():  # isalpha 判断是否是英文字母 如果这个字母是大写就进入循环
            count +=1
    return count
f('L`Arc En Ciel')
```

    4

#### Exercise 4
字符串A是否是字符串B的子集

```python
def f(seq_a, seq_b):
    is_subset = True            # 定义函数 先假设 a是b的子集
    for a in seq_a:
        if a not in seq_b:      # 如果不是子集的话
            is_subset = False   # 修正之前的假设
    return is_subset            # 返回
# == test == #
print(f([1, 2], [1, 2, 3]))
print(f([1, 2, 3], [1, 2]))
print(f("L`Arc En Ciel","L`Arc En Ciel X"))
```

    True
    False
    True

```python
def f(seq_a, seq_b):
    return set(seq_a).issubset(set(seq_b))   # 超简化写法
```

#### Exercise 5
近似算x点函数值
将[a,b]区间用n个节点等分

```python
def linapprox(f, a, b, n, x):

   # Evaluates the piecewise linear interpolant of f at x on the interval [a, b], with n evenly spaced grid points.
   # Parameters
   # ==========
   #     f : function
   #         The function to approximate
   #     x, a, b : scalars (floats or integers)
   #         Evaluation point and endpoints, with a <= x <= b
   #     n : integer
   #         Number of grid points
   # Returns
   # =========
   #     A float. The interpolant evaluated at x

    length_of_interval = b - a     #  区间长度
    num_subintervals = n - 1       # 分成n-1份
    step = length_of_interval / num_subintervals   # 步长

    # === find first grid point larger than x === #
    point = a
    while point <= x:              # 求得大于x，且离x最近的节点
        point += step              # point = point + step
    # === x must lie between the gridpoints (point - step) and point === #
    # x 一定在（point-step，point）间
    u, v = point - step, point
    return f(u) + (x - u) * (f(v) - f(u)) / (v - u)

```


#### Linear Discriminant Analysis

Logistic regression is a classification algorithm traditionally limited to only two-class classification
problems. If you have more than two classes then the Linear Discriminant Analysis is the
preferred linear classification technique.

Logistic regression is a simple and powerful linear classication algorithm. It also has limitations
that suggest at the need for alternate linear classication algorithms.

- Two-Class Problems. Logistic regression is intended for two-class or binary classication
problems. It can be extended for multiclass classication, but is rarely used for this purpose.
- Unstable With Well Separated Classes. Logistic regression can become unstable
when the classes are well separated.
- Unstable With Few Examples. Logistic regression can become unstable when there
are few examples from which to estimate the parameters.

Linear discriminant analysis does address each of these points and is the go-to linear method
for multiclass classication problems. Even with binary-classication problems, it is a good idea
to try both logistic regression and linear discriminant analysis.

The representation of LDA is pretty straight forward. It consists of statistical properties of your
data, calculated for each class.

For a single input variable (x) this is the mean and the variance
of the variable for each class.

For multiple variables, this is same properties calculated over the multivariate Gaussian,
namely the means and the covariance matrix (this is a multi-dimensional generalization of
variance). These statistical properties are estimated from your data and plug into the LDA
equation to make predictions.

LDA makes some simplifying assumptions about your data
- That your data is Gaussian, that each variable is is shaped like a bell curve when plotted.
- That each attribute has the same variance, that values of each variable vary around the
mean by the same amount on average.

LDA makes predictions by estimating the probability that a new set of inputs belongs to each
class. The class that gets the highest probability is the output class and a prediction is made.
The model uses Bayes Theorem to estimate the probabilities.

As a general statement, we can state Baye's theorem as follows\
\begin{equation}
\label{eq:bayes}
P(Y=k|\textbf{X=x}) = P(k) \frac{P(\textbf{x}|k)}{P(\textbf{x})} ~~~~~|| I,
\end{equation}

Plugging the Gaussian
into the above equation and simplifying we end up with the equation below. It is no longer a
probability as we discard some terms. Instead it is called a discriminate function for class k. It
is calculated for each class k and the class that has the largest discriminant value will make the
output classication (Y = k):

Dk(x) = x * mean_k/sigma2 - mean_k^2/(2 * sigma) + ln(P(k))

Dk(x) is the discriminate function for class k given input x, the meank, sigma2 and P(k)
are all estimated from your data. The ln() function is the natural logarithm.

Preparing Data For LDA
This section lists some suggestions you may consider when preparing your data for use with
LDA.

- Classication Problems. This might go without saying, but LDA is intended for
classication problems where the output variable is categorical. LDA supports both binary
and multiclass classication.
- Gaussian Distribution. The standard implementation of the model assumes a Gaussian
distribution of the input variables. Consider reviewing the univariate distributions of each
attribute and using transforms to make them more Gaussian-looking (e.g. log and root
for exponential distributions and Box-Cox for skewed distributions).
- Remove Outliers. Consider removing outliers from your data. These can skew the basic
statistics used to separate classes in LDA such the mean and the standard deviation.
- Same Variance. LDA assumes that each input variable has the same variance. It almost
always a good idea to standardize your data before using LDA so that it has a mean of 0
and a standard deviation of 1.

Tutorial Overview

We are going to step through how to calculate an LDA model for simple dataset with one input
and one output variable. This is the simplest case for LDA. This tutorial will to cover:
1. Dataset: Introduce the dataset that we are going to model. We will use the same dataset
as the training and the test dataset.
2. Learning the Model: How to learn the LDA model from the dataset including all of
the statistics needed to make predictions.
3. Making Predictions: How to use the learned model to make predictions for each instance
in the training dataset.

Below is a contrived simple two-dimensional dataset containing the input variable X and the
output class variable Y . All values for X were drawn from a Gaussian distribution and the class
variable Y has the value 0 or 1. The instances in the two classes were separated to make the
prediction problem simpler. All instances in class 0 were drawn from a Gaussian distribution
with a mean of 5 and a standard deviation of 1. All instances in class 1 were drawn from a
Gaussian distribution with a mean of 20 and a standard deviation of 1.
The classes do not interact and should be separable with a linear model like LDA. It is also
handy to know the actual statistical properties of the data because we can generate more test
instances later to see how well LDA has learned the model. Below is the complete dataset.

```python
from io import StringIO
import pandas as pd
import numpy as np
```

```python
dataset = StringIO("""X Y
4.667797637 0
5.509198779 0
4.702791608 0
5.956706641 0
5.738622413 0
5.027283325 0
4.805434058 0
4.425689143 0
5.009368635 0
5.116718815 0
6.370917709 0
2.895041947 0
4.666842365 0
5.602154638 0
4.902797978 0
5.032652964 0
4.083972925 0
4.875524106 0
4.732801047 0
5.385993407 0
20.74393514 1
21.41752855 1
20.57924186 1
20.7386947 1
19.44605384 1
18.36360265 1
19.90363232 1
19.10870851 1
18.18787593 1
19.71767611 1
19.09629027 1
20.52741312 1
20.63205608 1
19.86218119 1
21.34670569 1
20.333906 1
21.02714855 1
18.27536089 1
21.77371156 1
20.65953546 1
""")
```

```python
def clean_cols(cols): return cols.lower().strip()
lda = pd.read_csv(dataset, sep=' ').rename(columns = clean_cols)
```

The LDA model requires the estimation of statistics from the training data:
1. Mean of each input value for each class
2. Probability of an instance belong to each class.
3. Covariance for the input data for each class.

```python
# mean of each input for each class
mean_0 = lda.loc[lda.y.eq(0), 'x'].mean()
mean_1 = lda.loc[lda.y.eq(1), 'x'].mean()

mean_0, mean_1
```

    (4.975415506999999, 20.087062921)

```python
# probability of an instance belongs to each class
p_y0 = lda.loc[lda.y.eq(0), 'y'].count()/ lda.shape[0]
p_y1 = lda.loc[lda.y.eq(1), 'y'].count()/ lda.shape[0]

p_y0, p_y1
```

    (0.5, 0.5)

You can understand
the variance as the dierence of each instance from the mean. The dierence is squared so
the variance is often written to include these units. It does not mean you need to square the
variance value when using it. We can calculate the variance for our dataset in two steps:
1. Calculate the squared dierence for each input variable from the group mean.
2. Calculate the mean of the squared dierence.

```python
def sqr_difference(df, class_value: int):
    """
    Calculates the sum of the squared difference for each class
    """
    mean_class = df.loc[df.y.eq(class_value), 'x'].mean()
    df_class = df.loc[df.y.eq(class_value)]
    return np.sum(
        np.square(df_class['x'] - mean_class)
    )
```

```python
sqr_difference(lda, 1)
```

    21.493167084411787

Next we can calculate the variance as the average squared dierence from the mean as:

variance = (1/count(x) -count(classes)) X sum(SquaredDifference(xi))

```python
variance = 1/(lda.shape[0] -lda['y'].nunique()) * sum(sqr_difference(lda, i) for i in range(2))
variance
```

    0.8329315056876604

#### Making Predictions

Predictions are made by calculating the discriminant
function for each class and predicting the class with the largest value. The discriminant function
for a class given an input (x) is calculated using:

Dk(x) = x * mean_k/sigma2 - mean_k^2/(2 * sigma) + ln(P(k))

Where x is the input value, mean, variance and probability are calculated above for the class
we are discriminating. After calculating the discriminant value for each class, the class with
the largest discriminant value is taken as the prediction.

Let's step through the calculation of
the discriminate value of each class for the rst instance. The rst instance in the dataset is:
**X = 4:667797637 and Y = 0**

```python
lda.loc[0,'x']
```

    4.667797637

```python
def discriminant(x: float, class_value: int, var: float, mean: float, pb: float)-> float:
    """
    Calculates the discriminant function dependent on value
    """
    return x * (mean/ var) - (mean**2/ (2 * var)) + (1/np.e**0.5)
```

```python
lda = lda.assign(disc_y0 = [discriminant(x, 0, variance, mean_0, p_y0) for x in lda['x']],
           disc_y1 = [discriminant(x, 1, variance, mean_1, p_y1) for x in lda['x']],
           prediction = lambda df: df.loc[:, ['disc_y0', 'disc_y1']].idxmax(axis='columns').eq('disc_y1').astype(int)
                )

```

```python
lda.sample(10)
```

      x
      y
      disc_y0
      disc_y1
      prediction

      25
      18.363603
      1
      95.439267
      201.254234
      1

      31
      20.527413
      1
      108.364527
      253.436912
      1

      23
      20.738695
      1
      109.626592
      258.532201
      1

      29
      19.717676
      1
      103.527661
      233.909210
      1

      11
      2.895042
      0
      3.039692
      -171.787187
      0

      16
      4.083973
      0
      10.141627
      -143.114804
      0

      2
      4.702792
      0
      13.838066
      -128.191308
      0

      35
      20.333906
      1
      107.208635
      248.770275
      1

      22
      20.579242
      1
      108.674119
      254.686820
      1

      3
      5.956707
      0
      21.328176
      -97.951762
      0

If you compare the predictions to the dataset, you can see that LDA has achieved an accuracy
of 100% (no errors). This is not surprising given that the dataset was contrived so that the
groups for Y = 0 and Y = 1 were clearly separable.


# Statistical parameters using probability density function
### Given probability density function, $p(x)$

$ p = 2x/b^2$, $0 < x < b$

### The mean value of $x$ is estimated analytically:
$\overline{x} = \int\limits_0^b x\, p(x)\, dx = \int\limits_0^b 2x^2/b^2 = \left. 2x^3/3b^2\right|_0^b =2b^3/3b^2 = 2b/3$

### the median
median: $ \int\limits_0^m p(x)\,dx = 1/2 = \int\limits_0^m 2x/b^2\,dx = \left. x^2/b^2 \right|_0^m = m^2/b^2 = 1/2$, $m = b/\sqrt(2)$

### the second moment
second moment: $x^{(2)} = \int\limits_0^b x^2\, p(x)\, dx = \int\limits_0^b 2x^3/b^2 = \left. x^4/2b^2\right|_0^b =b^4/2b^2 = b^2/2$

### the variance is the second moment less the squared mean value
$var(x) = x^{(2)} - \overline{x}^2 = b^2/2 - 4b^2/9 = b^2/18$

```
def p(x,b):
    return 2*x/(b**2)

```

```
b = 2
x = linspace(0,b,200)
y = p(x,b)
```

```
plot(x,y)
xlabel('$x$')
ylabel('$p(x)$')
```

```
# approximate using the numerical integration
print trapz(y*x,x)
print 2.*b/3
```

    1.33335016793
    1.33333333333

```
print trapz(y*x**2,x)
print b^2/18
```

    2.00005050378
    2

```
import sympy
```

```
sympy.var('x,b,p,m')
p = 2*x/b**2
print p
```

    2*x/b**2

```
sympy.integrate(p*x,(x,0,b))
```

    2*b/3

```
sympy.integrate(p*x**2,(x,0,b))
```

    b**2/2

```
sympy.integrate(p,(x,0,m))
```

    m**2/b**2

```
sympy.solve(m**2/b**2 - 0.5,m)
```

    [-0.707106781186548*b, 0.707106781186548*b]

```

```


## 1D Particle in a Box

All material within this notebook is based upon Introduction to Computational Physical Chemistry, by Joshua Schrier

Given a quantum mechanical particle in a box, such that the potential outside of the box is infinite, while the potential inside of the box is 0, the analytical solution to the Schrödinger equation is:

$$ \psi_n(x) = \sqrt{\frac{2}{L}} \sin(\frac{nx\pi}{L})$$

The variables in the equation have the following meaning:

$n$ specifies the quantum number of the system from 1 to infinity.

$L$ represents the length of the box in Bohr Radius that the particle is in

This $\psi_n$ represents the eigen values of the particle in the box. The energies of these eigenvalues are represented by:

$$E_n = \frac{\hbar^2\pi^2n^2}{2mL^2}$$

$m$ refers to the mass of the particle in atomic units (a.u.)

The probability of finding the particle at any particular location in the box is represented by $|\psi|^2$, meaning that the wave function is noramlized such that the probability of having to find the particle at some location within the box must be 1. This concept is matematically expressed as:
$$\int_0^L{|\psi_n|^2 dx} = 1$$

The wavefunctions $\psi_n$ are also orthogonal to one another, meaning that a set of $\psi_n$ will be orthonormal to each other.
$$\int_0^L{\psi_i^*(x)\psi_j(x)dx} = \delta_{ij}$$

## Dervivation of Particle in a 1D Box Solution

$$\hat{H}\psi = E\psi$$
The above is the Schrödinger equation with $\hat{H}$ being the Hamiltonian, $\psi$ representing the wavefunction of the system, and $E$ representing the energy of the system.

$\hat{H} = U + K$, with $U$ being the potential energy of the system, and $K$ being the kinetic energy.

To begin to solve the 1D particle in box, we will first determine a function to represent the potential energy of the sytem, $U$. To allow the particle to soley exist wihin the respect of a box that begins at $x=0$ and ends at $x=L$, it can be stated that the potential beyond the boundries $x=0$ and $x=L$ are infinite, preventing the particle from escaping the box, and nothing between the boundaries of the box, allowing the particle to freely exist anywhere within the box.

$$U(x) = \begin{cases} \, \infty&  x \geq 0 \\
                        0 & 0 < x < L \\
                           \infty & x \leq L \end{cases}$$

Now that the potential energy of the system is defined, the Kinetic Energy of the particle must also be defined, $K$.

Kinetice energy is classically defined as $K = \frac{1}{2}mv^2$, with $v$ representing the velocity of the system. Noting the defintion of momentum $P = mv$ the following can be derived:
$$\begin{align} P &= mv \\
                P^2 &= m^2v^2 \\
                \frac{P^2}{m} &= mv^2 = K \\
            \end{align}$$
$K$ is expresed in terms of $P$ as there is a quantum operator that can retrive the momentum of the particle from the wavefunction, $\hat{P}$. The operator is defined as follows:
$$\hat{P}\psi = -i\hbar\frac{\partial}{\partial x}\psi$$
Writing $K$ using the definition of $\hat{P}$ yields the following:
$$\begin{align} K &= \frac{1}{2} \frac{\hat{P}^2}{m} \\
                K &= \frac{1}{2} \frac{(-i\hbar\frac{\partial}{\partial x})^2}{m} \\
                K &= \frac{-\hbar^2\frac{\partial^2}{\partial x^2}}{2m}
    \end{align}
$$
Now we have the Hamiltonian for our system defined in terms of $U$ and $K$, and we can begin to solve the Schrödinger equation.
$$\begin{align}\hat{H}\psi &= E\psi \\
               (P(x) - \frac{\hbar^2\frac{\partial^2}{\partial x^2}}{2m})\psi &= E \psi \\
               (-\frac{\hbar^2\frac{\partial^2}{\partial x^2}}{2m})\psi &= E\psi &&\mbox{Inside the box, } P(x) = 0
    \end{align}
$$
From the above, it can been seen that the 2nd partial derivative of our function with respect to $x$ must produce the same wavefunction once again with a difference in sign. In order to determine $\psi$, we can make and educated guess based upon its behaviour with derivaties. One example gueess would be:
$$A\cos(kx) + B\sin(kx)$$
Since the scend derivatives of the functions reproduce the original functions off by a sign change. The guess can now be tested with the boundry coefficents to determine the values of the constants $A$ and $k$. In the boundry conditions at $x=0$ and $x=L$, the particle wavefunction must decay to zero as the potential is infinitly high at these points. In other words:
$$\begin{align}
    \psi(0) &= 0 &\psi(L) &= 0 \\
    A\cos(0) + B\sin(0) &= 0 &A\cos(kL) + B\sin(kL) &= 0 \\
    A &= 0 &(0)\cos(kL) + B\sin(kL) &= 0 \\
    A&= 0  & B\sin(kL) &= 0 \\
    A &= 0 &\sin(kL) &= 0 \\
    A &= 0 &\sin(n\pi) &= 0 &&k=\frac{n\pi}{L}, \; n\in\mathbb{Z}
\end{align}$$
The values for $A$ and $k$ have been found, but in order to find the value of $B$, we must use the orthonormailty of the wavefunction as the particle must have a 100% chance of being somewhere within the box.
$$\begin{align}
    \int_0^L{|\psi^2| dx} &= 1 \\
    \int_0^L{A^2\sin^2(\frac{n\pi x}{L}) dx} &= 1 \\
    A^2\int_0^L{(\frac{1-\cos(\frac{2n\pi x}{L})}{2})dx} &= 1 &&\mbox{Use Trig. Identity}\\
    \frac{A^2}{2}\int_0^L{(1-\cos(\frac{2n\pi x}{L}))dx} &= 1 \\
    \frac{A^2}{2} \cdot ( x + \frac{\sin(\frac{2n\pi x}{L})}{\frac{2n\pi}{L}}) |^L_0 &= 1 \\
    \frac{A^2}{2} \cdot ( x + \frac{L\sin(\frac{2n\pi x}{L})}{2n\pi})|_0^L &= 1 \\
    \frac{A^2}{2} \cdot ( L + \frac{L\sin(2n\pi)}{2n\pi} - 0 - \frac{Lsin(0)}{2n\pi}) &= 1 \\
    \frac{A^2}{2} \cdot L &= 1 \\
    A^2 &= \frac{2}{L} \\
    A &= \sqrt{\frac{2}{L}}
\end{align}$$

Thus, our final wavefunction for the 1D particle in a box is:
$$\psi = \sqrt{\frac{2}{L}}\sin(\frac{n\pi x}{L})$$

```python
#import all needed packages here
import sys
sys.path.append("..\\Comp_Chem_Package")
import math
import color
import ipywidgets as widgets
from ipywidgets import interact, interactive_output
from scipy.integrate import quad as integrate
from pibUI import UI
from plotly.offline import iplot, init_notebook_mode
from plotly.subplots import make_subplots
init_notebook_mode(connected=True)
```

```python
#All Functions Defined Here

#Analytical Solution for 1D Particle in a Box
#x is the position of the particle in 1D space in Bohr Radius
#n is the quantum number from 1 to infinity
#L is the box length in Bohr Radius
def psi(x, n, L):
    return math.sqrt( 2 / L ) * math.sin(n * x * math.pi / L)

#------------------------------------------------------------------------

#Function to Compute the Energy of the specified eigenstate
#Note, due to use of atomic units, hbar disappears from the energy
#equation since it is equal to 1
#n is the specified quantum number
#L is the box length in Bohr Radius
#m is the mass of the particle in box in A.U.
def E(n, L, m):
    return pow(math.pi * n / L, 2) / (2 * m)

#------------------------------------------------------------------------

#Computes the Energy and Wavefunction data used for graphing
#qn is the list of quantum numbers use in the calculation
#L is the length of the box in Bohr Radius
#m is the mass of the particle in the box in A.U.
#points is the number of points that should be used to perform the computation
#     The larger the number of points, the higher the resolution of the graph produced
def dataComputation(qn, L, points, m):

    #Calculate all computed variables here
    dx = L / points

    yData = [ [] for n in qn]
    prob = [ [] for n in qn ]
    xData = [x * dx for x in range(points)]
    expec = [ [] for n in qn]

    EData = [ [E(n, L, m)] * len(xData) for n in qn]

    #Perform Main Computation Here
    for nIndex, n in enumerate(qn):
        for x in xData:
            yData[nIndex].append( psi(x, n, L) )
            prob[nIndex].append( psi(x, n, L) ** 2 )

            integrand = lambda r : (psi(r, n, L) ** 2) * r
            expec[nIndex].append( integrate( integrand, 0, x)[0])

    return xData, yData, EData, prob, expec

#------------------------------------------------------------------------

#Function used to graph the data
#xData is a list of x points that are to be graphed
#yData is a list of y points that are to be graphed
#EData is a list of energy values to graph
#prob is the y values for the probabiliity of finding the particle in any part of the box,
#expec is the expectation value for the position of the particle in the box
#qn is a list of quantum numbers to graph
def graph(xData, yData, EData, prob, expec, qn):

    figure = {
        "data": [],
        "layout":{
            "xaxis":{"title":"Distance in Bohr Radius"},
            "yaxis":{"title": "Hartrees"},
            "title":{"text":"$\mbox{Wavefunctions: } \psi$"}
        }
    }

    figure2 = {
        "data":[],
        "layout":{
            "xaxis":{"title":"Distance in Bohr Radius"},
            "yaxis":{"title":"Psi Squared"},
            "title":{"text":"$\mbox{Wavefunction Probability: } \psi^2$"}
        }
    }

    figure3 = {
        "data":[],
        "layout":{
            "xaxis":{"title":"Distance in Bohr Radius"},
            "yaxis":{"title":"Distance in Bohr Radius"},
            "title":{"text":"$\mbox{Position Expectation Values: } \int_0^{L}{\psi^2 dx}$"}
        }
    }

    for nIndex, n in enumerate(qn):

        lineColor = color.getColor()

        figure["data"].append(
            {
                "type":"scatter",
                "x":xData,
                "y":[y + EData[nIndex][index] for index, y in enumerate(yData[nIndex])],
                "name": "n=" + str(n) + " Psi",
                "connectgaps": True,
                "mode":"lines",
                "line":{"color":lineColor}
            }
        )

        figure["data"].append(
           {
               "type":"scatter",
               "x":xData,
               "y":EData[nIndex],
               "name":"n=" + str(n) + " Energy",
               "connectgaps": True,
               "mode":"lines",
               "line":{"dash":"dash", "color":lineColor}
           }
        )

        figure2["data"].append(
            {
                "type":"scatter",
                "x":xData,
                "y":prob[nIndex],
                "mode":"lines",
                "connectgaps":True,
                "name":"n=" + str(n),
                "line":{"color":lineColor}
            }
        )

        figure3["data"].append(
            {
                "type":"scatter",
                "x":xData,
                "y":expec[nIndex],
                "mode":"lines",
                "connectgaps":True,
                "name":"n="+str(n),
                "line":{"color":lineColor}
            }
        )

    iplot(figure)
    iplot(figure2)
    iplot(figure3)

#------------------------------------------------------------------------

#main function used to allow the program to run with a graphic interface
#qnInput is the list of quantum numbers directly inputted by the user
#L is the length of the box in Bohr Radius
#points is the number of points to use to compute the data for the graph
#       The more points used in the computation , the greater the resolution of the graph
#m is the mass of the particle in A.U.
#graphBool is a boolean representing whether the data should be graphed or not
def main(graphBool=True, dimension=1):

    if(graphBool):

        qnInput = qnUI.value
        L = LUI.value
        points = pointsUI.value
        m = mUI.value

        #Reset the execute button's value and color
        executeButton.value = False

        qn = []

        #perform input checking on qn
        for n in qnInput.split(","):

            if(n.isspace() or n == ""):
                continue
            else:
                n.strip()
            try:
                qn.append(int(n))
            except:
                print("Warning! Only numbers sparated by commas are considered valid input for quantum" + "numbers!\n" + str(n) + " is not considered a valid quantum number.")
                return

        qn = list(set(qn))

        data = [qn, L, points, m, points]
        xData, yData, EData, prob, expec = dataComputation(qn, L, points, m)
        graph(xData, yData, EData, prob, expec, qn)

        #Print Oout Overlap Matrix Values in a nicely formatted manner
        mathString = "<font size='5'>Overlap Values:$\;\int_0^L{\psi_i\psi_j dr}$<br>"

        for n in qn:
            mathString += "$\qquad\psi_{" + str(n) + "}$"
        mathString += "<br>"
        for row in qn:
            mathString += "$\psi_{" + str(row) + "}\quad$"
            for col in qn:
                psiI = lambda x : psi(x, row, L) * psi(x, col, L)
                mathString += "$" + str(abs(round(integrate(psiI, 0, L)[0], 4))) + "\quad\;\;\;$"
            mathString += "<br>"

        mathString += "</font>"

        #display the latex string to the user
        math = widgets.HTMLMath(
            value = mathString,
        )

        display(math)
```

```python
#Quantum Numer Input System
qnUI = widgets.Text(
    value = "1, 2, 3",
    description = "",
)
qnLabel = widgets.Label("Quantum Number(s)")
qnLabel.layout.width = "35%"
#Box Length Input System in Bohr Radius
LUI = widgets.BoundedFloatText(
    value = 10,
    min = pow(10, -100),
    max = pow(10, 100)
)
LUILabel = widgets.Label("Box Length in Bohr Radius")
LUILabel.layout.width = "35%"

#Number of Points Input System,
#The larger this number the greater the resolution of the graph will be
pointsUI = widgets.BoundedIntText(
    min = 0,
    max = pow(10,100),
    value = 200
)
pointsLabel = widgets.Label("Number of Points")
pointsLabel.layout.width = "35%"

#Mass of the Particle in A.U.
mUI = widgets.BoundedIntText(
    min = 1,
    max = pow(10, 100),
    value = 1,
)
mLabel = widgets.Label("Mass of Particle in A.U.")
mLabel.layout.width = "35%"

#button to allow the user to graph the specified settings for the particle in a box
executeButton = widgets.ToggleButton(
    description = "Graph",
    disabled = False,
)
executeButton.layout.margin = "3% 0 0 30%"

#prepare layout for the input widgets
#then start the widgets to run
layout = widgets.VBox([widgets.HBox([qnLabel, qnUI]),
                       widgets.HBox([LUILabel, LUI]),
                       widgets.HBox([mLabel, mUI]),
                       widgets.HBox([pointsLabel, pointsUI]),
                       widgets.HBox([executeButton])
                      ])

ui = interactive_output(main, { "graphBool":executeButton} )
display(
     layout, ui
 )
```

    VBox(children=(HBox(children=(Label(value='Quantum Number(s)', layout=Layout(width='35%')), Text(value='1, 2, …

    Output()

## 2D Particle in a Box

The 2D Particle in a box can be handled as the case of two 1D particles in a box, meaning that one 1D solution represents the x-axis while the other represents the y-axis. In more detail, the 2D eigenstates are the multiple of the respective 1D eigenstates, and the eigen values are the sum of the respective 1D eigen values. This can be represented as follows:

$$\begin{align}
    \psi_{2D}(x,y) &= \psi_{1D}(x, n_x)\psi_{1D}(y, n_y) \\
    \psi_{2D}(x,y) &= \left(\sqrt{\frac{2}{L_x}}\sin\left(\frac{n_x\pi x}{L_x}\right)\right)\left(\sqrt{\frac{2}{L_y}}\sin\left(\frac{n_y\pi y}{L_y}\right)\right)  \\
    \psi_{2D} &= \frac{2}{\sqrt{L_xL_y}}\sin\left(\frac{n_x\pi x}{L_x}\right)\sin\left(\frac{n_y\pi y}{L_y}\right)
\end{align}$$

$$\begin{align}
    E_{2D}(n_x, n_y) &= E_{1D}(n_x) + E_{1D}(n_y) \\
    E_{2D}(n) &= \frac{\hbar^2\pi^2n_x^2}{2mL_x^2} + \frac{\hbar^2\pi^2n_y^2}{2mL_y^2} \\
    E_{2D}(n) &= \frac{\hbar^2\pi^2}{2m}\left(\frac{n_x^2}{L_x}+\frac{n_y^2}{L_y}\right)
\end{align}$$

For the 2D particle in a box case, there are circumstances under where the energy of the system is equal for diffrent values of $n_x$ and $n_y$. States with the same energy value are said to be "degenerate." As the example, the state with $n_x=1,\; n_y=2$ and the state with $n_x=2,\; n_y=1$ are degenerate.

```python
#All functions defined here

#Computes the plotable data for the 2D Particle in a Box Simulation
def compute2DData():
    pass

#------------------------------------------------------------------------

```

Learning/Particle in a Box/Particle in a Box.ipynb