# Quantum State Discrimination

Date : December 15, 2021

This notebook contains material supporting a paper, currently titled *Five Starter Pieces: Quantum Information Science via Semi-definite Programs*, by Vikesh Siddhu (vsiddhu@protonmail.com) and Sridhar Tayur (stayur@cmu.edu). The paper is available on this **[arXiv](http://arxiv.org/abs/2112.08276)** link. The arXiv paper is released there is under the **[arXiv.org perpetual, non-exclusive license](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html)**, and this code is released under the **[MIT license](https://opensource.org/licenses/MIT)**.

This notebook depends upon various packages including [numpy](https://numpy.org/) >= 1.19.5, [picos](https://picos-api.gitlab.io/picos/index.html) >= 2.2.55, and [cvxopt](http://cvxopt.org/) >= 1.2.5.

[](https://colab.research.google.com/github/vsiddhu/SDP-Quantum-OR/blob/master/Notebook%201%20-%20Quantum%20State%20Discrimination.ipynb)

## Introduction

The maximum probability of successfully distinguishing states $\sigma_i$, created with probability
$p_i$, is given by

\begin{align}
    \begin{aligned}
        \text{maximize} \; &  \sum_{i = 1}^{n-1} p_i \rm Tr(E_i \rho_i) + \rm p_n Tr(E_n \rho_n) & \\
        \text{subject to} \; & E_j \succeq 0,& \forall 1 \leq j \leq n-1, \\
        \text{and} \; & I - \sum_{i = 1}^{n-1} E_i \succeq 0. &
    \end{aligned}
\end{align}

where $p_n := 1 - \sum_{i = 1}^{n-1} p_i$ and $E_n := I - \sum_{i = 1}^{n-1} E_i$.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
# For Google Colab use commands installing packages
try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False

# Install PICOS and CVXOPT in Google Colab
if IN_COLAB:
    !pip install -q picos
    !pip install -q cvxopt
```

```python
import picos as pic
import cvxopt as cvx
```

```python
print('Solvers supported on this installation of PICOS:', pic.solvers.all_solvers().keys())
```

    Solvers supported on this installation of PICOS: dict_keys(['cplex', 'cvxopt', 'ecos', 'glpk', 'gurobi', 'mosek', 'mskfsn', 'scip', 'smcp'])

```python
print('Solvers available to PICOS on this machine :', pic.solvers.available_solvers())
```

    Solvers available to PICOS on this machine : ['cvxopt', 'mosek', 'mskfsn']

### Example 1
Consider a case where $n=2$, density operators $\sigma_i$ have the form
    \begin{equation}
    \sigma_1 =
    \begin{pmatrix}
    1 & 0 \\
    0 & 0
    \end{pmatrix},
    \quad
    \sigma_2 =
    \begin{pmatrix}
    0 & 0 \\
    0 & 1
    \end{pmatrix},
    \end{equation}
  and $p_1 = q, p_2 = 1-q$ and $0 \leq q \leq 1$. In this case the SDP mentioned at the top of the notebook takes the form

\begin{align}
    \begin{aligned}
        \text{maximize} \; &  q \rm Tr(E_1 \sigma_1) + (1-q) \rm Tr\big( (I - E_1) \sigma_2\big) & \\
        \text{subject to} \; & E_1 \succeq 0, &, \\
        \text{and} \; & I - E_1 \succeq 0. &
    \end{aligned}
\end{align}

We solve this semidefinite program using PICOS.

```python
#Example 1

sig1 = np.array([[1.,0],[0.,0.]])
sig2 = np.array([[0.,0],[0.,1.]])

qVal = np.random.rand()
```

```python

#Constants
#----------
Sgs1 = pic.Constant("sg1", sig1)
Sgs2 = pic.Constant("sg2", sig2)

q = pic.Constant("q", qVal)

#Identity matrix
shp = np.shape(sig1)
iMat = pic.Constant('I', np.eye(shp[0],dtype='complex'))

#Variables
#----------
eVars = pic.HermitianVariable("E1", shp)

prob1 = pic.Problem()

#Constraint
#----------
prob1.add_constraint(eVars >> 0)
prob1.add_constraint(iMat - eVars >> 0)

#Objective
#----------
obj = q*(Sgs1 | eVars) + (1-q)*(Sgs2 | iMat - eVars)

prob1.set_objective('max',obj)

```

```python
#User readable view of the problem being composed in PICOS
print(prob1)
```

    ----------------------------------------------
    Complex Semidefinite Program
      maximize q·⟨sg1, E1⟩ + (1 - q)·⟨sg2, I - E1⟩
      over
        2×2 hermitian variable E1
      subject to
        E1 ≽ 0
        I - E1 ≽ 0
    ----------------------------------------------

```python
#Solve the problem using a solver of our choice and display progress
prob1.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
#The solver claims to have found an optimal solution. As a bonus, the SDP solution also gives the POVM element E1
pvm1 = np.array(eVars.value)
```

```python
prb = prob1.value
print('Probability of discriminating the inputs is = ', prb)
#For an explaination of why success probability is 1 even though the input p is random, see the text below
```

    Probability of discriminating the inputs is =  0.9999999994373823

* The probability of successful discriminating two qauntum states $\sigma_1$ and $\sigma_2$ each occuring with probability $p_1$ and $p_2$ respectively, is given by
 $$p^* = \frac{1}{2}( 1 + || p_1 \sigma_1 - p_2\sigma_2 ||_1 ).$$
 The example above numerically demonstrates this fact in a simple case where $\sigma_1$ and $\sigma_2$ are $2 \times 2$ matrices. In this example, $p_1 = q$ and $p_2 = 1 -q$, where $q \in [0,1]$ is random. Density operators $\sigma_1$ and $\sigma_2$ are orthogonal. These orthogonal density operators have $l_1$ distance, $|| \sigma_1 - \sigma_2 ||_1 = 2$ and thus $p^* = 1$, independent of $q$. In the following example, we choose $p_1,p_2,\sigma_1$ and $\sigma_2$ randomly and confirm our numerical SDP result with the algebraic result above.

### Example 2
Consider a case where $n=2, d = 6$, density operators $\sigma_1$ and $\sigma_2$ are chosen randomly, $p_1 = q$,
$p_2 = 1-q$ and $q$ chosen randomly between 0 and 1. In this case the SDP mentioned at the top of the notebook takes the form

\begin{align}
    \begin{aligned}
        \text{maximize} \; &  q \rm Tr(E_1 \sigma_1) + (1-q) \rm Tr\big( (I - E_1) \sigma_2\big) & \\
        \text{subject to} \; & E_1 \succeq 0, &, \\
        \text{and} \; & I - E_1 \succeq 0. &
    \end{aligned}
\end{align}

```python
#Example 2
d = 6

mt1 = np.random.rand(d,d) + 1j*np.random.randn(d,d)
mt1 = np.dot(mt1,mt1.conj().T)
sig1 = mt1/np.trace(mt1)

mt2 = np.random.rand(d,d) + 1j*np.random.randn(d,d)
mt2 = np.dot(mt2,mt2.conj().T)
sig2 = mt2/np.trace(mt2)

qVal = np.random.rand()
```

```python

#Constants
#----------
Sgs1 = pic.Constant("sg1", sig1)
Sgs2 = pic.Constant("sg2", sig2)

q = pic.Constant("q", qVal)

#Identity matrix
shp = np.shape(sig1)
iMat = pic.Constant('I', np.eye(shp[0],dtype='complex'))

#Variables
#----------
eVars = pic.HermitianVariable("E1", shp)

prob2 = pic.Problem()

#Constraint
#----------
prob2.add_constraint(eVars >> 0)
prob2.add_constraint(iMat - eVars >> 0)

#Objective
#----------
obj = q*(Sgs1 | eVars) + (1-q)*(Sgs2 | iMat - eVars)

prob2.set_objective('max',obj.real)

```

```python
#User readable view of the problem being composed in PICOS
print(prob2)
```

    --------------------------------------------------
    Complex Semidefinite Program
      maximize Re(q·⟨sg1, E1⟩ + (1 - q)·⟨sg2, I - E1⟩)
      over
        6×6 hermitian variable E1
      subject to
        E1 ≽ 0
        I - E1 ≽ 0
    --------------------------------------------------

```python
#Solve the problem using a solver of our choice and display progress
prob2.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
#Helstrom result computes using numpy
delOpt = qVal*sig1 - (1-qVal)*sig2
# print(np.linalg.eigh(delOpt))
(u,s,vH) = np.linalg.svd(delOpt, hermitian=False)
pStarAlg = (np.sum(s) + 1)/2
```

```python
prb2 = prob2.value
print('Probability of discriminating the input')
print('Using SDP = ', prb2)
print('Helstrom result', pStarAlg)
print('Difference between these values', abs(prb2 - pStarAlg))
```

    Probability of discriminating the input
    Using SDP =  0.9129937076702335
    Helstrom result 0.9129937078308089
    Difference between these values 1.6057544183212258e-10

### Example 3
Consider a case where $n=3$, the density operators $\sigma_i$ have the form
    \begin{equation}
    \sigma_1 =
    \begin{pmatrix}
    1 & 0 \\
    0 & 0
    \end{pmatrix}
    \quad
    \sigma_2 =
    \frac{1}{2}\begin{pmatrix}
    1 & 1 \\
    1 & 1
    \end{pmatrix}
    \quad
    \sigma_3 =
    \frac{1}{2}\begin{pmatrix}
    1 & 0 \\
    0 & 1
    \end{pmatrix}
    \end{equation}
 $p_1 = \frac{q}{2}, p_2 = \frac{q}{2}, p_3 = 1-q$, and $0 \leq q \leq 1$. In this case the SDP mentioned at the top of the notebook takes the form
\begin{align}
    \begin{aligned}
        \text{maximize} \; &  \frac{q}{2} \rm Tr(E_1 \sigma_1)+ \frac{q}{2} Tr(E_2 \sigma_2) + (1-q) \rm Tr\big( (I -E_1 - E_2) \sigma_3\big) & \\
        \text{subject to} \; & E_1 \succeq 0, &, \\
         & E_1 \succeq 0. & \\
        \text{and} \; & I - E_1 - E_2 \succeq 0. &
    \end{aligned}
\end{align}

```python
#Example 3

sig1 = np.array([[1.,0.],[0.,0.]])
sig2 = np.array([[0.,0.],[0.,1.]])
sig3 = np.array([[1.,0.],[0.,1.]])/2

qVal = np.random.rand()
```

```python
#Constants
#----------
Sgs1 = pic.Constant("sg1", sig1)
Sgs2 = pic.Constant("sg2", sig2)
Sgs3 = pic.Constant("sg2", sig3)

q = pic.Constant("q", qVal)

#Identity matrix
shp = np.shape(sig1)
iMat = pic.Constant('I', np.eye(shp[0],dtype='complex'))

#Variables
#----------
eVars1 = pic.HermitianVariable("E1", shp)
eVars2 = pic.HermitianVariable("E2", shp)

prob3 = pic.Problem()

#Constraint
#----------
prob3.add_constraint(eVars1 >> 0)
prob3.add_constraint(eVars2 >> 0)
prob3.add_constraint(iMat - eVars1 -eVars2 >> 0)

#Objective
#----------
obj3 = (q/2)*(Sgs1 | eVars1) + (q/2)*(Sgs2 | eVars2) + (1-q)*(Sgs3 | iMat - eVars1 - eVars2)

prob3.set_objective('max',obj3)

```

```python
#User readable view of the problem being composed in PICOS
print(prob3)
```

    ---------------------------------------------------------------------
    Complex Semidefinite Program
      maximize q/2·⟨sg1, E1⟩ + q/2·⟨sg2, E2⟩ + (1 - q)·⟨sg2, I - E1 - E2⟩
      over
        2×2 hermitian variable E1, E2
      subject to
        E1 ≽ 0
        E2 ≽ 0
        I - E1 - E2 ≽ 0
    ---------------------------------------------------------------------

```python
#Let us solve the problem using a solver of our choice
prob3.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
#The solver claims to have found an optimal solution. As a bonus, the SDP solution also gives the POVM element E
pov1 = np.array(eVars1.value)
pov2 = np.array(eVars2.value)
```

Using the structure of the SDP being solved, one can show its maximum value to be $\max(q, 1-q)$. The numerics above match this algebraic solution

```python
prb3 = prob3.value
print('Probability of discriminating the input given by SDP = ', prb3)
print('Algebraic solution = ', max(qVal,1-qVal))

```

    Probability of discriminating the input given by SDP =  0.7133864729887238
    Algebraic solution =  0.7133864734236646

Notebook 1 - Quantum State Discrimination.ipynb


# Speed by specialisation

In the previous notebooks we discussed Julia's elaborate type system and how multiple dispatch allows functions to decide on the method to dispatch really from looking at all the arguments. Intuitively **more specialised code**, i.e. code that is allowed to exploit known structure, **is faster.** In this notebook we will explore how Julia is able to exploit this while retaining generic code ... thanks to multiple dispatch.

## The speed of Julia

Now let us first try to address the question *Is Julia fast?*

To some extend this is a bit of a mismatching question, since one is able to write slow code in any language ... so let's try do address something else instead: *Can Julia be fast?*

## A simple example: Sums

Let's compare for a simple example ...

```julia
function mysum(v)
    result = zero(eltype(v))
    for i in 1:length(v)
        result += v[i]
    end
    result
end
```

```julia
v = randn(10^7);   # Large vector of numbers
```

```julia
using BenchmarkTools
```

```julia
# Compile some C code and call it from julia ...
using Libdl
code = """
#include <stddef.h>
double c_sum(size_t n, double *v) {
    double accu = 0.0;
    for (size_t i = 0; i < n; ++i) {
        accu += v[i];
    }
    return accu;
}
"""

# Compile to a shared library (with fast maths and machine-specific)
const Clib = tempname()
open(`gcc -fPIC -O3 -march=native -xc -shared -ffast-math -o $(Clib * "." * Libdl.dlext) -`, "w") do f
    print(f, code)
end

# define a Julia function that calls the C function:
c_sum(v::Array{Float64}) = ccall(("c_sum", Clib), Float64, (Csize_t, Ptr{Float64}), length(v), v)
```

```julia
bench = @benchmark c_sum($v)
```

```julia
times = Dict()
times["C (naive)"] = minimum(bench.times) / 1e6
times
```

An explicitly and fully vectorised C++ code:

```julia
# Compile some C code and call it from julia ...
using Libdl
code = """
#include <stddef.h>
#include <immintrin.h>

double c_sum_vector(size_t n, double *v)
{
    size_t i;
    double result;
    double tmp[4] __attribute__ ((aligned(64)));

    __m256d sums1 = _mm256_setzero_pd();
    __m256d sums2 = _mm256_setzero_pd();
    for ( i = 0; i + 7 < n; i += 8 )
    {
        sums1 = _mm256_add_pd( sums1, _mm256_loadu_pd(v+i  ) );
        sums2 = _mm256_add_pd( sums2, _mm256_loadu_pd(v+i+4) );
    }
    _mm256_store_pd( tmp, _mm256_add_pd(sums1, sums2) );

    return tmp[0] + tmp[1] + tmp[2] + tmp[3];
}
"""

# Compile to a shared library (with fast maths and machine-specific)
const Clib = tempname()
open(`gcc -fPIC -O2 -march=native -xc -shared -ffast-math -o $(Clib * "." * Libdl.dlext) -`, "w") do f
    print(f, code)
end

# define a Julia function that calls the C function:
c_sum(v::Array{Float64}) = ccall(("c_sum_vector", Clib), Float64, (Csize_t, Ptr{Float64}), length(v), v)
```

```julia
bench = @benchmark c_sum($v)
```

```julia
times["C (vectorised)"] = minimum(bench.times) / 1e6
```

How does our version do?

```julia
bench = @benchmark mysum($v)
```

```julia
times["Julia (naive)"] = minimum(bench.times) / 1e6
```

But unlike C we have not yet tried all tricks!

```julia
function fastsum(v)
    result = zero(eltype(v))
    @simd for i in 1:length(v)    # @simd enforces vectorisation in the loop
        @inbounds result += v[i]  # @inbounds suppresses bound checks
    end
    result
end

# Still nicely readable code ...
```

```julia
bench = @benchmark fastsum($v)
```

```julia
times["Julia (simd)"] = minimum(bench.times) / 1e6
```

Compare with python ...

```julia
using PyCall
numpysum(v) = pyimport("numpy").sum(v)

bench = @benchmark numpysum($v)
```

```julia
times["Numpy"] = minimum(bench.times) / 1e6
```

```julia
py"""
def py_sum(A):
    s = 0.0
    for a in A:
        s += a
    return s
"""

pysum = py"py_sum"

bench = @benchmark pysum($v)
```

```julia
times["Python (naive)"] = minimum(bench.times) / 1e6
```

Overview ...

```julia
times
```

## A more complicated example: Vandermonde matrices
(modified from [Steven's Julia intro](https://web.mit.edu/18.06/www/Fall17/1806/julia/Julia-intro.pdf))

\begin{align}V=\begin{bmatrix}1&\alpha _{1}&\alpha _{1}^{2}&\dots &\alpha _{1}^{n-1}\\1&\alpha _{2}&\alpha _{2}^{2}&\dots &\alpha _{2}^{n-1}\\1&\alpha _{3}&\alpha _{3}^{2}&\dots &\alpha _{3}^{n-1}\\\vdots &\vdots &\vdots &\ddots &\vdots \\1&\alpha _{m}&\alpha _{m}^{2}&\dots &\alpha _{m}^{n-1}\end{bmatrix}\end{align}

```julia
using PyCall
np = pyimport("numpy")
```

```julia
np.vander(1:5, increasing=true)
```

[The source code for this function](https://github.com/numpy/numpy/blob/v1.16.1/numpy/lib/twodim_base.py#L475-L563) calls `np.multiply.accumulate` [which is implemented in C](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/ufunc_object.c#L3678). However, this code doesn't actually perform the computation, it basically only checks types and stuff. The actual kernel is [implemented here](https://github.com/numpy/numpy/blob/deea4983aedfa96905bbaee64e3d1de84144303f/numpy/core/src/umath/loops.c.src#L1742), which is not even C code, but only a template that gets transformed to type-specific kernels. Still only a limited set of types `Float64`, `Float32`, and so forth are supported.

Here is a type-generic Julia implementation:

```julia
function vander(x::AbstractVector{T}) where T
    m = length(x)
    V = Matrix{T}(undef, m, m)
    for j = 1:m
        V[j,1] = one(x[j])
    end
    for i= 2:m
        for j = 1:m
            V[j,i] = x[j] * V[j,i-1]
            end
        end
    return V
end
```

```julia
vander(1:5)
```

### Quick speed comparison

```julia
using BenchmarkTools
using Plots
ns = exp10.(range(1, 4, length=30));

tnp = Float64[]
tjl = Float64[]
for n in ns
    x = collect(1:n)
    push!(tnp, @belapsed np.vander($x) samples=3 evals=1)
    push!(tjl, @belapsed vander($x)    samples=3 evals=1)
end
plot(ns, tnp./tjl, m=:circle, xscale=:log10, yscale=:log10, ylims=[1, Inf],
     xlab="matrix size", ylab="NumPy time / Julia time", legend=:false)
```

Notably the clean and concise Julia implementation is **faster than numpy**'s C implementation for small matrices and **as fast** for large matrix sizes. Still it works for **arbitrary types**.

```julia
vander(Int32[4, 8, 16, 32])
```

This includes non-numerical types ... the only assumption is that the type induces a multiplicative group, i.e. has a `one` function to yield the identity element and an apropriate `*` defined.

A rather unusual one is `String`, which works since `one(String) == ""`.

```julia
vander(["this", "is", "a", "test"])
```

## How does this work?

- AST = Abstract Syntax Tree
- SSA = Static Single Assignment
- [LLVM](https://de.wikipedia.org/wiki/LLVM) = Low Level Virtual Machine

Julia compiles **ahead of time**, i.e. Julia waits until the first function call for a particular set of input types is issued and then compiles the full call stack all the way down to machine code specialising as much as possible along the way.

```julia
x = [1, 3, 5]  # Vector{Int64}

@time mysum(x)
@time mysum(x)
```

As we saw if thes change, Julia compiles a new specialization of the function!

To see what happens under the hood, we have a bunch of inspection macros:
* The AST after parsing (**`@macroexpand`**)
* The AST after lowering (**`@code_typed`**, **`@code_warntype`**)
* The AST after type inference and optimization (**`@code_lowered`**)
* The LLVM IR (**`@code_llvm`**)
* The assembly machine code (**`@code_native`**)

```julia
@code_typed mysum(1)
```

```julia
@code_lowered mysum(1)
```

```julia
@code_llvm mysum(1)
```

We can remove the comments (lines starting with `;` using `debuginfo=:none`).

```julia
@code_llvm debuginfo=:none mysum(1.2)
```

```julia
@code_native debuginfo=:none mysum(1.2)
```

Let's compare this to `Float64` input.

```julia
@code_native debuginfo=:none mysum(1)
```

## Types and code specialisation

As a rule of thumb: The Julia compiler only uses the types to specialise, not the values. Therefore using special types that match the possible assumptions most closely is crucial.

As an example we consider the determinant of a 2x2 matrix.

```julia
# Fast custom implementation ... note: No type annotation
det_custom(M) = @inbounds (M[1, 1] * M[2, 2] - M[1, 2] * M[2, 1])
```

```julia
M = randn(2, 2)
@btime det_custom(M);
```

```julia
using LinearAlgebra
@btime det(M);
```

Now that is a drastic difference ... but we actually did not tell the compiler anything about the size of `M`:

```julia
typeof(M)
```

Now let's help along ... by using a static array type:

```julia
using StaticArrays

S = @SMatrix randn(2, 2)
```

```julia
@btime det(S);
```

Got it! Now ... what happens under the hood:

```julia
@code_typed debuginfo=:none det(S)
```

in other words
```
%2  = S[1, 1]
%4  = S[2, 2]
%5  = S[1, 1] * S[2, 2]
%7  = S[2, 1]
%9  = S[1, 2]
%10 = S[2, 1] * S[1, 2]
%11 = S[1, 1] * S[2, 2] - S[2, 1] * S[1, 2]
```
exactly what we had hand-optimised before!

Now one should add here that part of the magic has not been the compiler being smart, but much rather the `StaticArrays` package, which has a number of carfully optimised implementations for standard operations:

```
============================================
    Benchmarks for 3×3 Float64 matrices
============================================
Matrix multiplication               -> 5.9x speedup
Matrix multiplication (mutating)    -> 1.8x speedup
Matrix addition                     -> 33.1x speedup
Matrix addition (mutating)          -> 2.5x speedup
Matrix determinant                  -> 112.9x speedup
Matrix inverse                      -> 67.8x speedup
Matrix symmetric eigendecomposition -> 25.0x speedup
Matrix Cholesky decomposition       -> 8.8x speedup
Matrix LU decomposition             -> 6.1x speedup
Matrix QR decomposition             -> 65.0x speedup
```

##### More details
- https://github.com/JuliaCI/BenchmarkTools.jl
- https://github.com/JuliaArrays/StaticArrays.jl/
- https://docs.julialang.org/en/v1/devdocs/compiler/

## Takeaways

- Julia can be fast
- Code is compiled down to the metal
- Stages can be inspected using macros like `@code_warntype`
- Typing is crucial to exploit specialised structure
- Type annotations are irrelevant for performance


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import solve_triangular
import warnings
from functools import partial
```

## Finite differences

Consider the following one-dimensional PDE:
$$
-u_{xx}(x) = f(x)\quad\mathrm{ in }\ \Omega = (0, \pi)
$$
$$
u(x) = 0, \quad\mathrm{ on }\ \partial\Omega = \{0, \pi\}
$$

Given the following $4^{th}$ order finite difference approximation of the second order derivative:

$$u_{xx}(x_i) = \frac{-u_{i-2}+16u_{i-1}-30u_i+16u_{i+1}-u_{i+2}}{12h^2}$$

Implement a function that given the domain interval, the forcing function, the number of discretization points, the boundary conditions, returns the matrix $A$ and the the right hand side $b$.

```python
def finDif(omega,f,n,bc,second_order=False):
    """
    Compute the Finite Difference matrix for the differential
    equation :math:`-u'' = f` in the given domain `omega`, using
    a grid with `n` elements and the given boundary conditions
    `bc`.

    :return: The matrix for the FD problem and the RHS vector.
    """
    non_null_component = np.array([-1, 16, -30, 16, -1])
    f = np.vectorize(f)

    h = (omega[1] - omega[0]) / n

    A = np.zeros((n,n))
    rhs = f(np.linspace(*omega, n))

    if not second_order:
        # fourth order
        for i in range(2,n-2):
            A[i,i-2:i+3] = np.array(non_null_component) / 12
        # second order if the fourth order is not applicable
        A[1,0:3] = A[-2,-3:] = np.array([1,-2,1])
    else:
        for i in range(1,n-1):
            A[i,i-1:i+2] = np.array([1,-2,1])

    A /= h*h

    # boundary conditions
    A[0,0] = A[-1,-1] = 1
    rhs[0] = bc[0]
    rhs[-1] = bc[1]

    return A, rhs
```

Call the function using:

```python
omega = [0,np.pi]
f = lambda x : np.sin(x)
n=100
bc = [0,0]
A, rhs = finDif(omega, f, n, bc)
```

## LU and Cholesky factorization

Implement two functions that compute the LU and the Cholesky factorization of the system matrix $A$

```python
# row-partial pivoting
def gauss_factorization(A, row_pivoting=True):
    """
    Operate Gauss factorization on the matrix `A` (with optional
    row-pivoting depending on the value of the parameter
    `row_pivoting`).

    :return: The upper triangular matrix obtained at the end of
                the algorithm, the lower triangular Gauss coefficient
                matrix, and (if `row_pivoting` is `True`) the
                permutation matrix.
    """
    U = np.array(A)
    gauss_coeff_matrix = np.zeros_like(U)
    exchanged_rows = []

    for i in range(U.shape[0]-1):
        if row_pivoting:
            # we look for the biggest element in the column i. we need
            # to add i to row_max since the index given by argmax is
            # relative to the submatrix A[i:,i].
            row_max = i + np.argmax(np.abs(U[i:,i]))
            exchanged_rows.append((i, row_max))
            U[[i,row_max]] = U[[row_max,i]]
            gauss_coeff_matrix[[i,row_max]] = gauss_coeff_matrix[[row_max,i]]

        gauss_coeff_matrix[i+1:,i] = U[i+1:,i] / U[i,i]
        U[i+1:] -= np.multiply(gauss_coeff_matrix[i+1:,i][:,None], np.repeat(U[i][None], U.shape[0] - (i+1), axis=0))

    P = np.identity(A.shape[0])
    for a,b in exchanged_rows[::-1]:
        Pi = np.identity(U.shape[0])
        Pi[[a,b]] = Pi[[b,a]]
        P = P.dot(Pi)

    if row_pivoting:
        return U, gauss_coeff_matrix, P
    else:
        return U, gauss_coeff_matrix

def LU(A, row_pivoting=False):
    tp = gauss_factorization(A, row_pivoting)
    return (tp[1] + np.diag(np.ones(A.shape[0])), tp[0], *tp[2:])

# test gaussian elimination
temp_matrix = np.array([[4,2,3],[2,4,5],[7,8,9]])
np.testing.assert_almost_equal(np.dot(*LU(temp_matrix)), temp_matrix)

# test gaussian elimination with pivoting
L,U,P = LU(temp_matrix, row_pivoting=True)
np.testing.assert_almost_equal(L.dot(U), P.dot(temp_matrix))
```

```python
def cholesky(A):
    """
    Attemp Cholesky factorization on the matrix `A`. If the output
    contains `np.nan`, the factorization failed. No checks are
    performed.
    """
    warnings.filterwarnings('ignore')

    H = np.zeros_like(A)
    H[0,0] = np.sqrt(A[0,0])
    for j in range(1, A.shape[0]):
        for i in range(j):
            H[i,j] = (A[i,j] - H[:i,i].dot(H[:i,j])) / H[i,i]
        H[j,j] = np.sqrt(A[j,j] - np.einsum('i,i', H[:j,j], H[:j,j]))

    warnings.resetwarnings()
    return H.T, H

# test cholesky factorization on a positive definite matrix
temp_matrix = np.array([[2,-1,0],[-1,2,-1],[0,-1,2]], dtype=float)
np.testing.assert_almost_equal(np.dot(*cholesky(temp_matrix)), temp_matrix)
```

```python
def triangular_factorization(A):
    """
    Attempt a triangular factorization of the given matrix using
    Cholesky factorization, fallbacks to LU factorization with
    Gauss elimination method.
    """
    L,U = cholesky(A)
    if np.count_nonzero(np.isnan(L)) > 0:
        L,U = LU(A)
    return L,U

def check_triangular(T, tolerance=1.e-14, verbose=False):
    """
    Check that the given matrix is triangular (elements whose module is
    smaller than `tolerance` are assumed to be zero).

    ..warning

        Diagonal matrices are recognized as upper triangular matrices.

    :return: `'upper'` if `T` is upper triangular, `'lower'` if it is
                lower triangular, `None` otherwise.
    """
    if T.shape[0] != T.shape[1]:
        return False
    if T.shape[0] == 1:
        return True

    T = np.abs(T)

    upper = lower = True
    for i in range(T.shape[0]):
        uvec = T[i+1:,i]
        if verbose:
            print(i,uvec)
        if upper and np.count_nonzero(uvec > tolerance) != 0:
            upper = False
            if verbose:
                print('non upper (i={}): {}'.format(i,uvec))

        lvec = T[i,i+1:]
        if lower and np.count_nonzero(lvec > tolerance) != 0:
            lower = False
            if verbose:
                print('non lower (i={}): {}'.format(i,lvec))

    if upper:
        return 'upper'
    elif lower:
        return 'lower'
    return None

# test factorization on a non-(positive definite) matrix
temp_matrix = np.array([[-2,1,0],[-1,2,-1],[0,-1,2]], dtype=float)
np.testing.assert_almost_equal(np.dot(*triangular_factorization(temp_matrix)), temp_matrix)

# test check triangular
assert check_triangular(np.array([[1,1,1],[0,1,1],[0,0,0]])) == 'upper'
assert check_triangular(np.array([[1,0,0],[-1,1,0],[1,1,0]])) == 'lower'
assert check_triangular(np.array([[1,1,1],[0,1,1],[1,0,0]])) == None
```

## Direct methods for linear systems

Implement forward and backward substitution functions to exploit the developed factorization methods to solve the derived linear system of equations.

```python
def solve_triangular_system(M,rhs,lower):
    """
    Substitution method for triangular matrices (upper or lower depending
    on the parameter `lower`).
    """
    x = np.zeros_like(M[0])

    rng = range(M.shape[0]) if lower else range(M.shape[0]-1,-1,-1)
    for i in rng:
        if lower:
            mi = M[i,:i]
            xi = x[:i]
        else:
            mi = M[i,i+1:]
            xi = x[i+1:]
        x[i] = (rhs[i] - np.dot(xi, mi)) / M[i,i]

    np.testing.assert_almost_equal(x, solve_triangular(M,rhs,lower=lower))
    return x
```

```python
L_solve = partial(solve_triangular_system, lower=True)
U_solve = partial(solve_triangular_system, lower=False)
```

```python
def solve_direct(A,rhs,*,assertions=True,L=None,U=None,P=None):
    """
    Solve the given linear system using a direct
    method (i.e. LU or Cholesky).
    """
    if L is None or U is None:
        L,U = triangular_factorization(A)
    if P is None:
        P = np.identity(len(A))

    if assertions:
        np.testing.assert_almost_equal(P.dot(A), L.dot(U))
        assert check_triangular(L, tolerance=5.e-10) == 'lower'
        assert check_triangular(U, tolerance=5.e-10) == 'upper'

    y = L_solve(L,P.dot(rhs))
    if assertions:
        np.testing.assert_almost_equal(L.dot(y), P.dot(rhs))

    x = U_solve(U,y)
    if assertions:
        np.testing.assert_almost_equal(U.dot(x), y)

    return x
```

## Solution of the FD problem

Solve the derived linear system using the implemented functions and plot the computed solution:

```python
def solve_fin_diff_problem(omega, f, n, bc, solver=solve_direct, second_order=False):
    """
    Return the function `x` which satisfies the ODE
    :math:`-x'' = f`, and the discrete grid used to
    find the solution.

    By default the function uses a direct solver, but different
    solvers may be used by changing the parameter `solver`.

    :return: A tuple whose first item is the discrete grid used
                to compute the solution, the second item is the
                solution computed on the grid.
    """
    A, rhs = finDif(omega, f, n, bc, second_order=second_order)
    x = -solver(A, rhs)
    np.testing.assert_almost_equal(-A.dot(x), rhs)
    return np.linspace(*omega, n), x
```

```python
plt.xlabel('$\\Omega$')
plt.xticks(omega, ['0','$\\pi$'])
plt.plot(*solve_fin_diff_problem(omega, f, n, bc), label='IV order')
plt.plot(*solve_fin_diff_problem(omega, f, n, bc, second_order=True), label='II order')
plt.legend()
plt.show()
```

## Error plot

Considering the new domain $\Omega = (0,1)$ and the forcing term $f(x) = x(1-x)$ with B.C. $u(x) = 0$, on $\partial \Omega = {0,1}$ produce a plot and a table where you show the decay of the error w.r.t. the number of grid points.
(The analytical solution for the above problems is $u_{an} = \frac{x^4}{12} - \frac{x^3}{6} + \frac{x}{12}$)

```python
omega2 = [0,1]
f2 = lambda x : x*(1-x)
bc2 = [0,0]
ns = np.arange(10,1011,100)

exact = lambda x: np.power(x,4) / 12 - np.power(x,3) / 6 + x / 12
```

```python
def compute_error(n, error_function):
    xs, x = solve_fin_diff_problem(omega2, f2, n, bc2)
    return error_function(np.abs(exact(xs) - x))

error_infty_norm = partial(compute_error, error_function=np.max)
error_mean = partial(compute_error, error_function=np.mean)
err_funcs = [error_infty_norm, error_mean]

labels = ['$\\Vert \\cdot \\Vert_\\infty$', 'mean']

plt.title('Errors')
for err_func, label in zip(err_funcs, labels):
    err = list(map(lambda n: err_func(n), ns))
    plt.plot(ns, err, label=label)
plt.legend()

plt.xlabel('n')
plt.ylabel('Error')

plt.yscale('log')

plt.show()
```

## Inverse

Compute the inverse of the matrix A exploiting the derived LU factorization

```python
def inverse(A):
    """
    Compute the inverse matrix of `A` via triangular factorization.
    """
    L,U = triangular_factorization(A)
    A_inv = np.zeros_like(A)
    identity = np.identity(A.shape[0])

    for j in range(A.shape[0]):
        A_inv[:,j] = solve_direct(A, identity[:,j])
    return A_inv

# a quick test
plt.pcolormesh(inverse(A).dot(A));
```

## Condition number

Exploit the derived LU factorizations to compute the condition number of the system's matrix $A$ using the original problem formulation.

```python
def infty_norm(A):
    """
    Returns :math:`\\Vert A \\Vert_\\infty`.
    """
    return np.max(np.einsum('ij->i', np.abs(A)))

# infty norm
def condNumb(A):
    """
    Compute the condition number of `A` in the
    :math:`\\Vert \\cdot \\Vert_\\infty` norm."""
    return infty_norm(A) * infty_norm(inverse(A))

condNumb(A)
```

    11937.129794259363

## Preconditioned conjugate gradient method

Implement a preconditioned Conjugant Gradient method to solve the original linear system of equations using an iterative method:

```python
def conjugate_gradient(A, b, P=None, eps=1e-10):
    """
    Preconditioned gradient method with tolerance `eps` and
    preconditioner `P`. If `P` is `None`, the preconditioner
    is the diagonal matrix of `A`.
    """
    if P is None:
        P = np.identity(A.shape[0])

    x = np.zeros_like(A[0])
    r = b - A.dot(x)
    z = np.linalg.solve(P, r)
    p = z

    for i in range(len(A)):
        if np.linalg.norm(r) < eps:
            break

        a = p.dot(r) / A.dot(p).dot(p)
        x += a*p
        r -= a*A.dot(p)
        z = np.linalg.solve(P, r)
        beta = A.dot(p).dot(z) / A.dot(p).dot(p)
        p = z - beta*p

    if np.linalg.norm(A.dot(x) - b) > eps:
        warnings.warn("The given matrix may not be symmetric "
                      "positive definite, PCG did not converge.")

    return x
```

```python
xs, sol_conj = solve_fin_diff_problem(omega, f, 100, bc, solver=conjugate_gradient, second_order=True)
_, sol_direct = solve_fin_diff_problem(omega, f, 100, bc)

np.linalg.norm(sol_conj - sol_direct) / np.linalg.norm(sol_direct)
```

    8.390564146200464e-05

```python
plt.figure(figsize=(20,4))

plt.subplot(1,3,1)
plt.xlabel('$\\Omega$')
plt.plot(xs, sol_direct)
plt.xticks(omega, ['0','$\\pi$'])
plt.title('Direct')

plt.subplot(1,3,2)
plt.xlabel('$\\Omega$')
plt.plot(xs, sol_conj)
plt.xticks(omega, ['0','$\\pi$'])
plt.title('PCG')

plt.subplot(1,3,3)
plt.xlabel('$\\Omega$')
plt.plot(xs, np.abs(sol_direct - sol_conj))
plt.xticks(omega, ['0','$\\pi$'])
plt.title('Difference between PCG and direct solution')

plt.show()
```

## Forward euler

Consider the following time dependent variation of the PDE starting from the orginal problem formulation:
$$
u'(t)-u_{xx} = \alpha(t)f(x)
$$

for $t\in [0,T]$, with $\alpha(t) = \cos(t)$ and $T = 6\pi$

Use the same finite difference scheme to derive the semi-discrete formulation and solve it using a forward Euler's method.

Plot the time dependent solution solution at $x = \pi/2$, $x=1$,
$x=\pi$

### Forward Euler method
We discretize the time using the Forward Euler approximation for the first derivative with respect to $t$:
$$u_{xx} + \alpha(t)f(x) = \frac{\partial u}{\partial t}(x,t) \approx \frac{\partial u}{\partial t}(x,t_n) \approx \frac{u(x,t_{n+1}) - u(x,t_n)}{h_{\text{t}}}$$
$$\implies u(x,t_{n+1}) = u(x,t_n) + h_{\text{t}} u_{xx}(x,t_n) + h_{\text{t}} \alpha(t_n)f(x)$$

We discretize also the space:
$$u_{xx}(x,t) \approx u_{xx}(x_i,t) = \frac{-u(x_{i-2},t)+16u(x_{i-1},t)-30u(x_i,t)+16u(x_{i+1},t)-u(x_{i+2},t)}{12h_x^2}$$

And we obtain the following numerical scheme:
$$\begin{align} u(x_m,t_{n+1}) &= u(x_m,t_n) + h_{\text{t}} \frac{-u(x_{m-2},t_n)+16u(x_{m-1},t_n)-30u(x_m,t_n)+16u(x_{m+1},t_n)-u(x_{m+2},t_n)}{12h_x^2} + h_{\text{t}} \alpha(t_n)f(x_m)\\
&= h_{\text{t}} \frac{-u(x_{m-2},t_n)+16u(x_{m-1},t_n)-(30 - 12h_x^2/h_t)u(x_m,t_n)+16u(x_{m+1},t_n)-u(x_{m+2},t_n)}{12h_x^2} + h_{\text{t}} \alpha(t_n)f(x_m)\\
&= h_{\text{t}} \left(\frac{-u(x_{m-2},t_n)+16u(x_{m-1},t_n)-(30 - 12h_x^2/h_t)u(x_m,t_n)+16u(x_{m+1},t_n)-u(x_{m+2},t_n)}{12h_x^2} + \alpha(t_n)f(x_m)\right)
\end{align}$$

We can now write the problem in the following form for some matrix $A \in \mathbb{R}^{N_x \times N_x}$:
$$\begin{bmatrix} u(x_0)\\ \vdots \\ u(x_{N_x})\end{bmatrix}(t_{n+1}) = A \begin{bmatrix} u(x_0)\\ \vdots \\ u(x_{N_x})\end{bmatrix}(t_n) + \begin{bmatrix} f(x_0)\\ \vdots \\ f(x_{N_x})\end{bmatrix} h_t \alpha(t_n)$$
where $N_x$ is an integer such that $N_x h_x$ is equal to the right boundary of the space domain.

```python
def forward_euler_A(ht, hx, Nx):
    A = np.zeros((Nx,Nx), dtype=float)

    non_null_component = np.array([-1, 16, -30+12*hx**2/ht, 16, -1])

    # fourth order
    for i in range(2,Nx-2):
        A[i,i-2:i+3] = np.array(non_null_component) / 12
    # second order if the fourth order is not applicable
    A[1,0:3] = A[-2,-3:] = np.array([1,-2+hx**2/ht,1])

    A /= hx**2
    A *= ht

    return A

def forward_euler(u0, f, alpha, omega_x, omega_t, Nx, Nt):
    if not isinstance(u0, np.ndarray):
        u0 = np.repeat(u0, Nx)

    ht = (omega_t[1] - omega_t[0]) / Nt
    hx = (omega_x[1] - omega_x[0]) / Nx

    xs = np.linspace(*omega_x, Nx)
    ts = np.linspace(*omega_t, Nt)

    A = forward_euler_A(ht, hx, Nx)
    f = np.vectorize(f)(xs)

    u = np.zeros((Nx, Nt), dtype=float)
    u[:,0] = u0

    # boundary conditions
    u[[0,-1],0] = 0
    f[[0,-1]] = 0

    for i in range(1,Nt):
        u[:,i] = A.dot(u[:,i-1]) + ht * alpha(i*ht) * f

    return xs,ts,u
```

```python
def x_idx(xs, x):
    """
    Find the best value in xs to approximate the scalar x.
    """
    return np.argmin(np.abs(xs - x))

def show_solution(Nx, Nt, *, xs=None):
    xs, ts, u = forward_euler(0, np.sin, np.cos, (0,np.pi), (0,6*np.pi), Nx=Nx, Nt=Nt)

    pi_2_idx = x_idx(xs, np.pi/2)
    one_idx = x_idx(xs, 1)
    pi_idx = -1

    my_pi2 = np.round(xs[pi_2_idx],4)
    my_one = np.round(xs[one_idx],4)
    my_pi = np.round(xs[pi_idx], 4)

    time_slice = slice(None,None,100)

    plt.plot(ts[time_slice], u[pi_2_idx,time_slice], '-', label='x={}'.format(my_pi2))
    plt.plot(ts[time_slice], u[one_idx,time_slice], '-.', label='x={}'.format(my_one))
    plt.plot(ts[time_slice], u[pi_idx,time_slice], '--', label='x={}'.format(my_pi))

    plt.xticks([i*np.pi for i in range(7)], ['0', *('{}$\\pi$'.format(i) for i in range(1,7))])

    plt.xlabel('t')
    plt.ylabel('u(x,t)')

    plt.title('Time dependent solution (Nx={}, Nt={})'.format(Nx,Nt))
    plt.legend()

plt.figure(figsize=(20,4))

plt.subplot(1,3,1)
show_solution(50,10000)
plt.subplot(1,3,2)
show_solution(50,100000)
plt.subplot(1,3,3)
show_solution(100,100000)
plt.show()
```

## Eigenvalues and eigenvectors

Given the original $Au = b$ system, implement an algorithm to compute the eigenvalues and eigenvectors of the matrix $A$. Exploit the computed LU factorization

```python
def power_method(A, max_iter=1000, max_residual=1.e-10, inverse=False, verbose=False):
    x = np.ones_like(A[0])

    if inverse:
        L,U,P = LU(A, row_pivoting=True)
        if np.count_nonzero(np.isnan(L)) > 0 or np.count_nonzero(np.isnan(U)) > 0:
            return None,None
        np.testing.assert_almost_equal(L.dot(U), P.dot(A))

    for i in range(max_iter):
        y = x / np.linalg.norm(x)

        # update x
        if inverse:
            x = solve_direct(A, y, assertions=False, L=L, U=U, P=P)
        else:
            x = np.dot(A,y)

        # rayleigh equation
        eig = complex(y.dot(x))

        if np.linalg.norm(A.dot(y) - y * eig) < max_residual:
            break
    if verbose:
        print('Stopped after {} iterations'.format(i+1))

    return 1/eig if inverse else eig, y

def compute_some_eigenvalues(A, equal_tolerance=1.e-10):
    eigs = []
    eigenvalues = []

    # we look for the eigenvalues near the center of the gershgorin circles (i.e.
    # diagonal elements of A).
    for diag_element in [*np.diag(A)]:
        eig, vec = power_method(A - diag_element * np.identity(len(A)), inverse=True)
        if eig != None:
            eig += eig + diag_element
            eigs.append(eig)
            eigenvalues.append(vec)

    # add the smallest eigenvalue of A
    eig, vec = power_method(A, inverse=True)
    if eig != None:
        eigs.append(eig)
        eigenvalues.append(vec)
    # and the biggest
    eig, vec = power_method(A)
    eigs.append(eig)
    eigenvalues.append(vec)

    srt_idx = np.argsort(np.abs(eigs))
    eigs = np.array(eigs)[srt_idx]
    eigenvalues = np.array(eigenvalues)[srt_idx]

    # indexes of unique eigenvalues, since we may find the same eigenvalue
    # multiple times
    non_repeated_eigs_idxes = [0]
    for i in range(1,len(eigs)):
        if np.abs(eigs[non_repeated_eigs_idxes[-1]] - eigs[i]) > equal_tolerance:
            non_repeated_eigs_idxes.append(i)

    eigs = eigs[non_repeated_eigs_idxes]
    eigenvalues = eigenvalues[non_repeated_eigs_idxes]

    if len(eigs) > A.shape[0]:
        warnings.warn("The number of unique eigenvalues found is greater than the dimension "
                      "of the matrix, there are probably some spurious eigenvalues that "
                      "should be discarded.")
    return eigs, eigenvalues
```

```python
A, _ = finDif(omega, f, n, bc)
eigs, eigenvecs = compute_some_eigenvalues(A)
```

    /u/f/fandreuz/.local/lib/python3.6/site-packages/ipykernel_launcher.py:27: RuntimeWarning: invalid value encountered in true_divide

```python
true_eigs = np.linalg.eig(A)[0]
for eig, eigenvec in zip(eigs,eigenvecs):
    idx = np.argmin(np.abs(true_eigs - eig))
    print('Found {}, relative error={}, residual={}'.format(eig,
                                                            np.abs(true_eigs - eig)[idx] / np.abs(true_eigs[idx]),
                                                            np.linalg.norm(A.dot(eigenvec) - eig * eigenvec)))
```

    Found (0.9999999980238907+0j), relative error=1.9761092762138333e-09, residual=4.412018730890187e-09
    Found (-2083.933510491916+0j), relative error=0.01399144528808244, residual=28.75491882258004
    Found (-2589.5416917742764+0j), relative error=0.011031979370446372, residual=28.25605035791613
    Found (-5396.812957782051+0j), relative error=6.997473027363431e-06, residual=0.8857371160328106

```python
# a more refined approach may use this function to locate the area
# of the complex plane where the eigenvalues are located
def gershgorin_radius(A, row=True):
    # we ignore diagonal entries
    A = np.array(A) - np.diag(np.diag(A))
    return np.einsum('ij->{}'.format('i' if row else 'j'), np.abs(A))
```

## Cauchy problem with Backward Euler method

Consider the following Cauchy problem
$$
\begin{cases}
y'= -ty^2 \quad 0\le t \le 2\\
y(0) = 1
\end{cases}
$$
Implement a Backward Euler's method in a suitable function and solve the resulting non-linear equation using a Newton's method.

### Backward Euler method:
$$-ty_n^2 = y' \approx \frac{y_{n} - y_{n-1}}{h}$$
$$\implies thy_n^2 + y_n - y_{n-1} =: \phi(y_n) = 0$$
$$\phi'(x) = 2th + 1$$
### Newton:
$$\begin{align}x_{n+1} &= x_n - \frac{\phi'(x_{n})}{\phi(x_{n})}\\
&= x_n - \frac{2th + 1}{thx_n^2 + x_n - y_{n-1}}
\end{align}$$

```python
def newton(y, y_prev, n, h, phi, phi_prime, iterations=0, nmax=100, tolerance=1.e-12):
    """
    Apply Netwon method to the problem above to compute the n-th
    value of the solution y in the discretized time grid
    (i.e. y_n).

    y_prev is the value of the function computed in the time instant
    n-1 (i.e. y_{n-1}).
    """

    t = n*h
    y -= phi(y,y_prev,t) / phi_prime(t)

    if phi(y,y_prev,t) < tolerance or iterations >= nmax:
        return y
    else:
        return newton(y, y_prev, n, h, phi, phi_prime, iterations+1, nmax, tolerance)

def backward_euler(omega,n,bc):
    h = (omega[1] - omega[0]) / n

    def phi(x, y, t):
        return t*h*x**2 + x - y

    def phi_prime(t):
        return 2*t*h + 1

    y = np.zeros(n, dtype=float)
    y[0] = bc
    for i in range(1,n):
        y[i] = newton(y[i-1], y[i-1], i, h, phi, phi_prime)
    return y

def exact(x):
    return 1/(x**2/2+1)

n = 50
omega3 = (0,2)
plt.plot(np.linspace(*omega3,n), backward_euler(omega3, n, 1), '.')
plt.plot(np.linspace(*omega3,n), exact(np.linspace(*omega3,n)));
```


# Moguls of Chaos

This example was taken from the book **The Essence of Chaos** by *Edward Lorenz*. The aim of the model he presented in that chapter is to show how we can construct parsimonious models from real case phenomena which exhibit chaos.

The model consists of a board that goes down a regularly-bumpy slope or moguls without any control of the direction whatsoever. The dynamical system of this board is described by the system of ODE shown bellow

$$\frac{dx}{dt} = U, \quad \frac{dx}{dt} = V, \quad \frac{dz}{dt} = W.$$

\begin{align}
\frac{du}{dt} &= -F \partial_x H - cu, \\
\frac{dv}{dt} &= -F \partial_y H - cv, \\
\frac{dw}{dt} &= -g + F - cw.
\end{align}

$\vec{x} = (x,y,z)$ are the spacial coordinates and $\vec{u} = (u,v,w)$ are the components of the velocity of the board. And $H$ is the shape pf the slope that is parameterized by the following expression:

$$H(x,y) = -ax - b \ cos(px) \ cos(qy).$$

$a$ is the angle of the slope, $b$ is the height of the bumps, and $q$ and $p$ are spacial frequencies.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

We can plot the shape of the moguls to help us understand better the problem. For this we use the parameters that Lorenz suggest in his book, in which $a$ is the angle of the slope, $b$ is the depth of the moguls, and $p$ and $q$ are the spatial frequencies of the moguls in the $x$ and $y$ directions.

```python
a = 0.25
b = 0.4
q = (2*np.pi)/4.0
p = (2*np.pi)/10.0

def H_func(x,y):
    """
        H_func(x,y)
    Function that parametrizes the moguls.
    It takes a given x and y position as arguments, and returns
    the height of the slope.
    """
    return -a*x - b*np.cos(p*x)*np.cos(q*y)

def pits_n_crests(x,y):
    """
        pits_n_crests(x,y)

    Function that parametrizes the moguls without the slope.
    It takes a given x and y position as arguments, and returns
    the heights of the moguls as if they were in a flat surface.
    """
    return - b*np.cos(p*x)*np.cos(q*y)
```

```python
x_range = np.linspace(0, 20, 200)
y_range = np.linspace(-10, 10, 200)
XX, YY = np.meshgrid(x_range, y_range)
moguls = H_func(XX, YY)
```

```python
from mpl_toolkits.mplot3d import Axes3D
```

```python
fig = plt.figure(figsize=(10,6))
ax = fig.gca(projection='3d')

ax.set_ylabel('y (m)', fontsize = 15)
ax.set_xlabel('x (m)', fontsize = 15)
ax.set_zlabel('z (m)', fontsize = 15)
ax.set_title('Shape of the Moguls', fontsize = 15)

ax.plot_surface(XX, YY, moguls, cmap = 'plasma')
plt.savefig('fig/shape_of_moguls')
```

**Quite cool so far!** I ❤ matplotlib.

# Boards down the slope!

Our first goal is to solve the ODE system for a given initial condition using a numerical integration method. For this, we declare a function containing the right-hand side terms of the ODE system that we want to solve.

```python
def board(t, X_0):
    """
        board(t, X_0)

    Right-hand-side of the equations for a board going down a slope with moguls.

    X_0 is the the set of initial conditions containing [x, y, u, v], in that oder.
    t optional parameter.
    """

    x0 = np.copy(X_0[0])
    y0 = np.copy(X_0[1])
    u0 = np.copy(X_0[2])
    v0 = np.copy(X_0[3])

    g = 9.81
    c = 0.5
    a = 0.25
    b = 0.5
    p = (2*np.pi)/10.0
    q = (2*np.pi)/4.0

    H = -a*x0 - b*np.cos(p*x0)*np.cos(q*y0)
    H_x = -a + b*p*np.sin(p*x0)*np.cos(q*y0)
    H_xx = b*p**2 * np.cos(p*x0)*np.cos(q*y0)
    H_y = b*q*np.cos(p*x0)*np.sin(q*y0)
    H_yy = b*q**2 * np.cos(p*x0)*np.cos(q*y0)
    H_xy = -b*q*p*np.sin(p*x0)*np.sin(q*y0)

    F = (g + H_xx*u0**2 + 2*H_xy*u0*v0 + H_yy*v0**2)/(1 + H_x**2 + H_y**2)

    dU = -F*H_x - c*u0
    dV = -F*H_y - c*v0

    return np.array([u0, v0, dU, dV])
```

We use the fourth-order **Runge-Kutta method** as an integrator and we wrap it around the `solver` function.

```python
def runge_kutta_step(f, x0, dt, t=None):
    """
        runge_kutta_step(f, x0, dt, t=None)

    Computes a step using the Runge-Kutta 4th order method.

    f is a RHS function, x0 is an array of variables to solve,
    dt is the timestep, and t correspond to an extra paramameter of
    of the RHS.
    """

    k1 = f(t, x0) * dt
    k2 = f(t, x0 + k1/2) * dt
    k3 = f(t, x0 + k2/2) * dt
    k4 = f(t, x0 + k3) * dt
    x_new = x0 + (k1 + 2*k2 + 2*k3 + k4)/6

    return x_new

def solver(f, x0, y0, v0, u0, dt, N_t, N, b = 0.5):
    """
        solver(f, x0, y0, v0, u0, dt, N_t, N, b = 0.5)

    Function iterate the solution using runge_kutta_step and a RHS function f, for several points or initial
    conditions given by N.

    f is a RHS function
    x0, y0, v0, u0 are arrays containig the initial condition x,y,v,u; with N dimensions.
    dt is the timestep,
    N_t number of time steps.
    N number of initial conditions to iterate.
    b height of the moguls, parameter fixed.
    """

    solution = np.zeros((4, N_t+1, N))
    solution[0,0, :] = x0
    solution[1,0, :] = y0
    solution[2,0,: ] = u0
    solution[3,0, :] = v0

    for i in range(1, N_t + 1):
        for k in range(N):

            x_0_step = solution[:, i-1, k]
            solution[:, i, k] = runge_kutta_step(f, x_0_step, dt, b)

    return solution
```

```python
def plot_several(Solution):
    """
        plot_several_position(Solution)

    Function to plot several trajectories of boards or sleds.
    Solution is the array with the solutions of the ensemble of points.
    """

    N = Solution.shape[2]
    plt.figure(figsize=(5,10))

    for i in range(0,N):
        plt.plot(Solution[1, :, i], Solution[0, :, i])

    plt.gca().invert_yaxis()
    plt.gca().set_aspect('equal', adjustable='box')
    plt.grid()
    plt.ylabel('x - Downslope position (m)', fontsize =13)
    plt.xlabel('y - Crossslope position (m)', fontsize =13)
```

So now that we have our solver we can crush some snow with our boards in the moguls!

### Example 1

Initial conditions: $x = 0.0$, $y = [0,1]$ (randomly chosen), $U = 3.5$ and $V = 0$.

```python
n_sleds = 10
n_time = 1000
x_init = np.zeros(n_sleds)
y_init = np.random.rand(n_sleds)
v_init = np.zeros(n_sleds)
u_init = np.zeros(n_sleds) + 3.5

sol_ex1 = solver(board, x_init, y_init, v_init, u_init , 0.01, n_time, n_sleds)

plot_several(sol_ex1)
plt.title('10 boards')
```

### Example 2

Initial conditions: $x = 0.0$, $y = [0,1]$ scattered evenly, $U = 4.0$ and $V = 2.0$.

```python
n_sleds = 10
n_time = 1000
x_init = np.zeros(n_sleds)
y_init = np.linspace(0.1, 1, n_sleds)
v_init = np.zeros(n_sleds) + 2
u_init = np.zeros(n_sleds) + 4

sol_ex2 = solver(board, x_init, y_init, v_init, u_init , 0.01, n_time, n_sleds)

plot_several(sol_ex2)
plt.title('1 cm between boards')
```

We can see how the system behaves chaotically, i.e if we vary a little the initial conditions, the trajectories of the boards diverge after 10 m downslope.

### Example 3
Initial conditions just spaced $1$mm apart from $0.497$m to $0.503$m.

```python
n_sleds = 7
n_time = 2200
x_init = np.zeros(n_sleds)
y_init = np.linspace(0.497, 0.503, n_sleds)
v_init = np.zeros(n_sleds) + 2
u_init = np.zeros(n_sleds) + 4

sol_ex3 = solver(board, x_init, y_init, v_init, u_init , 0.01, n_time, n_sleds)

plot_several(sol_ex3)
plt.title('1 mm between boards')
```

The above shows that the system is sensitive to initial conditions but we would like to explore further the dynamics of the system. One of the technique for exploring the dynamics of the system is by plotting the phase space and see how the system behaves. The only problem is that for this system, there are 4 variables, which means that the phase space of the system lives in a four-dimensional space, which is not possible for us to visualize. We need to do some modifications to our board to reduce the dimensions of our system.

# Sleds down the slope!

To reduce the dimensions of our system, we can make the downslope velocity to be constants, or in the real world, by equipping our board with some brakes and an engine so it can maintain a constant velocity while going down the pits or up the bumps. In a mathematical sense, we say that $\partial_t U = 0$.

```python
def sled(t, X_0):
    """
        sled(t, X_0)

    Right-hand-side of the equations for a sled, with constant downward velocity,
    going down a slope with moguls.

    X_0 is the the set of initial conditions containing [x, y, u, v], in that oder.
    t optional parameter.
    """

    x0 = np.copy(X_0[0])
    y0 = np.copy(X_0[1])
    u0 = np.copy(X_0[2])
    v0 = np.copy(X_0[3])

    g = 9.81
    c = 0.5
    a = 0.25
    #b = 0.5
    p = (2*np.pi)/10.0
    q = (2*np.pi)/4.0

    H = -a*x0 - b*np.cos(p*x0)*np.cos(q*y0)
    H_x = -a + b*p*np.sin(p*x0)*np.cos(q*y0)
    H_xx = b*p**2 * np.cos(p*x0)*np.cos(q*y0)
    H_y = b*q*np.cos(p*x0)*np.sin(q*y0)
    H_yy = b*q**2 * np.cos(p*x0)*np.cos(q*y0)
    H_xy = -b*q*p*np.sin(p*x0)*np.sin(q*y0)

    F = (g + H_xx*u0**2 + 2*H_xy*u0*v0 + H_yy*v0**2)/(1 + H_x**2 + H_y**2)

    dU = 0
    dV = -F*H_y - c*v0

    return np.array([u0, v0, dU, dV])
```

Let's try it.

Initial conditions $x = 0$ m, $y = [0.1, 1]$ m, $v = 0$ m/s and $u = 3.5$ m/s.

```python
n_sleds = 10
n_time = 1000
x_init = np.zeros(n_sleds)
y_init = np.linspace(0.1, 1, n_sleds)
v_init = np.zeros(n_sleds)
u_init = np.zeros(n_sleds) + 3.5

sled_ex1 = solver(sled, x_init, y_init, v_init, u_init , 0.01, n_time, n_sleds)

plot_several(sled_ex1)
plt.title('10 sleds down the slope')
```

We see that the system behaves chaotically even though the downslope speed is maintained.

# Sleds and strange attractors

Now our system has 3 variables, which will allow us to plot it phase space. The only problem now is that the system is no compact, because $x$ and $y$ can grow or decrease infinitively. Although, the cross slope velocity $v$ is bounded between $-5$ m/s and $5$ m/s, because of friction.

A way to do this is reducing the slope to be $x \in [-5, 5]$ m and $y \in [-2, 2]$ m, and set the boundaries to be periodic. This way, we will be able to compact the position of the sled without altering the physics of the system.

```python
x_range_compact = np.linspace(-5, 5, 50)
y_range_compact = np.linspace(-2, 2, 50)
XX_c, YY_c = np.meshgrid(x_range_compact, y_range_compact)

moguls_compact = pits_n_crests(XX_c, YY_c)

plt.figure(figsize=(5,4))
plt.contourf(YY_c, XX_c, moguls_compact)
plt.colorbar(label = 'Height of bumps (m)')
plt.ylabel('x - Downslope position (m)', fontsize =13)
plt.xlabel('y - Crossslope position (m)', fontsize =13)
plt.savefig('fig/shape_of_moguls_compact')
```

```python
def compactor(x, lower_bound, upper_bound):
    """
        compactor(x, lower_bound, upper_bound)

    Funtion that maps any point outside the interval [lower_bound, upper_bound]
    to a point in the interval, preserving length. If the point is in the interval,
    it does not change.

    x point in question to be evaluated.
    lower_bound lower bound of the interval.
    upper_bound upper bound of the interval.
    """

    if x > upper_bound:
        return x%upper_bound - upper_bound
    elif x < lower_bound:
        return x%lower_bound - lower_bound
    else:
        return x
```

```python
def solver_compact(f, x0, y0, v0, u0, dt, N_t, N, b = 0.5):
    """
        solver_compact(f, x0, y0, v0, u0, dt, N_t, N, b = 0.5)

    Function iterate the solution using runge_kutta_step and a RHS function f, for several points or initial
    conditions given by N, in compact domain.

    f is a RHS function
    x0, y0, v0, u0 are arrays containig the initial condition x,y,v,u; with N dimensions.
    dt is the timestep,
    N_t number of time steps.
    N number of initial conditions to iterate.
    b height of the moguls, parameter fixed.
    """

    solution = np.zeros((3, N_t+1, N))
    solution[0,0, :] = x0
    solution[1,0, :] = y0
    solution[2,0, :] = v0

    for i in range(1, N_t + 1):
        for k in range(N):

            x_0_step = np.insert(solution[:, i-1, k], 2, u0)

            aux = runge_kutta_step(f, x_0_step, dt, b)
            solution[0, i, k] = compactor(aux[0], -5, 5)
            solution[1, i, k] = compactor(aux[1], -2, 2)
            solution[2, i, k] = aux[3]

    return solution
```

Let's try it!

```python
n_sleds = 1
n_time = 1000
x_init = np.zeros(n_sleds)
y_init = np.random.rand(n_sleds)*4 - 2
v_init = np.random.rand(n_sleds)*10 - 5

sled_compact_1 = solver_compact(sled, x_init, y_init, v_init, 3.5, 0.01, n_time, n_sleds)
plot_several(sled_compact_1)

plt.title('1 sled in compact domain')
```

We can see how the sled goes out of the domain and re-enters at the opposite side.

The main interest of this is to build a strange attractor. For this, we set a ensemble of sled all starting from $x = 0$ m, but with random $y$ and cross slope velocities $v$, and with $u$ set to 3.5 m.

```python
import time
```

```python
n_sleds = 1000
n_time = 1000
x_init = np.zeros(n_sleds)
y_init = np.random.rand(n_sleds)*4 - 2
v_init = np.random.rand(n_sleds)*10 - 5

start = time.time()
sled_compact_1000 = solver_compact(sled, x_init, y_init, v_init, 3.5, 0.01, n_time, n_sleds)
end = time.time()
print("Elapsed time:", (end - start)/60, "minutes...")
```

    Elapsed time: 6.451248653729757 minutes...

> Here is when I realized how bad I am in making my code to go fast...

Once that we have our solution, we can plot the phase space of our system!

```python
time_steps = range(0,1000, 180)

fig, axes = plt.subplots(3, 2, sharex=True, sharey= True, figsize = (6,8))
axes = axes.reshape(6)

for i, j in enumerate(time_steps):

    dist = round(j*(3.5*0.01))
    axes[i].scatter(sled_compact_1000[1, j, :], sled_compact_1000[2, j, :], s = 0.5, color = 'k')
    axes[i].text(1.8,-4.9, '{} m'.format(dist) , ha='center')

plt.tight_layout()
fig.text(0.5, -0.01, 'y - Cross slope position (m)', ha='center', fontsize = 13)
fig.text(-0.01, 0.5, 'v - Cross slope velocity (m/s)', va='center', rotation='vertical', fontsize = 13)

```

In the previous figure, we can see that a structure is being developed as the ensemble of sleds are going down the slope. This is a strange attractor.

The next step is to perform different simulations changing the parameter $b$ which is the height of the bumps, where we would see the transition between non-chaotic states (when $b$ is very small) towards chaotic states (above a critical value of $b$). This can be seen in a bifurcation diagram, but because of performance issues, it is not going to be done in this particular notebook (`solver_compact` function is too slow!), and because for computing a bifurcation diagram it is necessary to compute solutions for an array of $b$'s, for a long period of time to remove transient states before the system reaches its equilibrium, which will take a lot of time to solve using my code here.

The bifurcation diagram for the system and some animations can be seen in the Julia notebook, where my codes happen to be faster than the one presented here (I was lucky with Julia, I'm not implying that Julia is faster).

# References

- Lorenz, Edward N. *The Essence of Chaos*. Univ. of Washington Press, 1993.

----
Code by Claudio Pierard. *Master Environmental Fluid Mechanics*, Université Grenoble Alpes. January 2020.


```python
%matplotlib notebook
import matplotlib.pyplot as plt
from sympy import *
init_printing()
```

```python
x, y = var("x y")
```

```python
f = x**3 - 3 * x**2 -24 * x + +32
f
```

```python
plot(f)
```

    <IPython.core.display.Javascript object>

    <sympy.plotting.plot.Plot at 0x1635e543550>

```python
df=f.diff()
dff = df.diff()
df, dff
```

```python
pc = solve(df)
pc
```

```python
dff.subs(x,pc[0]), dff.subs(x,pc[1])
```

A(x,y) = x*y

P(x,y) = 2x + y =2400

```python
A=x*y
eq1=Eq(2*x+y,2400)
y_=solve(eq1,y)
y_
```

```python
f=A.subs(y,y_[0])
f
```

```python
df = f.diff()
ddf= df.diff()
pc = solve(df)
pc
```

```python
eq1.subs(x,600)
```

```python
solve(eq1.subs(x,600),y)
```

```python
plot(f)
```

    <IPython.core.display.Javascript object>

    <sympy.plotting.plot.Plot at 0x1635de17518>

```python
x=600
y=1200

print(x*y)

```

    720000

lata de un litro optimizar material

```python
r, h = var("r h", real="True")
A=2*pi*r**2 + 2*pi*r*h
eq1=Eq(pi*r**2*h,1000)
h_=solve(eq1,h)
h_

```

```python
f1=A.subs(h,h_[0])
f1
```

```python
df1 = f1.diff()
ddf1= df1.diff()

df1, ddf1
```

```python
pc1= solve(df1)
pc1
```

```python
N(pc1[0])
```

```python
ddf1.subs(r,pc1[0])
```

```python
eq1.subs(r,pc1[0])

```

```python
solve(eq1.subs(r,pc1[0]),h)[0]
```

```python
N(solve(eq1.subs(r,pc1[0]),h)[0])
```

f(x,y)= (x^2) + (y^2)

```python
x, y = var("x y")
f=(x**2) + (y**2)
dfx=f.diff(x)
dfy=f.diff(y)

print(dfx, dfy)
```

    2*x 2*y

```python
pc_xy=solve([dfx, dfy],[x,y])
dfx, dfy, pc_xy
```

```python
dfxx=dfx.diff(x)
dfxy=dfx.diff(y)
dfyx=dfy.diff(x)
dfyy=dfy.diff(y)

dfxx, dfxy, dfyx, dfyy
```

```python
D=dfxx*dfyy- dfxy**2
D
```

```python
dfxx.subs(pc_xy)
```

```python
nablaf=[f.diff(var) for var in [x,y]]
nablaf
```

```python
solve(nablaf)
```

```python
[[f.diff(varx).diff(vary) for varx in [x,y] for vary in [x,y]]]
```

```python
H = hessian(f,[x,y])
H
```

$$\left[\begin{matrix}2 & 0\\0 & 2\end{matrix}\right]$$

```python
H.eigenvals()
```


# Fitting a model to data with MCMC #

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import plot_helper as plot_helper
import pandas as pd
import emcee # 2.2.1
import corner
import progressbar
import scipy.optimize as op
import os
```

```python
# Make directories if required
plots_dir = './plots/'

if not os.path.exists(plots_dir):
    os.mkdir(plots_dir)
```

# What is MCMC?

This is a very short introduction to using MCMC for fitting a model to data; see references below for much more detailed examples.

### The ground truth

Let us suppose we are interested in some physical process which relates the quantities $x$ and $y$ as

\begin{equation}
y=y_{max}\frac{x}{x+K},
\end{equation}
with true parameter values $y_{max}=1$ and $K=2$.

```python
def model(x,ymax,K):
    return ymax*x/(x+K)
ymax=1
K=2
x=np.linspace(0,10,50)
y=model(x,ymax,K)
plot_helper.plot1(x,y,title='Ground truth')
```

Suppose we make some observations to measure $y_{max}$ and $K$.

```python
N=9
xobs=(np.random.rand(N))*10
yerrtrue=0.03*np.random.randn(N) # normally distributed errors
yobs=model(xobs,ymax,K)+yerrtrue
yerr=yerrtrue*1 # Our estimated error is not necessarily equal to the true error
plot_helper.plot2(x,y,xobs,yobs,yerr,title='Ground truth+observations')
```

We would like to estimate the posterior probability distribution for $y_{max}$ and $K$, given these observations. In other words, we want $P(model|data)$, the probability of our model parameters given the data. Bayes' theorem gives an expression for this quantity:

\begin{equation}
P(model|data)=\frac{P(data|model)P(model)}{P(data)}
\end{equation}

Let's unpack this equation.

### The prior

$P(model)$ is the prior; it is a description of the uncertainty we palce on the parameters in our model. For instance, let us assume that our parameters are initially normally distributed:

\begin{align}
y_{max}&=\mathcal{N}(1,0.2) \\
K&=\mathcal{N}(2,0.2)
\end{align}

so that our model becomes

\begin{equation}
\hat{y}=\mathcal{N}(1,0.2)\frac{x}{x+\mathcal{N}(2,0.2)}.
\end{equation}

The prior probability of our model given parameters $y_{max}$ and $K$ is

\begin{equation}
P(model)=\mathcal{N}(y_{max}-1,0.2)\mathcal{N}(\mu-2,0.2).
\end{equation}

Typically we express these probablities in terms of log-probabilities so that the terms become additive:

\begin{equation}
\ln P(model)=\ln\mathcal{N}(y_{max}-1,0.2)+\ln\mathcal{N}(\mu-2,0.2).
\end{equation}

```python
def prior(x,mu,sigma):
    return 1/np.sqrt(2*np.pi*sigma**2)*np.exp(-(x-mu)**2/(2*sigma**2))
mu1=1
mu2=2
sigma=0.2

xp=np.linspace(0,3,100)
y1=prior(xp,mu1,sigma)
y2=prior(xp,mu2,sigma)
plot_helper.plot3(xp,y1,xp,y2,title='Prior')
```

 ### The likelihood

$P(data|model)$ is known as the likelihood. It's a measure of how likely it is that our model generates the observed data. In order to calculate this term we need a measure of how far our model predictions are from the actual observed data; typically we assume that deviations are due to normally-distributed noise, in which case our likelihood takes the simple form of squared residuals for each of the data points $y_n$ with error $s_n$:

\begin{equation}
P(data|model)=\prod_n\frac{1}{2\pi s_n^2}\exp\left(-\frac{(y_n-\hat{y}_n)^2}{2s_n^2}\right)
\end{equation}

The negative log-likelihood is therefore

\begin{equation}
\ln P(data|model)=-\frac{1}{2}\sum_n \left(\frac{(y_n-\hat{y}_n)^2}{s_n^2}+\ln (2 \pi s_n^2) \right)
\end{equation}

### MCMC

What we want to do is determine the posterior probability distribution $\Pi(model|data)$. From this distribution we can determine probabilities as well as the expectation values of any quantity of interest by integrating. In other words, we would like to generate the probability landscape of likely model parameters, given our observations. In order to do this we must sample the landscape by varying the parameters. MCMC allows us to do this, without having to calculate the third term $P(data)$ in the Bayes formula which is nontrivial. The simplest MCMC algorithm is that of Metropolis:

#### The Metropolis algorithm

1) First we start at an initial point for the parameters $y_{max,0}$, $K_0$. We compute the probabilities

\begin{equation}
P(data|y_{max,0},K_0)P(y_{max,0},K_0).
\end{equation}

2) Then we move to a new location $y_{max,1}$, $K_1$. This new location is called the proposal, and it's generated by randomly moving to a new point with probability given by a normal distribution centered around the current location, and a fixed variance (the proposal width).

3) We calculate the new probabilities

\begin{equation}
P(data|y_{max,1},K_1)P(y_{max,1},K_1).
\end{equation}

4) We then calculate the acceptance ratio:

\begin{equation}
\alpha=\frac{P(data|y_{max,1},K_1)P(y_{max,1},K_1)}{P(data|y_{max,0},K_0)P(y_{max,0},K_0)}.
\end{equation}

If $\alpha$ is greater than 1, i.e. the probability at the new point is higher, we accept the new point and move there. If $\alpha$ is smaller than 1, then we accept the move with a probability equal to $\alpha$.

```python
def normalprior(param,mu,sigma):
    return np.log( 1.0 / (np.sqrt(2*np.pi)*sigma) ) - 0.5*(param - mu)**2/sigma**2

def like(pos,x,y,yerr):
    ymax=pos[0]
    K=pos[1]
    model=ymax*x/(x+K)
    inv_sigma2=1.0/(yerr**2)
    return -0.5*(np.sum((y-model)**2*inv_sigma2-np.log(inv_sigma2)))

def prior(pos):
    ymax=pos[0]
    K=pos[1]

    log_PRs=[normalprior(ymax,1,0.5), # Normal priors
            normalprior(K,2,0.5)]

    return np.sum(log_PRs)

def norm(pos,width):
    return pos+width*np.random.randn(2)

def metropolis(pos,MC,steps,width):
    for i in range(steps):
        proposal=norm(pos,width)
        newloglike=like(proposal,xobs,yobs,yerr)+prior(proposal)
        oldloglike=like(pos,xobs,yobs,yerr)+prior(pos)
        if newloglike>=oldloglike: # If new probability is higher then accept
            pos=proposal
        else:
            a=np.exp(newloglike-oldloglike)
            if np.random.rand()<a: # If old probability is higher than only accept with probability a.
                pos=proposal
            else:
                pos=pos

        MC[i]=pos
    return MC

# Run MCMC here:

steps=10000 # Set number of steps
width=0.1 # Set proposal width
MC=np.zeros(steps*2).reshape(steps,2)
pos=np.array([1,2])
MC=metropolis(pos,MC,steps,width)

plt.figure(figsize=(5,5))
plt.plot(MC[:,0],MC[:,1],'.-',alpha=0.3);
plt.show()
```

Our Markov chain samples positions in parameter space, spending proportionately more time in regions of high probability mass. While the Metropolis algorithm is intuitive and instructive it is not the most efficient MCMC algorithm, so for the next part we will apply a more efficient ensemble sampler.

#### A more efficient algorithm: Goodman and Weare affine-invariant ensemble samplers

```python
def normalprior(param,mu,sigma):
    return np.log( 1.0 / (np.sqrt(2*np.pi)*sigma) ) - 0.5*(param - mu)**2/sigma**2

def lnlike(theta,x,y,yerr):
    ymax,K=theta
    model=ymax*x/(x+K)
    inv_sigma2=1.0/(yerr**2)
    return -0.5*(np.sum((y-model)**2*inv_sigma2-np.log(inv_sigma2)))

def lnprior(theta):
    ymax,K=theta

    if not (0<ymax and 0<K) :
        return -np.inf # Hard-cutoff for positive value constraint

    log_PRs=[normalprior(ymax,1,0.5), # Normal priors
            normalprior(K,2,0.5)]

    return np.sum(log_PRs)

def lnprob(theta, x, y, yerr):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta, x, y, yerr)
```

```python
ndim,nwalkers,threads,iterations,tburn=2,20,8,1000,200
labels=["$y_{max}$","$K$"]
parametertruths=[1,2]
pos=[np.array([
    1*(1+0.05*np.random.randn()),
    1*(1+0.05*np.random.randn())]) for i in range(nwalkers)]
sampler=emcee.EnsembleSampler(nwalkers,ndim,lnprob,a=2,args=(xobs,yobs,yerr),threads=threads)
```

```python
### Start MCMC
iterations=iterations
bar=progressbar.ProgressBar(max_value=iterations)
for i, result in enumerate(sampler.sample(pos, iterations=iterations)):
    bar.update(i)
### Finish MCMC

samples=sampler.chain[:,:,:].reshape((-1,ndim)) # shape = (nsteps, ndim)
samplesnoburn=sampler.chain[:,tburn:,:].reshape((-1,ndim)) # shape = (nsteps, ndim)

df=pd.DataFrame(samples)
#df.to_csv(path_or_buf='samplesout_simple.csv',sep=',') # Uncomment to save

plot_helper.plottraces(samples,labels,parametertruths,nwalkers,iterations,1)
fig=corner.corner(samplesnoburn, labels=labels,truths=parametertruths,quantiles=[0.16, 0.5, 0.84],show_titles=True, title_fmt='.2e', title_kwargs={"fontsize": 10},verbose=False)
fig.savefig(plots_dir+"triangle_simple.pdf")

```

```python
plot_helper.plot4(xp,y1,xp,y2,samplesnoburn,title='Posterior')
```

The plots above are instructive: what MCMC does is it generates samples of the posterior distribution, which is a product of the prior distribution and the likelihood function as determined by the observations. For strong priors and weak data, the posterior will be very similar to the prior. However, if the data are very strong or the prior very weak, the posterior will be more simliar to the likelihood. Playing around with the strength of the priors, by varying the $\sigma$ of the Gaussians, lets you see this effect.

```python
plot_helper.plot5(x,y,xobs,yobs,yerr,samplesnoburn,xlabel='x',ylabel='y',legend=False,title=False)
```

Finally, we can plot our model 'fits' by sampling randomly from the posterior distribution. This gives us a sense of the spread in our model predictions.

**References**

* MacKay 2003 http://www.inference.org.uk/itprnn/book.html - the bible for MCMC and inferential methods in general

* Goodman and Weare 2010 https://projecteuclid.org/euclid.camcos/1513731992 - original paper describing affine-invariant ensemble sampling

* emcee http://dfm.io/emcee/current/user/line/ - Python implementation of the Goodman and Weare algorithm

* Fitting a model to data https://arxiv.org/abs/1008.4686 - excellent tutorial on how to 'properly' fit your data

* Hamiltonian Monte Carlo https://arxiv.org/abs/1701.02434 - a more efficient MCMC algorithm, as implemented in Stan (http://mc-stan.org)

* Another nice online tutorial http://twiecki.github.io/blog/2015/11/10/mcmc-sampling/

# MCMC on dynamical systems

```python
import tellurium as te
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import plot_helper as plot_helper
import pandas as pd
import emcee
import corner
import progressbar
```

Here is a more sophisticated example similar to what you might encounter in the lab. Suppose we have a dynamical system (here modelled using tellurium) and you want to determine its kinetic parameters. For example, a Michaelis-Menten kinetic model for enzyme catalysis. Experimentally, you might do a titration of concentrations followed by inference to get the parameters. Here we do this in silico.

```python
# Define model using tellurium, populate parameters, and run
# a simulation to see its typical behaviour.

def MM_genmodel():
    ''' Michaelis-Menten enzyme model '''

    rr = te.loada('''
        J1: E+S->ES ; k1*E*S-k2*ES ;
        J2: ES->E+S+P ; k3*ES ;

        k1=0;
        k2=0;
        k3=0;

        ''')

    return(rr)

def simulatemodel(rr,tmax,nsteps,paramdict):

    for j in rr.model.getGlobalParameterIds():
        rr[j]=paramdict[j] # set parameters

    for j in rr.model.getFloatingSpeciesIds():
        rr[j]=paramdict[j] # set concentrations

    out=rr.simulate(0,tmax,points=nsteps)

    return(out,rr)

# Generate model
rr=MM_genmodel()

# Simulate model
tmax=20
nsteps=51
keys=['k1','k2','k3','E','S','ES','P']
params=[1,1,1,1,10,0,0]
paramdict=dict(zip(keys,params))
out,_=simulatemodel(rr,tmax,nsteps,paramdict)
rr.plot()
```

Let's do a titration experiment and MCMC to extract kinetic parameters for this enzyme.

```python
np.random.seed(42)

# Define experiment settings and generate synthetic data
def titration_expt(titration,k1,k2,k3,tmax,nsteps,rr):

    Parr=np.zeros((nsteps,len(titration)))
    for j in range(len(titration)):
        keys=['k1','k2','k3','E','S','ES','P']
        params=[k1,k2,k3,1,titration[j],0,0]
        paramdict=dict(zip(keys,params))
        out,_=simulatemodel(rr,tmax,nsteps,paramdict)
        Parr[:,j]=out[:,4]
    return Parr

rr=MM_genmodel()
tmax=20
nsteps=51
titrationvals=[0,5,10,15,20]

Parr=titration_expt(titrationvals,1,10,1,tmax,nsteps,rr)
Parr+=0.2*np.random.randn(Parr.shape[0],Parr.shape[1])*Parr+0.0001*np.random.randn(Parr.shape[0],Parr.shape[1]) # Add noise
plt.plot(Parr,'o') ; plt.show()

# Put data into yobs, yerr
yobs=Parr
yerr=Parr*0.2

# Regenerate model
rr=MM_genmodel()

# Define titration experiments and time steps
inputkeys=['tmax','nsteps','titration','model','y','yerr']
inputvalues=[tmax,nsteps,titrationvals,rr,yobs,yerr]
inputs=dict(zip(inputkeys,inputvalues))
np.random.seed(42)

```

```python
# Define MCMC functions

def normalprior(param,mu,sigma):
    return np.log( 1.0 / (np.sqrt(2*np.pi)*sigma) ) - 0.5*(param - mu)**2/sigma**2

def lnlike(theta,inputs):
    k1,k2,k3=theta

    # DATA
    y=inputs['y']
    yerr=inputs['yerr']

    # MODEL INPUTS
    tmax=inputs['tmax']
    nsteps=inputs['nsteps']
    titration=inputs['titration']
    rr=inputs['model']

    ymodel=titration_expt(titration,k1,k2,k3,tmax,nsteps,rr)

    inv_sigma2=1.0/(yerr**2)
    return -0.5*(np.sum((y-ymodel)**2*inv_sigma2-np.log(inv_sigma2)))

def lnprior(theta):
    k1,k2,k3=theta

    if not (0<k1 and 0<k2 and 0<k3) :
        return -np.inf # Hard-cutoff for positive value constraint

    log_PRs=[normalprior(k1,5,10),
            normalprior(k2,10,10),
            normalprior(k3,1,0.01)]

    return np.sum(log_PRs)

def lnprob(theta,inputs):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta,inputs)

def gelman_rubin(chain):
    ''' Gelman-Rubin diagnostic for one walker across all parameters. This value should tend to 1. '''
    ssq=np.var(chain,axis=1,ddof=1)
    W=np.mean(ssq,axis=0)
    Tb=np.mean(chain,axis=1)
    Tbb=np.mean(Tb,axis=0)
    m=chain.shape[0]*1.0
    n=chain.shape[1]*1.0
    B=n/(m-1)*np.sum((Tbb-Tb)**2,axis=0)
    varT=(n-1)/n*W+1/n*B
    Rhat=np.sqrt(varT/W)
    return Rhat
```

The Gelman-Rubin diagnostic is a typical quantity used to tset convergence of MCMC chains. Technically this quantity cannot be used for emcee (ensemble sampling) as the walkers are not independent. But I've included it here as a reference.

First do maximum likelihood estimation using a nonlinear optimizer. This gives us some starting values for MCMC.

```python
# Maximum likelihood

# Set initial positions
pos=[
5,  # k1
10, # k2
1 # k3
]

nll= lambda *args: -lnlike(*args)
result=op.minimize(nll,pos,method='BFGS', args=(inputs))
paramstrue = result["x"]

k1_MLE=paramstrue[0]
k2_MLE=paramstrue[1]
k3_MLE=paramstrue[2]

print(k1_MLE,k2_MLE,k3_MLE)

tmax=20
nsteps=51
titration=[0,5,10,15,20]
ymodel=titration_expt(titration,k1_MLE,k2_MLE,k3_MLE,tmax,nsteps,rr)
plt.plot(yobs,'o')
plt.plot(ymodel,'k-',alpha=1) ; plt.show()

```

Now run MCMC using MLE values as initial positions.

```python
# Run MCMC

##### Set parameters here #####
ndim,nwalkers,threads,iterations,tburn=3,50,1,3000,1000
labels=["$k_1$","$k_2$","$k_3$"]
parametertruths=[1,10,1]
pos=[np.array([
    k1_MLE*(1+0.05*np.random.randn()),
    k2_MLE*(1+0.05*np.random.randn()),
    k3_MLE*(1+0.05*np.random.randn())]) for i in range(nwalkers)]
filename='MM'

##### The rest of the code is automatic #####
sampler=emcee.EnsembleSampler(nwalkers,ndim,lnprob,a=2,args=([inputs]),threads=threads)

### Start MCMC
iterations=iterations
bar=progressbar.ProgressBar(max_value=iterations)
for i, result in enumerate(sampler.sample(pos, iterations=iterations)):
    bar.update(i)
### Finish MCMC

samples=sampler.chain[:,:,:].reshape((-1,ndim)) # shape = (nsteps, ndim)
samplesnoburn=sampler.chain[:,tburn:,:].reshape((-1,ndim)) # shape = (nsteps, ndim)

df=pd.DataFrame(samples)
#df.to_csv(path_or_buf=plots_dir+'samplesout_'+filename+'.csv',sep=',') # UNCOMMENT TO SAVE OUTPUT

plot_helper.plottraces(samples,labels,parametertruths,nwalkers,iterations,1)
fig=corner.corner(samplesnoburn, labels=labels,truths=parametertruths,quantiles=[0.16, 0.5, 0.84],show_titles=True, title_fmt='.2e', title_kwargs={"fontsize": 10},verbose=False)
fig.savefig(plots_dir+'triangle_'+filename+'.pdf')

### Gelman-Rubin diagnostic
# NOT RELIABLE ESTIMATE FOR EMCEE AS WALKERS NOT INDEPENDENT!
plt.close("all")
figure_options={'figsize':(8.27,5.83)} #figure size in inches. A4=11.7x8.3.
font_options={'size':'12','family':'sans-serif','sans-serif':'Arial'}
plt.rc('figure', **figure_options)
plt.rc('font', **font_options)

chain=sampler.chain[:,tburn:,:] # shape = nwalkers, iterations-tburn, ndim
print('Mean acceptance fraction', np.mean(sampler.acceptance_fraction))
print('GR diagnostic for one walker', gelman_rubin(chain)[0]) # Change index to get a different walker

chain_length=chain.shape[1]
step_sampling=np.arange(int(0.2*chain_length),chain_length,50)
rhat=np.array([gelman_rubin(chain[:,:steps,:])[0] for steps in step_sampling])
plt.plot(step_sampling,rhat); ax=plt.gca(); ax.axhline(y=1.1,color='k'); ax.set_title('GR diagnostic');
plt.show()

```

```python
# Autocorrelation time analysis. 'c' should be as large as possible (default is 5)
tau = np.mean([emcee.autocorr.integrated_time(walker,c=1) for walker in sampler.chain[:,:,:]], axis=0)
print('Tau', tau)
```

    Tau [ 40.46367221  40.06771078  30.35263478]

```python
for k1,k2,k3 in samplesnoburn[np.random.randint(len(samplesnoburn), size=10)]:
    tmax=20
    nsteps=51
    titration=[0,5,10,15,20]
    ymodel=titration_expt(titration,k1,k2,k3,tmax,nsteps,rr)
    plt.plot(ymodel,'k-',alpha=0.1)
plt.plot(yobs,'o'); plt.show()
```

```python

```


# Lecture 16: Exponential Distribution, Memoryless Property

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## Exponential Distribution

### Description

Real-valued distribution describing wait times, survival times. Rate parameter $\lambda \gt 0$.

Continuous analog of the geometric distribution.

Unique in that the exponential distribution has the memoryless property.

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
from scipy.stats import expon

%matplotlib inline

plt.xkcd()

_, ax = plt.subplots(figsize=(12,8))

# seme Exponential parameters
lambdas = [2.0, 1.5, 1.0, 0.5]

# qualitative color scheme
colors = ['#66c2a5', '#fc8d62', '#8da0cb', '#e78ac3']

x = np.linspace(0, 4, 500)
for i,l in enumerate(lambdas):
    pdf = expon.pdf(x, scale=1/l)
    ax.plot(x, pdf, color=colors[i], lw=3.2, label=r'$\lambda = {}$'.format(l))

# legend styling
legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

# y-axis
ax.set_ylim([-0.01, 2.0])
ax.set_ylabel(r'$f(x)$')
ax.set_yticks(np.arange(0,2.1,.2))

# x-axis
ax.set_xlim([0.0, 3.0])
ax.set_xlabel(r'$x$')

# x-axis tick formatting
majorLocator = MultipleLocator(1)
majorFormatter = FormatStrFormatter('%d')
minorLocator = MultipleLocator(1)
ax.xaxis.set_major_locator(majorLocator)
ax.xaxis.set_major_formatter(majorFormatter)
ax.xaxis.set_minor_locator(minorLocator)

ax.grid(color='grey', linestyle='-', linewidth=0.3)

plt.suptitle(r'Exponential PDF: $f(x) = \lambda e^{-\lambda x}$ for $x \geq 0$')

plt.show()
```

### Notation

$X \sim \operatorname{Expo}(\lambda)$

### Parameters

$\lambda$ - rate parameter where $\lambda \gt 0$

### Probability density function

\begin{align}
  f(x) &=
  \begin{cases}
    \lambda e^{-\lambda x}, &\text{ if } x \ge 0 \\
    0, &\text{ otherwise }
  \end{cases}
\end{align}

### Cumulative distribution function

\begin{align}
  F(x) &= \int_{0}^{x}  \lambda e^{-\lambda t} \, dt \\
         &= \lambda \int_{0}^{x} e^{-\lambda t} \, dt &\text{ let } u = -\lambda t \text{, } du =  -\lambda \, dt \\
         &= \int - e^{u} \, du \\
         &= - e^{u} \\
         &= \left. - e^{-\lambda t} \right|_{0}^{x} \\
         &= \boxed{ 1 - e^{-\lambda x} }
\end{align}

### Standardized Exponential Distribution

If we let $Y = \lambda X$, then $Y \sim \operatorname{Expo}(1)$.

You can compare this with the standardized Normal.

Proof

\begin{align}
  P(Y \le y) &= P\left(X \le \frac{y}{\lambda} \right) \\
             &= 1 - e^{-\lambda \frac{y}{\lambda}} &\text{ just plugging } \frac{y}{\lambda} \text{ into the CDF above} \\
             &= 1 - e^{-y} &\text{ which is the CDF of } \operatorname{Expo}(1) ~~~~ \blacksquare \\
\end{align}

We will next find the mean and variance of $\operatorname{Expo}(1)$, and then derive the general case mean and variance afterwards.

### Mean and variance of $\operatorname{Expo}(1)$

Let $Y \sim \operatorname{Expo}(1)$, find $\mathbb{E}(Y)$ and $\operatorname{Var}(Y)$.

\begin{align}
  \mathbb{E}(Y) &= \int_{0}^{\infty} y\,e^{-y}\,dy & &\text{ let } u = y \text{, } du = dy \\
    & & &\text{ and let } dv = e^{-y}\,dy \text{, } v = -e^{-y} \\
    &= \underbrace{ \left. -y e^{^y} \right\vert_{0}^{\infty}}_{\text{evaluates to }0} + \underbrace{\int_{0}^{\infty} e^{-y}\,dy}_{\text{PDF of }\operatorname{Expo}(1)} \\
    &= \boxed{1} \\
    \\\\
  \operatorname{Var}(Y) &= \mathbb{E}(Y^2) - \mathbb{E}Y ^2\\
    &= \int_{0}^{\infty} y^{2}\,e^{-y}\,dy \,- 1^2 & &\text{ let } u = y^{2} \text{, } du = 2y\,dy \\
    & & &\text{ and let } dv = e^{-y}\,dy \text{, } v = -e^{-y} \\
    &= \left. -y^{2}\,e^{^y} \right\vert_{0}^{\infty} + \int_{0}^{\infty} 2y\,e^{-y}\,dy \,-1 \\
    &= 0 + 2 - 1 \\
    &= \boxed{1}
\end{align}

### Mean and variance of $\operatorname{Expo}(\lambda)$

We can derive the mean and variance of $\operatorname{Expo}(\lambda)$ from that of $\operatorname{Expo}(1)$.

From $Y = \lambda X$ we have $X = \frac{y}{\lambda}$.

\begin{align}
  \mathbb{E}(X) &= \mathbb{E}\left(\frac{Y}{\lambda}\right) \\
                &= \frac{1}{\lambda} \, \mathbb{E}(Y) \\
                &= \boxed{ \frac{1}{\lambda} } \\
  \\\\
  \operatorname{Var}(X) &= \operatorname{Var}\left(\frac{Y}{\lambda}\right) \\
                  &= \frac{1}{\lambda^2} \, \operatorname{Var}(Y) \\
                  &= \boxed{ \frac{1}{\lambda^2} } \\
\end{align}

### Memorylessness Property

If you have a random variable representing a wait-time (continuous), the memorylessness property means that no matter how long you have already waited, the probability that you will have to wait an _additional_ time $t$ is the same as if you were starting fresh from 0 (irrespective of the time you already spent waiting).

Fact: $\operatorname{Expo}(\lambda)$ is the only distribution with the memorylessness property.

\begin{align}
  P(X \ge s+t | X \ge s) &= P(X \ge t) \\
\end{align}

The survival function is the random variable that describes how long something might live/exist, in constrast to that for a waiting time. In other words, $P(X \ge s)$ is the probability that some object of interest _lasts longer than_ a continuous time $s$.

\begin{align}
  P(X \ge s) &= 1 - P(X \le s) \\
             &= 1 - (1 - e^{-\lambda s}) \\
             &= e^{-\lambda s} & \quad \text{ the survival function}
\end{align}

And so using this survival function in an equation using the definition of conditional probability, we have:

\begin{align}
  P(X \ge s+t | X \ge s) &= \frac{P(X \ge s+t \text{, }X \ge s)}{P(X \ge s)} \\
                         &= \frac{P(X \ge s+t)}{P(X \ge s)} & \quad \text{ since } P(X \ge s) \text{ is redundant} \\
                         &= \frac{e^{-\lambda (s+t)}}{e^{-\lambda s}} & \quad \text{ ratio of survival functions} \\
                         &= e^{-\lambda t} \\
                         &= P(X \ge t) & \quad \blacksquare
\end{align}

### Useful corollary of the Memorylessness Property

This is a brief introduction to conditional expectation, which is just like conditional probability.

Given $X \sim \operatorname{Expo}(\lambda)$, what is the expected wait-time if we have already waited for some time $a$?

\begin{align}
  \mathbb{E}(X | X \gt a) &= a + \mathbb{E}(X - a|X \gt a) & \quad \text{ by linearity} \\
                          &= a + \frac{1}{\lambda} & \quad \text{ by the memorylessness property}
\end{align}

----

View [Lecture 16: Exponential Distribution | Statistics 110](http://bit.ly/2CxRIfh) on YouTube.


# From straight beams to frames

A frame is obtained by assembling several straight beams with different orientations.
Different from the case of classes, the beams are clamped one to each other (and not simply joined).

The straight beam model illustrated in the `01-LinearBeam` notebook should be extended by including the possible different orientation of each element of the frame. This requires to consider a local and a global coordinate system.

The file `frame.py` contains the class `LinearFrame` proving a basic implementation of the matrix model of a linear frame. We will use it in this notebook by importing the content of the `frame.py` file (which consitutes a `python` module).

```python
from frame import *
```

```python
%matplotlib inline
from sympy.interactive import printing
printing.init_printing()
```

## Straight beam

Let us first consider a straight beam, and show how we can use the `LinearFrame` class to solve a beam problem discretised into several elements, as in the `01-LinearBeam` notebook.

We can define the frame as a set of nodes and elements, and start by plotting it

```python
nodes = np.array([[ 0.  ,  0.  ],
       [ 0.25,  0.  ],
       [ 0.5 ,  0.  ],
       [ 0.75,  0.  ],
       [ 1.  ,  0.  ]])

elements = np.array([[0, 1],
       [1, 2],
       [2, 3],
       [3, 4]])

frame = LinearFrame(nodes,elements)

frame.plot_with_label()
```

We set the stiffness, loading (distributed force per unit line), bcs

```python
ne = frame.nelements
ndof = frame.ndof
EI = np.ones(ne)*1.
ES = np.ones(ne)*100.
f_x = 0*np.ones(4)
f_y = -10*np.ones(4)
frame.set_distributed_loads(f_x, f_y)
frame.set_stiffness(EI, ES)
blocked_dof = np.array([0, 1, 2, ndof-3, ndof-2, ndof-1])
bc_values = np.array([0, 0, 0, 0, 0, 0])
```

Hence, we use the dedicated functions to assemble the linear system and impose bcs

```python
F = frame.assemble_F()
K = frame.assemble_K()
Kbc, Fbc = frame.bc_apply(K,F,blocked_dof,bc_values)
```

Finally we solve and represent the solution

```python
Usol = np.linalg.solve(Kbc,Fbc)
frame.set_displacement(Usol)
#frame.plot_with_label()
frame.plot_displaced()
```

## A simple frame

```python
nodes = np.array([[ 0.  ,  0.  ],
                    [ 0.,  1.  ],
                    [ 1.,  1.  ]
                 ])
elements = np.array([[0, 1],[1,2]])

frame = LinearFrame(nodes,elements)

frame.plot_with_label()
```

```python
ne = frame.nelements
# inputs
EI = np.ones(ne)*1.
ES = np.ones(ne)*100.
f_y = np.array([0.,0.])
f_x = np.array([1.,0.])
blocked_dof = np.array([0, 1, 2])
bc_values = np.array([0, 0, 0])
frame.set_stiffness(EI, ES)
frame.set_distributed_loads(f_x, -.5*f_y)
# Assemble
F = frame.assemble_F()
K = frame.assemble_K()
# Impose bcs
Kbc, Fbc = frame.bc_apply(K,F,blocked_dof,bc_values)
# Solve
Usol = np.linalg.solve(Kbc,Fbc)
# Plot
print(Usol)
frame.set_displacement(Usol)
frame.plot_with_label()
frame.plot_displaced()
```

We can note the use of the rotation from the local to global coordinate systems in the different steps inside the program (e.g. assembling).
The rotation matrix allowing to pass a vector (e.g. local displacement) from the local to the global coordinate system is

```python
alpha = sp.Symbol('alpha')
frame.rotation_matrix(alpha)
```

where $\alpha$ is the orientation angle of an element

## Exercise 1

Use the program to solve this problem
$$
E = 10^4 MPa, \quad I = 10^3  mm^4,\quad S = 10 mm^2\quad L = 100 mm\quad P= 10N\quad w=0.24N/mm
$$
You can verify against the solution provided below

## Exercice 2

The Young modulus of the silicon is approximately $E=1300$ KPa.
The cross section of the beams have a height $h=7.5$ mm and a width $b=20$ mm.
The total width of the frame is $55$ mm.
The length of the upper posts is $60$ mm.
The length of the lower posts is $45$ mm.
The frame is fixed in the soil.

The mass density is $\rho = 1100 kg/m^3$

Can you predict what is the mass that is hanged in the middle of the upper beam?

## Exercice 3

* Implement functions to extract the value of the normal force in each element

* Implement functions to plot N,T,M in each element

## References

You find below few references freely available online as pdfs:

* http://joegattas.com/wp-content/uploads/2015/08/3340-Lectures-7-to-12-S1-2015.pdf
* http://www.facweb.iitkgp.ernet.in/~baidurya/CE21004/online_lecture_notes/m4l30.pdf
* http://people.duke.edu/~hpgavin/cee421/
* http://people.duke.edu/~hpgavin/cee421/frame-element.pdf


# 1D Simple Harmonic Oscillator

## Imports

```python
from IPython.display import display, display_pretty
```

```python
from sympy import init_printing
init_printing(use_latex=True)
```

```python
from sympy import *
from sympy.physics.quantum import *
from sympy.physics.quantum.sho1d import *
from sympy.physics.quantum.tests.test_sho1d import *
```

## Printing Of Operators

Create a raising and lowering operator and make sure they print correctly

```python
ad = RaisingOp('a')
ad
```

```python
a = LoweringOp('a')
a
```

```python
print(latex(ad))
print(latex(a))
```

    a^{\dag}
    a

```python
display_pretty(ad)
display_pretty(a)
```

     †
    a

    a

```python
print(srepr(ad))
print(srepr(a))
```

    RaisingOp(Symbol('a'))
    LoweringOp(Symbol('a'))

```python
print(repr(ad))
print(repr(a))
```

    RaisingOp(a)
    a

## Printing of States

Create a simple harmonic state and check its printing

```python
k = SHOKet('k')
k
```

```python
b = SHOBra('b')
b
```

```python
print(pretty(k))
print(pretty(b))
```

    ❘k⟩
    ⟨b❘

```python
print(latex(k))
print(latex(b))
```

    {\left|k\right\rangle }
    {\left\langle b\right|}

```python
print(srepr(k))
print(srepr(b))
```

    SHOKet(Symbol('k'))
    SHOBra(Symbol('b'))

## Properties

Take the dagger of the raising and lowering operators. They should return each other:

```python
Dagger(ad)
```

```python
Dagger(a)
```

Check commutators of the raising and lowering operators

```python
Commutator(ad,a).doit()
```

```python
Commutator(a,ad).doit()
```

Take a look at the dual states of the bra and ket

```python
k.dual
```

```python
b.dual
```

Taking the inner product of the bra and ket will return the Kronecker delta function

```python
InnerProduct(b,k).doit()
```

Take a look at how the raising and lowering operators act on states. We use qapply to apply an operator to a state

```python
qapply(ad*k)
```

```python
qapply(a*k)
```

But the states may have an explicit energy level. Let's look at the ground and first excited states

```python
kg = SHOKet(0)
kf = SHOKet(1)
```

```python
qapply(ad*kg)
```

```python
qapply(ad*kf)
```

```python
qapply(a*kg)
```

```python
qapply(a*kf)
```

## Number operator and Hamiltonian

Let's look at the number operator and Hamiltonian operator:

```python
k = SHOKet('k')
ad = RaisingOp('a')
a = LoweringOp('a')
N = NumberOp('N')
H = Hamiltonian('H')
```

The number operator is simply expressed as `ad*a`:

```python
N.rewrite('a').doit()
```

The number operator expressed in terms of the position and momentum operators:

```python
N.rewrite('xp').doit()
```

It can also be expressed in terms of the Hamiltonian operator:

```python
N.rewrite('H').doit()
```

The Hamiltonian operator can be expressed in terms of the raising and lowering operators, position and momentum operators, and the number operator:

```python
H.rewrite('a').doit()
```

```python
H.rewrite('xp').doit()
```

```python
H.rewrite('N').doit()
```

The raising and lowering operators can also be expressed in terms of the position and momentum operators

```python
ad.rewrite('xp').doit()
```

```python
a.rewrite('xp').doit()
```

### Properties

Let's take a look at how the number operator and Hamiltonian act on states:

```python
qapply(N*k)
```

Apply the number operator to a state returns the state times the ket:

```python
ks = SHOKet(2)
qapply(N*ks)
```

```python
qapply(H*k)
```

Let's see how the operators commute with each other:

```python
Commutator(N,ad).doit()
```

```python
Commutator(N,a).doit()
```

```python
Commutator(N,H).doit()
```

## Representation

We can express the operators in number operator basis. There are different ways to create a matrix in Python, we will use 3 different ways.

Sympy:

```python
represent(ad, basis=N, ndim=4, format='sympy')
```

Numpy:

```python
represent(ad, basis=N, ndim=5, format='numpy')
```

    array([[ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ],
           [ 1.        ,  0.        ,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  1.41421356,  0.        ,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  1.73205081,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  0.        ,  2.        ,  0.        ]])

`scipy.sparse`:

```python
sparse_rep = represent(ad, basis=N, ndim=4, format='scipy.sparse', spmatrix='lil')
sparse_rep
```

    <4x4 sparse matrix of type '<class 'numpy.float64'>'
    	with 3 stored elements in Compressed Sparse Row format>

```python
print(sparse_rep)
```

      (1, 0)	1.0
      (2, 1)	1.41421356237
      (3, 2)	1.73205080757

The same can be done for the other operators

```python
represent(a, basis=N, ndim=4, format='sympy')
```

```python
represent(N, basis=N, ndim=4, format='sympy')
```

```python
represent(H, basis=N, ndim=4, format='sympy')
```

Bras and kets can also be represented:

```python
k0 = SHOKet(0)
k1 = SHOKet(1)
b0 = SHOBra(0)
b1 = SHOBra(1)
```

```python
represent(k0, basis=N, ndim=5, format='sympy')
```

```python
represent(k1, basis=N, ndim=5, format='sympy')
```

```python
represent(b0, basis=N, ndim=5, format='sympy')
```

```python
represent(b1, basis=N, ndim=5, format='sympy')
```


```python
from sympy import *
init_printing()
```

# 1. Símbolos (*Symbols*)

Variables de *Python* y *SymPy*

```python
a = 'stuff' # variable Python
```

La función `symbols` crea una variable *SymPy* `x` que es asignada a la variable Python `x`.

```python
x = symbols('x')
x
```

Lo siguiente funciona también, pero no se recomienda. La variable Python `crazy` se asigna a la variable *Sympy* `unrelated`. Por lo tanto, siempre es una buena idea nombrar sus variables igual que sus símbolos.

```python
crazy = symbols('unrelated')
crazy
```

# 2. Inmutabilidad (*Immutability*)

```python
x, y = symbols('x y')
a = x + y + 1
a
```

```python
x = 3
```

¿Qué es "` a` "ahora?

```python
a
```

Ahora, ¿cómo puedo sustituir x por 3?

```python
??
```

¿Qué es "` a` "ahora?

```python
a
```

Las expresiones de *SymPy* son inmutables. Nunca cambian en el lugar. Por lo tanto, cada vez que haces `subs`, `simplify`, etc. *SymPy* crea una nueva expresión.

# 3. Signo de igual

Comprobando la desigualdad matemática.

```python
x = symbols('x')
a = (x + 1)**2
a
```

```python
b = x**2 + 2*x + 1
```

Ahora, vamos a verificar si `a` y` b` son iguales o no. ¿Cuál crees que será el resultado de lo siguiente?

```python
a == b
```

```python
simplify(a - b)
```

Esa es la diferencia entre la igualdad estructural y matemática.

#### Representando una igualdad simbólica, como por ejemplo:
$$x^2 = y$$

```python
eq = Eq(x**2, y)
```

```python
eq.lhs
```

```python
eq.rhs
```

```python
solveset(eq, x)
```

# 4. Fracciones

```python
acos(1/2)
```

Estamos buscando resultados simbólicos, ¿verdad?

```python
Rational(1, 2)
```

```python
acos(Rational(1, 2))
```

```python
from fractions import Fraction
Fraction(1, 2)
```

```python
acos(Fraction(1, 2))
```

```python
type(sympify(Fraction(1, 2)))
```

# 5. Operador Lógico (XOR)

```python
x^2
```

```python
sympify('x^2')
```

Si estás haciendo lógica, y **no** quieres convertir el `XOR` al operador de potencia:

```python
sympify('x^2', convert_xor=False)
```