# Realization of Non-Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Quantization Effects

Numbers and numerical operations are represented with a finite numerical resolution in digital processors. The same holds for the amplitude values of signals and the algorithmic operations applied to them. Hence, the intended characteristics of an digital filter may deviate in practice due to the finite numerical resolution. The [double-precision floating point representation](https://en.wikipedia.org/wiki/Double-precision_floating-point_format) used in numerical environments like MATLAB or Python/numpy is assumed to be quasi-continuous. This representation serves therefore as reference for the evaluation of quantization effects.

This section investigates the consequences of quantization in non-recursive filters. We first take a look on the quantization of the filter coefficients, followed by the effects caused by a finite numerical resolution on the operations. The realization of non-recursive filters is subject to both effects.

### Quantization of Filter Coefficients

The output signal $y[k]$ of a non-recursive filter with a finite impulse response (FIR) $h[k]$ of length $N$ is given as

\begin{equation}
y[k] = h[k] * x[k] = \sum_{\kappa = 0}^{N-1} h[\kappa] \; x[k - \kappa]
\end{equation}

where $x[k]$ denotes the input signal. The quantized impulse response $h_Q[k]$ (quantized filter coefficients) is yielded by quantizing the impulse response $h[k]$

\begin{equation}
h_Q[k] = \mathcal{Q} \{ h[k] \} = h[k] + e[k]
\end{equation}

where $e[k] = h_Q[k] - h[k]$ denotes the [quantization error](../quantization/introduction.ipynb#Model-of-the-Quantization-Process). Introducing $h_Q[k]$ into above equation and rearranging results in

\begin{equation}
y_Q[k] = \sum_{\kappa = 0}^{N-1} h[\kappa] \; x[k - \kappa] + \sum_{\kappa = 0}^{N-1} e[\kappa] \; x[k - \kappa]
\end{equation}

The input signal $x[k]$ is filtered by the quantization noise $e[k]$ and superimposed to the desired output of the filter. The overall transfer function $H_Q(e^{j \Omega})$ of the filter with quantized filter coefficients is given as

\begin{equation}
H_Q(e^{j \Omega}) = \sum_{k=0}^{N-1} h[k] \; e^{-j \Omega k} + \sum_{k=0}^{N-1} e[k] \; e^{-j \Omega k}  = H(e^{j \Omega}) + E(e^{j \Omega})
\end{equation}

Hence, the quantization of filter coefficients results in a linear distortion of the desired frequency response. To some extend this distortion can be incorporated into the design of the filter. However, the magnitude of the quantization error $| E(e^{j \Omega}) |$ cannot get arbitrarily small for a finite quantization step $Q$. This limits the achievable attenuation of a digital filter with quantized coefficients. It is therefore important to normalize the filter coefficients $h[k]$ before quantization in order to keep the relative power of the quantization noise small.

#### Example

The coefficients of a digital lowpass filter with a cutoff frequency of $\Omega_0 = \frac{\pi}{2}$ are quantized in the following example.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

w = 8  # wordlength of quantized coefficients
A = 1  # attenuation of filter coefficients
N = 256  # number of coefficients for filter
Q = 1/(2**(w-1))  # quantization stepsize

def uniform_midtread_quantizer(x, Q):
    # limiter
    x = np.copy(x)
    idx = np.where(x <= -1)
    x[idx] = -1
    idx = np.where(x > 1 - Q)
    x[idx] = 1 - Q
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

# design lowpass
h = A * sig.firwin(N, .5)
# quantize coefficients
hQ = uniform_midtread_quantizer(h, Q)

# plot frequency response
Om, H = sig.freqz(h)
Om, HQ = sig.freqz(hQ)
Om, E = sig.freqz(hQ-h)

plt.figure(figsize=(10, 4))
plt.plot(Om, 20*np.log10(np.abs(H)), label=r'$| H(e^{j \Omega}) |$ in dB (Designed)')
plt.plot(Om, 20*np.log10(np.abs(HQ)), label=r'$| H_Q(e^{j \Omega}) |$ in dB (Quantized coefficients)')
plt.title('Magnitude of the transfer function w and w/o quantization of coefficients')
plt.xlabel(r'$\Omega$')
plt.axis([0, np.pi, -130, 10])
plt.legend(loc=3)
plt.grid()
```

**Exercise**

* Change the wordlength `w` of the quantized filter coefficients. How does the magnitude response $| H_Q(e^{j \Omega}) |$ of the quantized filter change?
* Change the attenuation `A` of the filter coefficients. What changes?
* Why does the magnitude response of the quantized filter $| H_Q(e^{j \Omega}) |$ deviate more from the magnitude response of the designed filter $| H(e^{j \Omega}) |$ in the frequency ranges with high attenuation?

### Quantization of Signals and Operations

Besides the quantization of filter coefficients $h[k]$, also the quantization of the signals, state variables and operations has to be considered in a practical implementation of filters. The computation of the output signal $y[k] = h[k] * x[k]$ of a non-recursive filter by the convolution involves multiplications and additions. In digital signal processors numbers are often represented in [fixed-point arithmetic](https://en.wikipedia.org/wiki/Fixed-point_arithmetic) using [two's complement](https://en.wikipedia.org/wiki/Two's_complement). When multiplying two numbers with a wordlength of $w$-bits in this representation the result would require $2w$-bits. Hence the result has to be requantized to $w$-bits. The rounding operation in the quantizer is often realized as truncation of the $w$ least significant bits. The resulting quantization error is known as [round-off error](https://en.wikipedia.org/wiki/Round-off_error). The addition of two numbers may fall outside the maximum/minimum values of the representation and may suffer from clipping. Similar considerations hold also for other number representations, like e.g. [floating point](https://en.wikipedia.org/wiki/Floating_point).

As for the [quantization noise](../quantization/linear_uniform_quantization_error.ipynb#Model-for-the-Quantization-Error), a statistical model for the round-off error in multipliers is used to quantify the average impact of round-off noise in a non-recursive filter.

#### Model for round-off errors in multipliers

As outlined above, multipliers require a requantization of the result in order to keep the wordlength constant. The multiplication of a quantized signal $x_Q[k]$ with a quantized factor $a_Q$ can be written as

\begin{equation}
y_Q[k] = \mathcal{Q} \{ a_Q \cdot x_Q[k] \} = a_Q \cdot x_Q[k] + e[k]
\end{equation}

where the round-off error $e[k]$ is defined as

\begin{equation}
e[k] = y_Q[k] - a_Q \cdot x_Q[k]
\end{equation}

This leads to the following model of a multiplier including round-off effects

The round-off error can be modeled statistically in the same way as quantization noise [[Zölzer](../index.ipynb#Literature)]. Under the assumption that the average magnitude of $a_Q \cdot x_Q[k]$ is much larger that the quantization step size $Q$, the round-off error $e[k]$ can be approximated by the following statistical model

1. The round-off error $e[k]$ is not correlated with the input signal $x_Q[k]$

2. The round-off error is [white](../random_signals/white_noise.ipynb)

    $$ \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sigma_e^2 $$

2. The probability density function (PDF) of the round-off error is given by the zero-mean [uniform distribution](../random_signals/important_distributions.ipynb#Uniform-Distribution)

    $$ p_e(\theta) = \frac{1}{Q} \cdot \text{rect} \left( \frac{\theta}{Q} \right) $$

The variance (power) of the round-off error is [derived from its PDF](../random_signals/important_distributions.ipynb#Uniform-Distribution) as

\begin{equation}
\sigma_e^2 = \frac{Q^2}{12}
\end{equation}

#### Round-off noise

Using above model of a multiplier and discarding clipping, a straightforward realization of the convolution with quantized signals would be to requantize after every multiplication

\begin{equation}
y_Q[k] = \sum_{\kappa = 0}^{N-1} \mathcal{Q} \{ h_Q[\kappa] \; x_Q[k - \kappa] \} = \sum_{\kappa = 0}^{N-1} h_Q[\kappa] \; x_Q[k - \kappa] + e[\kappa]
\end{equation}

The round-off errors for each multiplication are uncorrelated to each other. The overall power of the round-off error is then given as

\begin{equation}
\sigma_e^2 = N \cdot \frac{Q^2}{12}
\end{equation}

Many digital signal processors allow to perform the multiplications and additions in an internal register with double wordlength. In this case only the result has to be requantized

\begin{equation}
y_Q[k] = \mathcal{Q} \left\{ \sum_{\kappa = 0}^{N-1}  h_Q[\kappa] \; x_Q[k - \kappa] \right\}
\end{equation}

and the power of the round-off noise in this case is

\begin{equation}
\sigma_e^2 =  \frac{Q^2}{12}
\end{equation}

It is evident that this realization is favorable due to the lower round-off noise, especially for filters with a large number $N$ of coefficients.

#### Example

The following example simulates the round-off noise of a non-recursive filter when requantization is performed after each multiplication. Clipping is not considered. The input signal $x[k]$ is drawn from a [uniform distribution](../random_signals/important_distributions.ipynb#Uniform-Distribution) with $a=-1$ and $b=1-Q$. Both the input signal and filter coefficients are quantized. The output signal $y[k]$ without requantization of the multiplications is computed, as well as the output signal $y_Q[k]$ with requantization. The statistical properties of the round-off noise $e[k] = y_Q[k] - y[k]$ are evaluated.

```python
w = 16  # wordlength of quantized coefficients/operations
N = 32  # number of coefficients for filter
L = 8192  # length of input signal
Q = 1/(2**(w-1))  # quantization stepsize

def uniform_midtread_quantizer(x, Q):
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

# random impulse response
h = np.random.uniform(size=N, low=-1, high=1)
hQ = uniform_midtread_quantizer(h, Q)
# input signal
x = np.random.uniform(size=L, low=-1, high=1-Q)
xQ = uniform_midtread_quantizer(x, Q)
# output signal by convolution
y = np.zeros(L+N-1)
yQ = np.zeros(L+N-1)
for k in np.arange(L):
    for kappa in np.arange(N):
        if (k-kappa) >= 0:
            y[k] += hQ[kappa] * xQ[k-kappa]
            yQ[k] += uniform_midtread_quantizer(hQ[kappa] * xQ[k-kappa], Q)

# overall round-off error
e = yQ - y

# estimate power of round-off error
sx = 10*np.log10(np.var(e))
print('Power of overall round-off noise is %f dB' %sx)
# estimated PDF of round-off error
pe, bins = np.histogram(e, bins=50, normed=True, range=(-10*Q, 10*Q))
# estimate PSD of round-off error
nf, Pee = sig.welch(e, nperseg=128)

# plot statistical properties of error signal
plt.figure(figsize=(10,6))

plt.subplot(121)
plt.bar(bins[:-1]/Q, pe*Q, width = 20/len(pe))
plt.title('Estimated PDF of the round-off noise')
plt.xlabel(r'$\theta / Q$')
plt.ylabel(r'$\hat{p}_x(\theta) / Q$')
#plt.axis([-1, 1, 0, 1.2])

plt.subplot(122)
plt.plot(nf*2*np.pi, Pee*6/Q**2/N)
plt.title('Estimated PSD of the round-off noise')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\hat{\Phi}_{ee}(e^{j \Omega}) / \sigma_e^2$')
plt.axis([0, np.pi, 0, 2])
plt.grid();
```

**Exercise**

* Change the wordlength `w` and check if the $\sigma_e^2$ derived by numerical simulation is equal to its theoretic value derived above?
* Can you explain the shape of the estimated PDF for the round-off noise?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# Calculating the time dependent solution for $\eta$ and $\theta$ domain
\begin{equation}\label{p1introhpa} \Psi _{\eta \eta \tau }  = \frac{1}{{R_s }}\Psi _{\eta \eta \eta \eta }  + \Psi _{\eta \eta \eta } \Psi _\theta   - \Psi _{\eta \eta \theta } \Psi _\eta\\
-1 \le \eta \le1, \quad 0 \le \theta \le \pi
\end{equation}

The numerical solution found in this notebook is only stable for $0 < R_s \lesssim 55$

```julia
using DifferentialEquations
using Plots
using BenchmarkTools
using DelimitedFiles
using LaTeXStrings
using SparseArrays
using LinearAlgebra
```

In this code the vector $\Psi_j$ is used to store the solution at the current time step and $\Psi_{jnext}$ is psi value at the next time step

```julia
pyplot()
function Ψ_smallr(θ::Float64, η::Float64 ,r::Float64)
    return(-η*(η^2-1)*sin(θ)/2.0 + r*η*(η^2-1)^2.0*(2.0+η^2)*sin(2*θ)/560.0 +
        r^2.0*η*(η^2-1)^2*((-591 -2294*η^2 + 161*η^4 + 1428*η^6)*sin(θ) + 3.0*(423 + 166*η^2 - 553*η^4 - 84*η^6)sin(3*θ))/62092800)
end

function Ψ_smallr_d1(θ::Float64, η::Float64 ,r::Float64)
    return(-(3.0*η^2-1)*sin(θ)/2.0 + r*(7*η^6-9*η^2+2)*sin(2*θ)/560.0 +
        r^2.0*(3*(-197 - 1112*η^2 + 6930*η^4 - 2772*η^6 - 8085*η^8 + 5236*η^10)*sin(θ) + (1269 - 6120*η^2 - 6930*η^4 + 24948*η^6 - 10395*η^8 - 2772*η^10)*sin(3*θ))/62092800.0)
end

function  Ψ_smallr_d2(θ::Float64, η::Float64 ,r::Float64)
    return(-3.0*η*sin(θ) + r*(42*η^5-18*η)*sin(2*θ)/560.0 +
        r^2.0*(3*( - 2224*η + 27720*η^3 - 16632*η^5 - 64680*η^7 + 52360*η^9)*sin(θ) + (- 12240*η - 27720*η^3 + 149688*η^5 - 83160*η^7 - 27720*η^9)*sin(3*θ))/62092800.0)
end

function  Ψ_smallr_d3(θ::Float64, η::Float64 ,r::Float64)
    return(-3.0*sin(θ) + r*(210*η^4-18)*sin(2*θ)/560.0 +
        r^2.0*(3*( - 2224 + 83160*η^2 - 83160*η^4 - 452760*η^6 + 471240*η^8)*sin(θ) + (- 12240 - 83160*η^2 + 748440*η^4 - 582120*η^6 - 249480*η^8)*sin(3*θ))/62092800.0)
end

function Ψ_bigr(θ::Float64, η::Float64)
    return(sin(pi*η)*sin(θ)/pi)
end

function Ψ_bigr_d1(θ::Float64, η::Float64)
    return(cos(pi*η)*sin(θ))
end

function  Ψ_bigr_d2(θ::Float64, η::Float64)
    return(-sin(pi*η)*sin(θ)*pi)
end

function  Ψ_bigr_d3(θ::Float64, η::Float64)
     return(-cos(pi*η)*sin(θ)*pi^2)
end

```

    Ψ_bigr_d3 (generic function with 1 method)

```julia
#Change these to constants after
const R = Float64(50.0) # Reynolds number
const N = 200  # Total number of intervals (steps-1) in the eta direction
const M = 256  # Total number of intervals (steps-1) theta direction
const J = 5000    # Total number of time steps
ψ_current = zeros(N+1,M+1)
ψ_next =zeros(N+1,M+1)
ψ_j = ψ_current
ψ_jnext = ψ_next
const Δη = 2.0/N
const Δθ = pi/M
const Δt = 0.001
const α = 0
const c = Δt/Δη^2
const d3 =      -1.0/12.0 - c/R
const d2 =      4.0/3.0   + 4*c/R
const d1 =     -5.0/2.0   -6*c/R

A =spdiagm(-2=>ones(N-3)*d3 ,-1 => ones(N-2)*d2, 0 => ones(N-1)*d1, 1 => ones(N-2)*d2,2 => ones(N-3)*d3)

A[1,1] = d1 + d3 #Boundary condition psi@i=2 = 0, psi_eta@i=2 = -sin(theta) add -2d3\delta\eta\sine(theta) to RHS
A[end,end] = d1 + d3 #Boundary condition psi@i=N+1 = 0, psi_eta@i=N+1 = -sin(theta) add 2d3\delta\eta\sine(theta) to RHS
LUA = lu(A)
varinfo(r"A")
```

    WARNING: redefinition of constant R. This may fail, cause incorrect answers, or produce other errors.
    WARNING: redefinition of constant d3. This may fail, cause incorrect answers, or produce other errors.
    WARNING: redefinition of constant d2. This may fail, cause incorrect answers, or produce other errors.
    WARNING: redefinition of constant d1. This may fail, cause incorrect answers, or produce other errors.

| name                    |       size | summary                                       |
|:----------------------- | ----------:|:--------------------------------------------- |
| A                       | 17.172 KiB | 199×199 SparseMatrixCSC{Float64, Int64}       |
| LUA                     | 17.195 KiB | SuiteSparse.UMFPACK.UmfpackLU{Float64, Int64} |
| matrix_A_at_boundaries! |    0 bytes | typeof(matrix_A_at_boundaries!)               |
| matrix_A_for_coreflow!  |    0 bytes | typeof(matrix_A_for_coreflow!)                |

```julia
ψ_current[:,1] .=0.0
ψ_current[:,M+1] .=0.0
ψ_current[1,:] .= 0.0
ψ_current[N+1,:] .= 0.0
for n in 2:N
    for m in 2:M
        ψ_current[n,m] = Ψ_smallr(Δθ*(m-1),-1.0+Δη*(n-1),R)
        #ψ_current[n,m] = 0
    end
end
ψ_current
#n=1 is ficticous point at eta=-1-deltaeta, n=2 is at eta=-1 and n=N+2 and n=N+3 is ficticious at eta=1+deltaeta

```

    201×257 Matrix{Float64}:
     0.0   0.0           0.0          …   0.0           0.0          0.0
     0.0  -0.000123138  -0.000246257     -0.00023602   -0.000118018  0.0
     0.0  -0.000247023  -0.000494005     -0.000454142  -0.000227086  0.0
     0.0  -0.000371501  -0.000742936     -0.000655632  -0.000327836  0.0
     0.0  -0.000496412  -0.000992731     -0.000841677  -0.000420862  0.0
     0.0  -0.00062159   -0.00124306   …  -0.00101338   -0.000506719  0.0
     0.0  -0.000746868  -0.00149358      -0.00117179   -0.000585923  0.0
     0.0  -0.000872073  -0.00174396      -0.00131786   -0.000658959  0.0
     0.0  -0.00099703   -0.00199383      -0.00145249   -0.000726276  0.0
     0.0  -0.00112156   -0.00224286      -0.00157652   -0.000788292  0.0
     0.0  -0.00124549   -0.00249068   …  -0.00169073   -0.000845397  0.0
     0.0  -0.00136865   -0.00273694      -0.00179584   -0.000897952  0.0
     0.0  -0.00149084   -0.00298129      -0.00189252   -0.000946291  0.0
     ⋮                                ⋱                 ⋮
     0.0   0.00136865    0.00273694       0.00179584    0.000897952  0.0
     0.0   0.00124549    0.00249068   …   0.00169073    0.000845397  0.0
     0.0   0.00112156    0.00224286       0.00157652    0.000788292  0.0
     0.0   0.00099703    0.00199383       0.00145249    0.000726276  0.0
     0.0   0.000872073   0.00174396       0.00131786    0.000658959  0.0
     0.0   0.000746868   0.00149358       0.00117179    0.000585923  0.0
     0.0   0.00062159    0.00124306   …   0.00101338    0.000506719  0.0
     0.0   0.000496412   0.000992731      0.000841677   0.000420862  0.0
     0.0   0.000371501   0.000742936      0.000655632   0.000327836  0.0
     0.0   0.000247023   0.000494005      0.000454142   0.000227086  0.0
     0.0   0.000123138   0.000246257      0.00023602    0.000118018  0.0
     0.0   0.0           0.0          …   0.0           0.0          0.0

```julia
#wireframe(range(0,stop = pi,length = M+1),range(-1,stop = 1,length = N+1),ψ_j)

```

```julia
function make_ψ_jnext_on_boundaries!(ψ_j, ψ_jnext, parameters)
    (Δθ, Δη, R, N, M, rc, rd1, rd2, rd3, d1, d2, d3, A, LUA) = parameters
    for m = 2:M-2:M #For the first theta profile and last profile use second order difference formula for theta near boundaries

    #at n=1 psi[1,:] = 0 and we are only calculating the interior points from n=2 to n=N
    n=2
    bc=2*Δη*sin(Δθ*(m-1))
    bcp1=2*Δη*sin(Δθ*m)
    bcm1=2*Δη*sin(Δθ*(m-2))
    rhs = @view ψ_jnext[2:N,m]

    #Remember to fix the RHS so that the bc terms are still included

    rhs[1] = rd3*(ψ_j[2,m]+bc)    +    rd2*ψ_j[n-1,m] + rd1*ψ_j[n,m]+rd2*ψ_j[n+1,m] + rd3*ψ_j[n+2,m]+
                 rc*(-bc/2)*(-ψ_j[n,m-1]/2 + ψ_j[n,m+1]/2)-
                 rc*(-(-(ψ_j[2,m-1]+bcm1)/12 +   4*ψ_j[n-1,m-1]/3     -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/2+
                      (-(ψ_j[2,m+1]+bcp1)/12 +   4*ψ_j[n-1,m+1]/3     -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/2)*
                       (bc/12)- d3*bc

        for n = 3:N-1
            rhs[n-1] = rd3*ψ_j[n-2,m]    +    rd2*ψ_j[n-1,m] + rd1*ψ_j[n,m]+rd2*ψ_j[n+1,m] + rd3*ψ_j[n+2,m]
        end#End of the n loop

    #at n=N psi[N+1,:] = 0
    n=N
    rhs[N-1] = rd3*ψ_j[n-2,m]    +    rd2*ψ_j[n-1,m] + rd1*ψ_j[n,m]+rd2*ψ_j[n+1,m] + rd3*(ψ_j[N,m]-bc)+
                 rc*(-bc/2)*(-ψ_j[n,m-1]/2 + ψ_j[n,m+1]/2)-
                 rc*(-(-ψ_j[n-2,m-1]/12 + 4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -(ψ_j[N,m-1]-bcm1)/12)/2+
                      (-ψ_j[n-2,m+1]/12 + 4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -(ψ_j[N,m+1]-bcp1)/12)/2)*
                    (bc/12) + d3*bc

    matrix_A_at_boundaries!(ψ_j, m, parameters)
    lu!(LUA,A)
    ldiv!(LUA,rhs)
    end#End of the m loop
end

function make_ψ_jnext_in_core_region!(ψ_j, ψ_jnext, parameters)
    (Δθ, Δη, R, N, M, rc, rd1, rd2, rd3, d1, d2, d3, A, LUA) = parameters
    for m = 3:M-1

    #at n=1 psi[1,:] = 0 and we are only calculating the interior points from n=2 to n=N
    n=2
    bc=2*Δη*sin(Δθ*(m-1))
    bcp1=2*Δη*sin(Δθ*m)
    bcp2=2*Δη*sin(Δθ*(m+1))
    bcm1=2*Δη*sin(Δθ*(m-2))
    bcm2=2*Δη*sin(Δθ*(m-3))

    rhs = @view ψ_jnext[2:N,m]

    #Remember to fix the RHS so that the bc terms are still included

    rhs[1] = rd3*(ψ_j[2,m]+bc)    +    rd2*ψ_j[n-1,m] + rd1*ψ_j[n,m]+rd2*ψ_j[n+1,m] + rd3*ψ_j[n+2,m]+
             rc*(-bc/2)*( ψ_j[n,m-2]/12 -  2*ψ_j[n,m-1]/3  +2*ψ_j[n,m+1]/3 - ψ_j[n,m+2]/12)-
             rc*((-(ψ_j[2,m-2]+bcm2)/12 +  4*ψ_j[n-1,m-2]/3 -5*ψ_j[n,m-2]/2 + 4*ψ_j[n+1,m-2]/3 -ψ_j[n+2,m-2]/12)/12-
               2*(-(ψ_j[2,m-1]+bcm1)/12 +  4*ψ_j[n-1,m-1]/3 -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/3+
               2*(-(ψ_j[2,m+1]+bcp1)/12 +  4*ψ_j[n-1,m+1]/3 -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/3-
                 (-(ψ_j[2,m+2]+bcp2)/12 +  4*ψ_j[n-1,m+2]/3 -5*ψ_j[n,m+2]/2 + 4*ψ_j[n+1,m+2]/3 -ψ_j[n+2,m+2]/12)/12)*
                (bc/12) - d3*bc

        for n = 3:N-1
            rhs[n-1] = rd3*ψ_j[n-2,m] + rd2*ψ_j[n-1,m] + rd1*ψ_j[n,m]+rd2*ψ_j[n+1,m] + rd3*ψ_j[n+2,m]
        end#End of the n loop

    #at n=N psi[N+1,:] = 0
    n=N
    rhs[N-1] = rd3*ψ_j[n-2,m]    +    rd2*ψ_j[n-1,m] + rd1*ψ_j[n,m]+rd2*ψ_j[n+1,m] + rd3*(ψ_j[N,m]-bc)+
            rc*(-bc/2)*( ψ_j[n,m-2]/12 -  2*ψ_j[n,m-1]/3  +2*ψ_j[n,m+1]/3 - ψ_j[n,m+2]/12)-
            rc*((-ψ_j[n-2,m-2]/12 + 4*ψ_j[n-1,m-2]/3 -5*ψ_j[n,m-2]/2 + 4*ψ_j[n+1,m-2]/3 -(ψ_j[N,m-2]-bcm2)/12)/12-
              2*(-ψ_j[n-2,m-1]/12 + 4*ψ_j[n-1,m-1]/3 -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -(ψ_j[N,m-1]-bcm1)/12)/3+
              2*(-ψ_j[n-2,m+1]/12 + 4*ψ_j[n-1,m+1]/3 -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -(ψ_j[N,m+1]-bcp1)/12)/3-
                (-ψ_j[n-2,m+2]/12 + 4*ψ_j[n-1,m+2]/3 -5*ψ_j[n,m+2]/2 + 4*ψ_j[n+1,m+2]/3 -(ψ_j[N,m+2]-bcp2)/12)/12)*
                (bc/12) + d3*bc

    matrix_A_for_coreflow!(ψ_j, m, parameters)
    lu!(LUA,A)
    ldiv!(LUA,rhs)
    end#End of the m loop
end

function matrix_A_at_boundaries!( ψ_j, m, parameters)
    (Δθ, Δη, R, N, M, rc, rd1, rd2, rd3, d1, d2, d3, A, LUA) = parameters
    #This code should only be used to calculate matrix A for m= 2 or m = M because one sided theta derivatives are used at the boundaries
    bc=2*Δη*sin(Δθ*(m-1))
    bcp1=2*Δη*sin(Δθ*m)
    bcm1=2*Δη*sin(Δθ*(m-2))

    n=2
    ψ_θ = -ψ_j[n,m-1]/2 + ψ_j[n,m+1]/2
    ψ_ηηθ =  -(-(ψ_j[2,m-1]+bcm1)/12 +   4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/2+
              (-(ψ_j[2,m+1]+bcp1)/12 +   4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/2

    A[1,1]= d1+d3 -rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
    A[1,2]= d2    -rc*(-ψ_θ   -2*ψ_ηηθ/3)#-ve
    A[1,3]= d3    -rc*( ψ_θ/2 +  ψ_ηηθ/12)#-ve

    n=3
    ψ_θ = -ψ_j[n,m-1]/2 + ψ_j[n,m+1]/2
    ψ_ηηθ =  -(-ψ_j[n-2,m-1]/12 +   4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/2+
              (-ψ_j[n-2,m+1]/12 +   4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/2

    A[2,1]= d2 -rc*( ψ_θ   + 2*ψ_ηηθ/3)#-ve
    A[2,2]= d1
    A[2,3]= d2 -rc*(-ψ_θ   - 2*ψ_ηηθ/3)#-ve
    A[2,4]= d3 -rc*( ψ_θ/2 +   ψ_ηηθ/12)#-ve

    for n = 4:N-2
        ψ_θ = -ψ_j[n,m-1]/2 + ψ_j[n,m+1]/2
        ψ_ηηθ =  -(-ψ_j[n-2,m-1]/12 +  4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/2+
                  (-ψ_j[n-2,m+1]/12 +  4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/2

        A[n-1,n-3] = d3 -rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
        A[n-1,n-2] = d2 -rc*( ψ_θ   + 2*ψ_ηηθ/3)#-ve
        A[n-1,n-1] = d1
        A[n-1,n]   = d2 -rc*(-ψ_θ   - 2*ψ_ηηθ/3)#-ve
        A[n-1,n+1] = d3 -rc*( ψ_θ/2 +  ψ_ηηθ/12)#-ve

    end

    n=N-1
    ψ_θ = -ψ_j[n,m-1]/2 + ψ_j[n,m+1]/2
    ψ_ηηθ =  -(-ψ_j[n-2,m-1]/12 +  4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/2+
              (-ψ_j[n-2,m+1]/12 +  4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/2

    A[N-2,N-4]= d3 -   rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
    A[N-2,N-3]= d2    -rc*( ψ_θ   + 2*ψ_ηηθ/3)#-ve
    A[N-2,N-2]= d1
    A[N-2,N-1]= d2 -   rc*(-ψ_θ   - 2*ψ_ηηθ/3)#-ve

    n=N
    ψ_θ = -ψ_j[n,m-1]/2 + ψ_j[n,m+1]/2
    ψ_ηηθ =  -(-ψ_j[n-2,m-1]/12 + 4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -(-ψ_j[N,m-1]-bcm1)/12)/2+
              (-ψ_j[n-2,m+1]/12 + 4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -(-ψ_j[N,m+1]-bcp1)/12)/2

    A[N-1,N-3]= d3    -   rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
    A[N-1,N-2]= d2    -   rc*( ψ_θ   +2*ψ_ηηθ/3)#-ve
    A[N-1,N-1]= d1+d3 -   rc*( ψ_θ/2   +ψ_ηηθ/12)#-ve

end

function matrix_A_for_coreflow!(ψ_j, m, parameters)
    (Δθ, Δη, R, N, M, rc, rd1, rd2, rd3, d1, d2, d3, A, LUA) = parameters

    bc  =2*Δη*sin(Δθ*(m-1))
    bcp1=2*Δη*sin(Δθ*m)
    bcp2=2*Δη*sin(Δθ*(m+1))
    bcm1=2*Δη*sin(Δθ*(m-2))
    bcm2=2*Δη*sin(Δθ*(m-3))

    n=2
    ψ_θ =  ψ_j[n,m-2]/12 -  2*ψ_j[n,m-1]/3  +2*ψ_j[n,m+1]/3 - ψ_j[n,m+2]/12
    ψ_ηηθ =  (-(ψ_j[2,m-2]+bcm2)/12 +  4*ψ_j[n-1,m-2]/3  -5*ψ_j[n,m-2]/2 + 4*ψ_j[n+1,m-2]/3 -ψ_j[n+2,m-2]/12)/12-
           2*(-(ψ_j[2,m-1]+bcm1)/12 +  4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/3+
           2*(-(ψ_j[2,m+1]+bcp1)/12 +  4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/3-
             (-(ψ_j[2,m+2]+bcp2)/12 +  4*ψ_j[n-1,m+2]/3  -5*ψ_j[n,m+2]/2 + 4*ψ_j[n+1,m+2]/3 -ψ_j[n+2,m+2]/12)/12

    A[1,1]= d1+d3 -rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
    A[1,2]= d2    -rc*(-ψ_θ   -2*ψ_ηηθ/3)#-ve
    A[1,3]= d3    -rc*( ψ_θ/2 +  ψ_ηηθ/12)#-ve

    n=3
    ψ_θ =  ψ_j[n,m-2]/12 -  2*ψ_j[n,m-1]/3  +2*ψ_j[n,m+1]/3 - ψ_j[n,m+2]/12
    ψ_ηηθ =  (-ψ_j[n-2,m-2]/12 +  4*ψ_j[n-1,m-2]/3  -5*ψ_j[n,m-2]/2 + 4*ψ_j[n+1,m-2]/3 -ψ_j[n+2,m-2]/12)/12-
           2*(-ψ_j[n-2,m-1]/12 +  4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/3+
           2*(-ψ_j[n-2,m+1]/12 +  4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/3-
             (-ψ_j[n-2,m+2]/12 +  4*ψ_j[n-1,m+2]/3  -5*ψ_j[n,m+2]/2 + 4*ψ_j[n+1,m+2]/3 -ψ_j[n+2,m+2]/12)/12

    A[2,1]= d2 -rc*( ψ_θ   + 2*ψ_ηηθ/3)#-ve
    A[2,2]= d1
    A[2,3]= d2 -rc*(-ψ_θ   - 2*ψ_ηηθ/3)#-ve
    A[2,4]= d3 -rc*( ψ_θ/2 +   ψ_ηηθ/12)#-ve

    for n = 4:N-2
        ψ_θ = ψ_j[n,m-2]/12 -  2*ψ_j[n,m-1]/3  +2*ψ_j[n,m+1]/3 - ψ_j[n,m+2]/12
        ψ_ηηθ =  (-ψ_j[n-2,m-2]/12 +   4*ψ_j[n-1,m-2]/3  -5*ψ_j[n,m-2]/2 + 4*ψ_j[n+1,m-2]/3 -ψ_j[n+2,m-2]/12)/12-
               2*(-ψ_j[n-2,m-1]/12 +   4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/3+
               2*(-ψ_j[n-2,m+1]/12 +   4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/3-
                 (-ψ_j[n-2,m+2]/12 +   4*ψ_j[n-1,m+2]/3  -5*ψ_j[n,m+2]/2 + 4*ψ_j[n+1,m+2]/3 -ψ_j[n+2,m+2]/12)/12

        A[n-1,n-3]= d3 -rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
        A[n-1,n-2]= d2 -rc*( ψ_θ   + 2*ψ_ηηθ/3)#-ve
        A[n-1,n-1]= d1
        A[n-1,n]  = d2 -rc*(-ψ_θ   - 2*ψ_ηηθ/3)#-ve
        A[n-1,n+1]= d3 -rc*( ψ_θ/2 +  ψ_ηηθ/12)#-ve

    end

    n=N-1
    ψ_θ =  ψ_j[n,m-2]/12 -  2*ψ_j[n,m-1]/3  +2*ψ_j[n,m+1]/3 - ψ_j[n,m+2]/12
    ψ_ηηθ =  (-ψ_j[n-2,m-2]/12 +  4*ψ_j[n-1,m-2]/3  -5*ψ_j[n,m-2]/2 + 4*ψ_j[n+1,m-2]/3 -ψ_j[n+2,m-2]/12)/12-
           2*(-ψ_j[n-2,m-1]/12 +  4*ψ_j[n-1,m-1]/3  -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -ψ_j[n+2,m-1]/12)/3+
           2*(-ψ_j[n-2,m+1]/12 +  4*ψ_j[n-1,m+1]/3  -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -ψ_j[n+2,m+1]/12)/3-
             (-ψ_j[n-2,m+2]/12 +  4*ψ_j[n-1,m+2]/3  -5*ψ_j[n,m+2]/2 + 4*ψ_j[n+1,m+2]/3 -ψ_j[n+2,m+2]/12)/12

    A[N-2,N-4]= d3 -   rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
    A[N-2,N-3]= d2    -rc*( ψ_θ   + 2*ψ_ηηθ/3)#-ve
    A[N-2,N-2]= d1
    A[N-2,N-1]= d2 -   rc*(-ψ_θ   - 2*ψ_ηηθ/3)#-ve

    n=N
    ψ_θ =  ψ_j[n,m-2]/12 -  2*ψ_j[n,m-1]/3  +2*ψ_j[n,m+1]/3 - ψ_j[n,m+2]/12
    ψ_ηηθ =  (-ψ_j[n-2,m-2]/12 + 4*ψ_j[n-1,m-2]/3 -5*ψ_j[n,m-2]/2 + 4*ψ_j[n+1,m-2]/3 -(ψ_j[N,m-2]-bcm2)/12)/12-
           2*(-ψ_j[n-2,m-1]/12 + 4*ψ_j[n-1,m-1]/3 -5*ψ_j[n,m-1]/2 + 4*ψ_j[n+1,m-1]/3 -(ψ_j[N,m-1]-bcm1)/12)/3+
           2*(-ψ_j[n-2,m+1]/12 + 4*ψ_j[n-1,m+1]/3 -5*ψ_j[n,m+1]/2 + 4*ψ_j[n+1,m+1]/3 -(ψ_j[N,m+1]-bcp1)/12)/3-
             (-ψ_j[n-2,m+2]/12 + 4*ψ_j[n-1,m+2]/3 -5*ψ_j[n,m+2]/2 + 4*ψ_j[n+1,m+2]/3 -(ψ_j[N,m+2]-bcp2)/12)/12

    A[N-1,N-3]= d3    -   rc*(-ψ_θ/2 -  ψ_ηηθ/12)#-ve
    A[N-1,N-2]= d2    -   rc*( ψ_θ   +2*ψ_ηηθ/3)#-ve
    A[N-1,N-1]= d1+d3 -   rc*( ψ_θ/2   +ψ_ηηθ/12)#-ve

end
```

    matrix_A_for_coreflow! (generic function with 1 method)

```julia
const rd3 =      -1.0/12.0
const rd2 =       4.0/3.0
const rd1 =      -5.0/2.0
const rc    =   Δt/(Δη*Δθ)

parameters =  (Δθ, Δη, R, N, M, rc, rd1, rd2, rd3, d1, d2, d3, A, LUA);

```

```julia
#Forming the righthand side vector

percent = 5.0
old_norm = 0.0

#Only theta profiles from m =2 to m = M need to be calculated with one sided theta derivatives second order.
#11111111111111111111111111111111111111111111111111111111
@time for j =1:J

    make_ψ_jnext_on_boundaries!(ψ_j, ψ_jnext, parameters)
    make_ψ_jnext_in_core_region!(ψ_j, ψ_jnext, parameters)

    if( mod(j,2) == 1)
        ψ_j    = ψ_next
        ψ_jnext=ψ_current
    else
        ψ_j    = ψ_current
        ψ_jnext=ψ_next
    end #End of if statement

    if( j> round(percent*J/100))
        #Calculate a simple norm of psi
        psi_norm = norm(ψ_j[:,4])
        println(string(percent)*"% compelete T="*string(round(j*Δt; digits=2))*" Core flow norm = "*string(round(psi_norm; digits=6)))
        percent+=5
        if isnan(psi_norm )
           println("Psi is no longer a number, exiting... ")
           exit()
        elseif(abs(psi_norm-old_norm)<0.00001)
            println("Psi has converged, exiting... ")
            break
       end
    end

end #End of time loop j
println(string(percent)*"% compelete      Core flow norm = "*string(round(norm(ψ_j[:,4]); digits=6)))
println("Reynolds number= "*string(R))
#At this point inverse of A should be applied to the rhs vector to produce the next iterate of psi at the j+1 time step

```

    5.0% compelete T=0.25 Core flow norm = 0.117048
    10.0% compelete T=0.5 Core flow norm = 0.112857
    15.0% compelete T=0.75 Core flow norm = 0.109052
    20.0% compelete T=1.0 Core flow norm = 0.105983
    25.0% compelete T=1.25 Core flow norm = 0.103779
    30.0% compelete T=1.5 Core flow norm = 0.102386
    35.0% compelete T=1.75 Core flow norm = 0.10165
    40.0% compelete T=2.0 Core flow norm = 0.101388
    45.0% compelete T=2.25 Core flow norm = 0.101427
    50.0% compelete T=2.5 Core flow norm = 0.101629
    55.0% compelete T=2.75 Core flow norm = 0.101892
    60.0% compelete T=3.0 Core flow norm = 0.102148
    65.0% compelete T=3.25 Core flow norm = 0.102362
    70.0% compelete T=3.5 Core flow norm = 0.102518
    75.0% compelete T=3.75 Core flow norm = 0.102617
    80.0% compelete T=4.0 Core flow norm = 0.102668
    85.0% compelete T=4.25 Core flow norm = 0.102683
    90.0% compelete T=4.5 Core flow norm = 0.102675
    95.0% compelete T=4.75 Core flow norm = 0.102656
    146.286448 seconds (50.62 M allocations: 231.312 GiB, 2.94% gc time, 0.21% compilation time)
    100.0% compelete      Core flow norm = 0.102634
    Reynolds number= 50.0

```julia
open("psi_solution.txt", "w") do io
           writedlm(io, ψ_current)
       end
#wireframe(range(0,stop = pi,length = M+1),range(-1,stop = 1,length = N+1),ψ_j)
```

```julia
m=5
MatrixPsi=readdlm("psifin.m",Float64,);
xinterval=range(-1,stop=1,length=101)
size(MatrixPsi)
eta=range(-1, stop=1,length= N+1)
plot(eta,ψ_j[:,m],linecolor=:orange, label="Julia", linestyle=:solid)
plot!(x->Ψ_smallr(Δθ*(m-1),x,R),-1,1, label="Small R analytical", linestyle=:dot, linecolor=:blue,xlabel = L"\eta", ylabel = L"\Psi",size=(800,400))
plot!(xinterval,MatrixPsi[2:102,m], label="Spectral", linestyle=:dash, linecolor=:black,xlabel = L"\eta", ylabel = L"\Psi")
plot!(x->Ψ_bigr((m-1)*Δθ,x),-1,1, label="Large R analytical",  linestyle=:dash, linecolor=:red,xlabel = L"\eta", ylabel = L"\Psi",size=(800,400),yguidefontrotation=-90, legend=:outertopright)
```

```julia
eta_profile =0.6
eta_steps = size(MatrixPsi)[1]
delta_eta_steps = 2/(eta_steps-3)
theta=range(0, stop=pi,length= M+1)
plot(theta,ψ_j[round(Int,(eta_profile+1)/Δη)+1,:],linecolor=:orange, label="Julia", linestyle=:solid)
plot!(x->Ψ_smallr(x,eta_profile,R),0,pi, label="Small R analytical", linestyle=:dot, linecolor=:blue,xlabel = L"\theta", ylabel = L"\Psi",size=(800,400))
#plot!(x->Ψ_bigr(x,eta_profile),0,pi, label="Large R analytical",  linestyle=:dash, linecolor=:red,xlabel = L"\theta", ylabel = L"\Psi",size=(800,400),yguidefontrotation=-90, legend=:outertopright)
plot!(theta,MatrixPsi[round(Int,(eta_profile+1)/delta_eta_steps)+2,1:256])
```

```julia
round(Int,(eta_profile+1)/Δη)+1
```

    161

```julia
for n =2:N
    for m =2:M
            η = -1 + Δη*(n-1)
            θ = -1 + Δθ*(m-1)
            ψ_jnext[n,m] = ψ_j[n, m] - Ψ_smallr(θ,η,R)
    end
end

```

```julia
wireframe(range(0,stop = pi,length = M+1),range(-1,stop = 1,length = N+1),ψ_jnext)
```

```julia

```


```python
'''Shows how including vacuum mixing in the single photons used for photon catalysis
affects the resultant states after several steps'''
import cmath, random, numpy
import functools
import  matplotlib.pyplot as plt
import sys
import os
import math
from qutip import*

from sympy import*
#from sympsi import*
from scipy import optimize

from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
import time
import math
from qutip import *
from qutip.ipynbtools import plot_animation
import numpy as np
import matplotlib.pyplot as plt
import qutip
%matplotlib inline
import matplotlib.pylab as plt
import matplotlib as mpl
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from IPython.display import display, Math, Latex
import cmath
from mpl_toolkits.axes_grid1 import AxesGrid
from scipy.special import factorial
```

```python
"""Define the operators for MZI - we will deal in Heisenberg """
T = Symbol('T')
xvec = np.arange(-30.,30.)*5./40
X,Y = np.meshgrid(xvec, xvec)  ##Some plotting params
N_dim = 35##Dimenstion of the Hilbert spac

a1 = destroy(N_dim) ##This is for single-photon field
a2 = destroy(N_dim) ##for coherent field
a3 = destroy(N_dim) ##for vacuum field
def n_choose_k(n,k):
    return factorial(n)/(factorial(n-k)*factorial(k))
def D(state,alpha):
    Rho_new=displace(N_dim,alpha)*state*displace(N_dim,alpha).dag()
    return Rho_new
'''Define a rotation in phase space, or phase shifter operation'''
def Phase(theta):
    b=-1j*theta*a1.dag()*a1;
    return b.expm()
'''Squeezing operation, inputs a density matrix and outputs the squeezed density matrix for squeezing parameter r'''
def Sq(state,r):
    Rho_new=squeeze(N_dim,r)*state*squeeze(N_dim,r).dag();
    return Rho_new
'''The function below creates a beamsplitter operation that acts on two modes.
The value for k determines what number Fock state could be filtered out of the first state
based on a single photon input for the second BS port, followed by single photon detection.'''
def BS_operator_filtering(a1, a2, k):
    theta_k = np.arctan(1/np.sqrt(k))
    T = np.sin(theta_k)*np.sin(theta_k)
    R = np.cos(theta_k)*np.cos(theta_k)
    print('I am filtering', k, 'and:', theta_k*180/math.pi)
    print('BS T is : ', T, 'and : ', R)
    b = theta_k*(tensor(a1,a2.dag()) - tensor(a1.dag(),a2))
    return b.expm()
def SSV_plus(r,alpha):
    state = ket2dm((displace(N_dim,alpha)+displace(N_dim,-alpha))*squeeze(N_dim,r)*fock(N_dim,0))
    norm_state = state/state.tr()
    return norm_state
def SSV_minus(r,alpha):
    state = ket2dm((displace(N_dim,alpha)-displace(N_dim,-alpha))*squeeze(N_dim,r)*fock(N_dim,0))
    norm_state = state/state.tr()
    return norm_state
def cat_plus(alpha):
    cat = (1/(np.sqrt(2)*np.sqrt(1+np.e**(-alpha*alpha.conj()))))*(coherent(N_dim,-alpha)+(coherent(N_dim,alpha)))
    return cat
def cat_minus(alpha):
    cat = (1/(np.sqrt(2)*np.sqrt(1-np.e**(-alpha*alpha.conj()))))*(-coherent(N_dim,-alpha)+(coherent(N_dim,alpha)))
    return cat
def pnr_resolution_detector(eta, click, n_truc):
    pi_n = 0;
    l = np.arange(click,n_truc)
    for i in l:
        pi_n +=  n_choose_k(i,click)*math.pow((1-eta),(i-click))*math.pow(eta,click)*fock(N_dim,i)*fock(N_dim,i).dag()
        #print("The final Povm element is: ", pi_0)
    return Qobj(pi_n)
def Fock_Filter_povm(in1,in2,refl,num_det,eta,n_truc):
    Projector = tensor(pnr_resolution_detector(eta, num_det, n_truc),qeye(N_dim));
    Initial_state=tensor(in1,in2);
    theta_k=np.arccos(np.sqrt(refl));

    BS1= ((theta_k)*(tensor(a1,a2.dag()) - tensor(a1.dag(),a2))).expm()

    Rho=BS1*Initial_state*BS1.dag();

    Rho_filtered = ((Rho*Projector).ptrace(1))/((Rho*Projector).tr())
    '''The operation .ptrace(m) takes the partial trace over every mode EXCEPT m, where the numbering
    startes at 0.  So .ptrace(1) means you keep mode 1, which is actually the 2nd mode'''
    print('BS has reflectivity',refl,' and I am detecting the |',num,'> state, where my detector has efficiency', eta)
    return Rho_filtered
def Fock_Filter_prob(in_state,in_fock,refl,num_det,eta,n_truc):
    Projector = tensor(pnr_resolution_detector(eta, num_det, n_truc),qeye(N_dim));
    Initial_state=tensor(in_state,ket2dm(fock(N_dim,in_fock)));
    theta_k=np.arccos(np.sqrt(refl));

    BS1= ((theta_k)*(tensor(a1,a2.dag()) - tensor(a1.dag(),a2))).expm()

    Rho=BS1*Initial_state*BS1.dag();
    P=(Rho*Projector).tr()
    print('The probability of a sucessful detection is:',P)
    Rho_filtered = ((Rho*Projector).ptrace(1))/((Rho*Projector).tr())
    #Rho_filtered=Rho*Projector
    '''The operation .ptrace(m) takes the partial trace over every mode EXCEPT m, where the numbering
    startes at 0.  So .ptrace(1) means you keep mode 1, which is actually the 2nd mode'''
    print('BS has reflectivity',refl,' and I am detecting the |',num,'> state, where my detector has efficiency', eta)
    return Rho_filtered

def fid(state1,state2):
    F=np.absolute((state1.sqrtm()*state2*state1.sqrtm()).sqrtm().tr())
    return F
```

```python
#Variable definitions:
#delta = initial coherent state amplitude
# refl(1-3)= beamsplitter r^2 values at each step
# n(1-3) = number of photons detected at each detector
# beta = amplitude of final dispalcement to displace the state back
#alpha = amplitude of SSV state to be compared with.  Note, here
#this can be related to cat state amplitude by amp_cat=alpha/(Cosh[sq]-Sinh[sq])
# sq = the 'r' value, or squeezing parameter. This is the negative of the mathematica results.

'''Three-step check: P -> 0.00179605, 0.9836, r[3.] -> 0.520135, r[2.] -> 0.490869,
 r[1.] -> 0.637566, \[Alpha] -> 1.349,
 sq -> -0.476595, \[Beta] -> 2.03846, delta -> 3.53709, n1 -> 5,
 n2 -> 2, n3 -> 1}'''

refl1=0.637566**2; refl2=0.490869**2; refl3=0.520135**2;
delta=3.53709; beta=2.03846; sq=0.476595; alpha=1.349; n1=5;n2=2;n3=1;

#refl1=1/2; refl2=1/2; refl3=2/3;
#delta=np.sqrt(14); beta=np.sqrt(19/16); sq=0.476595; alpha=1.4; n1=5;n2=3;n3=1;
'''Two-step, check: {

 {P -> 0.0152557, 0.999653, r[2.] -> 0.852986,
  r[1.] -> 0.603152, \[Alpha] -> 0.901576,
  sq -> -0.224664, \[Beta] -> 0.5, delta -> 1.19655, n1 -> 1,
  n2 -> 2}
}'''
refl1=0.603152**2; refl2=0.852986**2; delta=1.19655; beta=0.5; sq=0.224664; alpha=0.901576; n1=1;n2=2

'''Four-step check: r[4.] = 0.42134; r[3.] = 0.69684; r[2.] = 0.55398;
r[1.] = .576813; \[Delta] = 4.6621868; n[4.] = 1; n[3.] = 2;
n[2.] = 4; n[1.] = 6;
sq= -0.51, alpha= 1.59, beta= 2.08625'''

refl1= .576813**2; refl2=0.55398**2; refl3=0.69684**2;refl4=0.42134**2;
delta=4.6621868; beta=2.08625; sq=0.476595; alpha=1.59; n1=6;n2=4;n3=2;n4=1;
```

```python
#TEST
#Make a loop to take max values after iterating and put into a single array
End=np.zeros(10);
for i in range(10):
    temp=np.zeros(31);
    for j in range(31):
        x=j/10+i;
        temp[j]=x;
    End[i]=max(temp);
```

```python
End
```

    array([ 3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.])

```python
'''Four-step check: r[4.] = 0.42134; r[3.] = 0.69684; r[2.] = 0.55398;
r[1.] = .576813; \[Delta] = 4.6621868; n[4.] = 1; n[3.] = 2;
n[2.] = 4; n[1.] = 6;
sq= -0.51, alpha= 1.59, beta= 2.08625'''
def mixture(purity):
    state=purity*ket2dm(fock(N_dim,1))+(1-purity)*ket2dm(fock(N_dim,0))
    return state

eta=1;
refl1= .576813**2; refl2=0.55398**2; refl3=0.69684**2;refl4=0.42134**2;
delta=4.6621868; beta=2.08625; sq=0.476595; alpha=1.59; n1=6;n2=4;n3=2;n4=1;

ssv=SSV_plus(sq,alpha)

four_step=np.zeros(21)
for k in range(21):
    first=Fock_Filter_povm(ket2dm(coherent(N_dim,delta)),mixture(1-0.02*k),refl1,n1,eta,N_dim)
    second=Fock_Filter_povm(first,mixture(1-0.02*k),refl2,n2,eta,N_dim);
    third=Fock_Filter_povm(second,mixture(1-0.02*k),refl3,n3,eta,N_dim);
    fourth=Fock_Filter_povm(third,mixture(1-0.02*k),refl4,n4,eta,N_dim);
    F=np.zeros(60)
    for i in range(60):
        final_state=D(fourth,beta*(-1)-0.3+0.01*i)
        F[i]=fid(final_state,ssv)
    four_step[k]=max(F)

plt.plot(range(21),four_step,'-*')
plt.grid()
plt.xlabel('vacuum mixing')
plt.ylabel('fidelity')
#set_xlim([40, 60])
plt.title('Fidelity as mixing with vacuum increases')
plt.show()

```

```python
'''Three-step check: P -> 0.00179605, 0.9836, r[3.] -> 0.520135, r[2.] -> 0.490869,
 r[1.] -> 0.637566, \[Alpha] -> 1.349,
 sq -> -0.476595, \[Beta] -> 2.03846, delta -> 3.53709, n1 -> 5,
 n2 -> 2, n3 -> 1}'''

refl1=0.637566**2; refl2=0.490869**2; refl3=0.520135**2;
delta=3.53709; beta=2.03846; sq=0.476595; alpha=1.349; n1=5;n2=2;n3=1;

ssv=SSV_minus(sq,alpha)

three_step=np.zeros(21)
for k in range(21):
    first=Fock_Filter_povm(ket2dm(coherent(N_dim,delta)),mixture(1-0.02*k),refl1,n1,eta,N_dim)
    second=Fock_Filter_povm(first,mixture(1-0.02*k),refl2,n2,eta,N_dim);
    third=Fock_Filter_povm(second,mixture(1-0.02*k),refl3,n3,eta,N_dim);
    F=np.zeros(60)
    for i in range(60):
        final_state=D(third,beta*(-1)-0.3+0.01*i)
        F[i]=fid(final_state,ssv)
    three_step[k]=max(F)

plt.plot(range(21),three_step,'-*')
plt.grid()
plt.xlabel('vacuum mixing')
plt.ylabel('fidelity')
#set_xlim([40, 60])
plt.title('Fidelity as mixing with vacuum increases')
plt.show()

```

```python
'''Two-step, check: {

 {P -> 0.0152557, 0.999653, r[2.] -> 0.852986,
  r[1.] -> 0.603152, \[Alpha] -> 0.901576,
  sq -> -0.224664, \[Beta] -> 0.5, delta -> 1.19655, n1 -> 1,
  n2 -> 2}
}'''
refl1=0.603152**2; refl2=0.852986**2; delta=1.19655; beta=0.5; sq=0.224664; alpha=0.901576; n1=1;n2=2

ssv=SSV_plus(sq,alpha)

two_step=np.zeros(21)
for k in range(21):
    first=Fock_Filter_povm(ket2dm(coherent(N_dim,delta)),mixture(1-0.02*k),refl1,n1,eta,N_dim)
    second=Fock_Filter_povm(first,mixture(1-0.02*k),refl2,n2,eta,N_dim);
    F=np.zeros(60)
    for i in range(60):
        final_state=D(second,beta*(-1)-0.3+0.01*i)
        F[i]=fid(final_state,ssv)
    two_step[k]=max(F)

plt.plot(range(21),two_step,'-*')
plt.grid()
plt.xlabel('vacuum mixing')
plt.ylabel('fidelity')
#set_xlim([40, 60])
plt.title('Fidelity as mixing with vacuum increases')
plt.show()

```

```python
from pylab import *
rc('axes', linewidth=2)
fontsize=14
array=np.arange(1,.58,-.02)
#array=[1,0.95,0.9,0.85,0.8]
plt.plot(array,two_step,'g-o',array,three_step,'-rv',array,four_step,'b-D')
plt.grid()
plt.legend((r'$N=2$', r'$N=3$',r'$N=4$'),
           loc='lower right', shadow=True, fontsize=14)
plt.xlabel('Single-photon purity',fontsize=16)
plt.ylabel('Fidelity',fontsize=16)
#set_xlim([40, 60])
#plt.title('Fidelity of filtered state with displaced single photon', fontsize=16)
plt.tick_params(axis='both', labelsize=12)

plt.show()
```

```python
two_step
```

    array([0.99982773, 0.97547229, 0.95286941, 0.93188413, 0.91240394,
           0.89432335, 0.87755267, 0.86200803, 0.84761995, 0.83432208,
           0.82205385, 0.81076378, 0.80040297, 0.7909238 , 0.78228823,
           0.77445457, 0.76738848, 0.76105703, 0.7554269 , 0.75047016,
           0.74615794])

```python
three_step
```

    array([0.99141659, 0.95642797, 0.92231563, 0.88915072, 0.85700033,
           0.82593688, 0.79601525, 0.7672917 , 0.73980125, 0.71357306,
           0.68861634, 0.66492573, 0.64249618, 0.62129855, 0.60127998,
           0.58237947, 0.56456402, 0.54773471, 0.53184398, 0.51683138,
           0.50269676])

```python
four_step
```

    array([0.98867933, 0.93652901, 0.88678155, 0.83951843, 0.79481757,
           0.75278407, 0.71341778, 0.67677286, 0.64280472, 0.61147745,
           0.5827516 , 0.55655054, 0.53285175, 0.51168623, 0.49322337,
           0.47789103, 0.46505077, 0.4529949 , 0.44160849, 0.43079168,
           0.42047334])

```python
two_step=[0.99982773, 0.97547229, 0.95286941, 0.93188413, 0.91240394,
       0.89432335, 0.87755267, 0.86200803, 0.84761995, 0.83432208,
       0.82205385, 0.81076378, 0.80040297, 0.7909238 , 0.78228823,
       0.77445457, 0.76738848, 0.76105703, 0.7554269 , 0.75047016,
       0.74615794];
three_step=[0.99141659, 0.95642797, 0.92231563, 0.88915072, 0.85700033,
       0.82593688, 0.79601525, 0.7672917 , 0.73980125, 0.71357306,
       0.68861634, 0.66492573, 0.64249618, 0.62129855, 0.60127998,
       0.58237947, 0.56456402, 0.54773471, 0.53184398, 0.51683138,
       0.50269676];
four_step=[0.98867933, 0.93652901, 0.88678155, 0.83951843, 0.79481757,
       0.75278407, 0.71341778, 0.67677286, 0.64280472, 0.61147745,
       0.5827516 , 0.55655054, 0.53285175, 0.51168623, 0.49322337,
       0.47789103, 0.46505077, 0.4529949 , 0.44160849, 0.43079168,
       0.42047334];
```

```python
F=np.zeros(40)
for i in range(40):
    final_state=D(fourth,beta*(-1)-0.2+0.01*i)
    F[i]=fid(final_state,ssv)
plt.plot(range(40),F,'-*')
plt.grid()
plt.xlabel('diplacement')
plt.ylabel('fidelity')
#set_xlim([40, 60])
plt.title('Fidelity of filtered state with displaced single photon')
plt.show()
```

```python

```


 Exam Questions to: Introduction to Python 3.6, May 2020

By Dr. Anders S. Christensen
`anders.christensen @ unibas.ch`

---

## ***Important:***
***The answers to the questions must be entered in the online system:***

https://www.chemie1.unibas.ch/~pythonprakt/

*    You should have received the username and password by email.

*    Use you @unibas.ch or @stud.unibas.ch email to login after you enter the username and password for the course.

## Exam Question 1: Debugging

The code block below contains a bug.

**To-do:**

1.   Fix the bug
2.   In the web-interface, enter what was wrong

```
number = 5

if number > 10:
  print("number is less than 10")
elif number <= 10
  print("number is less than or equal to 10")
```

## Exam Question 2: `for`-loops

In this exercises you will use a `for` loop to carry out summation as seen in the previous exercises in the course.

Use Python code to calculate the result, $x$, of the following summation:

\begin{equation}
x = \sum_{n=0}^{200} n^3
\end{equation}

**To-do:**

1.   Code the summation in the code block below
2.   In the web-interface, select the correct answer

```
## Insert the code here to calculate the sum and print the result

```

## Exam Question 3: Functions

In this exercise, you must write a function a function to calculate the followling polynomium:
\begin{equation}
p\left( x \right) = 3x^3 + 4x^2 - 8x -3
\end{equation}

**To-do:**
1.   Write a function to calculate the polynomial
2.   Use the code to calculate the value of:
    *   $p(5)$
    *   $p(13)$
    *   $p(72)$
3.   In the web-interface, select the correct value of the three numbers

```
## Write a function to calculate the ploynomium and calculate p(5), p(13), and p(72)

```

## Exam Question 4: Plotting

In this exercise, you will use plotting with `matplotlib.pyplot` to find out whether two sets of data have any correlation.

You are given two lists, `x` and `y`.

Similarly to the scatterplot in the previous exercises, make a scatter plot of the two lists in the code block below.

Once you have made the plot, you should be able to see whether there is any correlation between the numbers in the two lists.

**To-do:**

1.   Plot the scatterplot
2.   See if the two lists `x` and `y` correlate
3.   In the web-interface give the answer to (2.)

```
x = [0.4967, 0.3617, 1.6477, 3.023, 1.7658,
     2.2659, 4.5792, 4.2674, 3.5305, 5.0426,
     4.5366, 5.0343, 6.242, 4.5867, 5.2751,
     6.9377, 6.9872, 8.8142, 8.092, 8.0877]

y = [1.4656, 0.2742, 1.0675, 0.0753, 1.4556,
     2.6109, 1.849, 3.8757, 3.3994, 4.2083,
     4.3983, 7.3523, 5.9865, 5.4423, 7.8225,
     6.2792, 8.2089, 6.5403, 7.6718, 9.6969]

import matplotlib.pyplot as plt

# <-- Insert code to plot the scatter plot of 'x' and 'y' here

# Show the figure
plt.show()
```

## Exam Question 5: Numpy

In the example below, two random matrices are defined, `A` and `B`.

```
import numpy as np

np.random.seed(2020)

A = np.random.randint(10, size=(6,4))
B = np.random.randint(10, size=(4,5))

print("A:")
print(A)

print("B:")
print(B)
```

In this questions, you have to calculate the following matrix product:
\begin{equation}
\mathbf{C} = \mathbf{A}\mathbf{B}.
\end{equation}

**To-do:**
1.   Calculate the $\mathbf{C}$ matrix using Numpy, and print it
3.   In the web-interface select the correct output of the calculation

```
# Below, code the matrix product C=AB, print the answer, and select the answer in the web-interface

```


# The Iris Classification with Keras

In this project, we build and train a model to categorize the Iris flower base the sample data [1]. The algorithm we chose to implement for this project is multilayer perceptron. The Iris data set is available from the University of California at Irvine (UCI) Machine Learning Repository. The dataset consists of information on 150 Iris flowers [2]. We focus on learning three Iris species which are Setosa, Versicolour, and Virginica. The dataset is characterized by five attributes:
    1. Sepal Length in centimeters
    2. Sepal Width in centimeters
    3. Petal Length in centimeters
    4. Petal Width in centimeters
    5. Targeting Class (Setosa, Vericolour, Virginica)

||||
|:---|:---:|---:|
|Iris Setosa|Iris Versicolor|Iris Virginica|

## Learning Algorithm

### Multilayer Artificial Neural Network

We designed a Neural Network with very simple structure. The network contain 4 layers which are input, hidden, and output layers. The input layers consists of four nodes. The input for the network is a vector of Iris features which are Sepal Length, Sepal Width, Petal Length, and Petal Width [3]. There are two hidden layers with 10 nodes on each of the layers. We chose to use linear function, Relu, as the activation functions. The predicted probability of the Iris class will send to the output layer which represents Iris flower classes. The figure below shows the model Neuro Network [4].

There are various types of Neuro Networks. In this project, we chose to build the fully connected Neuro Networks, also known as Dense networks. The neurons from the input layer are connected each neurons in the first hidden layer. The same way applys to second and final output layers [5]. There is a single bias unit that is connected to each neurons in the hidden layers. The neurons in the hidden layers compute the weighted sum from the inputs to form the scalar net activation. We write the equation as in the figure below:

\begin{equation*}
\mathbf{y} = \mathbf{\sum_{i=1}^n x_i w_i} + \mathbf{bias}
\end{equation*}

The subscript i indexes neurons in the input layer. The w denotes the input to hidden layer weights at the hidden layer neurons. Such weights are also named synapses and the values of the connections the synaptic weights. The output y can be thought of as a function of input feature vector x. When there are k output neurons, we can think of the network as computing k discriminant functions and classify the input according to which discriminant function is the largest.

In this project, we express the Neuro Network to Mathematical functions as:

\begin{equation*}
\mathbf{\output} = \mathbf{\left(\sum_{j=1}^n w_j_m\left(\sum_{i=1}^n w_i_n\left(\sum_{q=1}^n w_q_o x + bias_q\right) + bias_n\right)+bias_m\right)}
\end{equation*}

The Wjm here denotes the weights in layer m and Win denotes the weights in layer n [6]. The bias_q, bias_n, and bias_m denote the bias vector in layer q, n, and m accordingly. The output is a vector of dimension of three which represents the probability of the three classes.

###  Training Error

The training error for the model that we build and train for this project to be the least mean square error which is sum over the output neurons of the squared difference between the actual desired value z and the actual ouput t. The equation is denoted below:

\begin{equation*}
\mathbf{E\left(w\right)} = \mathbf{\frac{1}{2}} \mathbf{\sum_{k=1}^n \left(z_k-t_k\right)^2}
\end{equation*}

where z and t are the desired value and actual network output. The purpose for applying least mean square error and gradient descent is to train the model learning weights. The weights are initialized with value zeros, and then changed in a direction that will reduce error:
\begin{align}
    \mathbf{\nabla w} = mathbf{-n} mathbf{\frac{\partial\mathbf{E}}{\partial\w}
\end{align}

where n is the learning rate which indicates the relative size of the change in weights [7]. The weight values is updated as follow:

\begin{align}
        \mathbf{W(k+1)}=\mathbf{W(k)} \mathbf{-n} \frac{\partial\mathbf{E}}{\mathbf{\partial w}}
\end{align}

Because the error function is given analytically, it is differentiable. The partial derivative with respect to w is:

\begin{align}
\frac{\partial{\mathbf{E}}}{\partial\mathbf{w}}=-\mathbf{(z-w^t x)x}
\end{align}

Substituting the result into the weight update equation, the resulting function is as below:

\begin{align}
\mathbf{W(k+1)}=\mathbf{W(k)} - \mathbf{n(z-w^tx)x}
\end{align}

where n is the learning rate, range 0<n<2 for the learning process to converge. In practice, we iterate the learning process to a specified threshold. The final computation in the hidden layers is the activation function. We chose the rectifier linear unit (ReLU) as the activation function for our network.

## Training Dataset Processing

In this project, we download the training dataset from "https://storage.googleapis.com/download.tensorflow.org/data/iris_training.csv" and store it onto the local hard drive.

We use the tf.data.experimental.make_csv_dataset to parse the training dataset [8]. The output is the file consisting a tuple of Iris features and corresponding labels. The size of the file is the number of rows in the batch.

```python
# Total 120 rows of Iris features.
# Chosing first 32 rows of data for training.

train_data_location = "C:/Users/JunnanLu123/iris_training.csv"
train_data_filePt = (train_file_fp, 32, column_names = column_names, label_name = label_name, num_epochs = 1)
```

```python
# The columns of the data set.

column_names = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species']

# The first four fields are flower features
# representing flower measurements.

feature_names = column_names[:-1]

# The last column is the flower label which we
# hope the model could predict in high accuracy.

label_name = column_names[-1]
```

```python
# Each class is associated with the string name such as setosa.
# The class name are mapped to then list representation such as
# 0, Iris setosa
# 1, Iris versicolor
# 2, virginica

class_names = ['Iris setosa', 'Iris versicolor', 'Iris virginica']
```

```python
# The input data is a file of CSV format. Therefore, we call
# tf.data.experimental.make_csv_dataset to make the training
# dataset. The make_csv_dataset returns a tf.data.Dataset which
# consists of feature and label pairs.

train_dataset = tf.data.experimental.make_csv_dataset(train_file_fp, 32, column_names = column_names, label_name = label_name, num_epochs = 1)
```

```python
# The train_dataset object is iterable. The return value is a tuple
# of features and label pair. The label here denotes the flower
# classes.

self.features, self.label = next(iter(train_dataset))
```

Each row of the sample data is corresponding to the feature array which is grouped together in a batch. We set the default batch size to 32.

```python
# We call tf.stack to create a combined tensor at
# the specified dimension. And we pack the features
# into a single array.

def pack(self, features, label):
    self.features = tf.stack(list(features.values()), axis=1)
return features, label
```

```python
# Each features such as (feature, label) pair mapped into training
# dataset by using tf.data.Dataset.map.

dataset = dataset.map(pack)
```

We plot the figure to visualize the flower features. The batch size is 32 rows of feature and label pairs.

```python
# Visualize the flower data from the dataset batch.
# The batch size is 32 rows of feature and label pairs.

plt.scatter(self.features['petal_length'], self.features['petal_width'], c=self.label, cmap='viridis')
plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.show()
```

The figure on the left displays the distribution of features for sepal length and sepal width. The figure on the right displays the features for petal length and petal width.

|||
|:---|:---:|
|Sepal Length and Width|Petal Length and Width|

## Model Building and Training With Keras

### Setup Tensorflow and Keras

We implement the model layers with tf.keras API. The API provides the way to handle the complexity of connections among model layers. The layer is the fundamental data structure for processing the Iris data for our project. In this project, our network topology is that layers are densly connected.

```python
# Setup tensorflow and Keras modules

import tensorflow as tf
from tensorflow.keras import layers
```

### Create Model with Keras

For each layer there is only one input tensor and one output tensor. If the topology is non-linear which there is a residual connection or needing layer sharing, then, our model will not produce the optimal results. Therefore, we apply the tf.keras.Sequential() model which is a linear model of layers.

```python
# We create layers that only accept input of 2D tensors
# where the input dimension is 4xNone. The axis 0 could
# be any value since it is not specified. The first
# hidden layer requires an input with dimension of
# 10. And the same way goes for the second hidden layer
# The final output layer will return a tensor with
# dimension of 3 denoting Setosa, Versicolor, and
# Verginica respectively.

model = tf.keras.Sequential()

# Here we chose Rectifier linear Unit, ReLu, as the activation
# function.

model.add(tf.keras.layers.Dense(10, activation=tf.nn.relu, input_shape=(4,)))
model.add(tf.keras.layers.Dense(10, activation=tf.nn.relu))
model.add(tf.keras.layers.Dense(3))
```

### The Loss and Gradient Function

As mentioned earlier, we use gradient descent algorithm to update weights of the layers to minimize the computing errors. We calculate the error by using tf.keras.losses.SparseCategoricalCrossentropy() function. The loss function returns the average loss across the training data.

```python
# Computes the sparse categorical crossentropy loss

loss_object = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
```

We use tf.GradientTape to record automatic differetiation. In our model, the trainable variable are the weights and bias which are being watched automatically.

```python
# GradientTape automatically watch trainable variable,
# the resource will be released when GradientTape.gradient()
# method is called. Using gradient to calculate and optimize
# gradients of the model.

with tf.GradientTape() as Tape:
   prediction = model(features, training=True)
   loss = loss_object(label, prediction)
grads = Tape.gradient(loss, model.trainable_variables)
```

### Create The Optimizer

The next step is to minimize the loss function defined above. We choose to use the tf.keras.optimizers.SGD() to update the set of weights and bias in our network. The gradient descent method is commonly used to update changes.

```python
# We create the optimizer to compute loss and set the learning
# rate to 0.01

optimizer = tf.keras.optimizers.SGD(learning_rate=0.01)
```

### Train The Model

The goal for our model is to learn from the training data and then is able to predict unseen data with high accuracy. Overfitting problem occurs when model is producing high accurate resutls when feeding with training data but performing poorly when feeding unseen data. The training process consists 100 epochs. We iterate through each epoch. And for each of the epochs, we then iterate through each of the 32 training dataset to let model to make predictions. Then, applying the loss and optimization methods mentioned earlier to update the weights and bias of the model.

```python
# Model Training Process Using Batch of 32 Samples
# Each epoch is one pass through the dataset

# Keep tracking the value for plotting.
loss_value_list=[]
model_accuracy_list=[]

for epochs in range(100):
    loss_value = tf.keras.metrics.Mean()
    accuracy = tf.keras.metrics.SparseCategoricalAccuracy()
    # Training the model through each of the training sample
    # Iterate through each example of the feature and label of
    # the dataset.

    for features, label in dataset:
    # GradientTape automatically watch trainable variable,
    # the resource will be released when GradientTape.gradient()
    # method is called. Using gradient to calculate and optimize
    # gradients of the model.

        with tf.GradientTape() as Tape:
             prediction = model(features, training=True)
             loss = loss_object(label, prediction)
        grads = Tape.gradient(loss, model.trainable_variables)

        # Optimizer implemented the Stochastic Gradient Decent method.

        optimizer.apply_gradients(zip(grads, model.trainable_variables))

        # Tracking the stats of the loss and accuracy value for visualization

        loss_value.update_state(loss)
        accuracy.update_state(label, model(features, training=True))

    # Append the loss value and accuracy result to the lists

    loss_value_list.append(loss_value.result())
    model_accuracy_list.append(accuracy.result())
```

### Visualize the Loss Function

```python
# Setup the plot by using matplotlib module.

import matplotlib.pyplot as plt
```

```python
# Visualize the loss in time elapsed fashion.
# TensorBoard can also help to visualize the training metrics.
# We are using matplotlib to create basic metrics chart.

fig1, axes = plt.subplots(2, sharex=True, figsize=(12,8))
fig1.suptitle("Train Metrics")
axes[0].set_ylabel("LOSS", fontsize=14)
axes[0].plot(loss_value_list)

axes[1].set_ylabel("ACCURACY", fontsize=14)
axes[1].set_xlabel("EPOCHS", fontsize=14)
axes[1].plot(model_accuracy_list)
plt.show()
```

Figures below present training loss and accuracy over time. We are happy that figures show no suprising behaviour from our model. The loss and accuracy values are satisifying our expactations.

|||
|:---|---:|
|Training loss|Training Accuracy|

## Model Testing

We trained our model. Now, we try to get the metrics by testing the model with testing data. Similar to model training, we download the test dataset and process the test data with tf.data.experimental.make_csv_dataset to parse the testing data. However, the difference is that, when in model testing phase, we set the training parameter in model to false.

### Parsing Testing Data

We download the test data from https://storage.googleapis.com/download.tensorflow.org/data/iris_test.csv. Processing testing data is similar to training data.

```python
# Model evaluation using test data
# Using single epoch to test the model.
# Dataset is downloaded and store in local machine.
# Parsing the test data.

dataset = self.makeDataCSV("C:/Users/JunnanLu123/iris_test.csv")
dataset = dataset.map(self.pack2)
```

Unlike training the model, we iterate through each row of the testing data in one epoch and compute the predictions. Then, we compare the model output to the actual value.

```python
# Calculates how often predictions equals labels.

accuracy = tf.keras.metrics.Accuracy()
test_accuracy_list=[]
```

```python
# We iterate every sample from the dataset and compare to the
# model's prediction against the actual to measure model's
# accuracy across the entire test dataset.

for features, label in dataset:
    prediction = model(features, training=False)
    prediction = tf.nn.softmax(prediction, axis=1,name='softmax')
    prediction = tf.argmax(prediction, axis=1, output_type=tf.dtypes.int32)
    print (prediction)
    accuracy.update_state(label, prediction)
    print (accuracy.result())
    test_accuracy_list.append(accuracy.result())
print ("TEST Accuracy {:1.2%}".format(accuracy.result()))
```

Our model archives test accuracy at 93.33% after 100 epchos of model training loops.

## Using Trained Model to Predict

Now, we are going to use our trained model to compute some unlabeled data. The dataset is downloaded from "https://gist.github.com/curran/a08a1080b88344b0c8a7#file-iris-csv". We parse and extract the feature data from the dataset similar to methods mentioned earlier. The only difference is that we choose first 10 rows of data to make a batch.

```python
# Making data from the CSV file with specified batch size
# The tf.data.experimental.make_csv_dataset method return
# parsed dataset

def makeDataFromCSV(self, name, batchSize):
    # Specified column names to make a suitable format
    column_names = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species']
    feature_names = column_names[:-1]
    label_name = column_names[-1]
    dataset = tf.data.experimental.make_csv_dataset(name, batch_size=batchSize, column_names = column_names, label_name = label_name, num_epochs = 1)
    return dataset
```

```python
# Extracting features from the dataset. In here,
# we take 10 samples from the features data
# letting our model to predict

predict_data = makeDataFromCSV("C:/Users/Byzantin/source/repos/Data1/Data1/iris1.csv",10)
predict_data = predict_data.map(self.pack2)
features, labels = unpackData(predict_data)
```

```python
# Helper function for making the feature and
# label batch

def unpackData(self, dataset):
    features, labels = next(iter(dataset))
    return features, labels
```

```python
# List of three class names
# Labeled data mapped with list index.

class_name = ['Iris Setosa', 'Iris Versicolor', 'Iris Virginica']
```

```python
# Using the trained model to predict unlabeled data.
# prediction stores model result.
prediction = model(features, training=False)

# Iterating through the model result.
for index, logits in enumerate(prediction):
    # Record the class index with maximum probability.
    class_index = tf.argmax(logits).numpy()
    # Calculate the class softmax probability.
    probability = tf.nn.softmax(logits)[class_index] * 10
    # Display the index, class name, and predicted class probability
    name = class_name[class_index]
    print ("index {}, class: {}, probability {:2.2%}".format(index,name,probability))
```

The figure below denotes the model predict results for 10 rows of input data.

The figure below presenting the loss surface in 3D plot.

## Summary

We build and train a machine learning model to predict the Iris flowers. The model, that we build, has four layers with 2 hidden layers. The final output layer consists of three outputs denoting three classes of Iris flower. We choose the m Our model is implemented by Python, Tensorflow, and Keras machine learning framework. The learning algorithm for our model to predict is forward multilayer Neuro Networks. We refine the learning algorithm by update learning metrics such as network weights and bias with Stochastic Gradient Descent algorithm. During testing phase, the model reaches the accuracy at 93.33%. We, then, are able to use our trained model to predict flowers on unlabeled data. In the future, we will need to test our model on noise data. Also, we intend to implement back propagation algorithm to train our model and compare the model performace with just feed forward training.

## References

[1] Billy Lanberta, Yash Katariya, Dimitry Volodin, Mingho Heo, Mark Daoust, Alan Chiao, Tensorflower Gardener Flower. https://www.tensorflow.org/tutorials/customization/custom_training_walkthrough

[2] Richard O. Duda, Peter E. Hart, David G. Stork. *Patter Classfication*. John Wiley & Sons Inc, United States, 2006.

[3] Rafael C. Gonzalez, Richard E. Woods. *Digital Image Processing*. Pearson Education Inc, United States, 2018.

[4] Pang Ning Tan, Michael Steinbach, Vipin Kumar. *Introduction to Data Mining*. Pearson Education Inc, United States, 2008.

[5] Junnan Lu, Alex Thomo. *An Experimental Evaluation of Giraph and Graphchi*. FAB-ASONAM, 2016.

[6] Henry D. Block And Simon A Levin. On the boundedness of an iterative procedure for solving a system of linear inequalities. *Proceedings of the American Mathematics Society,* 26:229-235, 1970.

[7] Federico Girosi, Michael Jones, and Tomaso Poggio. *Regularization Theory and Neuro Networks Architectures*. Neural Computation, 219-269, 1995.

[8] Yann Le Cun. *A learning scheme for asymmetric threshold networks*. *Proceeding of Cognitiva 85*, 599-604, France, 1985.


```python
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
import sympy as sy
sy.init_printing()
```

#  Null Space

The **null space**, denoted as $\text{Nul}A$ is the solution set of a homogeneous linear system, i.e. $Ax=0$.

A null space is a always a subspace of $\mathbb{R}^n$, why? Because the solution can always be at origin $(0, 0, ...)$.

As an example, consider a linear system.

$$
2x_1-x_2+x_3 = 0\\
x_1+2x_2+3x_3= 0
$$

The augmented matrix is

$$
\left[
\begin{matrix}
2 & -1 & 1 & 0\\
1 & 2 & 3 & 0
\end{matrix}
\right]
$$

Before solving the system, we have already known there is no unique solution since a free variable presents, due to fact that two equation with three variables.

Solve for the reduced echelon form.

```python
Aug = sy.Matrix([[2,-1,1,0],[1,2,3,0]])
Aug.rref()
```

$x_3$ is a free variable, the solution set can be written as

$$
\left[
\begin{matrix}
x_1 \\ x_2 \\ x_3
\end{matrix}
\right]=
\left[
\begin{matrix}
-x_3 \\ -x_3 \\ x_3
\end{matrix}
\right]=
x_3\left[
\begin{matrix}
-1 \\ -1 \\ 1
\end{matrix}
\right]
$$

which is a line passing both origin $(0, 0, 0)$ and $(-1, -1, 1)$, also a subspace of $\mathbb{R}^3$.

Consider another example, suppose we have an augmented matrix

```python
Aug = sy.Matrix([[-3,6,-1,1,-7,0],[1,-2,2,3,-1,0],[2,-4,5,8,-4,0]]);Aug
```

$\displaystyle \left[\begin{matrix}-3 & 6 & -1 & 1 & -7 & 0\\1 & -2 & 2 & 3 & -1 & 0\\2 & -4 & 5 & 8 & -4 & 0\end{matrix}\right]$

```python
Aug.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & -2 & 0 & -1 & 3 & 0\\0 & 0 & 1 & 2 & -2 & 0\\0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  2\right)\right)$

The solution can be written as:

$$
\left[
\begin{matrix}
x_1 \\ x_2 \\ x_3 \\x_4 \\ x_5
\end{matrix}
\right]=
\left[
\begin{matrix}
2x_2+x_4-3x_5 \\ x_2 \\ -2x_4+2x_5 \\x_4 \\ x_5
\end{matrix}
\right]=
x_2\left[
\begin{matrix}
2 \\ 1 \\ 0 \\0 \\ 0
\end{matrix}
\right]
+
x_4\left[
\begin{matrix}
1 \\ 0 \\ -2 \\1 \\ 0
\end{matrix}
\right]
+x_5\left[
\begin{matrix}
-3 \\ 0 \\ 2 \\0 \\ 1
\end{matrix}
\right]
$$

The $\text{Nul}A$ is a subspace in $\mathbb{R}^5$ with $\text{dim}A=3$.

#  Null Space vs Col Space

Consider matrix $A$

```python
A = sy.Matrix([[2,4,-2,1],[-2,-5,7,3],[3,7,-8,6]]);A
```

Column space is a subspace in $\mathbb{R}^n$, what is $n$? It is the number of rows, $n=3$.

Null space is a subspace in $\mathbb{R}^m$, what is $m$? It is the number of columns, $m=4$.

How to find any nonzero vector in $\text{Col}A$ and in $\text{Nul}A$?

Any column in a matrix can be a nonzero vector in $\text{Col}A$, for instance first column: $(2, -2, 3)^T$.

But to find a nonzero vector in null space requires some effort, construct the augmented matrix then turn it into rref.

```python
Aug = sy.Matrix([[2,4,-2,1,0],[-2,-5,7,3,0],[3,7,-8,6,0]]);Aug.rref()
```

The solution set with a free variable $x_3$ (because column 3 has no pivot) is

$$
\left[
\begin{matrix}
x_1 \\ x_2 \\ x_3\\x_4
\end{matrix}
\right]=
\left[
\begin{matrix}
-9x_3 \\ 5x_3 \\ x_3\\0
\end{matrix}
\right]
$$

If we pick $x_3 =1$, a nonzero vector in $\text{Nul}A$ is $(-9, 5, 1, 0)^T$

Now consider two vectors

$$
u = \left[
\begin{matrix}
3 \\ -2 \\ -1\\ 0
\end{matrix}
\right],\qquad
v = \left[
\begin{matrix}
3 \\ -1\\3
\end{matrix}
\right]\\
$$

Is $u$ in $\text{Nul}A$? It can be verified easily

```python
u = sy.Matrix([[3],[-2],[-1],[0]])
A*u
```

$Au\neq \mathbf{0}$, therefore $u$ is not in $\text{Nul}A$.

Is $v$ in $\text{Col}A$? Construct an augmented matrix with $v$, then solve it

```python
v = sy.Matrix([[3],[-1],[3]])
A.row_join(v).rref()
```

The augmented matrix show there are solutions, i.e. $v$ is a linear combination of its column space basis, so $v$ is in $\text{Col}A$.

#  Row Space

The **Row space** denoted as $\text{Row}A$, contains all linear combination of row vectors and subspace in $\mathbb{R}^n$.

If we perform row operations on $A$ to obtain $B$, both matrices have the same row space, because $B$'s rows are linear combinations of $A$'s. However, row operation will change the row dependence.

##  An Example

Find the row, column and null space of

```python
A = sy.Matrix([[-2, -5, 8, 0, -17],
               [1, 3, -5, 1, 5],
               [3, 11, -19, 7, 1],
               [1, 7, -13, 5, -3]]);A
```

```python
B = A.rref();B
```

The basis of the row space of $B$ is its first 3 rows: $(1,0,1,0,1), (0, 1, -2, 0, 3), (0, 0, 0, 1, -5)$ which are also the basis of the row space of $A$. However it does not necessarily mean that first 3 rows of $A$ forms the basis for row space, because the dependence among rows changed by row operation.

In constrast, the basis of col space of $A$ is $(-2, 1, 3, 1)^T, (-5, 3, 11, 7)^T, (0, 1, 7, 5)^T$.

```python
Aug = A.row_join(sy.zeros(4,1));Aug.rref()
```

The null space is

$$
\left[
\begin{matrix}
x_1 \\ x_2 \\ x_3\\x_4 \\x_5
\end{matrix}
\right]=
\left[
\begin{matrix}
-x_3-x_5 \\ 2x_3-3x_5 \\ x_3\\5x_5 \\x_5
\end{matrix}
\right]=
x_3\left[
\begin{matrix}
-1 \\ 2 \\ 1\\0 \\0
\end{matrix}
\right]+
x_5
\left[
\begin{matrix}
-1 \\ -3 \\ 0\\5 \\1
\end{matrix}
\right]
$$

#  Rank

Definition of rank:
The **rank** is the dimension of the column space of $A$. The **nullity** of $A$ is the dimension of the null space.

##  The Rank Theorem

The dimensions of the column space and the row space of an $m \times n$ matrix $A$ are equal that is why we only need to say rank is the dimension of the column space.

This common dimension, the rank of $A$, also equals the number of pivot positions in $A$ and satisfies the equation
$$
\operatorname{rank} A+\operatorname{dim} \mathrm{Nul} A=n
$$

The intuition is that when a matrix $A$ is converted into rref $B$, we can indirectly(matching the same column from $B$ to $A$) see the basis of column space, those columns in corresponding rref have pivots.

And in rref, we can also see the basis of row space directly, every row in the basis of row space must have a pivot. And those rows which does not have pivots are for free variables, which is the dimension of null space.

##  Example 1

If $A$ is $45 \times 50$ matrix with a $10$-dimension nullity, what is the rank of $A$?

$10$-$D$ nullity means 10 free variables, so the pivots are $50-10=40$, which is also the rank of $A$.

##  Example 2

The matrices below are row equivalent.
$$
A=\left[\begin{array}{rrrrr}
2 & -1 & 1 & -6 & 8 \\
1 & -2 & -4 & 3 & -2 \\
-7 & 8 & 10 & 3 & -10 \\
4 & -5 & -7 & 0 & 4
\end{array}\right], \quad B=\left[\begin{array}{rrrrr}
1 & -2 & -4 & 3 & -2 \\
0 & 3 & 9 & -12 & 12 \\
0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 0
\end{array}\right]
$$
1. Find rank $A$ and $\operatorname{dim}$ Nul $A$
2. Find bases for Col $A$ and Row $A$.
3. What is the next step to perform to find a basis for Nul $A$ ?
4. How many pivot columns are in a row echelon form of $A^{T} ?$

1. $rank(A)=2$, because $B$ has two pivots. And nullity is the number of free variables, there are 3, so $\text{dim Nul}A = 3$.

```python
A = sy.Matrix([[2,-1,1,-6,8,0],
               [1,-2,-4,3,-2,0],
               [-7,8,10,3,-10,0],
               [4,-5,-7,0,4,0]])
A.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 2 & -5 & 6 & 0\\0 & 1 & 3 & -4 & 4 & 0\\0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1\right)\right)$

2. Bases for $\text{Col}A$ is $(2,1,-7,4)^T, (-1,-2,8,-5)^T$, and for $\text{Row}A$ is $(1,-2,-4,3,-2),(0,3,9,-12,12)$.

The $\text{Nul}A$ and basis is

$$
\left[
\begin{matrix}
x_1 \\ x_2 \\ x_3\\x_4 \\x_5
\end{matrix}
\right]=
\left[
\begin{matrix}
-2x_3+5x_4-6x_5 \\ -3x_3+4x_4-4x_5 \\ x_3\\x_4 \\x_5
\end{matrix}
\right]=
x_3
\left[
\begin{matrix}
-2 \\ -3 \\ 1\\0 \\0
\end{matrix}
\right]+
x_4
\left[
\begin{matrix}
5 \\ 4 \\ 0\\1 \\0
\end{matrix}
\right]+
x_5
\left[
\begin{matrix}
-6 \\ -4 \\ 0\\0 \\1
\end{matrix}
\right]
$$

3. Perform rref on augmented $A$

4. Transpose $A$ then do rref.

```python
A.T.rref()
```

There are 2 pivot columns.

Actually, we don't need any calculation to know the rank of $A^T$, because

$$
rank(A)=rank(A^T)
$$

#  Orthogonality of $\text{Nul}A$ and $\text{Row}A$

##   $\text{Nul}A \perp \text{Row}A$

Here is the intersting connections of these subspaces we have discussed. Consider

```python
A = sy.Matrix([[5, 8, 2], [10, 16, 4], [3, 4, 1]]);A
```

```python
A.rref()
```

The basis of row space of $A$ is $(1, 0, 0)$ and $(0, 1, .25)$.And the $\text{Row}A$ is

$$
\text{Row}A=
s\left[
\begin{matrix}
1 \\ 0\\ 0
\end{matrix}
\right]+
t\left[
\begin{matrix}
0 \\ 1\\ 0.25
\end{matrix}
\right]
$$

The $\text{Nul}A$ is
$$
\left[
\begin{matrix}
x_1 \\ x_2\\ x_3
\end{matrix}
\right]=
x_3
\left[
\begin{matrix}
0 \\ -.25\\ 1
\end{matrix}
\right]
$$

Now we can visualize their relations geometrically. Again keep in mind that Matplotlib does not render 3D properly, so you need some imagination as well.

Here is what we observe.

The $\text{Row}A$ is a plane and $\text{Nul}A$ is a line which is perpendicular to the plane. It is easy to grasp the idea if you notice that in a homogeneous system $Ab = \mathbf{0}$, it breaks down into many dot products

$$
Ab =\left[
\begin{matrix}
A_{1i}\cdot b \\ A_{2i}\cdot b\\ A_{3i}\cdot b
\end{matrix}
\right]
$$

where $A_{1i}, A_{2i}, A_{3i}$ are the rows of $A$. In later chapters we will prove when the dot product of two vectors equals zero, which means geometrically they are perpendicular.

```python
%matplotlib inline
s = np.linspace(-1, 1, 10)
t = np.linspace(-1, 1, 10)
S, T = np.meshgrid(s, t)

X = S
Y = T
Z = T*.25

fig = plt.figure(figsize = (8,8))

ax = fig.add_subplot(111,projection='3d')
ax.plot_surface(X, Y, Z, alpha = .9, cmap=plt.cm.coolwarm)

x3 = np.linspace(-1, 1, 10)
x1 = 0*x3
x2 = -.25*x3
ax.plot(x1,x2,x3, lw = 5)

ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
ax.set_zlabel('z-axis', size = 18)

ax.axis([-1,1,-1,1])

ax.text(x = 1, y = -1, z = -.25, s = r'$Row\ A$', size = 17)
ax.text(0, -.25, 1, s = r'$Nul\ A$', size = 17)

ax.view_init(7, 20)
plt.show()
```

##   $\text{Nul}A^T \perp \text{Col}A$

The nullity of $A^T$ is

```python
A = sy.Matrix([[5, 8, 2], [10, 16, 4], [3, 4, 1]]);A.T.rref()
```

The $\text{Nul}A^T$ is

$$
\left[
\begin{matrix}
x_1 \\ x_2\\ x_3
\end{matrix}
\right]=
x_2
\left[
\begin{matrix}
-2 \\ 1\\ 0
\end{matrix}
\right]
$$

The  $\text{Col}A$ is

```python
A.rref()
```

$$
\text{Col}A=
s\left[
\begin{matrix}
5 \\ 10\\ 3
\end{matrix}
\right]+
t\left[
\begin{matrix}
8 \\ 16\\ 4
\end{matrix}
\right]
$$

$\text{Col}A$ is a plane and $\text{Nul}A^T$ is a line perpendicular to the plane. The intuition is similar to $\text{Nul}A \perp \text{Row}A$, here you can think of a system look like $b^TA = \mathbf{0}^T$.

```python
%matplotlib inline
s = np.linspace(-1, 1, 10)
t = np.linspace(-1, 1, 10)
S, T = np.meshgrid(s, t)

X = 5*S+8*T
Y = 10*S+16*T
Z = 3*S+4*T

fig = plt.figure(figsize = (8,8))

ax = fig.add_subplot(111,projection='3d')
ax.plot_surface(X, Y, Z, cmap=plt.cm.coolwarm)

x2 = np.linspace(-1, 1, 10)
x3 = x2*0
x1 = -2*x2

ax.plot(x1,x2,x3, lw = 3)

ax.set_xlabel('x-axis', size = 18)
ax.set_ylabel('y-axis', size = 18)
ax.set_zlabel('z-axis', size = 18)

ax.axis([-1,1,-1,1])

ax.view_init(-67, 35)
```

#  Rank Decomposition

Consider a matrix $A$, the purpose is to decompose it into the multiplication of $C$, $R$, which are the  bases of column space and row space respectively.

$$
A = CR
$$

```python
A = sy.Matrix([[2, 4, 1, -1], [4, 2, -4, 2], [2, -2, -5, 3], [1, 9, -3, 2]]);A
```

```python
Arref = A.rref();Arref
```

Get the basis of $\text{Col}A$.

```python
ColA_basis = A[:,:3];ColA_basis
```

Then get the $\text{Row}A$.

```python
RowA_basis = Arref[0][0:3,:];RowA_basis
```

Multiply $CR$, we are getting back $A$.

```python
ColA_basis*RowA_basis
```

Verify if $CR$ equals $A$.

```python
ColA_basis*RowA_basis == A
```

    True

Chapter 10 -Null Space vs Col Space, Row Space and Rank.ipynb


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 17: Exogenous Shocks in the New-Keynesian Model

In this notebook, we will compute impulse responses to exogenous shocks in the new-Keynesian model and interpret the results.

## The New-Keynesian Model

The most basic version of the New-Keynesian Model can be expressed as:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
i_{t} & = \bar{r} + \pi^T + \phi_{\pi}\big(\pi_t - \pi^T\big) + \phi_{y}\big(y_t - \bar{y}\big) + v_t\\
\pi_t -\pi^T & = \beta \left( E_t\pi_{t+1} - \pi^T\right)  + \kappa (y_t -\bar{y})+ u_t,
\end{align}

where: $y_t$ is (log) output, $r_t$ is the real interest rate, $i_t$ is the nominal interest rate, $\pi_t$ is the rate of inflation between periods $t-1$ and $t$, $\bar{r}$ is the long-run average real interest rate or the *natural rate of interest*, $\beta$ is the household's subjective discount factor, and $\pi^T$ is the central bank's inflation target. The coeffieints $\phi_{\pi}$ and $\phi_{y}$ reflect the degree of intensity to which the central bank *endogenously* adjusts the nominal interest rate in response to movements in inflation and output.

The variables $g_t$, $u_t$, and $v_t$ represent exogenous shocks to aggregate demand, inflation, and monetary policy. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}\\
v_{t+1} & = \rho_v v_{t} + \epsilon^v_{t+1}.
\end{align}

The goal is to compute impulse responses in the model to all three exogenous shocks. We will use the following parameterization:

| $\bar{y}$ | $\beta$ | $\bar{r}$    | $\kappa$ | $\pi^T$ | $\phi_{\pi}$ | $\phi_y$ | $\rho_g$ | $\rho_u$ | $\rho_v$ |
|-----------|---------|--------------|----------|---------|--------------|----------|----------|----------|----------|
| 0         | 0.995   | $-\log\beta$ | 0.1      | 0.02/4  | 1.5          | 0.5/4    | 0.5      | 0.5      | 0.5      |

## Solution

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series. CELL PROVIDED
parameters = pd.Series()
parameters['y_bar'] = 0
parameters['beta'] = 0.995
parameters['r_bar'] = -np.log(parameters.beta)
parameters['kappa'] = 0.1
parameters['pi_T'] = 0.02/4
parameters['phi_pi'] = 1.5
parameters['phi_y'] = 0.5/4
parameters['rho_g'] = 0.5
parameters['rho_u'] = 0.5
parameters['rho_v'] = 0.5

# Create variable called 'varNames' that stores the variable names in a list with state variables ordered first
varNames = ['g','u','v','y','pi','i','r']

# Create variable called 'shockNames' that stores an exogenous shock name for each state variable.
shockNames = ['e_g','e_u','e_v']

# Define a function that evaluates the equilibrium conditions of the model solved for zero.
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters
    p = parameters

    # Current variables
    cur = variables_current

    # Forward variables
    fwd = variables_forward

    # IS equation
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy
    monetary_policy = p.r_bar + p.pi_T + p.phi_pi*(cur.pi - p.pi_T) + p.phi_y*cur.y + cur.v - cur.i

    # Phillips curve
    phillips_curve = p.beta*(fwd.pi- p.pi_T) + p.kappa*cur.y + cur.u - (cur.pi-p.pi_T)

    # Demand process
    demand_process = p.rho_g*cur.g - fwd.g

    # Monetary policy process
    monetary_policy_process = p.rho_v*cur.v - fwd.v

    # Inflation process
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array
    return np.array([
            is_equation,
            fisher_equation,
            monetary_policy,
            phillips_curve,
            demand_process,
            monetary_policy_process,
            inflation_process
        ])

# Initialize the model into a variable named 'nk_model'
nk_model = ls.model(equations = equilibrium_equations,
                 nstates=3,
                 varNames=varNames,
                 shockNames=shockNames,
                 parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model
guess = [0,0,0,0,0.01,0.01,0.01]
nk_model.compute_ss(guess)
```

```python
# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'loglinear' to False
nk_model.approximate_and_solve(loglinear=False)

# Print the solution to the model
print(nk_model.solved())
```

    Solution to the linear system:

    g[t+1] = 0.5·g[t]+e_g[t+1]

    u[t+1] = 0.5·u[t]+e_u[t+1]

    v[t+1] = 0.5·v[t]+e_v[t+1]

      y[t] = 1.2136·g[t]-2.4151·u[t]-1.2136·v[t]

     pi[t] = 0.2415·g[t]+1.5094·u[t]-0.2415·v[t]

      i[t] = 0.514·g[t]+1.9623·u[t]+0.486·v[t]

      r[t] = 0.3932·g[t]+1.2075·u[t]+0.6068·v[t]

**Questions**

1. Will a positive shock to $g_t$ increase or decrease $y_t$?
2. Will a positive shock to $g_t$ increase or decrease $\pi_t$?
3. Will a positive shock to $u_t$ increase or decrease $y_t$?
4. Will a positive shock to $u_t$ increase or decrease $\pi_t$?

**Answers**

1. The coefficient on $g_t$ in the solution for $y_t$ is *positive* so the shock will *increase* $y_t$ <!-- answer -->
2. The coefficient on $g_t$ in the solution for $\pi_t$ is *positive* so the shock will *increase* $\pi_t$ <!-- answer -->
1. The coefficient on $u_t$ in the solution for $y_t$ is *negative* so the shock will *decrease* $y_t$ <!-- answer -->
4. The coefficient on $u_t$ in the solution for $\pi_t$ is *positive* so the shock will *increase* $\pi_t$ <!-- answer -->

## Impulse Responses

Compute a 21 period impulse response of the model's variables to a:

* 0.01 unit shock to $g_t$
* 0.01/4 unit shock to $u_t$
* 0.01/4 unit shock to $v_t$

```python
# Compute impulse responses
nk_model.impulse(T=21,t0=5,shocks=[0.01,0.01/4,0.01/4])
```

### Demand Shock

Plot the computed impulses responses of the nominal interest rate, the real interest rate, output, and inflation to the 0.01 unit demand shock. Express inflation and interest rates in *annualized* (e.g., multiplied by 4) terms.

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,8))

# Create upper-left axis. PROVIDED
ax1 = fig.add_subplot(2,2,1)
(nk_model.irs['e_g']['i']*400).plot(ax=ax1,lw=3,alpha=0.75,title='Nominal Interest',grid=True)
ax1.set_ylabel('% dev from steady state')

# Create upper-right axis. PROVIDED
ax2 = fig.add_subplot(2,2,2)
(nk_model.irs['e_g']['r']*400).plot(ax=ax2,lw=3,alpha=0.75,title='Real Interest',grid=True)
ax2.set_ylabel('% dev from steady state')

# Create lower-left axis. PROVIDED
ax3 = fig.add_subplot(2,2,3)
(nk_model.irs['e_g']['y']*100).plot(ax=ax3,lw=3,alpha=0.75,title='Output',grid=True)
ax3.set_ylabel('% dev from steady state')

# Create lower-right axis. PROVIDED
ax4 = fig.add_subplot(2,2,4)
(nk_model.irs['e_g']['pi']*400).plot(ax=ax4,lw=3,alpha=0.75,title='Inflation',grid=True)
ax4.set_ylabel('% dev from steady state')

ax1.set_ylim([-0.5,2.5])
ax2.set_ylim([-0.5,2.5])
ax3.set_ylim([-0.5,1.5])
ax4.set_ylim([-0.5,1.5])

fig.tight_layout()
```

### Inflation Shock

Plot the computed impulses responses of the nominal interest rate, the real interest rate, output, and inflation to the 0.01/4 unit inflation shock. Express inflation and interest rates in *annualized* (e.g., multiplied by 4) terms.

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,8))

# Create upper-left axis. PROVIDED
ax1 = fig.add_subplot(2,2,1)
(nk_model.irs['e_u']['i']*400).plot(ax=ax1,lw=3,alpha=0.75,title='Nominal Interest',grid=True)
ax1.set_ylabel('% dev from steady state')

# Create upper-right axis. PROVIDED
ax2 = fig.add_subplot(2,2,2)
(nk_model.irs['e_u']['r']*400).plot(ax=ax2,lw=3,alpha=0.75,title='Real Interest',grid=True)
ax2.set_ylabel('% dev from steady state')

# Create lower-left axis. PROVIDED
ax3 = fig.add_subplot(2,2,3)
(nk_model.irs['e_u']['y']*100).plot(ax=ax3,lw=3,alpha=0.75,title='Output',grid=True)
ax3.set_ylabel('% dev from steady state')

# Create lower-right axis. PROVIDED
ax4 = fig.add_subplot(2,2,4)
(nk_model.irs['e_u']['pi']*400).plot(ax=ax4,lw=3,alpha=0.75,title='Inflation',grid=True)
ax4.set_ylabel('% dev from steady state')

ax1.set_ylim([-.5,2.5])
ax2.set_ylim([-0.5,2.5])
ax3.set_ylim([-1,0.5])
ax4.set_ylim([-0.5,2])

fig.tight_layout()
```

### Monetary Policy Shock

Plot the computed impulses responses of the nominal interest rate, the real interest rate, output, and inflation to the 0.01/4 unit monetary policy shock. Express inflation and interest rates in *annualized* (e.g., multiplied by 4) terms.

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,8))

# Create upper-left axis. PROVIDED
ax1 = fig.add_subplot(2,2,1)
(nk_model.irs['e_v']['i']*400).plot(ax=ax1,lw=3,alpha=0.75,title='Nominal Interest',grid=True)
ax1.set_ylabel('% dev from steady state')

# Create upper-right axis. PROVIDED
ax2 = fig.add_subplot(2,2,2)
(nk_model.irs['e_v']['r']*400).plot(ax=ax2,lw=3,alpha=0.75,title='Real Interest',grid=True)
ax2.set_ylabel('% dev from steady state')

# Create lower-left axis. PROVIDED
ax3 = fig.add_subplot(2,2,3)
(nk_model.irs['e_v']['y']*100).plot(ax=ax3,lw=3,alpha=0.75,title='Output',grid=True)
ax3.set_ylabel('% dev from steady state')

# Create lower-right axis. PROVIDED
ax4 = fig.add_subplot(2,2,4)
(nk_model.irs['e_v']['pi']*400).plot(ax=ax4,lw=3,alpha=0.75,title='Inflation',grid=True)
ax4.set_ylabel('% dev from steady state')

ax1.set_ylim([-0.2,0.8])
ax2.set_ylim([-0.2,0.8])
ax3.set_ylim([-0.4,0.1])
ax4.set_ylim([-0.4,0.1])

fig.tight_layout()
```