# Bonds 3

This notebook presents a yield curve model (a simplified Vasicek model) and then uses it to improve upon the duration hedging approach.

## Load Packages and Extra Functions

```julia
using Printf

include("jlFiles/printmat.jl")
```

    printyellow (generic function with 1 method)

```julia
using Plots, LaTeXStrings

#pyplot(size=(600,400))
gr(size=(480,320))
default(fmt = :png)
```

# Predictions from an AR(1)

Consider an AR(1) with mean $\mu$

$r_{t+1}-\mu=\rho\left(  r_{t}-\mu\right)  +\varepsilon_{t+1}$

The forecast (based on information in $t$) for $t+s$ is

$\text{E}_{t}r_{t+s}=\left(  1-\rho^{s}\right)  \mu+\rho ^{s}r_{t}.$

In the the cells below, we code a function for these forecasts and then plot the results (for many forecasting horizons and also for several different starting values).

```julia
"""
    AR1Prediction(x0,ρ,μ,s)

Calculate forecast from AR(1)

x0,ρ,μ,s: scalars
"""
function AR1Prediction(x0,ρ,μ,s)
    E0xs = (1-ρ^s)*μ + ρ^s*x0
    return E0xs
end
```

    AR1Prediction

```julia
(μ,ρ) = (0.05,0.975)

printlnPs("Prediction for t+50, assuming current r=0.07: ",AR1Prediction(0.07,ρ,μ,50))
```

    Prediction for t+50, assuming current r=0.07:      0.056

```julia
sMax = 120                                   #make forecasts for many horizons and initial values
s_range = 1:sMax                             #different horizons
x0      = [0.03;0.05;0.07]

xPred = fill(NaN,(sMax,length(x0)))          #xPred[maturity,x0 value]
for s in s_range, j = 1:length(x0)
    xPred[s,j] = AR1Prediction(x0[j],ρ,μ,s)
end

txt = text(L"(\mu,\rho) = (%$μ,%$ρ)",8,:left)   #laTeX string to plot, notice %$
p1 = plot( s_range,xPred,
           linecolor = [:black :red :blue],
           linewidth = 2,
           linestyle = [:solid :dash :dot],
           label = ["3%" "5%" "7%"],
           title = "Forecast of short interest rate",
           xlabel = "forecast horizon",
           ylabel = "short interest rate",
           annotation = [(80,0.038,txt),
                    (15,0.068,text("(based on different values\nof current short rate)",8,:left))] )
display(p1)
```

# The Vasicek Model

The Vasicek model (without risk premia) implies that

$
y_{t}(n) =a(n)+b(n)r_{t} \: \text{, where }
$

$
b(n)     =(1-\rho^{n})/[(1-\rho)n],
$

$
a(n)     = \mu\left[  1-b(n)\right].
$

In the example, we think of the periods as months but the scale of the interest rate level is annualised. For instance, $y_{t}(36)$ which corresponds to `y[36]` in the code, is the (annualized) continuously compounded interest rate for a bond maturing in 3 years (36 months).

In the cells below, we code a function for this model and then plots the results for many different maturities (horizons) and several initial values of the short interest rate.

```julia
"""
    Vasicek(r,ρ,μ,n)

Vasicek model: calculate interest rate and (a,b) coeffs
r,ρ,μ,n: scalars

n is the maturity, but this could be measured in any unit (months, years, etc),
but changing the units require recalibrating ρ
"""
function Vasicek(r,ρ,μ,n)
    if ρ == 1.0
        b = 1.0
    else
        b = (1-ρ^n)/((1-ρ)*n)
    end
    a = μ*(1-b)
    y = a + b*r
    return y,a,b
end
```

    Vasicek

```julia
y = fill(NaN,(sMax,length(x0)))         #interest rates, different starting values in columns
ab = fill(NaN,(sMax,2))                 #a and b coefs
for n in s_range, j = 1:length(x0)
    #local a, y, b                      #local/global is needed in script
    (y_nj,a,b) = Vasicek(x0[j],ρ,μ,n)
    y[n,j] = y_nj
    if j == 1
        ab[n,:] = [a b]                 #the same across x0 values
    end
end

printblue("a and b for the first few horizons (months):")
printmat([ab[1:4,1]*100 ab[1:4,2]],colNames=["a*100","b"],rowNames=string.(s_range[1:4]),cell00="horizon")
```

    a and b for the first few horizons (months):
    horizon     a*100         b
    1           0.000     1.000
    2           0.062     0.988
    3           0.124     0.975
    4           0.184     0.963

```julia
p1 = plot( s_range,ab,
           linecolor = [:black :red],
           linestyle = [:dot :solid],
           linewidth = 2,
           label = [L"a(n)" L"b(n)"],
           title = "Vasicek model: coefficients",
           xlabel = "maturity (months)",
           annotation = [(80,0.2,txt),
                    (25,0.95,text(L"y(n) = a(n) + b(n)r",8,:left))] )
display(p1)
```

```julia
p1 = plot( s_range,y,
           linecolor = [:black :red :blue],
           linewidth = 2,
           linestyle = [:solid :dash :dot],
           label = ["3%" "5%" "7%"],
           title = "Vasicek model: yield curves",
           xlabel = "maturity (months)",
           ylabel = "interest rate",
           annotation = [(80,0.038,txt),
                    (25,0.069,text("for different current short rates",8,:left))] )
display(p1)
```

# Hedging Using the Vasicek Model

The change of the (value of the) hedge portfolio is

$\begin{equation}
\Delta V=v\Delta P_{H}-\Delta P_{L}
\end{equation}$

A. For an initial value of the short log interest rate $r$, use the Vasicek model to calculate all spot rates needed to discount the cash flows of the two bond portfolios. Repeat for a somewhat different short rate.

B. Calculate prices of both the liability $P_L$ and the hedge bond $P_H$.

C. Calculate the $v$ value that makes $\Delta V = 0$, that is, $v=\Delta P_{L}/\Delta P_{H}$.

## Bond Pricing Functions

```julia
"""
    BillPrice2(y,m)

Zero coupon bond price as a function of (y,m). m is typically measured in terms of years.
"""
BillPrice2(y,m) = exp(-m*y)

"""
    BondPrice3b(y,cf,m)

Calculate bond price from continously compounded interest rates, cash flows and time to cash flow.

y:  scalar or K vector of continuously compounded interest rates
cf: scalar or K vector of cash flows
m: K vector of times (years) of cash flows
"""
function BondPrice3b(y,cf,m)                #cf is a vector of all cash flows
    cdisc = cf./exp.(m.*y)
    P     = sum(cdisc)                      #price
    return P
end
```

    BondPrice3b

## Information about the Two Bonds

In the example below, the liability pays 0.2 every 12 months for 10 years. The hedge bond is a 3-year zero coupon bond.

```julia
nL   = 1:120                       #times of the cash flows of liability, months
nH   = 1:36                        #same, but for hedge bond

cfL = repeat([zeros(11);0.2],10)   #cash flow of liability
cfH = [zeros(35);1];               #cash flow of hedge bond, 3-year zero-coupon bond
```

## A. Calculate Yield Curves

at two different values of the current short rate: $r_0$ and $r_1$.

```julia
ρ = 0.975                       #experiment with this
μ = 0.05

r₀ = 0.05                         #initial (day 0) short log interest rate
r₁ = 0.03                         #another possible short rate

nMax = maximum([nL;nH])            #calculate yield curves the maximum maturity
nM   = 1:nMax                      #time to maturity (months)

(y₀,y₁) = [fill(NaN,nMax) for i=1:2]  #yield curve, before and after
 for n in nM
    y₀[n] = Vasicek(r₀,ρ,μ,n)[1]
    y₁[n] = Vasicek(r₁,ρ,μ,n)[1]
end

printred("compare with the previous figure of yield curves")
```

    compare with the previous figure of yield curves

## B. Calculate the Prices of the Hedge Bond and the Liability

With the two yield curves (different short rate values) from the Vasicek model, calculate prices of the hedge bond and the liability. In particular, the bond price $P$ is the present value of the future cash flows $cf_k$

$P = \sum_{k=1}^{K} \frac{cf_{k}}{\exp\left[m_{k} y(m_{k})\right]  }$,

where $y(m_k)$ is the continuously compounded (annualized) interest rate on a bond maturing in $m_k$ years.

```julia
B₀ = BillPrice2.(y₀,nM/12)           #prices of zero coupon bonds at y₀ curve, /12 to get years
B₁ = BillPrice2.(y₁,nM/12)           #at y₁ curve

p1 = plot( nM,[B₀ B₁],
           linecolor = [:red :black],
           linewidth = 2,
           linestyle = [:solid :dot],
           xlim = (0,120),
           label = ["at short rate of 5%" "at short rate of 3%"],
           title = "Prices of zero coupon bonds",
           xlabel = "maturity (months)",
           ylabel = "" )
scatter!(12:12:120,fill(0.6,9),label="time of liability cash flows")
scatter!([36],[0.62],label="time of hedge bond flow",markershape = :rect)
display(p1)
```

```julia
PL₀ = BondPrice3b(y₀[nL],cfL,nL/12)       #liability, before, /12 to get years
PL₁ = BondPrice3b(y₁[nL],cfL,nL/12)       #after
ΔPL = PL₁ - PL₀

PH₀ = BondPrice3b(y₀[nH],cfH,nH/12)       #hedge bond
PH₁ = BondPrice3b(y₁[nH],cfH,nH/12)
ΔPH = PH₁ - PH₀

printblue("Bond prices (according to the Vasicek model) at different r values")
xy = [PL₀ PH₀;PL₁ PH₁;(PL₁-PL₀) (PH₁-PH₀)]
printmat(xy,colNames=["PL","PH"],rowNames=["at $r₀","at $r₁","Δ"])
```

    Bond prices (according to the Vasicek model) at different r values
                   PL        PH
    at 0.05     1.535     0.861
    at 0.03     1.609     0.896
    Δ           0.074     0.035

## C. Calculate the Hedge Ratio

Calculate the hedge ratio as

$
v = \Delta P_L / \Delta P_H.
$

```julia
v = ΔPL/ΔPH                              #change PL/change PH

printblue("Hedge ratio from the Vasicek model:")
xy = [v;v*PH₀/PL₀]
printmat(xy,rowNames=["v","v*PH₀/PL₀"])

printred("Notice: ρ is important for the hedge ratio. Try also ρ=1 to see how it changes")
```

    Hedge ratio from the Vasicek model:
    v             2.106
    v*PH₀/PL₀     1.181

    Notice: ρ is important for the hedge ratio. Try also ρ=1 to see how it changes

```julia

```


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section 2.3, écoulement d'un film de fluide Newtonien dans une conduite

>  Le développement fait dans Transport Phenomena sera répété ici en développant les solutions avec le calculateur formel sympy et en traçant la solution avec sympy.plot. Nous n'irons pas ici autant dans le détail que dans la section 2.2, Les coordonnées cylindriques sont utilisées, évidemment. Remarquez bien comment la courbure de la géométrie a un effet sur le bilan.

>Vous pouvez suivre le développement détaillé à l'aide du livre et voir comment les équations de bilans initiales deviennent les profils de vitesse, de force, débit, etc...

```python
#
# Pierre Proulx
#
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Paramètres, variables et fonctions
#
r,delta_r,L,rho,g,mu,R,P_0,P_L = sp.symbols('r,delta_r,L,rho,g,mu,R,P_0,P_L')
C1 = sp.symbols('C1')
phi_rz,phi_zz = sp.symbols('phi_rz,phi_zz')
```

```python
#
# Équation du bilan des forces
#
eq  = 2*sp.pi*r*L*phi_rz(r)      -2*sp.pi*(r+delta_r)*L*phi_rz(r+delta_r)
eq += 2*sp.pi*delta_r*r*phi_zz(0)-2*sp.pi*r*delta_r*phi_zz(L)
eq += 2*sp.pi*r*delta_r*L*rho*g

#eq=r*phi_rz(r)/delta_r-(r+delta_r)*phi_rz(r+delta_r)/delta_r+r*P_0/L-r*P_L/L
#
# pose de la limite quand dr tends vers 0
#
eq /= (2*sp.pi*L*delta_r)
display(eq)
```

```python
eq1 = sp.limit(eq, delta_r, 0).doit()
display(eq1)
```

```python
eq1 = eq1.subs(phi_zz(0), P_0)     # les termes de convection s'annulent comme en 2.2
eq1 = eq1.subs(phi_zz(L), P_L)     # mais il reste le terme de pression
display(eq1)
```

```python
# Loi de Newton, le terme phi ne contient que le cisaillement, pas de vitesse radiale.
tau = sp.symbols('tau_rz')
eq2 = sp.Eq(eq1.subs(phi_rz(r),tau(r)))
display(eq2)
```

```python
eq2 = sp.dsolve(eq2)
display(eq2)
```

```python
vz     = sp.Function('v_z')(r)
newton = -mu*sp.Derivative(vz,r)
eq3 = eq2.subs(tau(r),newton)
eq3 = eq3.subs(C1,0)               # C1 doit être nulle sinon on a une force infinie en r=0
display(eq3)
```

```python
eq4=sp.dsolve(eq3,vz)      # rhs pour avoir la droite de l'équation vz=...
display(eq4)
```

```python
constante = sp.solve(sp.Eq(eq4.rhs.subs(r, R), 0), 'C1', dict=True)   # Condition de non-glissement à la paroi
display(constante)
constante = constante[0]
```

```python
vz = eq4.subs(constante)
print('vitesse en fonction de r')
display(vz.simplify())
```

```python
print('Force')
#
# collect et simplify simplifient l'expression, c'est purement cosmétique.
#
display(-mu*vz.lhs.diff(r)*2*sp.pi*r*L)
print('évalué en r=R devient')
display(-mu*vz.rhs.diff(r).subs(r, R)*2*sp.pi*R*L)
```

```python
print('débit et vitesse moyenne')
vz = vz.rhs # On enlève le terme de gauche et à partir d'ici on travaille seulement avec la partie droite
debit = sp.integrate(2*sp.pi*vz*r, (r, 0, R))
display(debit)
```

```python
vmax=vz.subs(r, 0)
display(debit/(sp.pi*R**2)/vmax)   # ce calcul donne quoi?
```

```python
# Maintenant traçons le profil en donnant des valeurs de paramètres réalistes
#
dico={
      'rho':1000,
      'mu':0.001,
      'R':0.05,
      'L':100,
      'P_0':200,
      'P_L':0,
      'g':9.81}

vz=vz.subs(dico)
vzmax=vz.subs(r, 0)
display(vz, vzmax)

plt.rcParams['figure.figsize']=10,8

goptions={
    'title':'Profil de vitesse parabolique',
    'ylabel':'V/Vmax',
    'xlabel':'r/R'}

sp.plot(vz.subs(dico)/vzmax, (r,0,R.subs(dico)), **goptions);
```

```python

```


# Characterization of Discrete Systems in the Spectral Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## The Transfer Function

Different time-domain representations are used to characterize linear time-invariant (LTI) discrete systems. For instance, the constant coefficients of [difference equations](../discrete_systems_time_domain/difference_equation.ipynb) and the [impulse response](../discrete_systems_time_domain/impulse_response.ipynb). The [transfer function](../discrete_systems_time_domain/eigenfunctions.ipynb#Transfer-Function) denotes the complex factor $H(z)$, which an eigenfunction $z^k$ at the input of an LTI system is weighted with, when passing through the system. Since the $z$-transform decomposes discrete signals with respect to these eigenfunctions, the transfer function constitutes a representation of an LTI system in the spectral domain (e.g. discrete (time) Fourier domain). The links between the transfer function and the time-domain representations of LTI systems are discussed in the following.

### Relation to Impulse Response and In-/Output Signal

It was already outlined in the context of the [convolution theorem of the $z$-transform](../z_transform/theorems.ipynb#Convolution-Theorem) and [discrete-time Fourier transform](../discrete_time_fourier_transform/theorems.ipynb#Convolution-Theorem), that an LTI system can be represented equivalently in the temporal and spectral domain. In the temporal domain the output $y[k]$ of a discrete LTI system is given by convolving the input signal $x[k]$ with the [impulse response](../discrete_systems_time_domain/impulse_response.ipynb) $h[k]$ of the system

\begin{equation}
y[k] = x[k] * h[k]
\end{equation}

Applying the convolution theorem of the $z$-transform yields

\begin{equation}
Y(z) = X(z) \cdot H(z)
\end{equation}

where $X(z) = \mathcal{Z} \{ x[k] \}$ and $Y(z) = \mathcal{Z} \{ y[k] \}$ denote the $z$-transforms of the input and output signal, respectively. The transfer function $H(z)$ is given as the $z$-transform of the impulse response $h[k]$

\begin{equation}
H(z) = \mathcal{Z} \{ h[k] \}
\end{equation}

This may also be concluded from the [previously derived link between the impulse response and the transfer function](../discrete_systems_time_domain/eigenfunctions.ipynb#Link-between-Transfer-Function-and-Impulse-Response). The same relation holds in the discrete-time Fourier domain if the region of convergence (ROC) of the signals and impulse response include the unit circle $|z| = 1$ of the $z$-plane

\begin{equation}
Y(e^{j \Omega}) = X(e^{j \Omega}) \cdot H(e^{j \Omega})
\end{equation}

where $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \}$, $Y(e^{j \Omega}) = \mathcal{F}_* \{ y[k] \}$ and $H(e^{j \Omega}) = \mathcal{F}_* \{ h[k] \}$ denote the discrete-time Fourier transforms (DTFTs) of the input/output signal and impulse response, respectively.

The derived results show that an LTI system can be fully characterized either in the temporal or spectral domain by its impulse response $h[k]$ or transfer function $H(z)$ respectively $H(e^{j \Omega})$.

It can furthermore be concluded that the transfer function is given as the quotient between the spectrum of the output $Y(z)$ and input signal $X(z)$

\begin{equation}
H(z) = \frac{Y(z)}{X(z)}
\end{equation}

for $X(z) \neq 0$. Hence, the transfer function may be derived by division of the spectra of the output and input signal.

**Example - Moving Average**

The transfer function $H(z) = \mathcal{Z} \{ h[k] \}$ of the moving average filter is derived from its [impulse response](../discrete_systems_time_domain/impulse_response.ipynb#Finite-Impulse-Response)

\begin{equation}
h[k] = \frac{1}{N} \cdot \text{rect}_N[k]
\end{equation}

using the $z$-transform of the rectangular signal as

\begin{equation}
H(z) =  \begin{cases}
\frac{1}{N} \frac{1 - z^{-N}}{1 - z^{-1}} & \text{for } z \neq 1 \\
1 & \text{for } z = 1
\end{cases}
\end{equation}

for $z \in \mathbb{C} \setminus \{ 0 \}$. Inspection of the transfer function $H(z)$ reveals that is has $N-1$ poles in the origin and $N-1$ zeros from the [roots of unity](https://en.wikipedia.org/wiki/Root_of_unity)

\begin{equation}
z_{0n} = e^{j \frac{2 \pi}{N} n} \quad \text{for } n=1,\dots,N-1
\end{equation}

The zeros are equiangularly distributed on the unit circle $|z| = 1$. Note that there is never a zero at $z=1$ which is a requirement for the here discussed filter characteristics. The transfer function may also be given in the discrete-time Fourier domain as the region of convergence includes the unit circle. Substituting $z$ with $e^{j \Omega}$ in above transfer function yields

\begin{equation}
H(e^{j \Omega}) =  \begin{cases}
\frac{1}{N} \frac{1 - e^{-j \Omega N}}{1 - e^{- j \Omega}} & \text{for } \Omega \neq 0 \\
1 & \text{for } \Omega = 0
\end{cases}
\end{equation}

The absolute value $|H(e^{j \Omega})|$ of the transfer function is plotted below for illustration using $N=5$, thus $N-1$ zeros in the plot can be observed between $0\leq \Omega < 2\pi$.

```python
import sympy as sym
sym.init_printing()
%matplotlib inline

N = 5
W = sym.symbols('Omega', real=True)

H = 1/N * (1 - sym.exp(-sym.I*W*N)) / (1 - sym.exp(-sym.I*W))
sym.plot(sym.Abs(H), (W, -sym.pi, +2*sym.pi),
         xlabel=r'$\Omega$', ylabel=r'$|H(e^{j \Omega})|$')
```

    <sympy.plotting.plot.Plot at 0x110c0a8d0>

**Exercises**

* Change the length `N` of the rectangular impulse response. How does the transfer function change?
* What overall character does the transfer function show (low-pass, high-pass, band-pass, band-stop)?
* Regarding the transfer function, can you explain the smoothing property of the [moving average filter](../discrete_systems_time_domain/difference_equation.ipynb#Moving-Average)?

### Relation to Differential Equations

Linear difference equations with constant coefficients quantifying the relation between input $x[k]$ and output $y[k]$ represent an LTI system. In this case a system is characterized by the constants $a_n, b_m \in \mathbb{R}$ of the difference equation

\begin{equation}
\sum_{n = 0}^{N} a_n \; y[k - n] = \sum_{m = 0}^{M} b_m \; x[k -m]
\end{equation}

where $a_0 \neq 0$ and $N$ denoting its degree. Initial values are discarded in the following. Applying a two-sided $z$-transform to the left- and right-hand-side and exploiting the [shift theorem](../z_transform/theorems.ipynb#Shift-Theorem) yields

\begin{equation}
Y(z) \sum_{n=0}^{N} a_n z^{-n} = X(z) \sum_{m=0}^{M} b_m z^{-m}
\end{equation}

Rearranging this result derives the transfer function $H(z)$ of the system as

\begin{equation}
H(z) = \frac{Y(z)}{X(z)} = \frac{\sum\limits_{m=0}^{M} b_m z^{-m}}{\sum\limits_{n=0}^{N} a_n z^{-n}}
\end{equation}

The transfer function of an LTI system whose in- and output relation can be characterized by a linear difference equation is given as rational function in $z^{-1}$. The coefficients of its numerator and denominator polynomial are given by the constant coefficients $b_m$ and $a_n$ of the difference equation. The conversion between both representations of an LTI system is therefore straightforward. Note that $H(z)$ is often normalized such that $a_0=1$.

Discrete systems may be classified into [recursive and non-recursive systems](../discrete_systems_time_domain/difference_equation.ipynb#Recursive-and-Non-Recursive-Systems). This has implications on the poles and zeros of the transfer function.

* a so called **non-recursive system** is given if $a_n = 0$ for $n > 0$
* a so called **recursive system** exhibits at least one $a_n \neq 0$ for $n > 0$

A non-recursive system has $M$ poles in the origin of the z-plane and $M$ zeros in the z-plane.
A recursive system has $N$ poles and $M$ zeros in the z-plane.
In order to obtain $a,b\in\mathbb{R}$ the poles/zeros occur either purely real or as complex-conjugated pairs.

**Example - Second-order system**

The transfer function $H(z)$ of the previously introduced [second-order recursive LTI system](difference_equation.ipynb#Second-Order-System) with the difference equation

\begin{equation}
y[k] - y[k-1] + \frac{1}{2} y[k-2] = x[k]
\end{equation}

is computed. The coefficients of the system can be derived from its difference equation as $a_0 = 1$, $a_1 = -1$, $a_2 = \frac{1}{2}$ and $b_0 = 1$. Introducing these into above equation results in

\begin{align}
&Y(z) - Y(z) z^{-1} + \frac{1}{2} Y(z) z^{-2} = X(z)\\
&H(z) = \frac{Y(z)}{X(z)} = \frac{1}{1 - z^{-1} + \frac{1}{2} z^{-2}}
\end{align}

The magnitude of the transfer function in the DTFT domain $H(e^{j \Omega}) = H(z) \vert_{z = e^{j \Omega}}$ is plotted for illustration of the system properties

```python
# define variables and transfer function
z = sym.symbols('z', complex=True)
W = sym.symbols('Omega', real=True)
H = 1 / (1 - z**(-1) + sym.Rational(1, 2)*z**(-2))

# plot transfer function
sym.plot(sym.Abs(H.subs(z, sym.exp(sym.I*W))), (W, -sym.pi, sym.pi),
         ylim=(0, 3), xlabel=r'$\Omega$', ylabel=r'$|H(e^{j \Omega})|$')
```

    <sympy.plotting.plot.Plot at 0x112186908>

It becomes evident from inspection of the magnitude response that the system has a low-pass character as higher normalized frequencies $\Omega$ are significantly attenuated.

It is common to numerically evaluate the transfer function $H(e^{j \Omega})$ of a discrete system at a number of discrete frequencies $\Omega_\mu = e^{j \frac{2 \pi}{N} \mu}$ for $\mu = 0,1,\dots,N-1$ in order to gain insights into its properties. The function [`scipy.signal.freqz`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.freqz.html) computes the transfer function for the given coefficients $a_n$ and $b_m$ numerically. The resulting sampled transfer function is plotted in the following example

```python
import scipy.signal as sig
import matplotlib.pyplot as plt
import numpy as np

# compute transfer function
a = (1, -1, 1/2)  # convention: a = (a0, a1, a2,...)
b = (1)  # convention: b = (b0, b1, b2,...)
W, H = sig.freqz(b, a, whole=True, worN=np.linspace(-np.pi, np.pi, 2**10))

# plot transfer function
plt.figure()
plt.plot(W, abs(H))
plt.ylabel(r'$|H(e^{j \Omega})|$')
plt.xlabel(r'$\Omega$')
plt.ylim([0, 3])
plt.xlim([-np.pi, np.pi])
plt.grid()
```

### Determination from Internal Structure

If known, the transfer function $H(z)$ may be determined from the internal structure of a discrete LTI system. The structure of a system is often visualized by a block-diagram or [signal-flow graph](https://en.wikipedia.org/wiki/Signal-flow_graph). For LTI systems these are composed from unit delays $z^{-1}$, multiplications and additions. From the analysis of the system structure, the linear difference equation can be derived. Once its coefficients are known it is straightforward to compute the transfer function $H(z)$. As alternative, the transfer function may also be derived by analysis of the system in the $z$-domain. The transfer function is then given as the quotient between the spectrum of the output signal $Y(z)$ and the input signal $X(z)$.

### Modeling a Continuous System by a Discrete System

It is often desired to model a continuous system by a discrete system showing similar properties. Applications include the characterization of continuous systems and digital realization of their discrete counter parts. For instance, the properties of the discrete counterpart can be investigated more conveniently by numerical methods than the continuous system.

In the following it is aimed at modeling the continuous system by a discrete LTI system with [finite impulse response](../discrete_systems_time_domain/impulse_response.ipynb#Finite-Impulse-Response) (FIR). The output signal $y(t) = \mathcal{H} \{ x(t) \}$ of the continuous system reads

\begin{equation}
y(t) = x(t) * h(t)
\end{equation}

where $h(t)$ denotes its impulse response. [Ideal sampling](../sampling/ideal.ipynb#Model-of-Ideal-Sampling) of all signals, e.g. $x[k] = x(k T)$ with the sampling interval $T$, yields their discrete-time counterparts

\begin{equation}
y_M[k] = x_L[k] * h_N[k]
\end{equation}

where its is assumed that the signals are of finite length given by the respective index. Due to the properties of the linear convolution for finite-length signals, the length $M$ of the signal $y_M[k]$ is $M = L+N-1$. In order to represent this relation in the DFT domain, the signals $x_L[k]$ and $h_N[k]$ have to be zero-padded to at least length $M$ in order to yield [equality of the linear and periodic convolution](../discrete_fourier_transform/fast_convolution.ipynb#Equivalence-of-Linear-and-Periodic-Convolution). For instance

\begin{equation}
x_M[k] = \begin{cases}
x_L[k] & \text{for } 0 \leq k < L \\
0 & \text{for } L \leq k < M
\end{cases}
\end{equation}

Introducing the zero-padded signals, performing a DFT of length M on the left- and right-hand side and exploiting the [convolution theorem](../discrete_fourier_transform/theorems.ipynb#Convolution-Theorem) yields

\begin{equation}
Y_M[\mu] = X_M[\mu] \cdot H_M[\mu]
\end{equation}

where for instance $Y_M[\mu] = \text{DFT}_M \{ y_M[k] \}$. Rearranging this result results in

\begin{equation}
H_M[\mu] = \frac{Y_M[\mu]}{X_M[\mu]}
\end{equation}

which requires for $X_M[\mu] \neq 0$.

The FIR $h_N[k]$ of the discrete system is given by the inverse DFT of the transfer function $H_M[\mu]$ followed by a truncation to $N$ samples.

Above relation requires the knowledge of the in- and output signal of the continuous system in order to derive its discrete model. Practically, the continuous system may be fed by the input signal $x(t)$ that has been reconstructed from its discrete counterpart $x[k]$ generated digitally. A Dirac impulse $x[k] = \delta[k]$ would be a natural first choice as it fulfills $X[\mu] \neq 0$. However, the resulting continuous signal after reconstruction $x(t) = \text{sinc} \left( \frac{\pi t}{T} \right)$ shows a very high [Crest factor](https://en.wikipedia.org/wiki/Crest_factor). In many applications this is undesirable as the major energy of the signal is concentrated in a very short period of time. This issue can be overcome by using specific measurement signals that fulfill $X[\mu] \neq 0$ while showing a low crest factor. For instance [chirp signals](https://en.wikipedia.org/wiki/Chirp).

The application example in the next section shows how the impulse response of a electroacoustic system can be measured following the outlined theory.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Complete Working Example

In this example we use the `CausalForest` class from `cforest` to fit a model on data. With the fitted model we predict effects on new data and visualize these predictions. On top we showcase how one can save and load a fitted model.

```python
import numpy as np
from pandas.testing import assert_frame_equal
from cforest.forest import CausalForest
```

```python
from example import simulate
from example import plot_treatment_effect
from example import plot_predicted_treatment_effect

%matplotlib inline
```

## Data

Here we use simulated data. For the sake of simplicity we chose a very simple data generating process using the [Neyman-Rubin potential outcome framework](https://en.wikipedia.org/wiki/Rubin_causal_model).

### Treatment effect:

In the Neyman-Rubin potential outcome framework we postulate the existence of so-called potential outcomes.
That is, for the outcome we want to measure we assume that, in theory, an individual $i$ has a (potential) outcome if it were not treated, which we call $y_i^0$, and a (potential) outcome if it were treated, which we call $y_i^1$.
These potential outcomes exist no matter if the individual was actually treated or not.
In reality we only observe one outcome, namely $y_i$; however, if the individual was treated we get $y_i = y_i^1$ and otherwise $y_i = y_i^0$.

The (individual) **treatment effect** is defined by $\tau_i = y_i^1 - y_i^0$, which again cannot be computed since at least one potential outcome is unobservable.
In the following we assume that the treatment effect is a smooth function which depends on two covariates and our goal will be to estimate this function. Analytically we can write the treatment effect function as

\begin{align}
\tau(x_1, x_2) = \gamma \left[1 + \exp(-\alpha \, (x_1 - 1/3))\right]^{-1} \left[1 + \exp(-\alpha \, (x_2 - 1/3))\right]^{-1} \,,
\end{align}

which (using $\gamma = 5$ and $\alpha = 0.9$) looks as

```python
plot_treatment_effect(alpha=0.9, scale=5, figsize=(14, 10))
```

### Simulation strategy:

For (independent) individuals $i=1,\dots,n$ do:

1. Simulate (independent) covariates: $x_{i1}, x_{i2} \sim \mathrm{Unif}(-15, 15)$
2. Simulate error terms: $\epsilon_i \sim \mathcal{N}(0, \sigma^2)$
3. Compute untreated potential outcome: $y_i^0 = 0.5 + 0.1 x_{i1} - 0.1 x_{i2} + \epsilon_i$
4. Compute treated potential outcome: $y_i^1 = y_i^0 + \tau(x_{i1}, x_{i2})$
5. Simulate treatment status: $T_i \sim \mathrm{Bern}(0.5)$
6. Compute (observed) outcomes: $y_i = T_i y_i^1 + (1 - T_i) y_i^0$
7. **Return**: Outcomes $y_i$, covariates $x_i = (x_{i1}, x_{i2})$ and treatment status $T_i$.

### Create the data:

```python
params = {
    'nobs': 5000,
    'nfeatures': 2,
    'coefficients': [0.5, 0.1, -0.1],
    'error_var': 0.1,
    'seed': 1,
    'alpha': 0.9,
}

X, t, y = simulate(**params)
```

## Estimation (fitting)

### Initiate a Causal Forest estimator:

```python
cfparams = {
    'num_trees': 40,
    'split_ratio': 1,
    'num_workers': 4,
    'use_transformed_outcomes': False,
    'min_leaf': 5,
    'max_depth': 20,
    'seed_counter': 1,
}

cf = CausalForest(**cfparams)
```

### Fit the model onto the training data:

```python
cf = cf.fit(X, t, y)
```

## Prediction

### Predict treatment effects for newly observed covariates:

```python
newx = np.array([
    [5, -10],
    [-10, 5],
    [-10, -10],
    [5, 5],
    [10, 10]
])

predictions = cf.predict(newx, num_workers=2)
predictions
```

    array([0.04468647, 0.1118838 , 0.12054378, 4.83699512, 5.01450885])

```python
newx = np.array([
    [5, -10],
    [-10, 5],
    [-10, -10],
    [5, 5],
    [10, 10]
])

predictions = cf.predict(newx, num_workers=2)
predictions
```

    array([0.04468647, 0.1118838 , 0.12054378, 4.83699512, 5.01450885])

### Visualize the predicted surface:

```python
plot_predicted_treatment_effect(cf, figsize=(14, 10), npoints=75, num_workers=4)
```

## Saving the Model

To save a (fitted) estimator model we simply call the `save` method with the filename.

```python
cf.save(filename="fitted-example-model.csv", overwrite=True)
```

## Loading a Fitted Model

To load a model we have to iniate a new CausalForest object or use an old one and overwrite the fitted_model their. Here we iniate a new one, load the model which we just saved and assert that the stored models in the two estimators are equal.

```python
cff = CausalForest(**cfparams)
cff = cff.load("fitted-example-model.csv")

assert_frame_equal(cff.fitted_model, cf.fitted_model)
```

## Bonus

How would the fit look like if we change the splitting criterion?
Currently implemented are two splitting criteria.

One, we split at a point on a certain axis if this split will lead to the highest sum over the resulting regions of squared treatment effects weighted by the number of observations.
That is, we maximize
$$criterion = n_{left} \tau_{left}^2 + n_{right}\tau_{right}^2 \,\,.$$

This criterion we get by setting ``use_transformed_outcomes=False``.
Second, we split at a point on a certain axis if this split will lead to the lowest sum of squared error loss in the resulting regions. Here we consider the squared individual deviations of the predicted effect, that is
$$
loss = \sum_{i \in left} (\tau_i - \tau_{left})^2 + \sum_{i \in right} (\tau_i - \tau_{right})^2 \,\,,
$$

where, of course, the $\tau_i$ are estimates to the individual treatment effect using the usual transformed outcome approximation $\tau_i = 2 D_i Y_i - 2 (1 - D_i) Y_i$ --for propensity scores other than 1/2 one has to estimate scores first.
We get this criterion by setting ``use_transformed_outcomes=True``.

```python
cfparams = {
    'num_trees': 40,
    'split_ratio': 1,
    'num_workers': 4,
    'use_transformed_outcomes': True,
    'min_leaf': 5,
    'max_depth': 20,
    'seed_counter': 1,
}

cf = CausalForest(**cfparams)
cf = cf.fit(X, t, y)
plot_predicted_treatment_effect(cf, figsize=(14, 10), npoints=75, num_workers=4)
```


```
d = 8
m = 7
p = max(d, m)
z = min(d, m)
rho = d + m
print "d: " + str(d) + ", m: " + str(m) + ", p: " + str(p) + ", z: " + str(z) + ", rho: " + str(rho)
```

    d: 8, m: 7, p: 8, z: 7, rho: 15

```
%matplotlib inline
from matplotlib.pyplot import plot, figure, gca, legend, style
```

```
from control import care, ss, forced_response, tf
from numpy import matrix, eye, linspace, sign, array, concatenate
from numpy.linalg import eig, det
```

```
from sympy import var, Matrix, eye, init_printing, roots, solve
init_printing()
```

```
var("s")
var("x:3")
```

```
A = Matrix([[0, 1], [0, 0]])
b = Matrix([[0], [1]])
c = Matrix([[-z], [1]])
d = 0
```

```
A
```

```
b
```

```
c
```

```
d
```

```
C = b.row_join(A*b)
C
```

```
C.det()
```

```
O = c.T.col_join(c.T*A)
O
```

```
O.det()
```

```
pol = (s + p)**2
pol.expand()
```

```
fp = Matrix([[-64], [-16]])
fp
```

```
Afp = A + b*fp.T
Afp
```

```
Afp.eigenvals()
```

```
Tp, Jp = Afp.jordan_form()
Jp
```

```
bJp = Tp.inv()*b
cJp = Tp.T*c
```

```
MSp = (s*eye(2) - Jp).row_join(bJp).col_join(-cJp.T.row_join(Matrix([[0]])))
MSp
```

```
1.0/15
```

```
Q = c*c.T
Q
```

```
Q.eigenvals()
```

```
X = Matrix([[x1], [x2]])
X
```

```
c.T*X
```

```
c.T*A*X
```

```
solve(Matrix([[(c.T*X)[0]-1], [(c.T*A*X)[0]-1]]))
```

```
x0 = Matrix([[-8.0/49], [-1.0/7]])
x0
```

```
A = matrix(A.tolist(), dtype=float)
b = matrix(b.tolist(), dtype=float)
c = matrix(c.tolist(), dtype=float)
Q = matrix(Q.tolist(), dtype=float)
x0 = matrix(x0.tolist(), dtype=float)
Afp = matrix(Afp.tolist(), dtype=float)
ceros = matrix([[0], [0]])
```

```
Q
```

    matrix([[ 49.,  -7.],
            [ -7.,   1.]])

```
from numpy import eye
```

```
r1 = 1.0/rho
P1, L1, G1 = care(A=A, B=b, Q=Q, R=r1, S=ceros, E=eye(2))
```

    /Users/roberto/miniconda/lib/python2.7/site-packages/control/mateqn.py:467: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
      if E != None and len(shape(E)) == 1:

```
P1
```

    array([[ 22.03743457,   1.80739223],
           [  1.80739223,   0.55466413]])

```
L1
```

    array([[-4.15998077+3.13136411j],
           [-4.15998077-3.13136411j]], dtype=complex64)

```
G1
```

    array([[ 27.11088342,   8.31996195]])

```
fo1 = matrix(G1.T)
fo1
```

    matrix([[ 27.11088342],
            [  8.31996195]])

```
r2 = 1
P2, L2, G2 = care(A=A, B=b, Q=Q, R=r2, S=ceros, E=eye(2))
```

```
P2
```

    array([[ 34.11088342,   7.        ],
           [  7.        ,   3.87298335]])

```
L2
```

    array([[-1.93649173+1.80277562j],
           [-1.93649173-1.80277562j]], dtype=complex64)

```
G2
```

    array([[ 7.        ,  3.87298335]])

```
fo2 = matrix(G2.T)
fo2
```

    matrix([[ 7.        ],
            [ 3.87298335]])

```
r3 = rho
P3, L3, G3 = care(A=A, B=b, Q=Q, R=r3, S=ceros, E=eye(2))
```

```
P3
```

    array([[ 59.01794475,  27.11088342],
           [ 27.11088342,  28.78066196]])

```
L3
```

    array([[-0.95935541+0.94182241j],
           [-0.95935541-0.94182241j]], dtype=complex64)

```
G3
```

    array([[ 0.,  0.]])

```
fo3 = matrix(G3.T)
fo3
```

    matrix([[ 0.],
            [ 0.]])

```
Afo1 = A - b*fo1.T
Afo1
```

    matrix([[  0.        ,   1.        ],
            [-27.11088342,  -8.31996195]])

```
Afo2 = A - b*fo2.T
Afo2
```

    matrix([[ 0.        ,  1.        ],
            [-7.        , -3.87298335]])

```
Afo3 = A - b*fo3.T
Afo3
```

    matrix([[ 0.,  1.],
            [ 0.,  0.]])

```
ssp = ss(Afp, b, c.T, 0)
sso1 = ss(Afo1, b, c.T, 0)
sso2 = ss(Afo2, b, c.T, 0)
sso3 = ss(Afo3, b, c.T, 0)
```

```
heavyside = lambda x: 0.5 * (sign(x) + 1)
t = linspace(0, 3, 1000)
v = heavyside(t)
```

```
_,  yp,  xp = forced_response(ssp, T=t, U=v, X0=x0)
_, yo1, xo1 = forced_response(sso1, T=t, U=v, X0=x0)
_, yo2, xo2 = forced_response(sso2, T=t, U=v, X0=x0)
_, yo3, xo3 = forced_response(sso3, T=t, U=v, X0=x0)

up  = -xp.T*fp
uo1 = xo1.T*fo1
uo2 = xo2.T*fo2
uo3 = xo3.T*fo3
```

```
f = figure(figsize=(18, 7))
style.use("ggplot")
p1, = plot(t, yp)
p2, = plot(t, yo1)
p3, = plot(t, yo2)
p4, = plot(t, yo3)
p5, = plot(t, v)

ax = f.gca()
ax.set_ylim(-1.5, 1.5);

legend([p1, p2, p3, p4, p5], [r"$y_p(t)$", r"$y_{O_1}(t)$", r"$y_{O_2}(t)$", r"$y_{O_3}(t)$", r"$\delta$"]);
```

```
f = figure(figsize=(18, 7))
style.use("ggplot")
p1, = plot(t, up)
p2, = plot(t, uo1)
p3, = plot(t, uo2)
p4, = plot(t, uo3)

ax = f.gca()
ax.set_ylim(-6, 3);

legend([p1, p2, p3, p4], [r"$u_p(t)$", r"$u_{O_1}(t)$", r"$u_{O_2}(t)$", r"$u_{O_3}(t)$"]);
```

```
up = array(up, dtype=float)
uo1 = array(uo1, dtype=float)
uo2 = array(uo2, dtype=float)
uo3 = array(uo3, dtype=float)
```

```
Jp = yp**2 + (up**2)[:, 0]
Jo1 = yo1**2 + r1*(uo1**2)[:, 0]
Jo2 = yo2**2 + r2*(uo2**2)[:, 0]
Jo3 = yo3**2 + r3*(uo3**2)[:, 0]
```

```
f = figure(figsize=(18, 7))
style.use("ggplot")
p1, = plot(t, Jp)
p2, = plot(t, Jo1)
p3, = plot(t, Jo2)
p4, = plot(t, Jo3)

ax = f.gca()
ax.set_ylim(-0.2, 200);

legend([p1, p2, p3, p4], [r"$J_p(t)$", r"$J_{O_1}(t)$", r"$J_{O_2}(t)$", r"$J_{O_3}(t)$"]);
```

```
Ao = Matrix([[0, 0], [1, 0]])
bo = Matrix([[1], [-7]])
co = Matrix([[0], [1]])
do = 0
Ao
```

```
Oo = co.T.col_join(co.T*Ao)
Oo
```

```
T = O.inv()*Oo
T
```

```
polob = (s+40)**2
polob.expand()
```

```
ko = Matrix([[-1600], [-80]])
ko
```

```
k = T*ko
k
```

```
A = Matrix([[0, 1], [0, 0]])
b = Matrix([[0], [1]])
c = Matrix([[-z], [1]])
d = 0
fp = matrix(fp.tolist(), dtype=float)
Afp = A + b*fp.T
```

```
Alcp = (A.row_join(b*fp.T)).col_join((-k*c.T).row_join(Afp + k*c.T))
blcp = b.col_join(b)
clcp = c.col_join(Matrix([[0], [0]]))
```

```
Alco1 = (A.row_join(b*fo1.T)).col_join((-k*c.T).row_join(Afo1 + k*c.T))
blco1 = b.col_join(b)
clco1 = c.col_join(Matrix([[0], [0], [0]]))
```

```
Alco2 = (A.row_join(b*fo2.T)).col_join((-k*c.T).row_join(Afo2 + k*c.T))
blco2 = b.col_join(b)
clco2 = c.col_join(Matrix([[0], [0], [0]]))
```

```
Alco3 = (A.row_join(b*fo3.T)).col_join((-k*c.T).row_join(Afo3 + k*c.T))
blco3 = b.col_join(b)
clco3 = c.col_join(Matrix([[0], [0], [0]]))
```

```
Alcp = matrix(Alcp.tolist(), dtype=float)
blcp = matrix(blcp.tolist(), dtype=float)
clcp = matrix(clcp.tolist(), dtype=float)
Alco1 = matrix(Alco1.tolist(), dtype=float)
Alco2 = matrix(Alco2.tolist(), dtype=float)
Alco3 = matrix(Alco3.tolist(), dtype=float)
x00 = concatenate((x0, x0))
x00
```

    matrix([[-0.16326531],
            [-0.14285714],
            [-0.16326531],
            [-0.14285714]])

```
sslcp  = ss(Alcp,  blcp, clcp.T, 0)
sslco1 = ss(Alco1, blcp, clcp.T, 0)
sslco2 = ss(Alco2, blcp, clcp.T, 0)
sslco3 = ss(Alco3, blcp, clcp.T, 0)
```

```
_,  ylcp,  xlcp = forced_response(sslcp, T=t, U=v, X0=x00)
_, ylco1, xlco1 = forced_response(sslco1, T=t, U=v, X0=x00)
_, ylco2, xlco2 = forced_response(sslco2, T=t, U=v, X0=x00)
_, ylco3, xlco3 = forced_response(sslco3, T=t, U=v, X0=x00)
```

```
f = figure(figsize=(18, 7))
style.use("ggplot")
p1, = plot(t, ylcp)
p2, = plot(t, ylco1)
p3, = plot(t, ylco2)
p4, = plot(t, ylco3)
p5, = plot(t, v)

ax = f.gca()
ax.set_ylim(-1.5, 1.5);

legend([p1, p2, p3, p4, p5], [r"$y_p(t)$", r"$y_{O_1}(t)$", r"$y_{O_2}(t)$", r"$y_{O_3}(t)$", r"$\delta$"]);
```

IPythonNotebooks/Teoria de Control I/Examen II - Control optimo.ipynb


$$
\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
\DeclareMathOperator{\Tr}{Tr}
\DeclareMathOperator*{\argmin}{arg\,min}
\def\norm#1{\left\lVert #1 \right\rVert}
$$

# What is adversarial attack and why we care about it?

Despite the effectiveness on a variety of tasks, deep neural networks can be very vulnerable to adversarial attacks. In security-critical domains like facial recognition authorization and autonomous vehicles, such vulnerability against adversarial attacks makes the model highly unreliable.

An adversarial attack tries to add an imperceptible perturbation to the sample so that a trained neural network would classify it incorrectly. The following is an example of adversarial attack.

## Adversarial Attacks

### Categories of Attacks
#### Poisoning Attack and Evasion Attack
Poisoning attacks involve manipulating the training process. The manipulation can happen on the training set (by replacing original samples or inserting fake samples) or the training algorithm itself (by changing the logic of the training algorithm). Such attacks either directly cause poor performance of the trained model or make the model fail on certain samples so as to construct a backdoor for future use.

Evasion attacks aim to manipulate the benign samples $\M x$ by a small perturbation $\M \delta$ so that a trained model can no longer classify it correctly. Usually, such perturbations are so small that a human observer cannot notice them. In other words, the perturbed sample "evades" from the classification of the model.

#### Targeted Attack and Non-Targeted Attack
A targeted attack tries to perturb the benign samples $\M x$ so that the trained model classifies it as a given certain class $t\in \mc Y$. A non-targeted attack only tries to perturb the benign samples $\M x$ so that the trained model classifies them incorrectly.

#### White-Box Attack and Black-Box Attack
For white-box attacks, the attacker has access to all the knowledge of the targeted model. For neural networks, the attacker knows all the information about the network structure, parameters, gradients, etc.

For black-box attacks, the attacker only knows the outputs of the model when feeding inputs into it. In practice, black-box attacks usually rely on generating adversarial perturbations from another model that the attacker has full access to.

Black-box attacks are more usual in applications, but the robustness against white-box attacks is the ultimate goal of a robust model because it reveals the fundamental weakness of neuron networks. Thus, most of the study of adversarial robustness focuses on white-box, non-targeted, evasion attacks.

### Examples of White-box Attacking Algorithms
Most white-box attacking algorithms are based on using the gradient calculated by the model to perturb the samples. Two typical examples are Fast Gradient Sign Method (FGSM) attack and its multi-step variant, Projected Gradient Descent (PGD) attack. FGSM attack generates the perturbation as
$$\begin{align}
    \M \delta = \epsilon\text{sgn}\nabla_{\M x}L(\M x, y),
\end{align}$$
where $\epsilon$ controls the perturbation size. The adversarial sample is
$$\begin{align}
    \M x' = \M x + \M \delta.
\end{align}$$
FGSM attack can be seen as trying to maximizing (single step) the loss of the model.

PGD attack tries perform the same task, but in a iterative way (at the cost of higher computational time):
$$\begin{align}
    \M x^{t+1} = \Pi_{\M x+\epsilon}\left(\M x^t + \alpha\text{sgn}\nabla_{\M x}L(\M x, y)\right),
\end{align}$$
where $\alpha$ is the step size. $\M x^t$ denote the generated adversarial sample in step $t$, with $\M x^0$ being the original sample. $\Pi$ refers to a projection operation that clips the generated adversarial sample into the valid region: the $\epsilon$-ball around $\M x$, which is $\{\M x':\norm{\M x'-\M x}\leq \epsilon \}$.

In practice, a PGD attack with a relatively small adversarial power $\epsilon$ (small enough to be neglected by human observers) is able to reduce the accuracy of a well-trained model to nearly zero. Because of such effectiveness, researchers often use PGD attacks as a basic check of the adversarial robustness of their models.

#### Implementation of FGSM attack

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
import numpy as np
import matplotlib.pyplot as plt

# NOTE: This is a hack to get around "User-agent" limitations when downloading MNIST datasets
#       see, https://github.com/pytorch/vision/issues/3497 for more information
from six.moves import urllib
opener = urllib.request.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
urllib.request.install_opener(opener)
```

##### Inputs
-  **epsilons** - List of epsilon values to use for the run. It is
   important to keep 0 in the list because it represents the model
   performance on the original test set. Also, intuitively we would
   expect the larger the epsilon, the more noticeable the perturbations
   but the more effective the attack in terms of degrading model
   accuracy. Since the data range here is $[0,1]$, no epsilon
   value should exceed 1.

-  **pretrained_model** - path to the pretrained MNIST model which was
   trained with [pytorch/examples/mnist ](https://github.com/pytorch/examples/tree/master/mnist).
   For simplicity, download the pretrained model [here](https://drive.google.com/drive/folders/1fn83DF14tWmit0RTKWRhPq5uVXt73e0h?usp=sharing).

-  **use_cuda** - boolean flag to use CUDA if desired and available.
   Note, a GPU with CUDA is not critical for this tutorial as a CPU will
   not take much time.

```python
epsilons = [0, .05, .1, .15, .2, .25, .3]
pretrained_model = "lenet_mnist_model.pth"
use_cuda=True
```

##### Model Under Attack

As mentioned, the model under attack is the same MNIST model from
[pytorch/examples/mnist](https://github.com/pytorch/examples/tree/master/mnist).
You may train and save your own MNIST model or you can download and use
the provided model. The *Net* definition and test dataloader here have
been copied from the MNIST example. The purpose of this section is to
define the model and dataloader, then initialize the model and load the
pretrained weights.

```python
# LeNet Model definition
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

# MNIST Test dataset and dataloader declaration
test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([
            transforms.ToTensor(),
            ])),
        batch_size=1, shuffle=True)

# Define what device we are using
print("CUDA Available: ",torch.cuda.is_available())
device = torch.device("cuda" if (use_cuda and torch.cuda.is_available()) else "cpu")

# Initialize the network
model = Net().to(device)

# Load the pretrained model
model.load_state_dict(torch.load(pretrained_model, map_location='cpu'))

# Set the model in evaluation mode. In this case this is for the Dropout layers
model.eval()
```

    CUDA Available:  True

    Net(
      (conv1): Conv2d(1, 10, kernel_size=(5, 5), stride=(1, 1))
      (conv2): Conv2d(10, 20, kernel_size=(5, 5), stride=(1, 1))
      (conv2_drop): Dropout2d(p=0.5, inplace=False)
      (fc1): Linear(in_features=320, out_features=50, bias=True)
      (fc2): Linear(in_features=50, out_features=10, bias=True)
    )

##### FGSM Attack

Now, we can define the function that creates the adversarial examples by
perturbing the original inputs. The ``fgsm_attack`` function takes three
inputs, *image* is the original clean image ($x$), *epsilon* is
the pixel-wise perturbation amount ($\epsilon$), and *data_grad*
is gradient of the loss w.r.t the input image
($\nabla_{x} L(\mathbf{\theta}, \mathbf{x}, y)$). The function
then creates perturbed image as

\begin{align}perturbed\_image = image + epsilon*sign(data\_grad) = x + \epsilon * sign(\nabla_{x} L(\mathbf{\theta}, \mathbf{x}, y))\end{align}

Finally, in order to maintain the original range of the data, the
perturbed image is clipped to range $[0,1]$.

```python
# FGSM attack code
def fgsm_attack(image, epsilon, data_grad):
    # Collect the element-wise sign of the data gradient
    sign_data_grad = data_grad.sign()
    # Create the perturbed image by adjusting each pixel of the input image
    perturbed_image = image + epsilon*sign_data_grad
    # Adding clipping to maintain [0,1] range
    perturbed_image = torch.clamp(perturbed_image, 0, 1)
    # Return the perturbed image
    return perturbed_image
```

##### Testing Function

Finally, the central result of this tutorial comes from the ``test``
function. Each call to this test function performs a full test step on
the MNIST test set and reports a final accuracy. However, notice that
this function also takes an *epsilon* input. This is because the
``test`` function reports the accuracy of a model that is under attack
from an adversary with strength $\epsilon$. More specifically, for
each sample in the test set, the function computes the gradient of the
loss w.r.t the input data ($data\_grad$), creates a perturbed
image with ``fgsm_attack`` ($perturbed\_data$), then checks to see
if the perturbed example is adversarial. In addition to testing the
accuracy of the model, the function also saves and returns some
successful adversarial examples to be visualized later.

```python
def test( model, device, test_loader, epsilon ):

    # Accuracy counter
    correct = 0
    adv_examples = []

    # Loop over all examples in test set
    for data, target in test_loader:

        # Send the data and label to the device
        data, target = data.to(device), target.to(device)

        # Set requires_grad attribute of tensor. Important for Attack
        data.requires_grad = True

        # Forward pass the data through the model
        output = model(data)
        init_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability

        # If the initial prediction is wrong, dont bother attacking, just move on
        if init_pred.item() != target.item():
            continue

        # Calculate the loss
        loss = F.nll_loss(output, target)

        # Zero all existing gradients
        model.zero_grad()

        # Calculate gradients of model in backward pass
        loss.backward()

        # Collect datagrad
        data_grad = data.grad.data

        # Call FGSM Attack
        perturbed_data = fgsm_attack(data, epsilon, data_grad)

        # Re-classify the perturbed image
        output = model(perturbed_data)

        # Check for success
        final_pred = output.max(1, keepdim=True)[1] # get the index of the max log-probability
        if final_pred.item() == target.item():
            correct += 1
            # Special case for saving 0 epsilon examples
            if (epsilon == 0) and (len(adv_examples) < 5):
                adv_ex = perturbed_data.squeeze().detach().cpu().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )
        else:
            # Save some adv examples for visualization later
            if len(adv_examples) < 5:
                adv_ex = perturbed_data.squeeze().detach().cpu().numpy()
                adv_examples.append( (init_pred.item(), final_pred.item(), adv_ex) )

    # Calculate final accuracy for this epsilon
    final_acc = correct/float(len(test_loader))
    print("Epsilon: {}\tTest Accuracy = {} / {} = {}".format(epsilon, correct, len(test_loader), final_acc))

    # Return the accuracy and an adversarial example
    return final_acc, adv_examples
```

##### Run Attack

The last part of the implementation is to actually run the attack. Here,
we run a full test step for each epsilon value in the *epsilons* input.
For each epsilon we also save the final accuracy and some successful
adversarial examples to be plotted in the coming sections. Notice how
the printed accuracies decrease as the epsilon value increases. Also,
note the $\epsilon=0$ case represents the original test accuracy,
with no attack.

```python
accuracies = []
examples = []

# Run test for each epsilon
for eps in epsilons:
    acc, ex = test(model, device, test_loader, eps)
    accuracies.append(acc)
    examples.append(ex)
```

    Epsilon: 0	Test Accuracy = 9810 / 10000 = 0.981
    Epsilon: 0.05	Test Accuracy = 9426 / 10000 = 0.9426
    Epsilon: 0.1	Test Accuracy = 8510 / 10000 = 0.851
    Epsilon: 0.15	Test Accuracy = 6826 / 10000 = 0.6826
    Epsilon: 0.2	Test Accuracy = 4303 / 10000 = 0.4303
    Epsilon: 0.25	Test Accuracy = 2087 / 10000 = 0.2087
    Epsilon: 0.3	Test Accuracy = 871 / 10000 = 0.0871

##### Results

Accuracy vs Epsilon

The first result is the accuracy versus epsilon plot. As alluded to
earlier, as epsilon increases we expect the test accuracy to decrease.
This is because larger epsilons mean we take a larger step in the
direction that will maximize the loss. Notice the trend in the curve is
not linear even though the epsilon values are linearly spaced. For
example, the accuracy at $\epsilon=0.05$ is only about 4% lower
than $\epsilon=0$, but the accuracy at $\epsilon=0.2$ is 25%
lower than $\epsilon=0.15$. Also, notice the accuracy of the model
hits random accuracy for a 10-class classifier between
$\epsilon=0.25$ and $\epsilon=0.3$.

```python
plt.figure(figsize=(5,5))
plt.plot(epsilons, accuracies, "*-")
plt.yticks(np.arange(0, 1.1, step=0.1))
plt.xticks(np.arange(0, .35, step=0.05))
plt.title("Accuracy vs Epsilon")
plt.xlabel("Epsilon")
plt.ylabel("Accuracy")
plt.show()
```

##### Sample Adversarial Examples

Remember the idea of no free lunch? In this case, as epsilon increases
the test accuracy decreases **BUT** the perturbations become more easily
perceptible. In reality, there is a tradeoff between accuracy
degredation and perceptibility that an attacker must consider. Here, we
show some examples of successful adversarial examples at each epsilon
value. Each row of the plot shows a different epsilon value. The first
row is the $\epsilon=0$ examples which represent the original
“clean” images with no perturbation. The title of each image shows the
“original classification -> adversarial classification.” Notice, the
perturbations start to become evident at $\epsilon=0.15$ and are
quite evident at $\epsilon=0.3$. However, in all cases humans are
still capable of identifying the correct class despite the added noise.

```python
# Plot several examples of adversarial samples at each epsilon
cnt = 0
plt.figure(figsize=(8,10))
for i in range(len(epsilons)):
    for j in range(len(examples[i])):
        cnt += 1
        plt.subplot(len(epsilons),len(examples[0]),cnt)
        plt.xticks([], [])
        plt.yticks([], [])
        if j == 0:
            plt.ylabel("Eps: {}".format(epsilons[i]), fontsize=14)
        orig,adv,ex = examples[i][j]
        plt.title("{} -> {}".format(orig, adv))
        plt.imshow(ex, cmap="gray")
plt.tight_layout()
plt.show()
```

## Adversarial Training with Perturbed Examples
To defend adversarial attacks, a direct idea is to add perturbations during the training process. Mardy et al. \cite{madry2017} proposed to formulate a robust optimization problem to minimize the adversarial risk instead of the usual empirical risk:
$$\begin{align}
    \min_{\theta} \E_{(\M x, y)\in \mc D} \left[\max_{\norm{\M\delta}_p<\epsilon}L(\M x+\M\delta, y)\right].
\end{align}$$
The inner maximization tries to find perturbed samples that produce a high loss, which is also the goal of PGD attacks. The outer minimization problem tries to find the model parameters that minimize the adversarial loss given by the inner adversaries.

This robust optimization approach effectively trains more robust models. It has been a benchmark for evaluating the adversarial robustness of models and is often seen as the standard way of adversarial training. Based on this method, many variants were proposed in the following years. They involve using a more sophisticated regularizer or adaptively adjusting the adversarial power. Those methods that add perturbations during training share a common disadvantage of high computational cost.

## Adversarial Training with Stochastic Networks
Stochastic networks refer to the neuron networks that involve random noise layers. Liu et al. proposed Random Self Ensemble (RSE). Their method injects spherical Gaussian noise into different layers of a network and uses the ensemble of multiple forward pass as the final output. The variance of their added noise is treated as a hyper-parameter to be tuned. RSE shows good robustness against PGD attack and C\&W attack.

Similar to RSE, He et al. proposed Parametric Noise Injection (PNI). Rather than fixed variance, they applied an additional intensity parameter to control the variance of the noise. This intensity parameter is trained together with the model parameters.

Inspired by the idea of trainable noise, Eustratiadis et al. proposed Weight-Covariance Alignment (WCA) method \cite{wca}. This method adds trainable Gaussian noise to the activation of the penultimate layer of the network. Let $\M g_{\theta}:\mathcal X\rightarrow \mathbb R^D$ be the neural network parameterized by $\theta$ except the final layer and $f_{\M W, \M b}:\mathbb R^D \rightarrow \mathbb R^K$ be the final linear layer parameterized by weight matrix $\M W^{K\times D}$ and bias vector $\M b^{K\times 1}$, where $K=\abs{\mc Y}$ is the number of classes . This WCA method adds a Gaussian noise $\M u \sim \mathcal N_{0, \M\Sigma}$ to the output of penultimate layer $\M g_{\theta}(x)$, where $\M\Sigma^{D\times D}$ is the covariance matrix. Thus, the final output becomes
  $$\begin{align}
  f_{\M W, \M b}\left(\M g_\theta(\M x)\right) = \M W\left(\M g_\theta (\M x)+\M u\right) + \M b.
  \end{align}$$

The loss function is defined as
  $$\begin{align}
  L=L_{\text{CE}} + L_{\text{WCA}}+ \lambda \sum_{y\in \mathcal{Y}}\M W_y^{\intercal} \M W_y,
  \end{align}$$
where $L_{\text{CE}}$ is the usual cross-entropy loss, and $L_{\text{WCA}}$ is a term that encourage the noise and the weight of last layer to be aligned with each other. The third term is gives $l^2$ penalty to $\M W_y$ with large magnitude. The WCA regularizer is defined as
  $$\begin{align}
  L_{\text{WCA}} = -\log\sum_{y\in \mathcal{Y}}\M W_y \M\Sigma\M W_y^\intercal .
  \end{align}$$
where $\M W_y$ is the weight vector of the last layer that is associated with class $y$.

The WCA regularizer encourages the weights associated with the last layer to be well aligned with the covariance matrix of the noise. Larger trained variance corresponding to one feature means this feature is harder to perturb, so putting more weight on such features will force the final layer to focus more on these robust features.

Models trained with WCA show better performance against PGD attacks on various datasets comparing with the aforementioned approaches. In addition, because of the fact that WCA does not involve generating adversarial samples, the computational time is significantly lower than adversarial training with perturbations. The method we propose is inspired by this WCA method. But instead of adding noise to the penultimate layer, we directly add noise to the output of the final layer.

## Training a neural network with noisy logits

We consider training a model with a noisy representation $\R{Z}$ satisfying the Markov chain:

$$\R{X}\to \R{Z} \to \hat{\R{Y}}$$

Hence, the estimate for $P_{\R{Y}|\R{X}}$ is given by $P_{\R{Y}|\R{Z}}$ and $P_{\R{Z}|\R{X}}$ as
$$P_{\hat{\R{Y}}|\R{X}} (y|x) = E\left[\left.P_{\hat{\R{Y}}|\R{Z}}(y|\R{Z}) \right|\R{X}=x\right].$$

In particular, we propose to set $\R{Z}$ to be the noisy logits of $\hat{\R{Y}}$, i.e.,
$P_{\hat{Y}|\R{Z}}$ is defined by the pmf obtained with the usual softmax function
$$
p_{\hat{\R{Y}}|\RM{z}} (y|\M{z}) := \frac{\exp(z_y)}{\sum_{y'\in \mathcal{Y}} \exp(z_{y'})},
$$
so $z_y$ is the logit for class $y$.

The noisy logit is defined as
$$
\R{Z} = \RM{z}:=[g(y|\R{X})+\R{u}_y]_{y\in \mathcal{Y}}
$$
where
$$g(y|x)\in \mathbb{R}$$
for $(x,y)\in \mathcal{X}\times \mathcal{Y}$
is computed by a neural network to be trained, and $\R{u}_y\sim \mathcal{N}_{0,\sigma_y^2}$ for $y\in \mathcal{Y}$ are independent gaussian random variables with variance $\sigma_y^2>0$. For simplicity,
$$
$$\begin{align}
\M{g}(x)&:= [g(y|x)]_{y\in \mathcal{Y}}\\
\RM{u}&:=[\R{u}_y]_{y\in \mathcal{Y}}\\
\M{\Sigma}&:=\M{\sigma} \M{I} \M{\sigma}^\intercal \quad \text{with }\M{\sigma}:=[
\sigma_y]_{y\in \mathcal{Y}},
\end{align}$$
$$ which are referred to as the (noiseless) logits, additive noise (vector) and its (diagonal) covariance matrix respectively. Hence, $P_{\R{Z}|\R{X}}$ is defined by the multivariate gaussian density function
$$
p_{\RM{z}|\R{X}}(\M{z}|x) = \mathcal{N}_{\M{g}(x),\Sigma}(\M{z})
$$
for $x\in \mathcal{X}$ and $\M{z}\in \mathbb{R}^{\abs{\mathcal{Y}}}$.

The loss function used for training the neural network is derived from

$$
L := E\left[-\log p_{\hat{\R{Y}}|\R{X}}(\R{Y}|\R{X})\right] - \log \sum_{y\in \mathcal{Y}}\sigma_y^2 + \lambda\left(\sum_{y\in \mathcal{Y}}\sigma_y^2\right)
$$

```python

```


# Canonical Clustering

## $k$ Nearest-Neighbor
### Distance Measure
The $k$ nearest neighbor is based on the simple idea that when a sample is close to other samples, changes are that they share the same label. To know how close one sample is from another sample, there needs to be a notion of close. For this application we will use euclidian distance also known as the $L^2$ norm

$$ \left\| x \right\|_2 \overset{\textrm{def}}= \left( \sum_{i=1}^n x_i^2 \right)^{1/2}$$

### Classification
In classification we would like to assign certain samples to a class. To do so we would like to implement an algorithm that assigns the class based on the classes of the $k$ nearest neighbors. Try to do so in `.code/knn.py`

```python
import sys
sys.path.append('code/')
import knn

import numpy as np

from sklearn import datasets, model_selection
from sklearn.metrics import adjusted_rand_score

# import some data to play with
X, y = datasets.load_iris(return_X_y=True)

# inspect the shape of the data
print(X.shape)
print(y.shape)

# split data into train and test
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y)

# and predict
k=10
y_predicted = knn.predict(X_train, X_test, y_train, k)

# evaluate predictions using y_predicted and y_test

```

    (150, 4)
    (150,)

## Naive Bayes
#### TODO: probably this exercise is to extensive, rewrite it with using libraries.
### Dataset
Naive Bayes classifier is often used to filter emails for spam, such as in [this article](https://www.aclweb.org/anthology/E03-1059.pdf). For this exercise we will use a dataset containing samples from email messages. This dataset is available on [openml](https://www.openml.org/d/42673) and can be read in using sklearn:

```python
from sklearn import datasets
import numpy as np
spam = datasets.fetch_openml('auml_eml_1_a')

# lets inspect the dataset
np.set_printoptions(precision=3, suppress=True)
print("Dataset Dictionary Keys: \n{0}\n".format(spam.keys()))
print("Dataset Feature Names: \n{0}\n".format(spam['feature_names']))

# print a few examples of spam and non-spam
print("Example spam data entries: \n{0}\n".format(spam['data'][spam['target']==0][:8,:].T))
print("Example spam targets: \n{0}\n".format(spam['target'][spam['target']==0][:8].T))
print("Example non-spam data entries: \n{0}\n".format(spam['data'][spam['target']==1][:8,:].T))
print("Example non-spam targets: \n{0}\n".format(spam['target'][spam['target']==1][:8].T))
```

    Dataset Dictionary Keys:
    dict_keys(['data', 'target', 'frame', 'categories', 'feature_names', 'target_names', 'DESCR', 'details', 'url'])

    Dataset Feature Names:
    ['F1_text_html_found', 'F1_number_of_links', 'F1_contains_javascript']

    Example spam data entries:
    [[0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 5. 4. 3. 3. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0.]]

    Example spam targets:
    [0. 0. 0. 0. 0. 0. 0. 0.]

    Example non-spam data entries:
    [[0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0.]]

    Example non-spam targets:
    [1. 1. 1. 1. 1. 1. 1. 1.]

This dataset contains 3 features. The first is a binary feature named 'F1_text_html_found', the second is a nominal feature named 'F1_number_of_links', and the last is again a binary feature named 'F1_contains_javascript'. Using these three features, we are going to try to predict the change of a sample being spam. To do so we use:

\begin{align}
P(Y=y_k | X=x_i) = \frac{P(Y=y_k) P(X=x_i | Y=y_k)}{P(X=x_i)}
\end{align}

This is an alternative way of writing:

\begin{align}
P(Y=y_k | X=x_i) = \frac{P(Y=y_k) P(X_1=x_{i,1} \wedge \ldots \wedge X_d=x_{i,d}) | Y=y_k)}{P(X_1=x_{i,1} \wedge \ldots \wedge X_d=x_{i,d})}
\end{align}

The nominator is often omitted, because it scales all classes the same so when comparing classes it does not matter to omit it. Furthermore, it simplifies the calculation. Thus we ultimately we will use:

\begin{align}
P(Y=y_k | X=x_i) = P(Y=y_k) P(X_1=x_{i,1} \wedge \ldots \wedge X_d=x_{i,d}) | Y=y_k)
\end{align}

To use this formula, we first need to define what the probality distribution is. For a binary this is relatively simple. For the counted feature 'F1_number_of_links', we will use a probability distribution based on a histogram with bins with a width of $10$. Implement these probabilities in `code/utils.py`.

```python
import sys
sys.path.append('code/')
import naive_bayes

import pprint
from sklearn.model_selection import train_test_split

X = spam.data
y = spam.target

X_train, X_test, y_train, y_test = train_test_split(X,y, stratify=y)

feature_types = ['binary', 'histogram', 'binary']

model = naive_bayes.fit(X_train, y_train, feature_types)

# inspect model
#pprint.pprint(model)

y_predict = naive_bayes.predict(X_test, model)
```


```python
import numpy as np
import sympy as sym
import pydae.build as db
from pydae.grid_bpu import bpu
```

```python
bpu_obj = bpu(data_input='oc_3bus_vsg_pi.json')

g_list = bpu_obj.dae['g']
h_dict = bpu_obj.dae['h_dict']
f_list = bpu_obj.dae['f']
x_list = bpu_obj.dae['x']
params_dict = bpu_obj.dae['params_dict']

P_2 = sym.Symbol("P_B2", real=True)

h_dict.update({'P_B2':P_2})

sys = {'name':'oc_3bus',
       'params_dict':params_dict,
       'f_list':f_list,
       'g_list':g_list,
       'x_list':x_list,
       'y_ini_list':bpu_obj.dae['y_ini'],
       'y_run_list':bpu_obj.dae['y_run'],
       'u_run_dict':bpu_obj.dae['u_run_dict'],
       'u_ini_dict':bpu_obj.dae['u_ini_dict'],
       'h_dict':h_dict}

sys = db.system(sys)
db.sys2num(sys)
```

```python
sys['f'].T
```

$\displaystyle \left[\begin{matrix}- K_{\delta B1} \delta_{B1} + \Omega_{b B1} \left(\omega_{B1} - \omega_{coi}\right)\\- i_{d B1} \left(R_{v B1} i_{d B1} + V_{B1} \sin{\left(\delta_{B1} - \theta_{B1} \right)}\right) - i_{q B1} \left(R_{v B1} i_{q B1} + V_{B1} \cos{\left(\delta_{B1} - \theta_{B1} \right)}\right) + p_{m B1}\\\frac{K_{q B1} \left(- q_{g B1} + q_{ref B1}\right) - e_{qv B1} + v_{ref B1}}{T_{v B1}}\\- K_{\delta B3} \delta_{B3} + \Omega_{b B3} \left(\omega_{B3} - \omega_{coi}\right)\\- i_{d B3} \left(R_{v B3} i_{d B3} + V_{B3} \sin{\left(\delta_{B3} - \theta_{B3} \right)}\right) - i_{q B3} \left(R_{v B3} i_{q B3} + V_{B3} \cos{\left(\delta_{B3} - \theta_{B3} \right)}\right) + p_{m B3}\\\frac{K_{q B3} \left(- q_{g B3} + q_{ref B3}\right) - e_{qv B3} + v_{ref B3}}{T_{v B3}}\\1 - \omega_{coi}\end{matrix}\right]$

```python
sys['x'].T
```

$\displaystyle \left[\begin{matrix}\delta_{B1}\\\xi_{p B1}\\e_{qv B1}\\\delta_{B3}\\\xi_{p B3}\\e_{qv B3}\\\xi_{freq}\end{matrix}\right]$

```python

```


# One-Electron Atoms
Just as the touchstone of chemistry is the periodic table, the touchstone of quantum chemistry is the atomic wavefunction. As we shall see, our intuition about many-electron atoms is built up from our knowledge of 1-electron atoms, mainly because many-electron atoms are mathematically intractable, while 1-electron atoms are not appreciably more complicated than an electron confined to a spherical ball. A detailed mathematical exposition on 1-electron atoms--which are often called hydrogenic atoms--is provided as a [pdf](https://github.com/QC-Edu/IntroQM2022/blob/master/documents/Hatom.pdf). This is only a brief summary.

## Schrödinger Equation for One-Electron Atoms
Denoting the mass of the electron as $m_e$, the charge of the electron as $=e$, and the [permittivity of free space](https://en.wikipedia.org/wiki/Vacuum_permittivity) as $\epsilon_0$, the Hamiltonian for the attraction of an electron to a atomic nucleus with atomic number $Z$ (and charge $+Ze$) at the origin, $(x,y,z) = (0,0,0)$, is:

$$
\hat{H}_{\text{1 el. atom}} = -\frac{\hbar^2}{2m_e} \nabla^2 - \frac{Z e^2}{4 \pi \epsilon_0 r}
$$

where $r = \sqrt{x^2+y^2+z^2}$ is the distance of the electron from the nucleus. In [atomic units](https://en.wikipedia.org/wiki/Hartree_atomic_units), the Hamiltonian is:

$$
\hat{H}_{\text{1 el. atom}} = -\tfrac{1}{2} \nabla^2 - Zr^{-1}
$$

Just as we did for the electron confined to a spherical ball, we rewrite the Schrödinger equation in spherical coordinates,

$$
\left(-\frac{1}{2} \left( \frac{d^2}{dr^2}
+ \frac{2}{r} \frac{d}{dr}\right)
      + \frac{\hat{L}^2}{2r^2} - \frac{Z}{r} \right)
      \psi_{n,l,m_l}(r,\theta,\phi)
= E_{n,l,m_l}\psi_{n,l,m_l}(r,\theta,\phi)
$$

and use the fact the eigenvalues of the squared-magnitude of the angular momentum, $\hat{L}^2$ are the [spherical harmonics](https://en.wikipedia.org/wiki/Spherical_harmonics)

$$
\hat{L}^2 Y_l^{m} (\theta, \phi) = l(l+1)Y_l^{m} (\theta, \phi) \qquad l=0,1,2,\ldots m=0, \pm 1, \ldots, \pm l
$$

and the technique of separation of variables to deduce that the wavefunctions of one-electron atoms have the form

$$
\psi_{n,l,m}(r,\theta,\phi) = R_{n,l}(r) Y_l^{m}(\theta,\phi)
$$

where the radial wavefunction, $R_{n,l}(r)$ is obtained by solving the radial Schrödinger equation

$$
\left(-\frac{1}{2} \left( \frac{d^2}{dr^2}
+ \frac{2}{r} \frac{d}{dr}\right)
      + \frac{l(l+1)}{2r^2} - \frac{Z}{r} \right)
      R_{n,l}(r)
= E_{n,l}R_{n,l}(r)
$$

## The Radial Equation for One-Electron Atoms
To solve the radial Schrödinger equation, we rewrite it as a [homogeneous linear differential equation](https://en.wikipedia.org/wiki/Homogeneous_differential_equation)

$$
\left( \frac{d^2}{dr^2}
+ \frac{2}{r} \frac{d}{dr}
      - \frac{l(l+1)}{r^2} + \frac{2Z}{r} + 2E_{n,l}\right)
      R_{n,l}(r) = 0
$$

It is (quite a bit) more involved than the previous cases we have considered, but the same basic technique reveals that the eigenenergies are

$$
E_n = -\frac{Z^2}{2n^2}
$$

and the radial wavefunctions are the product of an [associated Laguerre polynomial](https://en.wikipedia.org/wiki/Laguerre_polynomials#Generalized_Laguerre_polynomials) and an exponential,

$$
R_{n,l}(r) \propto \left(\frac{2Zr}{n}\right)^l L_{n-1-l}^{2l+1}\left(\frac{2Zr}{n}\right) e^{-\frac{Zr}{n}}
$$

with

$$
n=1,2,3,\ldots \\
l=0,1,\ldots,n-1 \\
m = 0,\pm 1, \pm2, \ldots, \pm l
$$

## Eigenenergies and Wavefunctions for One-Electron Atoms
The eigenenergies of the Hydrogenic wavefunctions do not depend on $m$ or $l$. So there are $(n+1)^2$ degenerate eigenfunctions, with energies

$$
E_n = -\frac{Z^2}{2n^2}
$$

The energy eigenfunctions are:

$$
\psi_{nlm}(r,\theta,\phi) \propto \left(\frac{2Zr}{n}\right)^l L_{n-1-l}^{2l+1}\left(\frac{2Zr}{n}\right) e^{-\frac{Zr}{n}}  Y_l^{m} (\theta, \phi)
$$

These eigenfunctions are complex-valued, because the spherical harmonics are complex-valued. Like all other one-electron wavefunctions, these eigenfunctions are referred to as *orbitals*. For historical reasons, orbitals are labelled by their principle quantum number $n$ (which specifies their energy), their total angular momentum quantum number $l$, and the quantum number that specifies their angular momentum around the $z$ axis, $m$,

$$
\hat{L}_z Y_l^{m} (\theta, \phi) = \hbar m Y_l^{m} (\theta, \phi)
$$

The $l$ quantum number is stored by a letter code that dates back to the pre-history of quantum mechanics, where certain spectral lines were labelled as **s**harp ($l=0$ indicated no spatial degeneracy that could be broken by an external field), **p**rinciple ($l=1$ lines were still relatively sharp), **d**iffuse ($l=2$ lines were quite diffuse due to the 5-fold degeneracy of d orbitals), and **f**undamental ($l=3$).

Note that the orbital images that appear above do not look that much like the usual orbital pictures, with the exception of the $m=0$ orbitals. This is because of the complex-valuedness. We often instead use the *real* spherical harmonics, which are defined simply as:

$$
\begin{align}
S_l^{m>0}(\theta,\phi) &= \frac{1}{\sqrt{2}} \left(Y_l^{-m} (\theta, \phi) + (-1)^{m} Y_l^{m} (\theta, \phi) \right) \\
S_l^{m=0}(\theta,\phi) &= Y_l^{m=0} (\theta, \phi) \\
S_l^{m<0}(\theta,\phi) &= \frac{i}{\sqrt{2}} \left(Y_l^{-m} (\theta, \phi) - (-1)^{m} Y_l^{m} (\theta, \phi) \right)
\end{align}
$$

The following animations shows one can take linear combinations of the (complex) spherical harmonics to form the $p_x$, $p_y$, etc. orbitals one generally uses in chemistry.

Using the [orbitron](https://winter.group.shef.ac.uk/orbitron/atomic_orbitals/7i/index.html), you can visualize the (real, Cartesian) spherical harmonics and the [radial wavefunctions](https://winter.group.shef.ac.uk/orbitron/atomic_orbitals/7f/7f_wave_function.html) for hydrogenic orbitals. Most orbitals have very complicated formulas, but a few have simple equations, including:

$$
\psi_{\text{1s}}(r) = \psi_{100}(r) = \sqrt{\frac{Z^3}{\pi}}e^{-Zr}  \\
\psi_{n,n-1,m} \propto r^{n-1} e^{\tfrac{-Zr}{n}} Y_l^{m_l} (\theta,\phi)
$$

### 📝 What is the expectation value of $r^k$ for the $l=n-1$ orbital of a hydrogenic atom.

### 📝 Show that the energy of the Hydrogen atom in arbitrary units (not necessarily atomic units) can be written as:
$$
E_n = -\frac{Z^2}{2n^2} \cdot \left(\frac{e^2}{4 \pi \epsilon_0}\right)^2 \cdot \left( \frac{m_e}{\hbar^2}\right)
$$

## 🪞 Self-Reflection
- Using the conversion from atomic units to traditional chemical units of kJ/mol, what is the energy of the Hydrogen atom? How accurately, in atomic units, must one determine the energy of a one-electron atom in order to attain "chemical accuracy" of ~1 kJ/mol?
- Write a small Python script to evaluate the expectation value of the radius, $r$, for a one-electron atom.
- Test to confirm that the Heisenberg uncertainty principle for position and momentum holds for the ground state of a Hydrogenic atom.
- To what extent is the shape of the spherical harmonic intuitive, especially the doughnut shapes associated with an electron's angular momentum around the z axis.

## 🤔 Thought-Provoking Questions
- In one-electron atoms, the eigenenergies depend only on the principle quantum number, $n$, and not the angular momentum quantum number, $l$. Why are $s$ orbitals lower in energy than $p$ orbitals in real multielectron atoms, but not one-electron atoms? It turns out this is *not* an accidental degeneracy, but a hidden symmetry of the Hydrogen atom.
- Suppose electrons did not repel each other. Can you write the wavefunction for a many-electron atom in that case?
- Why do you think solving the Schrödinger equation for the one-electron molecule is more complicated than solving the Schrödinger equation for the one-electron atom?
- For what $Z$ is the energy of a one-electron atom comparable to the rest-mass energy of an electron, $mc^2$? For atomic numbers close to this value, relativistic effects become extremely important.
- The Kratzer-Fues potential, $V_{\text{Kratzer-Fues}}(r) = \frac{a}{r^2} - \frac{b}{r}$ (here $a>0$ and $b>0$) is a reasonable model for a diatomic molecule rotating and vibrating in 3 dimensions, or even a ion-pair complex (e.g., an single ion pair from an ionic solvent in the gas phase). What are the solutions to the Schrödinger equation for the Kratzer-Fues potential? [Solution](problems/KratzerFues.md)

$$
\left(-\frac{\hbar^2}{2m}\nabla^2 +\frac{a}{r^2} - \frac{b}{r} \right) \psi(\mathbf{r}) = E\psi(\mathbf{r})
$$

## 🔁 Recapitulation
- What are eigenfunctions for the $n=l+1$ state of a one-electron atom?
- What are the energy eigenfunctions and eigenvalues for a one-electron atom?
- How does the energy increase as the atomic number increases?

## 🔮 Next Up...
- Multielectron systems
- Approximate methods.

## 📚 References
My favorite sources for this material are:
- [Randy's book](https://github.com/PaulWAyers/IntroQChem/blob/main/documents/DumontBook.pdf?raw=true)
- D. A. MacQuarrie, Quantum Chemistry (University Science Books, Mill Valley California, 1983)
- [One-electron atoms](https://github.com/PaulWAyers/IntroQChem/blob/main/documents/Hatom.pdf?raw=true) (my notes).
- [Davit Potoyan's Jupyterbook course](https://dpotoyan.github.io/Chem324/Lec5-0.html).

There are also some excellent wikipedia articles:
- [Hydrogen-like atoms](https://en.wikipedia.org/wiki/Hydrogen-like_atom)
- [Atomic orbitals](https://en.wikipedia.org/wiki/Atomic_orbital)

```python

```


```python
from IPython.display import display
import numpy as np
from sympy import *

# -----------------------------------------------------------------
### Globlas ###
init_printing()
LIMIT = 100  # im Fall von kreisen
ein, ver = var('eintretende verlassende')

# Hilfsfunktionen
base_zero = lambda X: [i - 1 for i in X]  # Wandel eine Liste in Basis 1 um (für die LP-Basis)

# Eine Klasse um ein LP-Problem zu repräsentieren
class LP:
    def __init__(self, A, b, c, B):
        self.A = A
        self.b = b
        self.c = c
        self.B = B  # zur Basis 1 (nicht 0)

# -----------------------------------------------------------------
def sympy_simplex(lp):
    ### define the variables ###
    A, b, c, B = lp.A, lp.b, lp.c, lp.B
    m , n = A.shape[0], A.shape[1]
    z = symbols('z')
    x = symbols(' '.join(['x' + str(i) for i in range(1, n+m+1)])) # 1 -> n+m

    ### create rows ###
    rows = []
    for i in range(m):
        row = Eq(x[i + n], int(b[i]))
        nbv = [-1 * int(A[i,j]) * x[j] for j in range(n)]
        for k in range(len(nbv)):
            row = Eq(row.lhs, row.rhs + nbv[k])
        rows.append(row)

    ### create ziel ###
    ziel = Eq(z, 0)
    for i in range(len(c)):
        ziel = Eq(ziel.lhs, ziel.rhs + int(c[i]) * x[i])

    ### B and N ###
    # assuming B at the start is the slack variables (i.e. Dict feasible)
    N = [x[i] for i in range(n)]
    B = [x[i + n] for i in range(m)]

    ### show the given problem ###
    print("given LP")
    for row in rows:
        display(row)
    display(ziel)
    print("-" * 30)

    # -----------------------------------------------------------------
    ### Solving ###
    for i in range(LIMIT):
        # eintretende Variable finden
        # auswaehlen nach dem Teknik in der Vorlesung
        # (d.h. var mit grosstem Koeffizeint)
        eintretende = None
        max_eintretende = -oo
        for var, coeff in ziel.rhs.as_coefficients_dict().items():
            # 1 is the first coeff i.e. the value of the ziel function
            if var != 1 and coeff > 0 and coeff > max_eintretende:
                max_eintretende = coeff
                eintretende = var

        # falls keien positive costs => optimal
        if eintretende == None:
            break

        # verlassende Variable finden
        verlassende = None
        min_wert = +oo
        min_row = None

        for row in rows:
            if row.has(eintretende):
                new_row = row
                for nbv in N:
                    if nbv != eintretende:
                        new_row = new_row.subs(nbv, 0)
                wert = solve(new_row.rhs >= 0).as_set().right
                if wert < min_wert:
                    min_wert = wert
                    min_row = row
                    verlassende = row.lhs

        # display eintretende und verlassende Variablen
        print("Step ", i)
        if i == 0: print("Unzulaessiges Dict")
        display(Eq(ein, eintretende))
        display(Eq(ver, verlassende))

        # die Formlen umsetzen und rows updaten
        new_formel = Eq(eintretende, solve(min_row, eintretende)[0])
        new_rows = [new_formel]
        for row in rows:
            if row.lhs != verlassende:
                new_rows.append(
                    Eq(row.lhs, row.rhs.subs(eintretende, new_formel.rhs))
                )
        rows = new_rows

        # new ziel
        ziel = Eq(z, ziel.rhs.subs(eintretende, new_formel.rhs))

        # update B, N
        B.remove(verlassende);
        B.append(eintretende)
        N.remove(eintretende);
        N.append(verlassende)

        # show current solution
        for row in rows:
            display(row)
        display(ziel)
        print("-" * 30)

# -------------------------------------------------------------------------------
blatt2_aufgabe1 = LP(
    np.matrix('2 0 6; -2 8 4; 3 6 5'),
    np.matrix('10; 12; 20'),
    np.matrix('2; 1; 3; 0; 0; 0'),
    [4, 5, 6])

sympy_simplex(blatt2_aufgabe1)
```


## 8.4 Principal Component Regression

- We can determine the principal components of our data set from a singular value decomposition (SVD) of the
coveriance matrix $X^{T}X$:

$$X^{T}X=V\Sigma V^{T}$$

- Projecting the data onto the principal components defines a new data set

$$Z=XV^{T}$$

- We can truncate the new set of vectors to exclude components with small eigenvalues (features/dimensions that contribute little variance)

- Principal component regression:

$$ \vec{w}_{\text{PCA}}=\left( \begin{array}{c}
1 \\
1 \\
1 \\
0 \\
\vdots\\
0 \end{array}\right)
\hskip{1in}
M_{\text{z}}=\left( \begin{array}{ccc}
1 & z_{01} & z_{02} & z_{03} & 0 & \cdots & 0 \\
1 & z_{11} & z_{12} & z_{13} & 0 & \cdots & 0 \\
1 & z_{21} & z_{22} & z_{23} & 0 & \cdots & 0 \\
\vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\
1 & z_{N1} & z_{N2} & z_{N3} & 0 & \cdots & 0 \end{array}\right)
$$

- PCA has an advantage over ridge regression when the data contains many independent collinear variables (high covariance)

- In this case the regression coefficients $\theta_{i}$ have high variance and their solutions can become unstable

```python
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
import random as ra
from scipy.stats import multivariate_normal
import scipy.interpolate
```

```python
for covar in [0,0.95]:

    #Generate bivariate norm distribution
    var = multivariate_normal(mean = [0,0], cov = [[1,covar],[covar,1]])

    #Randomly draw x1, x2 values from pdf
    x1 = []
    x2 = []
    for i in range(1,30000):
        x1rand = 10.*ra.random()-5
        x2rand = 10.*ra.random()-5
        norm_chance = var.pdf([x1rand, x2rand])
        const_chance = ra.random()
        if const_chance <= norm_chance:
            x1.append(x1rand)
            x2.append(x2rand)

    #Plot (x1, x2; y) data
    alpha = 1
    beta = 0.3
    l = 0.2
    y = [1*x1[i] + 1*x2[i] + beta*(ra.random()-.5) for i in range(len(x1))]
    cm = plt.cm.get_cmap('RdYlBu')
    plt.scatter(x1,x2, c = y, cmap = cm)
    plt.colorbar()
    plt.xlabel('x1')
    plt.ylabel('x2')
    plt.show()

    #Generate cost contour
    th1, th2 = np.linspace(0, 2, 100), np.linspace(0, 2, 100)

    cost = np.zeros((len(th1),len(th2)))
    for i in range(len(th1)):
        for j in range(len(th2)):
            summ=0
            for k in range(len(x1)):
                summ+=(th1[i]*x1[k]+th2[j]*x2[k]-y[k]+l*(th1[i]**2.0+th2[j]**2.0))**2.0
            cost[i,j]=summ

    th1, th2 = np.meshgrid(th1, th2)

    # Interpolate
    rbf = scipy.interpolate.Rbf(th1, th2, cost, function='linear')
    cost_inter = rbf(th1, th2)

    plt.imshow(cost_inter, vmin=cost_inter.min(), vmax=cost_inter.max(), origin='lower', extent=[-5, 5, -5, 5])
    plt.colorbar()
    plt.xlabel('Theta 1')
    plt.ylabel('Theta 2')
    plt.title('Mean-square error')
    plt.show()

```

## 8.5 Kernel Regression

- Define a kernel $K\left(x_{i}, x\right)$ local to each data point with the amplitude of the kernel depending onn the distance from the local point to all other points in the sample

- e.g. top-hat or Gaussian kernel

- Nadaraya-Watson estimate of the regression function:
$$f\left(x|K\right)=\frac{\sum_{i=1}^{N}K\left(\frac{||x_{i}-x||}{h}\right)y_{i}}{\sum_{i=1}^{N}K\left(\frac{||x_{i}-x||}{h}\right)}$$

- i.e., the predicted value of the function at $x_{i}$ is a weighted average of the y-values of all the points, with the individual weights given by the values of the kernel at that position

- Rule of thumb: bandwidth is more important than shape of kernel used

- Optimal bandwidth can be found by minimizing the cost with respect to the bandwidth on a cross-validation set
$$CV_{L_{2}}\left(h\right)=\frac{1}{N}\sum_{i=1}^{N}\left(y_{i}-f\left(x_{i}|K\left(\frac{||x_{i}-x_{j}||}{h}\right)\right)\right)^{2}$$

#### Author: Jake VanderPlas
#### License: BSD
####   The figure produced by this code is published in the textbook
####   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
####   For more information, see http://astroML.github.com
####   To report a bug or issue, use the following forum:
####    https://groups.google.com/forum/#!forum/astroml-general

## 8.6 Locally Linear Regression

- Related to kernel regression, but we solve a separate weighted least-squares problem at each point x, finding the weight $w\left(x\right)$ which minimizes
$$\sum_{i=1}^{N}K\left(\frac{||x-x_{i}||}{h}\right)\left(y_{i}-w\left(x\right)x_{i}\right)^{2}$$
- Assume regression function can be approximated as a Taylor series about any local point; truncation at first term (locally constant solution) leads to Kernel regression
- The estimate for locally linear regression is
$$\begin{align}\\
f\left(x|K\right)&=\theta\left(x\right)x\\
&=x^{T}\left(\mathbf{X}^{T}W\left(x\right)\mathbf{X}\right)^{-1}\mathbf{X}^{T}W\left(x\right)\mathbf{Y}\\
&=\sum_{i=1}^{N}w_{i}\left(x\right)y_{i}\\
\end{align}
$$
- where $W\left(x\right)$ is an $N\times N$ diag. matrix with the $i\text{th}$ diaganol element given by $K||x_{i}-x||/h$

```python

```

```python

```


# Speed of symbolic computation of stability polynomials

Symbolic computation of stability polynomials and internal stability polynomials in SymPy can get very slow for methods with many stages.  The stability polynomial can be computed in a variety of ways, and different ways seem to work better for different classes of RK methdods.  Let's investigate.

```
%pylab inline
import sympy
from nodepy import rk
import time
import matplotlib.pyplot as plt
import numpy as np
```

    Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.kernel.zmq.pylab.backend_inline].
    For more information, type 'help(pylab)'.

Next we have 3 functions for computing the stability polynomial in different ways.  The first uses the matrix-inverse formula, the second uses the ratio of determinants, and the third uses a power series for the matrix inverse.  They are implemented for both the Shu-Osher and Butcher forms.

Note that the ratio of determinants in the Shu-Osher form is still not implemented correcly.

Also note that for implicit methods the ratio of determinants is the preferred method, since it gives formulas for the numerator and denominator.  Here we focus on explicit methods, for which the denominator is 1.

```
def lower_triangular_solve(rk,SO=False): # Using lower_triangular_solve, which is always faster than LUsolve
    start = time.clock()
    z=sympy.var('z')
    I = sympy.matrices.eye(len(rk))
    if SO:
        v = 1 - rk.alpha.sum(1)
        vstar = sympy.matrices.Matrix(v[:-1]).T
        v_mp1 = sympy.Rational(v[-1])
        alphastar=sympy.matrices.Matrix(rk.alpha[:-1,:])
        betastar=sympy.matrices.Matrix(rk.beta[:-1,:])
        alpha_mp1 = sympy.matrices.Matrix(rk.alpha[-1,:])
        beta_mp1 = sympy.matrices.Matrix(rk.beta[-1,:])
        p1=(alpha_mp1 + z*beta_mp1)*(I-alphastar-z*betastar).lower_triangular_solve(vstar)
        p1=p1[0].expand()+v_mp1
    else:
        Asym=sympy.matrices.Matrix(rk.A)
        bsym=sympy.matrices.Matrix(rk.b)
        e = sympy.matrices.ones((len(rk),1))
        p1=z*bsym*(I-z*Asym).lower_triangular_solve(e)
        p1=p1[0].expand()+1
    p1=p1.as_poly(z).all_coeffs()
    p1=p1[::-1]
    q1=[sympy.Rational(1)]
    p=np.poly1d(p1[::-1])    # Numerator
    q=np.poly1d(q1[::-1])    # Denominator
    t = time.clock()-start
    return p,q,t

def determinants(rk,SO=False): #Using charpoly
    start = time.clock()
    z=sympy.var('z')
    if SO:
        I = sympy.matrices.eye(len(rk))
        v = 1 - rk.alpha.sum(1)
        vstar = sympy.matrices.Matrix(v[:-1]).T
        v_mp1 = sympy.Rational(v[-1])
        alphastar=sympy.matrices.Matrix(rk.alpha[:-1,:])
        betastar=sympy.matrices.Matrix(rk.beta[:-1,:])
        alpha_mp1 = sympy.matrices.Matrix(rk.alpha[-1,:])
        beta_mp1 = sympy.matrices.Matrix(rk.beta[-1,:])
        xsym = I - alphastar - z*betastar + vstar/v_mp1 * (alpha_mp1+z*beta_mp1)
        p1=v_mp1*xsym.charpoly(z).coeffs()
    else:
        Asym=sympy.matrices.Matrix(rk.A)
        bsym=sympy.matrices.Matrix(np.tile(rk.b,(len(rk),1)))
        xsym=Asym-bsym
        p1=xsym.charpoly(z).coeffs()

    q1=[sympy.Rational(1)]
    p=np.poly1d(p1[::-1])    # Numerator
    q=np.poly1d(q1[::-1])    # Denominator
    t = time.clock()-start
    return p,q,t

def series(rk,SO=False): # Using power series
    start = time.clock()
    s = rk.num_seq_dep_stages()
    z=sympy.var('z')
    I = sympy.matrices.eye(len(rk))
    if SO:
        alphastarsym = sympy.matrices.Matrix(rk.alpha[0:-1,:])
        betastarsym  = sympy.matrices.Matrix(rk.beta[0:-1,:])

        matsym = alphastarsym + betastarsym*z
        vecsym = sympy.matrices.Matrix(rk.alpha[-1,:]+z*rk.beta[-1,:])

    else:
        Asym=sympy.matrices.Matrix(rk.A)
        bsym=sympy.matrices.Matrix(rk.b)
        matsym = z*sympy.matrices.Matrix(rk.A)
        vecsym = z*sympy.matrices.Matrix(rk.b)

    # Compute (I-zA)^(-1) = I + zA + (zA)^2 + ... + (zA)^(s-1)
    matpow = I
    matsum = I
    for i in range(1,s):
        matpow = matpow*matsym
        matsum = matsum + matpow
    p1 = vecsym*matsum
    if SO:
        v = 1 - rk.alpha.sum(1)
        vstar = sympy.matrices.Matrix(v[:-1]).T
        v_mp1 = sympy.Rational(v[-1])
        p1 = p1*vstar
        p1=p1[0].expand()+v_mp1
    else:
        e = sympy.matrices.ones((len(rk),1))
        p1 = p1*e
        p1=p1[0].expand()+1

    p1=p1.as_poly(z).all_coeffs()
    p1=p1[::-1]
    q1=[sympy.Rational(1)]
    p=np.poly1d(p1[::-1])    # Numerator
    q=np.poly1d(q1[::-1])    # Denominator
    t = time.clock()-start
    return p,q,t
```

The following function just runs each of the above approaches on a given family of methods and plots the run-times for each approach:

```
def time_stability_polynomial_calculation(method,pmin=2,pmax=5,approaches=[lower_triangular_solve,determinants,series],SO=False):
    norders = pmax-pmin+1
    orders = range(pmin,pmax+1)

    times = []
    for j in range(len(approaches)):
        times.append(np.zeros((norders,1)))

    for i, order in enumerate(orders):
        myrk = method(order)
        for j, approach in enumerate(approaches):
            p,q,t = approach(myrk,SO=SO)
            times[j][i] = t

    for j in range(len(approaches)):
        plt.plot(orders,times[j],linewidth=3)
        plt.hold(True)

    leg_text = [approach.__name__ for approach in approaches]
    plt.legend(leg_text,loc='best')
    plt.ylabel('time (s)')
    plt.xlabel('order')
    plt.title('Computation of stability polynomial for '+method.__name__)
    plt.hold(False)
```

```
time_stability_polynomial_calculation(rk.extrap,pmax=7)
```

```
time_stability_polynomial_calculation(rk.extrap,pmax=7,SO=True,approaches=[lower_triangular_solve,series])
```

Clearly the `lower_triangular_solve` method is the fastest for extrapolation methods.  The other two methods will quickly reach unreasonable run-times if we increase $p$ further.  Using the Shu-Osher form is noticeably faster for all methods.

```
time_stability_polynomial_calculation(rk.DC,pmax=5)
```

```
time_stability_polynomial_calculation(rk.DC,pmax=5, )
```

For deferred correction methods, we see roughly the opposite behavior: `lower_triangular_solve` is extremely slow compared to either of the other two.  Series has a slight edge over charpoly.  Again, using the Shu-Osher form is significantly faster for some methods.

```
time_stability_polynomial_calculation(rk.RKC1,pmax=12)
```

```
time_stability_polynomial_calculation(rk.RKC1,pmax=12,SO=True,approaches=[lower_triangular_solve,series])
```

For RKC methods, things are complicated.  The `determinants` approach is fastest if using the Butcher form, but the other two approaches are reasonably fast if using the Shu-Osher form (I really need to implement the Shu-Osher ration of determinants!) (I should add that the x-axis label here is wrong; it should really be "stages")

```
time_stability_polynomial_calculation(rk.SSPRK2,pmax=12)
```

```
time_stability_polynomial_calculation(rk.SSPRK2,pmax=12,SO=True,approaches=[lower_triangular_solve,series])
```

SSPRK2 methods behave mostly like RKC methods, which is unsurprising since they have a very similar structure.  But they show much better results using `lower_triangular_solve` on the SO form.

So, we can implement all three options.  Which should be the default?  Let's try an "ordinary" Runge-Kutta method, not belonging to any of these families (but with a moderately large number of stages):

```
approaches=[lower_triangular_solve,determinants,series]
SO=False
times = []
myrk = rk.loadRKM('DP8')
for j, approach in enumerate(approaches):
    p,q,t = approach(myrk,SO=SO)
    times.append(t)
print zip([approach.__name__ for approach in approaches], times)
```

    [('lower_triangular_solve', 6.044034000000011), ('determinants', 0.4058479999998781), ('series', 1.3429089999999633)]

`lower_triangular_solve` is the clear loser.  So we'll go with `determinants` as the default if Butcher coefficients are used, `lower_triangular_solve` as the default if Shu-Osher arrays are used, but allow the user to pick any of the three.

For the implementation of `internal_stability_polynomials`, it seems not worthwhile to work out and implement a ratio of determinants formula, so we'll just provide the other two and default to Shu-Osher with `lower_triangular_solve`.