```julia
using CSV, DataFrames
using Plots
using Distributions, Statistics
using Optim, NLopt
```

# Time Series Analysis

**Definition (Stochastic process)**
A stochastic process
is a sequence of random variables, indexed by time: $\{Y_{t}\}_{t=-\infty}^{\infty}$

**Definition (Times series)**
A time series is one
observation of a stochastic process, over a specific interval: $\{y_{t}\}_{t=1}^{n}$.

So a time series is a sample of size $n$ from a stochastic process.
It's important to keep in mind that conceptually, one could draw another
sample, and that the values would be different.

**Definition (Autocovariance)**
The $j^{th}$ autocovariance of a stochastic
process is $\gamma_{jt}=\mathcal{E}(y_{t}-\mu_{t})(y_{t-j}-\mu_{t-j})$
where $\mu_{t}=\mathcal{E}\left(y_{t}\right).$

**Definition (Covariance (weak) stationarity)**
A stochastic process is covariance
stationary if it has time constant mean and autocovariances of all
orders:

\begin{eqnarray*}
\mu_{t} & =\mu, & \forall t\\
\gamma_{jt} & =\gamma_{j}, & \forall t
\end{eqnarray*}

As we've seen, this implies that $\gamma_{j}=\gamma_{-j}:$ the autocovariances
depend only one the interval between observations, but not the time
of the observations.

**Definition (Strong stationarity)**
A stochastic process is strongly stationary
if the joint distribution of an arbitrary collection of the $\left\{ Y_{t}\right\} $,
e.g., $(Y_{t-j},Y_{t-k},...,Y_{t},...,Y_{t+l},Y_{t+m}\}$, doesn't
depend on $t.$

Since moments are determined by the distribution, strong stationarity$\Rightarrow$weak
stationarity.

The time series
is one sample from the stochastic process, and each of the random
variables over the sample interval is sampled only once. One could
think of $M$ repeated samples from the stoch. proc., e.g., $\left\{ y_{tm}\right\} _{m=1}^{M}$
By a LLN, we would expect that
$$
\frac{1}{M}\sum_{m=1}^{M}y_{tm}\overset{p}{\rightarrow}\mathcal{E}(Y_{t})
$$
as $M$ gets large. The problem is, we have only one sample to work
with, since we can't go back in time and collect another. How can
$\mathcal{E}(Y_{t})$ be estimated then? It turns out that **ergodicity**
is the needed property.

**Definition (Ergodicity)**
A stationary stochastic process is ergodic (for
the mean) if the time average converges to the mean
\begin{equation}
\frac{1}{n}\sum_{t=1}^{n}y_{t}\overset{p}{\rightarrow}\mu
\end{equation}

A sufficient condition for ergodicity is that the autocovariances
be absolutely summable:
$$
\sum_{j=0}^{\infty}|\gamma_{j}|<\infty
$$

This implies that the autocovariances die off.

**Definition (Autocorrelation)**
The $j^{th}$ autocorrelation, $\rho_{j}$ is
just the $j^{th}$ autocovariance divided by the variance:

\begin{equation}
\rho_{j}=\frac{\gamma_{j}}{\gamma_{0}}
\end{equation}

$\,$
**Definition (White noise)**
White noise is just the time series literature term
for a classical error. $\epsilon_{t}$ is white noise if i) $\mathcal{E}(\epsilon_{t})=0,\forall t,$
ii) $V(\epsilon_{t})=\sigma^{2},\forall t$ and iii) $\epsilon_{t}$
and $\epsilon_{s}$ are independent, $t\neq s.$ Gaussian white noise
just adds a normality assumption.

## AR(p)

An AR(p) process can be represented as
$$
y_{t}=c+\phi_{1}y_{t-1}+\phi_{2}y_{t-2}+\cdots+\phi_{p}y_{t-p}+\varepsilon_{t}
$$
where $\epsilon_{t}$ is white noise. This is just a linear regression
model, and assuming stationarity, we can estimate the parameters by
OLS.

What is needed for stationarity?

The dynamic behavior of an AR(p) process can be studied by writing
this $p^{th}$ order difference equation as a vector first order difference
equation (this is known as the companion form):
$$
\left[\begin{array}{l}
y_{t}\\
y_{t-1}\\
\vdots\\
y_{t-p+1}
\end{array}\right]=\left[\begin{array}{l}
c\\
0\\
\vdots\\
0
\end{array}\right]+\left[\begin{array}{lllll}
\phi_{1} & \phi_{2} & \cdots &  & \phi_{p}\\
1 & 0 & 0 &  & 0\\
0 & 1 & 0 & \ddots & 0\\
\vdots & \ddots & \ddots & \ddots & 0\cdots\\
0 & \cdots & 0 & 1 & 0
\end{array}\right]\left[\begin{array}{l}
y_{t-1}\\
y_{t-2}\\
\vdots\\
y_{t-p}
\end{array}\right]+\left[\begin{array}{l}
\varepsilon_{t}\\
0\\
\vdots\\
0
\end{array}\right]
$$
 or
$$
Y_{t}=C+FY_{t-1}+E_{t}
$$

With this, we can recursively work forward in time:
\begin{eqnarray*}
Y_{t+1} & = & C+FY_{t}+E_{t+1}\\
 & = & C+F\left(C+FY_{t-1}+E_{t}\right)+E_{t+1}\\
 & = & C+FC+F^{2}Y_{t-1}+FE_{t}+E_{t+1}
\end{eqnarray*}
 and
\begin{eqnarray*}
Y_{t+2} & = & C+FY_{t+1}+E_{t+2}\\
 & = & C+F\left(C+FC+F^{2}Y_{t-1}+FE_{t}+E_{t+1}\right)+E_{t+2}\\
 & = & C+FC+F^{2}C+F^{3}Y_{t-1}+F^{2}E_{t}+FE_{t+1}+E_{t+2}
\end{eqnarray*}
 or in general
$$
Y_{t+j}=C+FC+\cdots+F^{j}C+F^{j+1}Y_{t-1}+F^{j}E_{t}+F^{j-1}E_{t+1}+\cdots+FE_{t+j-1}+E_{t+j}
$$

Consider the impact of a shock in period $t$ on $y_{t+j}.$
This is simply
$$
\frac{\partial Y_{t+j}}{\partial E_{t}^{\prime}}_{(1,1)}=F_{(1,1)}^{j}
$$
 If the system is to be stationary, then as we move forward in time
this impact must die off. Otherwise a shock causes a permanent change
in the mean of $y_{t}.$ Therefore, stationarity requires that
$$
\lim_{j\rightarrow\infty}F_{(1,1)}^{j}=0
$$
Save this result, we'll need it in a minute.

Consider the eigenvalues of the matrix $F.$ These are the $\lambda$
such that
$$
|F-\lambda I_{P}|=0
$$
 The determinant here can be expressed as a polynomial. For example,
for $p=1,$ the matrix $F$ is simply
$$
F=\phi_{1}
$$
 so
$$
|\phi_{1}-\lambda|=0
$$
 can be written as
$$
\phi_{1}-\lambda=0
$$
 When $p=2,$ the matrix $F$ is
$$
F=\left[\begin{array}{ll}
\phi_{1} & \phi_{2}\\
1 & 0
\end{array}\right]
$$
 so
$$
F-\lambda I_{P}=\left[\begin{array}{ll}
\phi_{1}-\lambda & \phi_{2}\\
1 & -\lambda
\end{array}\right]
$$
 and
$$
|F-\lambda I_{P}|=\lambda^{2}-\lambda\phi_{1}-\phi_{2}
$$
 So the eigenvalues are the roots of the polynomial
$$
\lambda^{2}-\lambda\phi_{1}-\phi_{2}
$$
 which can be found using the quadratic equation. This generalizes.
For a $p^{th}$ order AR process, the eigenvalues are the roots of
$$
\lambda^{p}-\lambda^{p-1}\phi_{1}-\lambda^{p-2}\phi_{2}-\cdots-\lambda\phi_{p-1}-\phi_{p}=0
$$

Supposing that all of the roots of this polynomial are distinct,
then the matrix $F$ can be factored as, using Eigen decomposition,}
$$
F=T\Lambda T^{-1}
$$
 where $T$ is the matrix which has as its columns the eigenvectors
of $F,$ and $\Lambda$ is a diagonal matrix with the eigenvalues
on the main diagonal. Using this decomposition, we can write
$$
F^{j}=\left(T\Lambda T^{-1}\right)\left(T\Lambda T^{-1}\right)\cdots\left(T\Lambda T^{-1}\right)
$$
 where $T\Lambda T^{-1}$ is repeated $j$ times. This gives
$$
F^{j}=T\Lambda^{j}T^{-1}
$$
 and
$$
\Lambda^{j}=\left[\begin{array}{llll}
\lambda_{1}^{j} & 0 &  & 0\\
0 & \lambda_{2}^{j}\\
 &  & \ddots\\
0 &  &  & \lambda_{p}^{j}
\end{array}\right]
$$

Supposing that the $\lambda_{i}$ $i=1,2,...,p$ are all
real valued, it is clear that
$$
\lim_{j\rightarrow\infty}F_{(1,1)}^{j}=0
$$
 requires that
$$
|\lambda_{i}|<1,i=1,2,...,p
$$
 e.g., the eigenvalues must be less than one in absolute value.
- It may be the case that some eigenvalues are complex-valued. The previous
result generalizes to the requirement that the eigenvalues be less
than one in **modulus**, where the modulus of a complex number
$a+bi$ is
$$
mod(a+bi)=\sqrt{a^{2}+b^{2}}
$$
 This leads to the famous statement that "stationarity requires
the roots of the determinantal polynomial to lie inside the complex
unit circle."

- When there are roots on \ the unit circle (unit roots) or outside
the unit circle, we leave the world of stationary processes.
- Dynamic multipliers: $\partial y_{t+j}/\partial\varepsilon_{t}=F_{(1,1)}^{j}$
is a \emph{dynamic multiplier} or an **impulse-response** function.
Real eigenvalues lead to steady movements, whereas complex eigenvalues
lead to oscillatory behavior. Of course, when there are multiple eigenvalues
the overall effect can be a mixture.

### Moments of AR(p)  process

The AR(p) process is
$$
y_{t}=c+\phi_{1}y_{t-1}+\phi_{2}y_{t-2}+\cdots+\phi_{p}y_{t-p}+\varepsilon_{t}
$$
 Assuming stationarity, $\mathcal{E}(y_{t})=\mu,\forall t,$ so
$$
\mu=c+\phi_{1}\mu+\phi_{2}\mu+...+\phi_{p}\mu
$$
 so
$$
\mu=\frac{c}{1-\phi_{1}-\phi_{2}-...-\phi_{p}}
$$
 and
$$
c=\mu-\phi_{1}\mu-...-\phi_{p}\mu
$$
 so
\begin{eqnarray*}
y_{t}-\mu & = & \mu-\phi_{1}\mu-...-\phi_{p}\mu+\phi_{1}y_{t-1}+\phi_{2}y_{t-2}+\cdots+\phi_{p}y_{t-p}+\varepsilon_{t}-\mu\\
 & = & \phi_{1}(y_{t-1}-\mu)+\phi_{2}(y_{t-2}-\mu)+...+\phi_{p}(y_{t-p}-\mu)+\varepsilon_{t}
\end{eqnarray*}

With this, the second moments are easy to find: The variance
is
$$
\gamma_{0}=\phi_{1}\gamma_{1}+\phi_{2}\gamma_{2}+...+\phi_{p}\gamma_{p}+\sigma^{2}
$$
 The autocovariances of orders $j\geq1$ follow the rule
\begin{eqnarray*}
\gamma_{j} & = & \mathcal{E}\left[\left(y_{t}-\mu\right)\left(y_{t-j}-\mu)\right)\right]\\
 & = & \mathcal{E}\left[\left(\phi_{1}(y_{t-1}-\mu)+\phi_{2}(y_{t-2}-\mu)+...+\phi_{p}(y_{t-p}-\mu)+\varepsilon_{t}\right)\left(y_{t-j}-\mu\right)\right]\\
 & = & \phi_{1}\gamma_{j-1}+\phi_{2}\gamma_{j-2}+...+\phi_{p}\gamma_{j-p}
\end{eqnarray*}
 Using the fact that $\gamma_{-j}=\gamma_{j},$ one can take the $p+1$
equations for $j=0,1,...,p$, which have $p+1$ unknowns ($\sigma^{2},$
$\gamma_{0},\gamma_{1},...,\gamma_{p})$ and solve for the unknowns.
With these, the $\gamma_{j}$ for $j>p$ can be solved for recursively.

## MA(q)

A $q^{th}$ order moving average (MA) process is
$$
y_{t}=\mu+\varepsilon_{t}+\theta_{1}\varepsilon_{t-1}+\theta_{2}\varepsilon_{t-2}+\cdots+\theta_{q}\varepsilon_{t-q}
$$
 where $\varepsilon_{t}$ is white noise. The variance is
\begin{eqnarray*}
\gamma_{0} & = & \mathcal{E}\left(y_{t}-\mu\right)^{2}\\
 & = & \mathcal{E}\left(\varepsilon_{t}+\theta_{1}\varepsilon_{t-1}+\theta_{2}\varepsilon_{t-2}+\cdots+\theta_{q}\varepsilon_{t-q}\right)^{2}\\
 & = & \sigma^{2}\left(1+\theta_{1}^{2}+\theta_{2}^{2}+\cdots+\theta_{q}^{2}\right)
\end{eqnarray*}
 Similarly, the autocovariances are
\begin{eqnarray*}
\gamma_{j} & = & \mathcal{E}\left[\left(y_{t}-\mu\right)\left(y_{t-j}-\mu\right)\right]\\
 & = & \sigma^{2}(\theta_{j}+\theta_{j+1}\theta_{1}+\theta_{j+2}\theta_{2}+\cdots+\theta_{q}\theta_{q-j}),j\leq q\\
 & = & 0,j>q
\end{eqnarray*}
 Therefore an MA(q) process is necessarily covariance stationary and
ergodic, as long as $\sigma^{2}$ and all of the $\theta_{j}$ are
finite.

For example, if we have an MA(1) model, then $E(y_{t})=\mu$, $V(y_{t})=\sigma^{2}(1+\theta_{1}^{2})$,
and $\gamma_{1}=\sigma^{2}\theta_{1}$. The higher order autocovariances
are zero.
- Thus, if the model is MA(1) with normally distributed shocks, the
density of the vector of $n$ observations, $y$, is
\begin{align}
f_{Y}(y|\rho) & =\frac{\text{1}}{\sqrt{\left(2\pi\right)^{n}\left|\Sigma\right|}}\exp\left(-\frac{1}{2}\left(y-\mu\right)^{\prime}\Sigma^{-1}\left(y-\mu\right)\right)\label{eq:MA1likelihood}
\end{align}
where
$$
\Sigma=\sigma^{2}\left[\begin{array}{ccccc}
1+\theta_{1}^{2} & \theta_{1} & 0 & \cdots & 0\\
\theta_{1} & \ddots & \ddots & \ddots & \vdots\\
0 & \ddots & \ddots & \ddots & 0\\
\vdots & \ddots & \ddots & \ddots & \theta_{1}\\
0 & \cdots & 0 & \theta_{1} & 1+\theta_{1}^{2}
\end{array}\right].
$$
With this, it is very easy to program the log-likelihood function.
For higher order MA models, the only difference is the structure of
$\Sigma$ becomes more complicated. In this form, one needs a lot
of computer memory. A more economical approach uses the Kalman filter,
which we'll see in the discussion of state space models.

- If we don't make assumptions on the distribution of the shocks, then
method of moments estimation can be used.

Notes:

- An issue to be aware of is that MA models are not identified, in that
there exist multiple parameter values that give the same value of
the likelihood function.
- For example, the MA(1) model with $\tilde{\sigma}^{2}=\theta^{2}\sigma^{2}$
and $\tilde{\theta}_{1}=\frac{1}{\theta_{1}}$ has identical first
and second moments to the original model, so the likelihood function
has the same value.
- Normally, the parameterization that leads to an invertible
MA model is the one that is selected. An invertible MA model is one
that has a representation as a AR($\infty)$ model. For the MA(1)
model, the invertible parameterization has $\left|\theta_{1}\right|<1$.
- This implies that parameter restrictions will need to be imposed when
estimating the MA model, to enforce selection of the invertible model.
- Maximization of the conditional likelihood is also used for estimation,
sometimes. Assuming that $\epsilon_{0}$ is known (for example, equal
to zero), then one can compute $\epsilon_{1}$, given the parameters.
Then one works forward recursively to get all of the $\epsilon_{t}$.
With these, the likelihood function is very easy to compute. This
is a convenient shortcut, but it's not recommended if the sample is
not large, especially since it's not hard to compute the exact likelihood
function.

## ARMA(p,q) Model

An ARMA($p,q$) model is $(1+\phi_{1}L+\phi_{2}L^{2}+...+\phi_{p}L^{p})y_{t}=c+(1+\theta_{1}L+\theta_{2}L^{2}+...+\theta_{q}L^{q})\epsilon_{t}$.
These are popular in applied time series analysis. A high order AR
process may be well approximated by a low order MA process,
and a high order MA process may be well approximated by a low
order AR process. By combining low order AR and MA processes in the
same model, one can hope to fit a wide variety of time series using
a parsimonious number of parameters. There is much literature on how
to choose $p$ and $q,$ which is outside the scope of this course.
Estimation can be done using the Kalman filter, assuming that the
errors are normally distributed.

## ARCH and GARCH

ARCH (autoregressive conditionally heteroscedastic) models appeared
in the literature in 1982, in Engle, Robert F. (1982). Autoregressive
Conditional Heteroskedasticity with Estimates of Variance of United
Kingdom Inflation, Econometrica 50:987-1008. This paper
stimulated a very large growth in the literature for a number of years
afterward. The related GARCH (generalized ARCH) model is now one of
the most widely used models for financial time series.

Financial time series often exhibit several type of behavior:
- volatility clustering: periods of low variation can be followed by
periods of high variation
- fat tails, or excess kurtosis,:
the marginal density of a series is more strongly peaked and has fatter
tails than does a normal distribution with the same mean and variance.
- leverage (negative correlation between returns and volatility), which
often shows up as negative skewness
of returns
- perhaps slight autocorrelation within the bounds allowed by arbitrage

The data set "nysewk" provides
an example. If we compute 100 times the growth rate of the series,
using log differences, we can obtain the plots below:

```julia
# weekly close price of NSYE, data provided with GRETL

data = DataFrame(CSV.File("../data/nysewk.csv",header=false))
first(data,6)
```

	Column1

	Float64

6 rows × 1 columns

1	531.0

2	533.0

3	538.0

4	538.0

5	531.0

6	540.0

```julia
# compute weekly percentage growth
data = convert(Array,data)
y = 100.0 * log.(data[2:end] ./ data[1:end-1]);
```

```julia
p1 = plot(y,legend=false)
```

```julia
p2 = histogram(y, normed=true,legend=false,bins=500)
```

In the first we clearly see volatility clusters, and in the second, we see
excess kurtosis, skew, and tails fatter than the normal distribution.

### ARCH

A basic Autoregressive conditional heteroskedasticity (ARCH) specification is
\begin{align*}
y_{t} & =\mu+\rho y_{t-1}+\epsilon_{t}\\
 & \equiv g_{t}+\epsilon_{t}\\
\epsilon_{t} & =\sigma_{t}u_{t}\\
\sigma_{t}^{2} & =\omega+\sum_{i=1}^{q}\alpha_{i}\epsilon_{t-i}^{2}
\end{align*}
where the $u_{t}$ are Gaussian white noise shocks. The ARCH variance
is a moving average process. Previous large shocks to the series cause
the conditional variance of the series to increase. There is no leverage:
negative shocks have the same impact on the future variance as do
positive shocks..

- for $\sigma_{t}^{2}$ to be positive for all realizations of $\left\{ \epsilon_{t}\right\} $,
we need $\omega>0$, $\alpha_{i}\ge0$, $\forall i$.
- to ensure that the model is covariance stationary, we need $\sum_{i}\alpha_{i}<1$.
Otherwise, the variances will explode off to infinity.

To find the likelihood
in terms of the observable $y_{t}$ instead of the unobservable $\epsilon_{t}$,
first note that the series $u_{t}=\left(y_{t}-g_{t}\right)/\sigma_{t}=\frac{\epsilon_{t}}{\sigma_{t}}$
is iid Gaussian, so the likelihood is simply the product of standard
normal densities.
\begin{eqnarray*}
u & \sim & N(0,I),\textrm{ so}\\
f(u) & = & \prod_{t=1}^{n}\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{u_{t}^{2}}{2}\right)
\end{eqnarray*}
 The joint density for $y$ can be constructed using a change of variables:

We have $u_{t}=\left(y_{t}-\mu-\rho y_{t-1}\right)/\sigma_{t}$, so
$\frac{\partial u_{t}}{\partial y_{t}}=\frac{1}{\sigma_{t}}$ and
$|\frac{\partial u}{\partial y^{\prime}}|=\prod_{t=1}^{n}\frac{1}{\sigma_{t}},$  doing a change of variables,
$$
f(y;\theta)=\prod_{t=1}^{n}\frac{1}{\sqrt{2\pi}}\frac{1}{\sigma_{t}}\exp\left(-\frac{1}{2}\left(\frac{y_{t}-\mu-\rho y_{t-1}}{\sigma_{t}}\right)^{2}\right)
$$
where $\theta$ is the vector of all parameters (the parameters in
$g_{t}$, and the $\omega$ and alpha parameters of the ARCH specification.
Taking logs,
$$
\ln L(\theta)=-n\ln\sqrt{2\pi}-\sum_{t=1}^{n}\ln\sigma_{t}-\frac{1}{2}\sum_{t=1}^{n}\left(\frac{y_{t}-\mu-\rho y_{t-1}}{\sigma_{t}}\right)^{2}.
$$

In principle, this is easy to maximize. Some complications
can arise when the restrictions for positivity and stationarity are
imposed. Consider a fairly short data series with low volatility in
the initial part, and high volatility at the end. This data appears
to have a nonstationary variance sequence. If one attempts to estimate
and ARCH model with stationarity imposed, the data and the restrictions
are saying two different things, which can make maximization of the
likelihood function difficult.

### GARCH

Note that an ARCH model specifies the variance process as a moving
average. For the same reason that an ARMA model may be used to parsimoniously
model a series instead of a high order AR or MA, one can do the same
thing for the variance series. A basic GARCH(p,q) specification is
\begin{align*}
y_{t} & =\mu+\rho y_{t-1}+\epsilon_{t}\\
\epsilon_{t} & =\sigma_{t}u_{t}\\
\sigma_{t}^{2} & =\omega+\sum_{i=1}^{q}\alpha_{i}\epsilon_{t-i}^{2}+\sum_{i=1}^{p}\beta_{i}\sigma_{t-i}^{2}
\end{align*}
The idea is that a GARCH model with low values of p and q may fit
the data as well or better than an ARCH model with large q.
 The model also requires restrictions for positive variance and stationarity,
which are:

- $\omega>0$
- $\alpha_{i}\ge0,\,i=1,...,q$
- $\beta_{i}\ge0,\,i=1,...,p$
- $\sum_{i=1}^{q}\alpha_{i}$+$\sum_{i=1}^{p}\beta_{i}<1$.

To estimate a GARCH model, you need to initialize $\sigma_{0}^{2}$
at some value. The sample unconditional variance is one possibility.
Another choice could be the sample variance of the initial elements
of the sequence. One can also ''backcast'' the conditional variance.

Notes:
- The GARCH model also requires restrictions on the parameters to ensure
stationarity and positivity of the variance.
- A useful modification is the EGARCH model (exponential GARCH). This model treats the logarithm of the variance as an ARMA process, so the variance will be positive without restrictions on the parameters.
- There are many variants that introduce asymmetry (leverage) and non-normality.
- GARCH(1,1) is a highly popular model in financial analysis.

```julia
function garch11(theta, y)
    # dissect the parameter vector
    mu = theta[1]
    rho = theta[2]
    omega = theta[3]
    alpha = theta[4]
    beta = theta[5]
    resid = y[2:end] .- mu .- rho*y[1:end-1]
    n = size(resid,1)
    h = zeros(n)
    # initialize variance; either of these next two are reasonable choices
    h[1] = var(y[1:10])
    #h[1] = var(y)
    rsq = resid.^2.0
    for t = 2:n
        h[t] = omega + alpha*rsq[t-1] + beta*h[t-1]
    end
    logL = -log(sqrt(2.0*pi)) .- 0.5*log.(h) .- 0.5*rsq./h
end
```

    garch11 (generic function with 1 method)

```julia
thetastart = [mean(y); 0.0; var(y); 0.1; 0.1]
obj = theta -> -sum(garch11(theta, y))
thetahat, logL, junk  = fmincon(obj, thetastart, [], [], [-Inf, -1.0, 0.0, 0.0, 0.0], [Inf, 1.0, Inf, 1.0, 1.0])

```

    ([0.17509446603259488, -0.0017974081950996854, 0.1565082410267794, 0.11201475043965671, 0.8548995338432239], 4402.20524546603, :FTOL_REACHED)

```julia
model = θ -> garch11(θ, y)
avg_obj = θ -> -mean(vec(model(θ))) # average log likelihood
thetahat, objvalue, converged = fminunc(avg_obj, thetahat) # do the minimization of -logL
```

    ([0.17509289904293293, -0.0017974048437616162, 0.15650738859226337, 0.11200543356269112, 0.8549074823073503], 2.081420919751293, true)

How do you select p and q?

## Model Selection

Generally, as you add more variables to a regression, the bias of the
predictions decreases and the variance increases. Too few covariates yields high
bias; this called **underfitting**. Too many covariates yields high variance; this
called **overfitting**. Good predictions result from achieving a good balance
between bias and variance.

In model selection there are two problems:
1. assigning a "score" to each
model which measures, in some sense, how good the model is, and
2. searching
through all the models to find the model with the best score.

Let $S \subset {1,\dots,k}$ and let $\mathcal{X}_S = {X_j:j\in S}$ denote a subset of the covariates. Let $\beta_S$ denote the coefficients of the corresponding set of covariates. Define $\hat{r_S}(x)$ to be the estimated regression function. The fitted/predicted values from model $S$ are denoted by $\hat{Y_i}(S)=\hat{r_S}(X_i)$. The **prediction risk** is defined to be
$$R(S)=\sum_{i=1}^{n}\mathbb{E}(\hat{Y_i}(S)-Y_i^*)^2$$
where $Y_i^*$ denotes the value of a future observation of Y_i at covariate value $X_i$. Our goal is to choose $S$ to make $R(S)$ small.

The **training error** is defined to be
$$\hat{R_{tr}}(S)=\sum_{i=1}^{n}\mathbb{E}(\hat{Y_i}(S)-Y_i)^2$$

**Finding a good model involves trading off fit and complexity.**

### AIC (Akaike Information Criterion)
The idea is to choose S to maximize:
$$AIC(S) = l_S-\vert S \vert$$
where $l_S$ is the log-likelihood of the model evaluated at the MLE. This can be thought of "goodness of fit" minus "complexity."

### BIC (Bayesian information criterion)
$$BIC(S) = l_S - \frac{\vert S \vert}{2}\log n$$
The BIC score has a Bayesian interpretation. Let $\mathcal{S} = {S_1, ... , S_m}$ denote
a set of models. Suppose we assign the prior $P(S_j) = 1/m$ over the models.
Also, assume we put a smooth prior on the parameters within each model. It
can be shown that the posterior probability for a model is approximately,
$$P(S_j \vert \text{data}) \approx \frac{e^{BIC(S_j)}}{\sum_r e^{BIC(S_r)}}$$ Hence, choosing the model with highest BIC is like choosing the model with
highest posterior probability. The BIC score also has an information-theoretic
interpretation in terms of something called minimum description length.

### Leave-one-out cross-validation
The risk estimator is
$$\hat{R}_{CV}(S)=\sum_{i=1}^{n}(Y_i-\hat{Y_{(i)}})^2$$
where $\hat{Y_{(i)}}$ is the prediction for $Y_i$ obtained by fitting the model $Y_i$ with omitted.
It can be shown that
$$\hat{R}_{CV}(S)=\sum_{i=1}^{n}\Big(\frac{Y_i-\hat{Y_i}(S)}{1-U_{ii}(S)}\Big)^2$$
where $U_{ii}(S)$ is the $i^{\text{th}}$ diagonal element of the matrix
$$U(S)=X_S(X_S^TX_S)^{-1}X_S^T$$
Thus, one need not actually drop each observation and re-fit the model. A
generalization is **k-fold cross-validation**. Here we divide the data into k
groups; often people take k = 10. We omit one group of data and fit the
models to the remaining data. We use the fitted model to predict the data
in the group that was omitted. We then estimate the risk by $\sum_i (Y_i-\hat{Y_i})^2$
where the sum is over the the data points in the omitted group. This process is
repeated for each of the k groups and the resulting risk estimates are averaged.

Now let us turn to the problem of model search. If there are k covariates
then there are $2^k$ possible models. We need to search through all these models,
assign a score to each one, and choose the model with the best score. If k is
not too large we can do a complete search over all the models. When k is large,
this is infeasible. In that case we need to search over a subset of all the models.
Two common methods are forward and backward stepwise regression.
In forward stepwise regression, we start with no covariates in the model. We
then add the one variable that leads to the best score. We continue adding
variables one at a time until the score does not improve. Backwards stepwise
regression is the same except that we start with the biggest model and drop
one variable at a time. Both are greedy searches; nether is guaranteed to
find the model with the best score. Another popular method is to do random
searching through the set of all models. However, there is no reason to expect
this to be superior to a deterministic search.

# Function Load:

```julia
function fminunc(obj, x; tol = 1e-10)
    results = Optim.optimize(obj, x, LBFGS(),
                            Optim.Options(
                            g_tol = tol,
                            x_tol=tol,
                            f_tol=tol))
    return results.minimizer, results.minimum, Optim.converged(results)
    #xopt, objvalue, flag = fmincon(obj, x, tol=tol)
    #return xopt, objvalue, flag
end
```

    fminunc (generic function with 1 method)

```julia
function fmincon(obj, startval, R=[], r=[], lb=[], ub=[]; tol = 1e-10, iterlim=0)
    # the objective is an anonymous function
    function objective_function(x::Vector{Float64}, grad::Vector{Float64})
        obj_func_value = obj(x)[1,1]
        return(obj_func_value)
    end
    # impose the linear restrictions
    function constraint_function(x::Vector, grad::Vector, R, r)
        result = R*x .- r
        return result[1,1]
    end
    opt = Opt(:LN_COBYLA, size(startval,1))
    min_objective!(opt, objective_function)
    # impose lower and/or upper bounds
    if lb != [] lower_bounds!(opt, lb) end
    if ub != [] upper_bounds!(opt, ub) end
    # impose linear restrictions, by looping over the rows
    if R != []
        for i = 1:size(R,1)
            equality_constraint!(opt, (theta, g) -> constraint_function(theta, g, R[i:i,:], r[i]), tol)
        end
    end
    xtol_rel!(opt, tol)
    ftol_rel!(opt, tol)
    maxeval!(opt, iterlim)
    (objvalue, xopt, flag) = NLopt.optimize(opt, startval)
    return xopt, objvalue, flag
end
```

    fmincon (generic function with 5 methods)


$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\L}{\mathcal{L}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Parents}{\mathrm{Parents}}
\newcommand{\NonDesc}{\mathrm{NonDesc}}
\newcommand{\I}{\mathcal{I}}
\newcommand{\dsep}{\text{d-sep}}
\newcommand{\Cat}{\mathrm{Categorical}}
\newcommand{\Bin}{\mathrm{Binomial}}
$$

```python
from __future__ import division

# scientific
%matplotlib inline
from matplotlib import pyplot as plt;
import matplotlib as mpl;
import numpy as np;
import sklearn as skl;
import sklearn.datasets;
import sklearn.cluster;
import sklearn.mixture;

# ipython
import IPython;

# python
import os;
import random;

#####################################################

# image processing
import PIL;

# trim and scale images
def trim(im, percent=100):
    print("trim:", percent);
    bg = PIL.Image.new(im.mode, im.size, im.getpixel((0,0)))
    diff = PIL.ImageChops.difference(im, bg)
    diff = PIL.ImageChops.add(diff, diff, 2.0, -100)
    bbox = diff.getbbox()
    if bbox:
        x = im.crop(bbox)
        return x.resize(((x.size[0]*percent)//100,
                         (x.size[1]*percent)//100), PIL.Image.ANTIALIAS);

#####################################################

# daft (rendering PGMs)
import daft;

# set to FALSE to load PGMs from static images
RENDER_PGMS = True;

# decorator for pgm rendering
def pgm_render(pgm_func):
    def render_func(path, percent=100, render=None, *args, **kwargs):
        print("render_func:", percent);
        # render
        render = render if (render is not None) else RENDER_PGMS;

        if render:
            print("rendering");
            # render
            pgm = pgm_func(*args, **kwargs);
            pgm.render();
            pgm.figure.savefig(path, dpi=300);

            # trim
            img = trim(PIL.Image.open(path), percent);
            img.save(path, 'PNG');
        else:
            print("not rendering");

        # error
        if not os.path.isfile(path):
            raise Exception("Error:  Graphical model image %s not found.\
                You may need to set RENDER_PGMS=True." % path);

        # display
        return IPython.display.Image(filename=path);

    return render_func;

######################################################
```

# EECS 445:  Machine Learning

## Lecture 18:  GMMs & Hidden Markov Models

- Instructor:  **Jacob Abernethy**
- Date:  November 15, 2016

*Lecture Exposition:*  Benjamin Bray

# Gaussian Mixture Models

> Review

### Gaussian Mixture Models:  Specification

Recall from earlier the **Gaussian Mixture Model**:
$$
\begin{align}
\theta &= (\vec\pi, \vec\mu, \vec\Sigma) && \text{model parameters} \\
z_n &\sim \Cat[\pi]  && \text{cluster indicators} \\
x_n | z_n, \theta &\sim \mathcal{N}(\mu_{z_n}, \Sigma_{z_n}) && \text{base distribution}
\end{align}
$$

### GMM:  Complete Log Likelihood

The complete data log-likelihood for a single datapoint $(x_n, z_n)$ is
$$
\begin{align}
\log p(x_n, z_n | \theta)
&= \log \prod_{k=1}^K \pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k)^{\mathbb{I}(z_n = k)} \\
&= \sum_{k=1}^K \mathbb{I}(z_n = k) \log \pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k)
\end{align}
$$

### GMM:  Hidden Posterior

The hidden posterior for a single point $(x_n, z_n)$ can be found using Bayes' rule:
$$
\begin{align}
p(z_n = k | x_n, \theta)
&= \frac{P(z_n=k | \theta) p(x_n | z_n=k, \theta)}{p(x_N | \theta)} \\
&= \frac{\pi_k \mathcal{N}(x_n | \mu_k, \Sigma_k)}{\sum_{k'=1}^K \pi_{k'} \mathcal{N}(x_n | \mu_{k'}, \Sigma_{k'})}
\end{align}
$$

### GMM: Visual example

## Gaussian Mixture Model EM in Action

```python
from IPython.display import YouTubeVideo
YouTubeVideo('v-pq8VCQk4M')
```

### GMM:  E-Step

The **expected complete likelihood** is
$$
\begin{align}
E_q[ \log p(\X,Z|\theta)]
&= \sum_{n=1}^N \sum_{k=1}^K
        E_q\big[ \mathbb{I}(z_n = k) \big]
        \log \pi_k \mathcal{N}(x_n \mid \mu_k, \Sigma_k) \\
&= \sum_{n=1}^N \sum_{k=1}^K
        r_{nk} \log \pi_k
 + \sum_{n=1}^N \sum_{k=1}^K
        r_{nk} \log \mathcal{N}(x_n \mid \mu_k, \Sigma_k)
\end{align}
$$

where $r_{nk} \equiv p(z_n = k \mid x_n,\theta_t)$ is the **responsibility** that cluster $k$ takes for datapoint $x_n$ after step $t$.

### GMM:  M-Step

During the M-Step, we optimize the lower bound with respect to $\theta=(\pi,\mu,\Sigma)$.  Verify that the correct updates are
$$
\begin{gather}
\pi_k = \frac{1}{N} \sum_{n=1}^N r_{nk} = \frac{r_k}{N}
\quad
\mu_k = \frac{\sum_{n=1}^N r_{nk} x_n}{r_k}
\\
\Sigma_k = \frac{\sum_{n=1}^N r_{nk} x_n x_n^T}{r_k} - \mu_k \mu_k^T
\end{gather}
$$

where $r_k = \sum_{n=1}^N r_{nk}$ is the *effective* sample size for cluster $k$.

# Markov Models

> Uses material from **[MLAPP]**

### Sequential Data

Some data has intrinsic sequential structure
- **Time Series:** Speech, EKGs, stock market, robot sensors, etc.
- **Spatial Data:** DNA, natural language, etc.

We could treat data points as iid samples
- e.g. "bag of words" assumption for spam classification
- But this is false!  We are ignoring valuable constraints in the data.

### Markov Models

A **Markov chain** is a series of random variables $X_1, \dots, X_N$ satisfying the *Markov property*:
> The future is independent of the past, given the present.
    $$
    p(x_n | x_1, \dots, x_{n-1}) = p(x_n | x_{n-1})
    $$

A chain is **stationary** if the transition probabilities do not change with time.

### Example:  [Random Shakespeare](http://www.cs.princeton.edu/courses/archive/spr05/cos126/assignments/markov.html)

```
DUKE SENIOR	Now, my co-mates and thus bolden'd, man, how now, monsieur Jaques,
	Unclaim'd of his absence, as the holly!
	Though in the slightest for the fashion of his absence, as the only wear.

TOUCHSTONE	I care not for meed!
	This I must woo yours: your request than your father: the time,
	That ever love I broke
	my sword upon some kind of men
	Then, heigh-ho! sing, heigh-ho! sing, heigh-ho! sing, heigh-ho! unto the needless stream;
	'Poor deer,' quoth he,
	'Call me not so keen,
	Because thou the creeping hours of the sun,
	As man's feasts and women merely players:
	Thus we may rest ourselves and neglect the cottage, pasture?

[Exit]
```

### Markov Models:  PGM

```python
@pgm_render
def pgm_markov_chain():
    pgm = daft.PGM([6, 6], origin=[0, 0])

    # Nodes
    pgm.add_node(daft.Node("x1", r"$X_n$", 2, 2.5))
    pgm.add_node(daft.Node("x2", r"$X_2$", 3, 2.5))
    pgm.add_node(daft.Node("ellipsis", r" . . . ", 3.7, 2.5,
                           offset=(0, 0), plot_params={"ec" : "none"}))
    pgm.add_node(daft.Node("ellipsis_end", r"", 3.7, 2.5,
                           offset=(0, 0), plot_params={"ec" : "none"}))
    pgm.add_node(daft.Node("xN", r"$X_N$", 4.5, 2.5))

    # Add in the edges.
    pgm.add_edge("x1", "x2", head_length=0.08)
    pgm.add_edge("x2", "ellipsis", head_length=0.08)
    pgm.add_edge("ellipsis_end", "xN", head_length=0.08)

    return pgm;
```

```python
%%capture
pgm_markov_chain("images/pgm/markov-chain.png")
```

### Markov Models:  Joint Distribution

If a sequence has the Markov property, then the joint distribution factorizes according to
    $$
    p(x_1, \dots, x_N) = p(x_1) \prod_{n=2}^N p(x_n | x_{n-1})
    $$

### Example: Language Modeling

One important application is statistical **language models**.
- Bag-of-words assumption is inadequate
- Assume each word $X_t$ depends on the previous $n$ words

Can model longer-range dependencies

*Zero-Order:* $P(X_t)$

*First-Order:* $P(X_t | X_{t-1})$

*Second-Order:* $P(X_t | X_{t-1}, X_{t-2})$

One we've learned an $n$th order Markov chain, we can use it to generate text!

### Example:  [Garkov Chain](http://joshmillard.com/garkov/)

### Markov Models:  Transition Matrix

Suppose $X_t \in \{1,\dots,K\}$ is discrete.  Then, a stationary chain with $N$ states can be described by a **transition matrix**, $A \in \R^{N \times N}$ where
    $$
    a_{ij} = p(X_t=j \mid X_{t-1}=i)
    $$

is the probability of transitioning from state $i$ to $j$.

> Each row sums to one, $\sum_{j=1}^K A_{ij} = 1$, so $A$ is a *row-stochastic matrix*.

### Markov Models:  Transition Diagram

Transitions between states can be represented as a graph:

> The graph above is *not* a Bayesian network!
> Instead, the nodes in this graph represent states that each $X_i$ can take.

(Figure 17.4b from **[MLAPP]**)

### Markov Models:  State Vectors

Consider a row vector $x_t \in \R^{1 \times K}$ with entries $x_{tj} = p(X_t = j)$.  Then,

$$
    \begin{align}
    p(X_t = j)
    &= \sum_{i=1}^K p(X_t = j \mid X_{t-1} = i) p(X_{t-1} = i) \\
    &= \sum_{i=1}^K A_{ij} x_{t-1,i} \\
    \end{align}
$$

Therefore, we conclude $x_t = x_{t-1} A$.

(Note $\sum_{j=1}^K x_{tj} = 1$.)

### Markov Models:  Matrix Powers

Since $x_t = x_{t-1} A$, this suggests that in general,
    $$
    x_{t} = x_{t-1} A = x_{t-2} A^2 = \cdots x_{0} A^t
    $$
If we know the initial state probabilities $x_0$, we can find the probabilities of landing in any state at time $t > 0$.

### Example:  Weather

Suppose the weather is either $R=\text{Rainy}$ or $S=\text{Sunny}$,
    $$
    A = \begin{bmatrix}
        0.9 & 0.1 \\
        0.5 & 0.5
    \end{bmatrix}
    $$

(Taken from [Wikipedia](https://en.wikipedia.org/wiki/Examples_of_Markov_chains#A_very_simple_weather_model))

### Example:  Weather

Suppose today is sunny, $x_0 = \begin{bmatrix} 1 & 0 \end{bmatrix}$.  We can predict tomorrow's weather,
    $$
    x_1 = x_0 A = \begin{bmatrix} 0.9 & 0.1 \end{bmatrix}
    $$
The weather over the next several days will be
    $$
    \begin{align}
    x_2 &= x_1 A = \begin{bmatrix} 0.86 & 0.14 \end{bmatrix} \\
    x_3 &= x_2 A = \begin{bmatrix} 0.844 & 0.156 \end{bmatrix} \\
    x_4 &= x_3 A = \begin{bmatrix} 0.8376 & 0.1624 \end{bmatrix}
    \end{align}
    $$

> **Question:** What happens to $x_0 A^n$ as $n \rightarrow \infty$?

### Markov Chains:  Stationary Distribution

If we ever reach a stage $x$ where
    $$
    x = xA
    $$
then we have reached the **stationary distribution** of the chain.
- To find $x=v^T$, solve the eigenvalue problem $A^T v = v$
- Under certain conditions, the limiting distribution $\lim_{n \rightarrow \infty} x_0 A^n = x$
- Stationary distribution $x$ does not depend on the starting state $x_0$

## Break time!

# Hidden Markov Models

> Uses material from **[MLAPP]**

### Hidden Markov Models

Noisy observations $X_k$ generated from *discrete* hidden Markov chain $Z_k$.
$$
P(\vec{X}, \vec{Z}) = P(Z_1) P(X_1 \mid Z_1) \prod_{k=2}^T P(Z_k \mid Z_{k-1}) P(X_k \mid Z_k)
$$

```python
@pgm_render
def pgm_hmm():
    pgm = daft.PGM([7, 7], origin=[0, 0])

    # Nodes
    pgm.add_node(daft.Node("Z1", r"$Z_1$", 1, 3.5))
    pgm.add_node(daft.Node("Z2", r"$Z_2$", 2, 3.5))
    pgm.add_node(daft.Node("Z3", r"$\dots$", 3, 3.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("Z4", r"$Z_T$", 4, 3.5))

    pgm.add_node(daft.Node("x1", r"$X_1$", 1, 2.5, observed=True))
    pgm.add_node(daft.Node("x2", r"$X_2$", 2, 2.5, observed=True))
    pgm.add_node(daft.Node("x3", r"$\dots$", 3, 2.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("x4", r"$X_T$", 4, 2.5, observed=True))

    # Add in the edges.
    pgm.add_edge("Z1", "Z2", head_length=0.08)
    pgm.add_edge("Z2", "Z3", head_length=0.08)
    pgm.add_edge("Z3", "Z4", head_length=0.08)

    pgm.add_edge("Z1", "x1", head_length=0.08)
    pgm.add_edge("Z2", "x2", head_length=0.08)
    pgm.add_edge("Z4", "x4", head_length=0.08)

    return pgm;
```

```python
%%capture
pgm_hmm("images/pgm/hmm.png")
```

### HMM:  Parameters

For a Hidden Markov Model with $N$ hidden states and $M$ observed states, there are three *row-stochastic* parameters $\theta=(A,B,\pi)$,
- Transition matrix  $A \in \R^{N \times N}$
    $$
    A_{ij} = p(Z_t = j | Z_{t-1} = i)
    $$
- Emission matrix $B \in \R^{N \times M}$
    $$
    B_{jk} = p(X_t = k | Z_t = j)
    $$
- Initial distribution $\pi \in \R^N$,
    $$
    \pi_j = p(Z_1 = j)
    $$

### HMM:  Filtering Problem

**Filtering** means to compute the current *belief state* $p(z_t | x_1, \dots, x_t,\theta)$.
    $$
    p(z_t | x_1,\dots,x_t) = \frac{p(x_1,\dots,x_t,z_t)}{p(x_1,\dots,x_t)}
    $$
- Given observations $x_{1:t}$ so far, infer $z_t$.
- *Example*:  Estimate robot position given previous sensor readings.

> Solved by the **forward algorithm**.

### HMM:  Filtering Problem

As an example of filtering, consider **robot localization**, in which we
- estimate the current location $z_t$
- given **noisy** sensor readings $x_1,\dots,x_t$.

(Figure 18.3a from **[MLAPP]**)

## How do we infer values of hidden variables?

- One of the most challenging part of HMMs is to try to "predict" what are the values of the hidden variables $z_t$, having observed all the $x_1, \ldots, x_T$.
- Computing $p(z_t \mid \X)$ is known on *smoothing*. More on this soon.
- But it turns out that this probability can be computed from two other quantities:
    - $p(x_1,\dots,x_t,z_t)$, which we are going to label $\alpha_t(z_t)$
    - $p(x_{t+1},\dots,x_{T} | z_t)$, which we are going to label $\beta_t(z_t)$

### HMM:  Forward Algorithm

The **forward algorithm** computes $\alpha_t(z_t) \equiv p(x_1,\dots,x_t,z_t)$.

> Looks kinda hard to compute this probability, there are a lot of variables! Maybe it can be computed recursively? **Yes!**

$$
    \begin{align}
    \alpha_t(z_t)
    &= \sum_{z_{t-1}} p(x_1, \dots, x_t, z_{t-1}, z_t) \\
    &= \sum_{z_{t-1}} p(x_1, \dots, x_{t-1}, z_{t-1}) p(z_t | z_{t-1}) p(x_t | z_t) \\
    &= p(x_t | z_t) \sum_{z_{t-1}} \alpha_{t-1}(z_{t-1}) p(z_t | z_{t-1})\\
    &= B_{z_t,x_t} \sum_{z_{t-1}} \alpha_{t-1}(z_{t-1}) A_{z_{t-1}, z_t}
    \end{align}
$$

> Recursion starts from the *front* of the chain.
> (Suggestion: you need to work out the above recursion on your own, it's a really important exercise)

### HMM:  Backward Algorithm

The **backward algorithm** computes $\beta_t(z_t) \equiv p(x_{t+1},\dots,x_{T} | z_t)$,
> Again, maybe we can try to compute this recursively?

$$
    \begin{align}
    \beta(z_t)
    &= \sum_{z_{t+1}} p(x_{t+1},\dots,x_{T},z_{t+1} | z_t) \\
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}, \dots, x_T |z_{t+1}, z_t)\\
        (\text{since } x_{t+1:T} \perp z_t \mid z_{t+1}) \quad
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}, \dots, x_T |z_{t+1})\\
        (\text{since } x_{t+2:T} \perp x_{t+1} \mid z_{t+1}) \quad
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}|z_{t+1})  p(x_{t+2}, \dots, x_T |z_{t+1})\\
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}|z_{t+1}) \beta_{t+1}(z_{t+1}) \\
    &= \sum_{z_{t+1}} A_{z_t, z_{t+1}} B_{z_{t+1}, x_{t+1}} \beta_{t+1}(z_{t+1})
    \end{align}
$$

> Recursion starts from the *back* of the chain.
> (Suggestion: you need to work out the above recursion on your own, it's a really important exercise)

### HMM:  Smoothing Problem

Compute $p(z_t | \X)$ offline, given all observations.
- Retroactively infer $z_t$.  (Hindsight!)

We can break the chain into two parts, the *past* and *future*:
    $$
    \begin{align}
    p(z_t | \X)
    &= p(x_{1:t}, z_t, x_{t+1:T}) \frac{1}{p(\X)} \\
    &= p(x_{1:t}, z_t) p(x_{t+1:T} | x_{1:t}, z_t)\frac{1}{p(\X)} \\
    &= p(x_{1:t}, z_t) p(x_{t+1:T} | z_t)\frac{1}{p(\X)}
    \end{align}
    $$
**Exercise**: How did we get that last line?

### HMM:  Smoothing Problem

Overall, the **smoothing problem** is to compute
$$
\begin{align*}
    \gamma_t(j) \equiv p(z_t = j \mid \X)
    & = \frac{p(\X \mid z_t = j)p(z_t = j)}{p(\X)} \\
    & = \frac{p(\X_{1:t} \mid z_t = j) p(\X_{t+1:T} | z_t=j) p(z_t = j)}{p(\X)} \\
    & = \frac{\alpha_t(j) \beta_t(j)}{p(\X)} = \frac{\alpha_t(j) \beta_t(j)}{\sum_k \alpha_t(k)\beta_t(k)}
\end{align*}
$$
It is an easy exercise to check that, for any $t$, $p(\X) = \sum_k \alpha_t(k)\beta_t(k)$.

We solve this via the **forward-backward algorithm**, where
- $\alpha_t(z_t) \equiv p(x_1,\dots,x_t,z_t)$ is found with the **forward algorithm**
- $\beta_t(z_t) \equiv p(x_{t+1},\dots,x_{T} | z_t)$ is found with the **backward algorithm**

### HMM:  Decoding Problem

**Decoding** computes the most probable state sequence, given observations.
    $$
    \vec{z}^* = \arg\max_{z_1,\dots,z_T} p(z_1,\dots,z_T | x_1, \dots, x_T, \theta)
    $$

> The decoding problem is solved by the **Viterbi algorithm**, which uses dynamic programming.  See **[PRML]** or **[MLAPP]** for more details.

### Viterbi Algorithm

* Viterbi is another recursive procedure for computing the most likely sequence of hidden states.
* We define $V_t(z_t)$ to be the probability of the most likely sequence of states up to time $t$ *that ended in state* $z_t$, given observed data
* These $V(z_t)$ values satisfies the recursion:

\begin{align}
V_1(j) & = p(x_1 \mid z_1=j) = B_{x_1, j}\\
V_{t+1}(j) & = p(x_{t+1} \mid z_{t+1} = j) \max_{k} \bigg\{ V_{t}(k) p(z_{t+1} = j \mid z_t = k) \bigg\}\\
& = B_{x_{t+1},j} \max_{k} \bigg\{ V_{t}(k) A_{j,k} \bigg\}
\end{align}

* Can be computed very quickly!

### HMM:  Part-of-Speech Tagging

In English, some words can have multiple parts of speech.  For instance,

- Business is going **well** (*Adverb*).
- All is **well** with us (*Adjective*).
- **Well**, who would have thought he could do it? (*Interjection*)
- The **well** was drilled fifty meters deep. (*Noun*)
- Tears **well** up in my eyes. (*Verb*)

(Example taken from [here](http://english.stackexchange.com/questions/46277/what-word-can-fulfill-the-most-parts-of-speech).)

### HMM:  Part-of-Speech Tagging

We can use a Hidden Markov Model to **disambiguate** the part of speech using context clues!
- Hidden states $z_t$ are parts-of-speech
- Observed states $x_t$ are words

Certain sequences of POS tags are unlikely.  This allows us to infer the correct tags!

### HMM:  Learning Problem

It is usually necessary to **learn** the model parameters $\theta = (A,B,\pi)$ from data.
- Given observations $\mathcal{X} = \{ x_1, \dots, x_T \}$
- Given model dimensions $N$ and $M$
- Find parameters that best fit the data

> The learning problem is solved by the **Baum-Welch algorithm**, a special case of expectation maximization.

### Recall:  Expectation-Maximization

**E-Step:** Write down an expression for
    $$
    Q(\theta_t, \theta) = E_{q}[ \log p(\X,Z | \theta) ] \quad \quad \text{ where } q = q(\cdot | \theta_t)
    $$

**M-Step:** Maximize the auxiliary function,
    $$
    \theta_{t+1}
    = \arg\max_\theta Q(\theta_t, \theta)
    $$

> Recall $q_t(Z) = p(Z|\X,\theta_t)$

### HMM:  Complete-Data Log-Likelihood

The joint likelihood of the hidden and observed states is

$$
\begin{align}
\log p(x_{1:T}, z_{1:T} | \theta)
&= \log \left[ p(z_1|\pi)
    \prod_{t=2}^T p(z_t | z_{t-1}, A)
    \prod_{t=1}^T p(x_t | z_t, B) \right] \\
&=  \log p(z_1|\pi)
    + \sum_{t=2}^T \log p(z_t | z_{t-1}, A) \\
    & \qquad \qquad + \sum_{t=1}^T \log p(x_t | z_t, B)
\end{align}
$$

### HMM:  Complete-Data Log-Likelihood

Each term of the complete-data log-likelihood is:

$$
\begin{align}
\log p(z_1 | \pi)
    &= \sum_{j=1}^N \mathbb{I}(z_t=j) \log \pi_j \\
\log p(z_t | z_{t-1}, A)
    &= \sum_{i=1}^N \sum_{j=1}^N
        \mathbb{I}(z_{t-1}=i)\mathbb{I}(z_t=j) \log A_{ij} \\
\log p(x_t | z_t, B)
    &= \sum_{j=1}^N
        \mathbb{I}(z_t=j) \log B_{j,x_t}
\end{align}
$$

### HMM:  Expected Complete Likelihood

The expected complete likelihood $Q(\theta_t, \theta)$ is

$$
\begin{align}
Q(\theta_t,\theta)
&= E_q[ \log p(\X,Z|\theta) ] \\
&= E_q[ \log p(z_1|\pi) ]
    + E_q\left[ \sum_{t=2}^T \log p(z_t | z_{t-1}, A) \right]\\
    &\qquad + E_q\left[ \sum_{t=1}^T \log p(x_t | z_t, B) \right]
\end{align}
$$

### HMM:  Expected Complete Likelihood

Fixing $t > 1$, and taking expectations with respect to $q(Z) = p(Z|\X,\theta)$,

$$
\begin{align*}
E_q[\log p(z_1 | \pi)]
    &= \sum_{j=1}^N q(z_1=j) \log \pi_j \\
E_q[\log p(z_t | z_{t-1}, A)]
    &= \sum_{i=1}^N \sum_{j=1}^N
        q(z_{t-1}=i, z_t=j) \log A_{ij} \\
E_q[\log p(x_t | z_t, B)]
    &= \sum_{j=1}^N
        q(z_t=j) \log B_{j, x_t}
\end{align*}
$$

### HMM:  Baum-Welch

- The **E-Step** consists of computing the $q$ terms from the previous slide, which can all be computed using the **forward-backward algorithm**!

- The **M-Step** consists of normalizing the expected transition and emission counts
    - Similar to MLE for complete data
    - Requires some careful calculations. See details on next slide.

You must compute the key quantities for $q()$ using the forward-backward algorithm. In the E step you can assume the parameters $\theta = (A, B, \pi)$, so we'll drop dependence on $\theta$.
1. First note that $q(z_t=j) = p(z_t = j \mid \X, \theta) = \gamma_t(j)$ which we computed above.
1. We then compute $q(z_{t-1}=i, z_t=j) = p(z_{t-1}=i, z_t=j | \X, \theta)$.
$$
\begin{align*}
    p(z_{t-1}, z_t | \X, \theta)
    & = \frac{p(\X \mid z_t, z_{t-1})p(z_t, z_{t-1})}{p(\X)} \\
    & = \frac{p(\X_{1:t-1} \mid z_t, z_{t-1}) p(\X_{t:T} | z_t, z_{t-1}) p(z_t, z_{t-1})}{p(\X)} \\
    & = \frac{p(\X_{1:t-1} \mid z_{t-1}) p(\X_{t:T} | z_t) p(z_t, z_{t-1})}{p(\X)} \\
    & = \frac{p(\X_{1:t-1}, z_{t-1}) p(x_t \mid z_t) p(\X_{t+1:T} | z_t) p(z_t, z_{t-1})}{p(z_{t-1})p(\X)} \\
    & = \frac{\alpha_{t-1}(z_{t-1}) \beta_t(z_t) p(z_t \mid z_{t-1}) p(x_t \mid z_t)}{p(\X)} \\
    & = \frac{\alpha_{t-1}(z_{t-1}) \beta_t(z_t) A_{z_{t-1}, z_t} B_{z_t, x_t}}{\sum_k \alpha_t(k)\beta_t(k)}
\end{align*}
$$


[Sascha Spors](https://orcid.org/0000-0001-7225-9992),
Professorship Signal Theory and Digital Signal Processing,
[Institute of Communications Engineering (INT)](https://www.int.uni-rostock.de/),
Faculty of Computer Science and Electrical Engineering (IEF),
[University of Rostock, Germany](https://www.uni-rostock.de/en/)

# Tutorial Signals and Systems (Signal- und Systemtheorie)

Summer Semester 2021 (Bachelor Course #24015)

- lecture: https://github.com/spatialaudio/signals-and-systems-lecture
- tutorial: https://github.com/spatialaudio/signals-and-systems-exercises

WIP...
The project is currently under heavy development while adding new material for the summer semester 2021

Feel free to contact lecturer [frank.schultz@uni-rostock.de](https://orcid.org/0000-0002-3010-0294)
## Übung / Exercise 5 Bode Plot of LTI Systems

### References

* Norbert Fliege (1991): "*Systemtheorie*", Teubner, Stuttgart (GER), cf. chapter 4.3.5

* Alan V. Oppenheim, Alan S. Willsky with S. Hamid Nawab (1997): "*Signals & Systems*", Prentice Hall, Upper Saddle River NJ (USA), 2nd ed., cf. chapter 6

* Bernd Girod, Rudolf Rabenstein, Alexander Stenger (2001): "*Signals and Systems*", Wiley, Chichester (UK), cf. chapter 10

* Bernd Girod, Rudolf Rabenstein, Alexander Stenger (2005/2007): "*Einführung in die Systemtheorie*", Teubner, Wiesbaden (GER), 3rd/4th ed., cf. chapter 10

Let's import required packages and define some helping routines for plotting in the following.

```python
import matplotlib.pyplot as plt
import numpy as np
import scipy.signal as signal
```

```python
def lti_bode_plot(sys, txt):
    w, mag, phase = sys.bode(np.logspace(-2, 2, 2**8))

    fig = plt.figure(figsize=(6, 5), tight_layout=True)
    plt.subplot(2, 1, 1)
    plt.title(txt)
    plt.semilogx(w, mag, 'C0', linewidth=3)
    plt.grid(True)
    plt.xlabel('$\omega$ / (rad/s)')
    plt.ylabel('Level in dB')
    plt.xlim(w[0], w[-1])

    plt.subplot(2, 1, 2)
    plt.semilogx(w, phase, 'C0', linewidth=3)
    plt.grid(True)
    plt.xlabel('$\omega$ / (rad/s)')
    plt.ylabel('Phase in deg')
    plt.xlim(w[0], w[-1])
```

```python
# plot LTI system characteristics for Laplace domain
def plot_LTIs(sys):
    z = np.squeeze(sys.zeros)
    p = np.squeeze(sys.poles)
    k = sys.gain

    w = np.logspace(-2, 2, 2**8)
    w, H = signal.freqs_zpk(z, p, k, w)
    th, h = signal.impulse(sys)
    the, he = signal.step(sys)

    plt.figure(figsize=(7, 9), tight_layout=True)

    # pole / zero plot
    plt.subplot(325)

    # clear poles / zeros that compensate each other, TBD: numerical robust,
    # works for didactical purpose
    sz_tmp = z
    sp_tmp = p
    zp = np.array([1])
    while zp.size != 0:
        zp, z_ind, p_ind = np.intersect1d(sz_tmp, sp_tmp, return_indices=True)
        sz_tmp = np.delete(sz_tmp, z_ind)
        sp_tmp = np.delete(sp_tmp, p_ind)
    z = sz_tmp
    p = sp_tmp

    zu, zc = np.unique(z, return_counts=True)  # find and count unique zeros
    for zui, zci in zip(zu, zc):  # plot them individually
        plt.plot(np.real(zui), np.imag(zui), ms=10,
                 color='C0', marker='o', fillstyle='none')
        if zci > 1:  # if multiple zeros exist then indicate the count
            plt.text(np.real(zui), np.imag(zui), zci, color='C0',
                     fontsize=14, fontweight='bold')

    pu, pc = np.unique(p, return_counts=True)  # find and count unique poles
    for pui, pci in zip(pu, pc):  # plot them individually
        plt.plot(np.real(pui), np.imag(pui), ms=10,
                 color='C3', marker='x')
        if pci > 1:  # if multiple poles exist then indicate the count
            plt.text(np.real(pui), np.imag(pui), pci, color='C3',
                     fontsize=14, fontweight='bold')

    plt.axis("equal")
    plt.xlabel(r'$\Re\{s\}$')
    plt.ylabel(r'$\Im\{s\}$')
    plt.grid(True)
    plt.title("Pole/Zero/Gain Map, gain=%f" % k)

    # Nyquist plot
    plt.subplot(326)
    plt.plot(H.real, H.imag, "C0", label="$\omega>0$")
    plt.plot(H.real, -H.imag, "C1", label="$\omega<0$")
    plt.plot(H.real[0], H.imag[0], marker='$w=0$', markersize=25, color="C0")
    plt.axis("equal")
    plt.legend()
    plt.xlabel(r'$\Re\{H(s)\}$')
    plt.ylabel(r'$\Im\{H(s)\}$')
    plt.grid(True)
    plt.title("Nyquist Plot")

    # magnitude response
    plt.subplot(321)
    plt.semilogx(w, 20*np.log10(np.abs(H)))
    plt.xlabel('$\omega$ / (rad/s)')
    plt.ylabel(r'$A$ / dB')
    plt.grid(True)
    plt.title("Level")
    plt.xlim((w[0], w[-1]))
    #plt.ylim((-60, 6))
    #plt.yticks(np.arange(-60, +12, 6))

    # phase response
    plt.subplot(323)
    plt.semilogx(w, np.unwrap(np.angle(H))*180/np.pi)
    plt.xlabel('$\omega$ / (rad/s)')
    plt.ylabel(r'$\phi$ / deg')
    plt.grid(True)
    plt.title("Phase")
    plt.xlim((w[0], w[-1]))
    plt.ylim((-180, +180))
    plt.yticks(np.arange(-180, +180+45, 45))

    # impulse response
    plt.subplot(322)
    plt.plot(th, h)
    plt.xlabel('t / s')
    plt.ylabel('h(t)')
    plt.grid(True)
    plt.title("Impulse Response")

    # step response
    plt.subplot(324)
    plt.plot(the, he)
    plt.xlabel('t / s')
    plt.ylabel('h$_\epsilon$(t)')
    plt.grid(True)
    plt.title("Step Response")

# plot LTI system characteristics for z-domain

def plot_LTIz(b, a):
    TBD = true
```

# Example: Unity Gain

```python
sz = []
sp = []
H0 = +1
txt = 'Unity Gain'
sys = signal.lti(sz, sp, H0)
lti_bode_plot(sys, txt)
```

# Example: Gain and Polarity

```python
sz = []
sp = []
H0 = -10
txt = '20 dB Gain with Inverted Polarity'
sys = signal.lti(sz, sp, H0)
lti_bode_plot(sys, txt)
```

# Example: Poles / Zeros in Origin

```python
sz = 0, 0  # note: more zeros than poles is not a causal system!
sp = 0,
H0 = 1
sys = signal.lti(sz, sp, H0)
txt = str(len(sz)) + ' Zeros / ' + str(len(sp)) + ' Poles in Origin'
txt1 = (': ' + str((len(sz)-len(sp))*20) + ' dB / decade')
lti_bode_plot(sys, txt+txt1)
```

```python
sz = 0,
sp = 0, 0, 0
H0 = 1
sys = signal.lti(sz, sp, H0)
txt = str(len(sz)) + ' Zeros / ' + str(len(sp)) + ' Poles in Origin'
txt1 = (': ' + str((len(sz)-len(sp))*20) + ' dB / decade')
lti_bode_plot(sys, txt+txt1)
```

# Example: Single Real Pole, PT1

```python
sz = []
sp = -1
H0 = 1
sys = signal.lti(sz, sp, H0)
txt = 'Single Real Pole, decreasing slope, -20 dB / decade'
lti_bode_plot(sys, txt)
```

# Example: Single Real Zero

```python
sz = -1
sp = []
H0 = 1
sys = signal.lti(sz, sp, H0)
txt = 'Single Real Zero, increasing slope, + 20 dB / decade'
lti_bode_plot(sys, txt)
```

# Example: Complex Conjugate Zero Pair

```python
sz = -3/4-1j, -3/4+1j
sp = []
H0 = 16/25
sys = signal.lti(sz, sp, H0)
txt = 'Conjugate Complex Zero'
lti_bode_plot(sys, txt)
```

# Examples: Complex Conjugate Pole Pair, PT2

```python
sz = []
sp = -3/4-1j, -3/4+1j
H0 = 25/16
sys = signal.lti(sz, sp, H0)
txt = 'Conjugate Complex Pole, -3/4$\pm$1j'
lti_bode_plot(sys, txt)
```

```python
sz = []
sp = -1/2-1j, -1/2+1j
H0 = 5/4
sys = signal.lti(sz, sp, H0)
txt = 'Conjugate Complex Pole, -1/2$\pm$1j'
lti_bode_plot(sys, txt)
```

```python
sz = []
sp = -1/4-1j, -1/4+1j
H0 = 17/16
sys = signal.lti(sz, sp, H0)
txt = 'Conjugate Complex Pole, -1/4$\pm$1j'
lti_bode_plot(sys, txt)
```

```python
sz = []
sp = -1/8-1j, -1/8+1j
H0 = 65/64
sys = signal.lti(sz, sp, H0)
txt = 'Conjugate Complex Pole, -1/8$\pm$1j'
lti_bode_plot(sys, txt)
```

# Example: Integrator, I

```python
sz = []
sp = 0
H0 = 1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_i_element.pdf')
```

# Example: Lowpass 1st Order, PT1

```python
sz = []
sp = -1
H0 = np.abs(sp)
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_pt1_element.pdf')
```

# Example: Highpass 1st Order, DT1

```python
sz = 0
sp = -1
H0 = 1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_dt1_element.pdf')
```

# Example: Allpass 1st Order

```python
sz = +1
sp = -1
H0 = -1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_ap1_element.pdf')
```

# Example: Lowpass 2nd Order, PT2

```python
sz = []
sp = 1*np.exp(+1j*3*np.pi/4), 1*np.exp(-1j*3*np.pi/4)
H0 = 1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_pt2_element.pdf')
```

# Example: Highpass 2nd Order, DT2

```python
sz = 0, 0
sp = 1*np.exp(+1j*3*np.pi/4), 1*np.exp(-1j*3*np.pi/4)
H0 = 1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_dt2_element.pdf')
```

# Example: Bandpass 2nd Order

```python
sz = 0
# sp = 1*np.exp(+1j*2.1*np.pi/4), 1*np.exp(-1j*2.1*np.pi/4)  # high Q
sp = 1*np.exp(+1j*3*np.pi/4), 1*np.exp(-1j*3*np.pi/4)
# sp = 1*np.exp(+1j*3.9*np.pi/4), 1*np.exp(-1j*3.9*np.pi/4)  # low Q
H0 = np.sqrt(2)
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_bp2_element.pdf')
```

# Example: Bandstop 2nd Order

```python
sz = -1j, +1j
# sp = 1*np.exp(+1j*2.1*np.pi/4), 1*np.exp(-1j*2.1*np.pi/4)  # high Q
sp = 1*np.exp(+1j*3*np.pi/4), 1*np.exp(-1j*3*np.pi/4)
# sp = 1*np.exp(+1j*3.9*np.pi/4), 1*np.exp(-1j*3.9*np.pi/4)  # low Q
H0 = 1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_bs2_element.pdf')
```

# Example: Allpass 2nd Order

```python
sz = 1*np.exp(+1j*1*np.pi/4), 1*np.exp(-1j*1*np.pi/4)
sp = 1*np.exp(+1j*3*np.pi/4), 1*np.exp(-1j*3*np.pi/4)
H0 = -1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
plt.savefig('bodeplot_examples_ap2_element.pdf')
```

# Example: PIT1

```python
sz = -1
sp = 0, -10
H0 = 1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
```

# Example: PIDT

```python
sz = -2, -5/2
sp = 0, -10
H0 = 1
sys = signal.lti(sz, sp, H0)
plot_LTIs(sys)
```

# Example: Lowpass 2nd Order, PT2

The 2nd order lopwass
\begin{align}
H_\mathrm{Low}(s) = \frac{1}{\frac{16}{25}s^2+\frac{24}{25}s +1} = [\frac{16}{25}s^2+\frac{24}{25}s +1]^{-1}
\end{align}
is to be characterized by the pole/zero map, the Nyquist plot, the bode plot, the impulse response and the step response.

```python
w0 = 5/4
D = 3/5
Q = 1/(2*D)
print("D = %4.3f, Q = %4.3f" % (D, Q))

# these are all the same cases:
A = (1/w0**2, 2*D/w0, 1)
print("A = ", A)
A = (1/w0**2, 1/(Q*w0), 1)
print("A = ", A)
A = (16/25, 24/25, 1)
print("A = ", A)

B = (0, 0, 1)

z, p, k = signal.tf2zpk(B, A)
sys = signal.lti(z, p, k)
plot_LTIs(sys)
```

# Example: Lowpass to Highpass Transform
The Laplace transfer function of a lowpass is transformed to a highpass by exchanging
\begin{align}
s \rightarrow \frac{1}{s}
\end{align}

The 2nd order lopwass $H_\mathrm{Low}(s) = [\frac{16}{25}s^2+\frac{24}{25}s +1]^{-1}$ yields the
following 2nd order highpass filter. It is to be characterized by the pole/zero map, the Nyquist plot, the bode plot, the impulse response and the step response.

```python
# highpass 2nd order, from lowpass 2nd order with s -> 1/s
A = (1, 24/25, 16/25)
B = (1, 0, 0)
z, p, k = signal.tf2zpk(B, A)
sys = signal.lti(z, p, k)
plot_LTIs(sys)
```

# Example: Lowpass to Bandpass Transform

The Laplace transfer function of a lowpass is transformed to a bandpass by exchanging
\begin{align}
s \rightarrow s + \frac{1}{s}
\end{align}
The 2nd order lopwass $H_\mathrm{Low}(s) = [\frac{16}{25}s^2+\frac{24}{25}s +1]^{-1}$ yields to the
following 4th order bandpass filter. It is to be characterized by the pole/zero map, the Nyquist plot, the bode plot, the impulse response and the step response.

```python
# bandpass 4th order, from lowpass 2nd order with s -> s + 1/s
A = (16, 24, 57, 24, 16)
B = (0, 0, 25, 0, 0)
z, p, k = signal.tf2zpk(B, A)
sys = signal.lti(z, p, k)
plot_LTIs(sys)
```

# Example: Lowpass to Bandstop Transform

The Laplace transfer function of a lowpass is transformed to a bandstop filter by exchanging
\begin{align}
s \rightarrow \frac{1}{s + \frac{1}{s}}
\end{align}

The 2nd order lopwass $H_\mathrm{Low}(s) = [\frac{16}{25}s^2+\frac{24}{25}s +1]^{-1}$ yields to the
following 4th order bandstop filter. It is to be characterized by the pole/zero map, the Nyquist plot, the bode plot, the impulse response and the step response.

```python
# bandstop 4th order, from lowpass 2nd order with s -> 1 / (s + 1/s)
A = (25, 24, 66, 24, 25)
B = (25, 0, 50, 0, 25)
z, p, k = signal.tf2zpk(B, A)
sys = signal.lti(z, p, k)
plot_LTIs(sys)
```

# Example: Bandpass from Real Poles
* zero in origin $s_0=0$
* pole at $s_{\infty,1}=-0.1$
* pole at $s_{\infty,2}=-10$
* $H_0$ = 10

\begin{align}
H(s) = H_0\frac{s-s_{0,1}}{(s-s_{\infty,1})(s-s_{\infty,2})} = 10\frac{(s-0)}{(s+0.1)(s+10)}
=\frac{100 s}{10 s^2 + 101 s + 10}
\end{align}

```python
if True:
    sz = 0
    sp = -0.1, -10
    H0 = 10
    sys = signal.lti(sz, sp, H0)
else:
    B = (0, 100, 0)
    A = (10, 101, 10)
    sys = signal.lti(B, A)
txt = 'Bandpass'
lti_bode_plot(sys, txt)
```

# Example: Highpass with Slight Resonance

\begin{align}H(s) = \frac{s^2+10 s}{s^2+\sqrt{2} s +1}\end{align}

```python
sz = 0, 10
sp = (-1+1j)/np.sqrt(2), (-1-1j)/np.sqrt(2)
H0 = 1
sys = signal.lti(sz, sp, H0)
txt = 'Highpass with Slight Resonance'
lti_bode_plot(sys, txt)
```

## Copyright

This tutorial is provided as Open Educational Resource (OER), to be found at
https://github.com/spatialaudio/signals-and-systems-exercises
accompanying the OER lecture
https://github.com/spatialaudio/signals-and-systems-lecture.
Both are licensed under a) the Creative Commons Attribution 4.0 International
License for text and graphics and b) the MIT License for source code.
Please attribute material from the tutorial as *Frank Schultz,
Continuous- and Discrete-Time Signals and Systems - A Tutorial Featuring
Computational Examples, University of Rostock* with
``main file, github URL, commit number and/or version tag, year``.


Table of Contents

1  Pytorch Introduction

1.1  Linear Regression

1.2  Linear Regression Version 2

1.3  Logistic Regression

1.4  Recurrent Neural Network (RNN)

1.4.1  Vanilla RNN

1.4.2  LSTM

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(css_style='custom2.css', plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

%watermark -a 'Ethen' -d -t -v -p torch,numpy,matplotlib
```

    Ethen 2020-09-26 21:24:32

    CPython 3.6.4
    IPython 7.15.0

    torch 1.6.0
    numpy 1.18.5
    matplotlib 3.1.0

# Pytorch Introduction

```bash
# installation on a mac
# for more information on installation refer to
# the following link:
# http://pytorch.org/
conda install pytorch torchvision -c pytorch
```

At its core, PyTorch provides two main features:

- An n-dimensional Tensor, similar to numpy array but can run on GPUs. PyTorch provides many functions for operating on these Tensors, thus it can be used as a general purpose scientific computing tool.
- Automatic differentiation for building and training neural networks.

Let's dive in by looking at some examples:

## Linear Regression

```python
# make up some trainig data and specify the type to be float, i.e. np.float32
# We DO not recommend double, i.e. np.float64, especially on the GPU. GPUs have bad
# double precision performance since they are optimized for float32
X_train = np.asarray([3.3, 4.4, 5.5, 6.71, 6.93, 4.168, 9.779, 6.182, 7.59,
                      2.167, 7.042, 10.791, 5.313, 7.997, 5.654, 9.27, 3.1], dtype = np.float32)
X_train = X_train.reshape(-1, 1)
y_train = np.asarray([1.7, 2.76, 2.09, 3.19, 1.694, 1.573, 3.366, 2.596, 2.53,
                      1.221, 2.827, 3.465, 1.65, 2.904, 2.42, 2.94, 1.3], dtype = np.float32)
y_train = y_train.reshape(-1, 1)

# Convert numpy array to Pytorch Tensors
X = torch.FloatTensor(X_train)
y = torch.FloatTensor(y_train)
```

Here we start defining the linear regression model, recall that in linear regression, we are optimizing for the squared loss.

\begin{align}
L = \frac{1}{2}(y-(Xw + b))^2
\end{align}

```python
# with linear regression, we apply a linear transformation
# to the incoming data, i.e. y = Xw + b, here we only have a 1
# dimensional data, thus the feature size will be 1
model = nn.Linear(in_features=1, out_features=1)

# although we can write our own loss function, the nn module
# also contains definitions of popular loss functions; here
# we use the MSELoss, a.k.a the L2 loss, and size_average parameter
# simply divides it with the number of examples
criterion = nn.MSELoss(size_average=True)

# Then we use the optim module to define an Optimizer that will update the weights of
# the model for us. Here we will use SGD; but it contains many other
# optimization algorithms. The first argument to the SGD constructor tells the
# optimizer the parameters that it should update
learning_rate = 0.01
optimizer = optim.SGD(model.parameters(), lr=learning_rate)
```

```python
# start the optimization process
n_epochs = 100
for _ in range(n_epochs):
    # torch accumulates the gradients, thus before running new things
    # use the optimizer object to zero all of the gradients for the
    # variables it will update (which are the learnable weights of the model),
    # think in terms of refreshing the gradients before doing the another round of update
    optimizer.zero_grad()

    # forward pass: compute predicted y by passing X to the model
    output = model(X)

    # compute the loss function
    loss = criterion(output, y)

    # backward pass: compute gradient of the loss with respect to model parameters
    loss.backward()

    # call the step function on an Optimizer makes an update to its parameters
    optimizer.step()
```

```python
# plot the data and the fitted line to confirm the result

# change default style figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 14

# convert a torch FloatTensor back to a numpy ndarray
# here, we also call .detach to detach the result from the computation history,
# to prevent future computations on it from being tracked
y_pred = model(X).detach().numpy()
plt.plot(X_train, y_train, 'ro', label='Original data')
plt.plot(X_train, y_pred, label='Fitted line')
plt.legend()
plt.show()
```

```python
# to get the parameters, i.e. weight and bias from the model,
# we can use the state_dict() attribute from the model that
# we've defined
model.state_dict()
```

    OrderedDict([('weight', tensor([[0.3563]])), ('bias', tensor([0.0564]))])

```python
# or we could get it from the model's parameter
# which by itself is a generator
list(model.parameters())
```

    [Parameter containing:
     tensor([[0.3563]], requires_grad=True),
     Parameter containing:
     tensor([0.0564], requires_grad=True)]

## Linear Regression Version 2

A better way of defining our model is to inherit the `nn.Module` class, to use it all we need to do is define our model's forward pass and the `nn.Module` will automatically define the backward method for us, where the gradients will be computed using autograd.

```python
class LinearRegression(nn.Module):

    def __init__(self, in_features, out_features):
        super().__init__()  # boilerplate call
        self.in_features = in_features
        self.out_features = out_features
        self.linear = nn.Linear(in_features, out_features)

    def forward(self, x):
        out = self.linear(x)
        return out

# same optimization process
n_epochs = 100
learning_rate = 0.01
criterion = nn.MSELoss(size_average=True)
model = LinearRegression(in_features=1, out_features=1)

# when we defined our LinearRegression class, we've assigned
# a neural network's component/layer to a class variable in the
# __init__ function, and now notice that we can directly call
# .parameters() on the class we've defined due to some Python magic
# from the Pytorch devs
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

for epoch in range(n_epochs):
    # forward + backward + optimize
    optimizer.zero_grad()
    output = model(X)
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()

    # print the loss per 20 epoch this time
    if (epoch + 1) % 20 == 0:
        # starting from pytorch 0.4.0, we use .item to get a python number from a
        # torch scalar, before loss.item() looks something like loss.data[0]
        print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch + 1, n_epochs, loss.item()))
```

    Epoch [20/100], Loss: 0.1608
    Epoch [40/100], Loss: 0.1602
    Epoch [60/100], Loss: 0.1596
    Epoch [80/100], Loss: 0.1591
    Epoch [100/100], Loss: 0.1586

After training our model, we can also [save the model's parameter and load it back into the model in the future](https://stackoverflow.com/questions/42703500/best-way-to-save-a-trained-model-in-pytorch)

```python
checkpoint_path = 'model.pkl'
torch.save(model.state_dict(), checkpoint_path)
model.load_state_dict(torch.load(checkpoint_path))

y_pred = model(X).detach().numpy()
plt.plot(X_train, y_train, 'ro', label='Original data')
plt.plot(X_train, y_pred, label='Fitted line')
plt.legend()
plt.show()
```

## Logistic Regression

Let's now look at a classification example, here we'll define a logistic regression that takes in a bag of words representation of some text and predicts over two labels "English" and "Spanish".

```python
# define some toy dataset
train_data = [
    ('me gusta comer en la cafeteria'.split(), 'SPANISH'),
    ('Give it to me'.split(), 'ENGLISH'),
    ('No creo que sea una buena idea'.split(), 'SPANISH'),
    ('No it is not a good idea to get lost at sea'.split(), 'ENGLISH')
]

test_data = [
    ('Yo creo que si'.split(), 'SPANISH'),
    ('it is lost on me'.split(), 'ENGLISH')
]
```

The next code chunk create words to index mappings. To build our bag of words (BoW) representation, we need to assign each word in our vocabulary an unique index. Let's say our entire corpus only consists of two words "hello" and "world", with "hello" corresponding to index 0 and "world" to index 1. Then the BoW vector for the sentence "hello world hello world" will be [2, 2], i.e. the count for the word "hello" will be at position 0 of the array and so on.

```python
idx_to_label = ['SPANISH', 'ENGLISH']
label_to_idx = {"SPANISH": 0, "ENGLISH": 1}

word_to_idx = {}
for sent, _ in train_data + test_data:
    for word in sent:
        if word not in word_to_idx:
            word_to_idx[word] = len(word_to_idx)
print(word_to_idx)

VOCAB_SIZE = len(word_to_idx)
NUM_LABELS = len(label_to_idx)
```

    {'me': 0, 'gusta': 1, 'comer': 2, 'en': 3, 'la': 4, 'cafeteria': 5, 'Give': 6, 'it': 7, 'to': 8, 'No': 9, 'creo': 10, 'que': 11, 'sea': 12, 'una': 13, 'buena': 14, 'idea': 15, 'is': 16, 'not': 17, 'a': 18, 'good': 19, 'get': 20, 'lost': 21, 'at': 22, 'Yo': 23, 'si': 24, 'on': 25}

Next we define our model using the inherenting from `nn.Module` approach and also two helper functions to convert our data to torch Tensors so we can use to during training.

```python
class BoWClassifier(nn.Module):

    def __init__(self, vocab_size, num_labels):
        super().__init__()
        self.linear = nn.Linear(vocab_size, num_labels)

    def forward(self, bow_vector):
        """
        When we're performing a classification, after passing
        through the linear layer or also known as the affine layer
        we also need pass it through the softmax layer to convert a vector
        of real numbers into probability distribution, here we use
        log softmax for numerical stability reasons.
        """
        return F.log_softmax(self.linear(bow_vector), dim = 1)
```

```python
def make_bow_vector(sentence, word_to_idx):
    vector = torch.zeros(len(word_to_idx))
    for word in sentence:
        vector[word_to_idx[word]] += 1

    return vector.view(1, -1)

def make_target(label, label_to_idx):
    return torch.LongTensor([label_to_idx[label]])
```

We are now ready to train this!

```python
model = BoWClassifier(VOCAB_SIZE, NUM_LABELS)

# note that instead of using NLLLoss (negative log likelihood),
# we could have used CrossEntropyLoss and remove the log_softmax
# function call in our forward method. The CrossEntropyLoss docstring
# explciitly states that this criterion combines `LogSoftMax` and
# `NLLLoss` in one single class.
criterion = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

n_epochs = 100
for epoch in range(n_epochs):
    for instance, label in train_data:
        bow_vector = make_bow_vector(instance, word_to_idx)
        target = make_target(label, label_to_idx)

        # standard step to perform the forward and backward step
        model.zero_grad()
        log_probs = model(bow_vector)
        loss = criterion(log_probs, target)
        loss.backward()
        optimizer.step()

# we can also wrap the code block in with torch.no_grad(): to
# prevent history tracking, this is often used in model inferencing,
# or when evaluating the model as we won't be needing the gradient during
# this stage

with torch.no_grad():
    # predict on the test data to check if the model actually learned anything
    for instance, label in test_data:
        bow_vec = make_bow_vector(instance, word_to_idx)
        log_probs = model(bow_vec)

        y_pred = np.argmax(log_probs[0].numpy())
        label_pred = idx_to_label[y_pred]
        print('true label: ', label, ' predicted label: ', label_pred)
```

    true label:  SPANISH  predicted label:  SPANISH
    true label:  ENGLISH  predicted label:  ENGLISH

## Recurrent Neural Network (RNN)

The idea behind RNN is to make use of sequential information that exists in our dataset. In feedforward neural network, we assume that all inputs and outputs are independent of each other. But for some tasks, this might not be the best way to tackle the problem. For example, in Natural Language Processing (NLP) applications, if we wish to predict the next word in a sentence (one business application of this is [Swiftkey](https://en.wikipedia.org/wiki/SwiftKey)), then we could imagine that knowing the word that comes before it can come in handy.

### Vanilla RNN

The input $x$ will be a sequence of words, and each $x_t$ is a single word. And because of how matrix multiplication works, we can't simply use a word index like (36) as an input, instead we represent each word as a one-hot vector with a size of the total number of vocabulary. For example, the word with index 36 have the value 1 at position 36 and the rest of the value in the vector would all be 0's.

```python
torch.manual_seed(777)

# suppose we have a
# one hot encoding for each char in 'hello'
# and the sequence length for the word 'hello' is 5
seq_len = 5
h = [1, 0, 0, 0]
e = [0, 1, 0, 0]
l = [0, 0, 1, 0]
o = [0, 0, 0, 1]

# here we specify a single RNN cell with the property of
# input_dim (4) -> output_dim (2)
# batch_first explained in the following
rnn_cell = nn.RNN(input_size=4, hidden_size=2, batch_first=True)

# our input shape should be of shape
# (batch, seq_len, input_size) when batch_first=True;
# the input size basically refers to the number of features
# (seq_len, batch_size, input_size) when batch_first=False (default)
# thus we reshape our input to the appropriate size, torch.view is
# equivalent to numpy.reshape
inputs = torch.Tensor([h, e, l, l, o])
inputs = inputs.view(1, 5, -1)

# our hidden is the weights that gets passed along the cells,
# here we initialize some random values for it:
# (batch, num_layers * num_directions, hidden_size) for batch_first=True
# disregard the second argument as of now
hidden = torch.zeros(1, 1, 2)
out, hidden = rnn_cell(inputs, hidden)
print('sequence input size', inputs.size())
print('out size', out.size())
print('sequence size', hidden.size())

# the first value returned by the rnn cell is all
# of the hidden state throughout the sequence, while
# the second value is the most recent hidden state;
# hence we can compare the last slice of the the first
# value with the second value to confirm that they are
# the same
print('\ncomparing rnn cell output:')
print(out[:, -1, :])
hidden[0]
```

    sequence input size torch.Size([1, 5, 4])
    out size torch.Size([1, 5, 2])
    sequence size torch.Size([1, 1, 2])

    comparing rnn cell output:
    tensor([[-0.7762,  0.8319]], grad_fn=<SliceBackward>)

    tensor([[-0.7762,  0.8319]], grad_fn=<SelectBackward>)

In the next section, we'll teach our RNN to produce "ihello" from "hihell".

```python
# create an index to character mapping
idx2char = ['h', 'i', 'e', 'l', 'o']

# Teach hihell -> ihello
x_data = [[0, 1, 0, 2, 3, 3]]    # hihell
x_one_hot = [[[1, 0, 0, 0, 0],   # h 0
              [0, 1, 0, 0, 0],   # i 1
              [1, 0, 0, 0, 0],   # h 0
              [0, 0, 1, 0, 0],   # e 2
              [0, 0, 0, 1, 0],   # l 3
              [0, 0, 0, 1, 0]]]  # l 3

x_one_hot = np.array(x_one_hot)
y_data = np.array([1, 0, 2, 3, 3, 4])  # ihello

# As we have one batch of samples, we will change them to variables only once
inputs = torch.Tensor(x_one_hot)
labels = torch.LongTensor(y_data)

# hyperparameters
seq_len = 6      # |hihell| == 6, equivalent to time step
input_size = 5   # one-hot size
batch_size = 1   # one sentence per batch
num_layers = 1   # one-layer rnn
num_classes = 5  # predicting 5 distinct character
hidden_size = 4  # output from the RNN
```

```python
class RNN(nn.Module):
    """
    The RNN model will be a RNN followed by a linear layer,
    i.e. a fully-connected layer
    """
    def __init__(self, seq_len, num_classes, input_size, hidden_size, num_layers):
        super().__init__()
        self.seq_len = seq_len
        self.num_layers = num_layers
        self.input_size = input_size
        self.num_classes = num_classes
        self.hidden_size = hidden_size
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.linear = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        # assuming batch_first = True for RNN cells
        batch_size = x.size(0)
        hidden = self._init_hidden(batch_size)
        x = x.view(batch_size, self.seq_len, self.input_size)

        # apart from the output, rnn also gives us the hidden
        # cell, this gives us the opportunity to pass it to
        # the next cell if needed; we won't be needing it here
        # because the nn.RNN already computed all the time steps
        # for us. rnn_out will of size [batch_size, seq_len, hidden_size]
        rnn_out, _ = self.rnn(x, hidden)
        linear_out = self.linear(rnn_out.view(-1, hidden_size))
        return linear_out

    def _init_hidden(self, batch_size):
        """
        Initialize hidden cell states, assuming
        batch_first = True for RNN cells
        """
        return torch.zeros(batch_size, self.num_layers, self.hidden_size)
```

```python
# Set loss, optimizer and the RNN model
torch.manual_seed(777)
rnn = RNN(seq_len, num_classes, input_size, hidden_size, num_layers)
print('network architecture:\n', rnn)

# train the model
num_epochs = 15
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(rnn.parameters(), lr=0.1)
for epoch in range(1, num_epochs + 1):
    optimizer.zero_grad()
    outputs = rnn(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # check the current predicted string
    # max gives the maximum value and its
    # corresponding index, we will only
    # be needing the index
    _, idx = outputs.max(dim = 1)
    idx = idx.detach().numpy()
    result_str = [idx2char[c] for c in idx]
    print('epoch: {}, loss: {:1.3f}'.format(epoch, loss.item()))
    print('Predicted string: ', ''.join(result_str))
```

    network architecture:
     RNN(
      (rnn): RNN(5, 4, batch_first=True)
      (linear): Linear(in_features=4, out_features=5, bias=True)
    )
    epoch: 1, loss: 1.756
    Predicted string:  eeeeee
    epoch: 2, loss: 1.626
    Predicted string:  ehhhhh
    epoch: 3, loss: 1.485
    Predicted string:  elllll
    epoch: 4, loss: 1.405
    Predicted string:  llllll
    epoch: 5, loss: 1.293
    Predicted string:  illlll
    epoch: 6, loss: 1.217
    Predicted string:  iiilll
    epoch: 7, loss: 1.057
    Predicted string:  iollll
    epoch: 8, loss: 0.967
    Predicted string:  ielllo
    epoch: 9, loss: 0.837
    Predicted string:  ihlllo
    epoch: 10, loss: 0.696
    Predicted string:  ihello
    epoch: 11, loss: 0.615
    Predicted string:  ihello
    epoch: 12, loss: 0.535
    Predicted string:  ihhllo
    epoch: 13, loss: 0.452
    Predicted string:  ihhllo
    epoch: 14, loss: 0.387
    Predicted string:  ihello
    epoch: 15, loss: 0.322
    Predicted string:  ihello

### LSTM

The example below uses an LSTM to generate part of speech tags. The usage of LSTM API is essentially the same as the RNN we were using in the last section. Expect in this example, we will prepare the word to index mapping ourselves and as for the modeling part, we will add an embedding layer before the LSTM layer, this is a common technique in NLP applications. So for each word, instead of using the one hot encoding way of representation the data (which can be inefficient and it treats all words as independent entities with no relationships amongst each other), word embeddings will compress them into a lower dimension that encode the semantics of the words, i.e. how similar each word is used within our given corpus.

```python
# These will usually be more like 32 or 64 dimensional.
# We will keep them small for this toy example
EMBEDDING_SIZE = 6
HIDDEN_SIZE = 6

training_data = [
    ("The dog ate the apple".split(), ["DET", "NN", "V", "DET", "NN"]),
    ("Everybody read that book".split(), ["NN", "V", "DET", "NN"])
]

idx_to_tag = ['DET', 'NN', 'V']
tag_to_idx = {'DET': 0, 'NN': 1, 'V': 2}

word_to_idx = {}
for sent, tags in training_data:
    for word in sent:
        if word not in word_to_idx:
            word_to_idx[word] = len(word_to_idx)

word_to_idx
```

    {'The': 0,
     'dog': 1,
     'ate': 2,
     'the': 3,
     'apple': 4,
     'Everybody': 5,
     'read': 6,
     'that': 7,
     'book': 8}

```python
def prepare_sequence(seq, to_idx):
    """Convert sentence/sequence to torch Tensors"""
    idxs = [to_idx[w] for w in seq]
    return torch.LongTensor(idxs)

seq = training_data[0][0]
inputs = prepare_sequence(seq, word_to_idx)
inputs
```

    tensor([0, 1, 2, 3, 4])

```python
class LSTMTagger(nn.Module):

    def __init__(self, embedding_size, hidden_size, vocab_size, tagset_size):
        super().__init__()
        self.embedding_size = embedding_size
        self.hidden_size = hidden_size
        self.vocab_size = vocab_size
        self.tagset_size = tagset_size

        self.embedding = nn.Embedding(vocab_size, embedding_size)
        self.lstm = nn.LSTM(embedding_size, hidden_size)
        self.hidden2tag = nn.Linear(hidden_size, tagset_size)

    def forward(self, x):
        embed = self.embedding(x)
        hidden = self._init_hidden()

        # the second dimension refers to the batch size, which we've hard-coded
        # it as 1 throughout the example
        lstm_out, lstm_hidden = self.lstm(embed.view(len(x), 1, -1), hidden)
        output = self.hidden2tag(lstm_out.view(len(x), -1))
        return output

    def _init_hidden(self):
        # the dimension semantics are [num_layers, batch_size, hidden_size]
        return (torch.rand(1, 1, self.hidden_size),
                torch.rand(1, 1, self.hidden_size))
```

```python
model = LSTMTagger(EMBEDDING_SIZE, HIDDEN_SIZE, len(word_to_idx), len(tag_to_idx))
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

epochs = 300
for epoch in range(epochs):
    for sentence, tags in training_data:
        model.zero_grad()

        sentence = prepare_sequence(sentence, word_to_idx)
        target = prepare_sequence(tags, tag_to_idx)

        output = model(sentence)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
```

```python
inputs = prepare_sequence(training_data[0][0], word_to_idx)
tag_scores = model(inputs)

# validating that the sentence "the dog ate the apple".
# the correct tag should be DET NOUN VERB DET NOUN
print('expected target: ', training_data[0][1])

tag_scores = tag_scores.detach().numpy()
tag = [idx_to_tag[idx] for idx in np.argmax(tag_scores, axis = 1)]
print('generated target: ', tag)
```

    expected target:  ['DET', 'NN', 'V', 'DET', 'NN']
    generated target:  ['DET', 'NN', 'V', 'DET', 'NN']

# Reference

- [Github: Simple PyTorch Tutorials Zero to ALL!](https://github.com/hunkim/PyTorchZeroToAll)
- [Github: PyTorch Tutorial for Deep Learning Researchers](https://github.com/yunjey/pytorch-tutorial)
- [PyTorch Documentation: Deep Learning for NLP with Pytorch](http://pytorch.org/tutorials/beginner/deep_learning_nlp_tutorial.html)
- [PyTorch Documentation: Deep Learning with PyTorch: A 60 Minute Blitz](http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html)


# Announcements
- Problem 3 posted on D2L, due on 9/21/2020. Submission of notebooks encouraged(one per problem set, each problem labeled)
- Solutions to Monday's breakout room exercise posted on D2L in ``Lecture 9``
- Markdown cheat sheet posted on D2L in ``Course Notes``

# Ordinary Differential Equations - higher order methods

Based on notes and notebooks by Niels Henrik Aase, Thorvald Ballestad, Vasilis Paschalidis and Jon Andreas Støvneng

## Recap

Physics equations of motion require the solution of ordinary
differential equations (ODE). While the vast majority of equations in
Physics are partial differential equations, ordinary differential
equations pop-up in scenarios where a high degree of symmetry and/or
stationarity (time-independence) is imposed (e.g., stationary and
spherically symmetric). For example, in Newtonian gravity the
equation for the gravitational potential is

$$
\nabla^2 \Phi = 4\pi G \rho,
$$
but in spherical symmetry this becomes an ODE

$$
\frac{1}{r^2}\frac{d}{dr}\left(r^2\frac{d}{dr} \Phi\right) = 4\pi G \rho.
$$

The Newtonian equation of motion is also a second-order ODE in time (acceleration=force/mass),
$$
\frac{d^2 x}{dt^2} = F/m
$$
For a very complex density distribution or complicated
(time-dependent) forces, these last equations are not easily solved
analytically. So, in such cases we need to solve the ODEs numerically.
There are multiple ways one can numerically solve ODEs, and we will
review here some of these methods including the Euler, and Runge-Kutta
methods.

### Initial value problems
An initial value problem (IVP) for ODEs is described by the following
equations
$$
\frac{d}{dt} y(t)=g(y(t), t), a \leq t \leq b,\;\; \mbox{ Initial conditions: } y(a)=\alpha
$$
for a single ODE, and
$$
\frac{d}{dt} \vec y=\vec G(y(t), t), a \leq t \leq b,\;\; \mbox{ Initial conditions: } \vec y(a)=\vec\alpha,
$$
where
$$
\vec y =(y_1, y_2, \ldots y_n), \ \vec G =(g_1(\vec y(t), t), g_2(\vec y(t), t) \ldots g_n(\vec y(t), t)).
$$
for a system of coupled ODEs.

Before we decide to code any differential equation in a computer it is
always important to be able to answer the following questions:
- (1) Does the ODE have a solution?
- (2) Is the ODE solution unique for a given initial conditions?
- (3) Does the solution depend continuously on the initial conditions?

All three questions point to the concept of
well-posedness of the IVP. *If the answer is NO to any of these 3
questions, the IVP is not well-posed, and it is redundant to try to
solve the ODE as an IVP, as the numerical algorithm will fail.* In this
first section we will introduce the mathematical background for
answering the above questions.

#### Definition: Well-posedness of the IVP
The IVP
$$\frac{dy}{dt}=g(y,t), a \leq t \leq b, \mbox{ Initial conditions: } y(a)=\alpha
$$
is __well-posed__ if
- A unique solution $y(t)$ exists
- There exist constants $\epsilon_0>0$, $k>0$ such that $\forall
  \epsilon >0$ with $\epsilon_0 > \epsilon >0$, whenever $\delta(t)$
  is continuous with $|\delta(t)| < \epsilon$ $\forall t \in [a,b]$,
  and $\delta_0 < \epsilon$, the IVP
$$    \frac{dz}{dt}=g(z,t)+\delta(t), a \leq t \leq b, \mbox{ Initial conditions: } z(a)=\alpha+\delta_0 \label{pert}\tag{1}
$$  has a unique solution $z(t)$ that satisfies
$$    |z(t)-y(t)| < k \epsilon_0, \ \forall t \in [a,b].\label{stab}\tag{2}$$

#### Theorem
Assume $D = \{(y,t)\ |-\infty < y < \infty,\ a \leq t \leq b\}$, where $g(y,t)$ is continuous and is
Lipschitz continuous in $y$ on $D$, then the IVP
$$\frac{dy}{dt}=g(y,t), a \leq t \leq b, \mbox{ Initial conditions: } y(a)=\alpha$$
is well-posed.

Well-posedness is a fundamental property of initial value problems for
ordinary and partial differential equations. If a differential
equation does not admit a well-posed initial value problem, then we
cannot integrate it numerically.If a small perturbation either in
initial conditions or in the right-hand-side of Eq.\ref{pert} leads
the difference between the solution $z(t)$ and $y(t)$ to grow without
bound, then this problem is ill-posed and cannot be integrated in a
computer. The reason is that computers will always have finite error,
i.e., will always apply small perturbations in the form of either
truncation or round-off error. *More specifically well-posedness
implies continuous dependence on the initial data.*

## Algorithms for initial value problem ODEs

Assume we have a first-order differential equation which can be expressed in the form

$$ \frac{dy}{dt} = g(y,t) $$

We will solve this on constant-interval mesh of the independent variable $t$ defined by

$$ t_n = t_0 + n h $$

### Forward-Euler method

In Lecture 10 we derived Euler's method, which simply solves the first-order forward difference approximation to $dy/dt$

$$ \frac{y_{i+1}-y_i}{h} = g(y_i,t_i)$$

as

$$ y_{i+1} = y_i + h g(y_i,t_i) \label{Euler_fwd}\tag{3}$$

```python
# Importing the necessary libraries
import numpy as np # NumPy is used to generate arrays and to perform some mathematical operations
import matplotlib.pyplot as plt # Used for plotting results

```

```python
def forwardEuler_step(t, y, h, g, *P):
    """
    Implements a single step of the forward-Euler finite-difference scheme
    Parameters:
            t: time t
            y: Numerical approximation of y at time t
            h: Step size
            g: RHS of our ODE (RHS = Right hand side). Can be any function with signature g(t,y,*P).
            *P: tuple of parameters, arguments to g
        Returns:
            next_y: Numerical approximation of y at time t+h
    """
    next_y = y + h*g(t, y, *P)
    return next_y

```

We now need some sort of framework which will take this function and do the integration for us. Let's rewrite `full_Euler` from Lecture 10 to be more general:

```python
def odeSolve(t0, y0, tmax, h, g, method, *P):
    """ A full numerical aproximation of an ODE in a set time interval. Performs consecutive steps of `method`
    with step size h from start time until the end time. Also takes into account the initial values of the ODE

    Parameters:
            t0: start time
            y0 : Initial condition for y at t = t0
            tmax: The end of the interval where the `method` is integrated, t_N
            h: Step size
            g: RHS of our ODE (RHS = Right hand side). Can be any function with signature g(t,y,*P).
            *P: tuple of parameters, arguments to g
        Returns:
            t_list: Evenly spaced discrete list of time with spacing h.
                    Starting time = start_t, and end time = end_t
            y_list: Numerical approximation of y at times t_list
    """
    # make the t-mesh; guarantees we stop precisely at tmax
    t_list = np.arange(t0,tmax+h,h)
    # allocate space for the solution
    y_list = np.zeros_like(t_list)
    # set the initial condition
    y_list[0] = y0

    # find out the size of the t-mesh, and then integrate forward one meshpoint per iteration of the loop
    n, = t_list.shape
    for i in range(0,n-1):
        y_list[i+1] = method(t_list[i], y_list[i], h, g, *P)

    # return the solution
    return t_list,y_list
```

Armed with this machinery, let's set up another simple problem and try it out.

Last time, we looked at exponential growth, let's solve exponential decay this time:

$$ \frac{dy}{dt} = - c y, \quad y[0] = 1 $$

First, we provide a function to implement the RHS:

```python
def expRHS(t, y, c):
    """
    Implements the RHS (y'(x)) of the DE
    """
    return -c*y
```

Now we set up the problem to compute and plot the result, along with a plot of the magnitude of the fractional error

```python
# set up problem
c = 1.0
h = 0.5
t0 = 0.0
y0 = 1.0
tmax = 5.0

# call the solver
t, y = odeSolve(t0, y0, tmax, h, expRHS, forwardEuler_step, c)

# plot the result
fig,ax = plt.subplots(1,2)
ax[0].plot(t,y,'o')
ans = np.exp(-c*t)
ax[0].plot(t,ans,'r')
ax[0].set_xlabel('t')
ax[0].set_ylabel('y')

# along with the errors
err = np.abs((ans-y)/ans)
ax[1].plot(t, err, 'r')
ax[1].plot(t, err, 'o')
ax[1].set_xlabel('t')
ax[1].set_ylabel('fractional error')

# this gives better spacing between axes
plt.tight_layout()
plt.show()
```

In developing the forward Euler method, we expanded the solution about time $t$,
the beginning of the interval; we can just as easily expand backwards about
$t+h$
\begin{equation}
  f(t) = f(t+h) - h f'(t+h) + \frac{h^2}{2} f''(t+h) + \dots
\end{equation}
Proceeding as before, keeping terms to $\mathcal{O}(h)$, we have
\begin{equation}
  \frac{f(t+h)-f(t)}{h} = f'(t+h)
\end{equation}
giving the update formula
\begin{equation}
  f_{i+1} = f_i + f'_{i+1}
\end{equation}
In this case, we may not be able to solve explicitly for $f_{i-1}$ since $f'$ is evaluated at $t+h$
on the RHS; this is known as an _implicit_ method. While we may not be able to solve for
$f_{n+1}$ analytically, we can always employ some of the numerical techniques for solving nonlinear
equations we have already seen.

The Euler method as implemented above corresponds to Riemann integration for integral - we use the value of the function at $t_i$ to predict its value at $t_{i+1}$. Similar to integration of functions, we can improve convergence by turning to higher order methods that include information on the function behavior within $[t_i,t_{i+1}]$.

##### Explicit midpoint method
By averaging the forward- and backward-Euler scheme, it can be shown that
\begin{equation}
  f_{i+1} = f_i + \frac{h}{2} f'_{i+\frac{1}{2}} + \mathcal{O}(h^3)
\end{equation}
but now we need to evaluate the derivative at the half time step!

With some Taylor expansion tricks, it can be shown that the explict midpoint method can be written as

\begin{align}
\begin{split}
y_{i+\frac{1}{2}} &= y_i + \frac{h}{2} RHS(t_i, y_i) \\
y_{i+1} &= y_i + h RHS(t_i + \frac{1}{2}h, y_{i+\frac{1}{2}})\,,
\end{split}
\end{align}
which is second-order in $h$.

### Runge-Kutta Schemes

The idea of the Runge-Kutta schemes is to take advantage of derivative information at the times between $t_i$ and $t_{i+1}$ to increase the order of accuracy.

For example, in the midpoint method, the derivative at the initial time is used to approximate the derivative at the midpoint of the interval, $f(y_i+\frac{1}{2}hf(y_i,t_i), t_i+\frac{1}{2}h)$. The derivative at the midpoint is then used to advance the solution to the next step. The method can be written in two stages $k_i$,

$$ \begin{aligned} \begin{array}{l} k_1 = h f(y_i,t_i)\\ k_2 = h f(y_i+\frac{1}{2}k_1, t_n+\frac{1}{2}h)\\ y_{i+1} = y_i + k_2 \end{array} \end{aligned}\label{RK2}\tag{4} $$

The midpoint method is known as a __2nd-order Runge-Kutta__ formula.

In general, an explicit 2-stage Runge-Kutta method can be written as,
$$ \begin{array}{l} k_1 = h f(y_n,t_n)\\ k_2 = h f(y_n+b_{21}k_1, t_n+a_2h)\ \\ y_{n+1} = y_n + c_1k_1 +c_2k_2 \label{explicitrk2}\tag{5}\end{array} $$

The scheme is said to be *explicit* since a given stage does not depend *implicitly* on itself, as in the backward Euler method , or on a later stage.

Other explicit second-order schemes can be derived by comparing Eq.(\ref{explicitrk2}) to other second-order expansions and matching terms to determine the coefficients $a_2$, $b_{21}$, $c_1$ and $c_2$.

### Explicit Fourth-Order Runge-Kutta Method

Explicit Runge-Kutta methods are popular as each stage can be calculated with one function evaluation. In contrast, implicit Runge-Kutta methods usually involves solving a non-linear system of equations in order to evaluate the stages. As a result, explicit schemes are much less expensive to implement than implicit schemes.

The higher-order Runge-Kutta methods can be derived by in manner similar to the midpoint formula. An s-stage method is compared to a Taylor method and the terms are matched up to the desired order.

Methods of order $M > 4$ require $M+1$ or $M+2$ function evaluations or stages, in the case of explicit Runge-Kutta methods. As a result, fourth-order Runge-Kutta methods have achieved great popularity over the years as they require only four function evaluations per step. In particular, there is the classic fourth-order Runge-Kutta formula:
$$ \begin{array}{l} k_1 = h f(y_n,t_n)\\ k_2 = h f(y_n+\frac{k_1}{2}, t_n+\frac{h}{2})\\ k_3 = h f(y_n+\frac{k_2}{2}, t_n+\frac{h}{2})\\ k_4 = h f(y_n+k_3, t_n+h)\\ y_{n+1} = y_n + \frac{k_1}{6}+ \frac{k_2}{3}+ \frac{k_3}{3} + \frac{k_4}{6} \label{RK4}\tag{6}\end{array} $$

### Break-out Practice
 Implement the 2-nd order Runge-Kutta step `RK2_step(t, y, h, g, *P)` using the Equations(\ref{RK2}).
 Implement the 4-th order Runge-Kutta step `RK4_step(t, y, h, g, *P)` using the Equations(\ref{RK4}).

 Rerun the example problem from above and compare the errors from RK2, RK4 and forward-Euler

```python
def RK2_step(t, y, h, g, *P):
    """
    Implements a single step of the second-order, explicit midpoint method
    """

```

```python
def RK4_step(t, y, h, g, *P):
    """
    Implements a single step of a fourth-order, explicit Runge-Kutta scheme
    """

```

```python
# set up problem
c = 1.0
h = 0.5
t0 = 0.0
y0 = 1.0
tmax = 5.0

# call the solver for RK2
t, y2 = odeSolve(t0, y0, tmax, h, expRHS, RK2_step, c)
# call the solver for RK4
t, y4 = odeSolve(t0, y0, tmax, h, expRHS, RK4_step, c)

# plot the result
fig,ax = plt.subplots(1,2)
ans = np.exp(-c*t)
ax[0].plot(t,ans,'r')
ax[0].set_xlabel('t')
ax[0].set_ylabel('y')

ax[0].plot(t,y2,'o','RK2')

ax[0].plot(t,y4,'o','RK4')

# along with the errors
err = np.abs((ans-y)/ans)
ax[1].plot(t, err, 'o')
ax[1].set_xlabel('t')
ax[1].set_ylabel('fractional error')
ax[1].legend()
# now also overplot the error we calculated for forward-Euler

# this gives better spacing between axes
plt.tight_layout()
plt.show()
```

```python

```