```scala211
import Resolvers._
interp.resolvers() = interp.resolvers() :+ Resolver.Http(
    "cibotech",
    "https://dl.bintray.com/cibotech/public/",
    MavenPattern,
    true
)
```

    import Resolvers._

```scala211
import $ivy.`com.cibo::evilplot:0.4.1`
import $ivy.`com.cibo::evilplot-repl:0.4.1`
import $ivy.`org.ruivieira::scala-ssm:0.1`
```

    import $ivy.$

    import $ivy.$

    import $ivy.$

```scala211
import breeze.linalg.{DenseVector, DenseMatrix}
import org.ruivieira.ssm.univariate.{
  StateGenerator,
  UnivariateGenerator,
  UnivariateStructure
}
```

    import breeze.linalg.{DenseVector, DenseMatrix}

    import org.ruivieira.ssm.univariate.{
      StateGenerator,
      UnivariateGenerator,
      UnivariateStructure
    }

# Univariate models

## Gaussian observations

### Locally constant (random walk)

First, we start by creating a simple univariate Gaussian random walk. This will correspond to a dynamic linear model in the form

\begin{align} y_t &\sim \mathcal{N}\left(\theta_t,V\right) \\
\theta_t &\sim \mathcal{N}\left(\theta_{t-1},W\right)
\end{align}

We start by definind the variance of the latent states as $W=1.5$.

```scala211
val structure = UnivariateStructure.createLocallyConstant(W = 1.5)
```

    structure: UnivariateStructure = UnivariateStructure(1.0  , 1.0  , 1.5  )

And generate a chain of $n=1000$ states with an initial state of $\theta_0 = 0$.

```scala211
val states = StateGenerator.states(nobs = 1000,
                                   structure = structure,
                                   state0 = DenseVector[Double](0.0))
```

    Sep 26, 2018 9:57:40 PM com.github.fommil.netlib.BLAS <clinit>
    WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
    Sep 26, 2018 9:57:40 PM com.github.fommil.netlib.BLAS <clinit>
    WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
    Sep 26, 2018 9:57:40 PM com.github.fommil.netlib.LAPACK <clinit>
    WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
    Sep 26, 2018 9:57:40 PM com.github.fommil.netlib.LAPACK <clinit>
    WARNING: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK

    states: Vector[org.ruivieira.ssm.package.State[Double]] = Vector(
      DenseVector(-1.504710151447631),
      DenseVector(-4.005465847587514),
      DenseVector(-4.354553846657136),
      DenseVector(-4.090964429230846),
      DenseVector(-3.8886790204564536),
      DenseVector(-2.1425874155478892),
      DenseVector(-1.1682184564618354),
      DenseVector(-3.3981356166143737),
      DenseVector(-3.6590137781214365),
      DenseVector(-1.727922543731949),
      DenseVector(-0.5518069589923127),
    ...

We can now generate the observations from the states, using an observation variance $V=4.0$.

```scala211
val observations = UnivariateGenerator.gaussian(states = states,
                                                structure = structure,
                                                V = 4.0)
```

    observations: Vector[Double] = Vector(
      -9.813130907813472,
      -7.990584023190113,
      2.1316191824548856,
      -6.067755532423735,
      -7.957126356837044,
      -4.1117732863679155,
      -8.524885531215773,
      -5.355052151115001,
      -3.995888534432344,
      -3.025142484729351,
      1.2021989398740822,
    ...

```scala211
import com.cibo.evilplot._
import com.cibo.evilplot.plot._
import com.cibo.evilplot.plot.aesthetics.DefaultTheme._
import com.cibo.evilplot.numeric.Point

val states_plot = ScatterPlot(Seq.tabulate(100) { i => Point(i.toDouble, states(i)(0)) })
val obs_plot = LinePlot(Seq.tabulate(100) { i => Point(i.toDouble, observations(i)) })
val plot = Overlay(states_plot, obs_plot).render()
publish.png(plot.asBufferedImage)
```

### A stock exchange example

Let's assume we are following the end of day stock price of company Foo Ltd.
We will simulate a stock price history for 365 days (each timepoint is a day), for a relatively stable stock price, with no trend or seasonality but with natural fluctuation. We also assume that the stock's initial price at day 0 is around \$100. As such we will set an initial state of $\theta_0 = 100$ and low underlying variance ($W=0.01$) with some noise in the data ($V=1.0$).

```scala211
val stock_structure = UnivariateStructure.createLocallyConstant(W = 0.01)
val states = StateGenerator.states(nobs = 365,
                                   structure = structure,
                                   state0 = DenseVector[Double](100.0))
val observations = UnivariateGenerator.gaussian(states = states,
                                                structure = structure,
                                                V = 1.0)
```

    stock_structure: UnivariateStructure = UnivariateStructure(1.0  , 1.0  , 0.01  )
    states: Vector[org.ruivieira.ssm.package.State[Double]] = Vector(
      DenseVector(99.78680062288154),
      DenseVector(100.26624884558552),
      DenseVector(102.03388255437703),
      DenseVector(105.625201972566),
      DenseVector(105.33415776002083),
      DenseVector(102.91081019847178),
      DenseVector(102.42751587310697),
      DenseVector(101.19916482365566),
      DenseVector(102.76492508146914),
      DenseVector(102.93087490118525),
      DenseVector(102.03033335393255),
    ...
    observations: Vector[Double] = Vector(
      99.38015637063377,
      100.61438467289895,
      101.49950588987667,
      106.05296404728294,
      105.1691874217793,
      104.133872379809,
      102.64554389788702,
      100.34367642683276,
      102.13975530656289,
      103.91538916619982,
      102.41639264335903,
    ...

```scala211
Scatter((1 until 365), observations.toSeq).plot()
```

    res7: String = "plot-1426975349"

Now (ignoring the complexities of the stock market), let's suppose that on day $t=100$, this company announces a revolutionary breakthrough. We want to incorporate in our simulated data a jump of _twice_ is stock price, regardless of the value at $t=100$.

We can do this by changing the data at the _state_ level at any point. First we create a chain for a "normal" random walk and then we append another one with a starting value of $\theta_{100} = 2\theta_{100}$.

```scala211
val states_pre = StateGenerator.states(nobs = 100, structure = structure, state0 = DenseVector[Double](100.0))
val states_post = StateGenerator.states(nobs = 265, structure = structure, state0 = states_pre.last * 2.0)
val states = states_pre ++ states_post
```

    states_pre: Vector[org.ruivieira.ssm.package.State[Double]] = Vector(
      DenseVector(98.82503499510088),
      DenseVector(100.75347140076735),
      DenseVector(102.61400348742461),
      DenseVector(103.60993699902369),
      DenseVector(104.10731988767311),
      DenseVector(105.11693765961462),
      DenseVector(107.44395834806993),
      DenseVector(107.55526831805163),
      DenseVector(107.22946098825858),
      DenseVector(106.19430875830102),
      DenseVector(106.42933230702745),
    ...
    states_post: Vector[org.ruivieira.ssm.package.State[Double]] = Vector(
      DenseVector(256.4542820043453),
      DenseVector(255.40719942962374),
      DenseVector(255.0710954720491),
      DenseVector(255.22458300974884),
      DenseVector(255.85937114794743),
      DenseVector(255.02410719219338),
      DenseVector(253.91240527847032),
      DenseVector(254.98412682672023),
      DenseVector(252.64928398740798),
      DenseVector(251.05195463972834),
      DenseVector(251.05243215367832),
    ...
    states: Vector[org.ruivieira.ssm.package.State[Double]] = Vector(
      DenseVector(98.82503499510088),
      DenseVector(100.75347140076735),
      DenseVector(102.61400348742461),
      DenseVector(103.60993699902369),
      DenseVector(104.10731988767311),
      DenseVector(105.11693765961462),
      DenseVector(107.44395834806993),
      DenseVector(107.55526831805163),
      DenseVector(107.22946098825858),
      DenseVector(106.19430875830102),
      DenseVector(106.42933230702745),
    ...

It is important to note that due to the Markovian nature of the SSM, changes at the state level will _propagate_ to future states. This means that the jump in stock price will be propagated to future values.

```scala211
val observations = UnivariateGenerator.gaussian(states = states,
                                                structure = structure,
                                                V = 1.0)
Scatter((1 until 365), observations.toSeq).plot()
```

    observations: Vector[Double] = Vector(
      98.76791478327466,
      99.44480224605408,
      102.18322819215143,
      104.08455394844155,
      103.49572904914345,
      103.71264224462338,
      105.63841898163986,
      106.87033432445901,
      105.89336819861931,
      107.556757017998,
      104.55721302982165,
    ...
    res9_1: String = "plot-960957936"

### Locally linear (mean and trend)

For a locally linear model, we assume the state and observations matrices to be, respectively

$$ \mathsf{F} = \begin{bmatrix} 1 & 0 \end{bmatrix},\qquad \mathsf{G} = \begin{bmatrix} 1 & 1 \\ 0 & 1\end{bmatrix}.$$

The latent states, will then correspond to $\theta_t = \left(\mu, \tau\right)$, that is two components, representing the mean and the trend, respectively. The model will then take the form

\begin{align}
y_t \sim \mathcal{N}\left(\mathsf{F}\theta_t,V\right) \\
\theta_t \sim \mathcal{N}\left(\mathsf{G}\theta_{t-1},\mathsf{W}\right)
\end{align}

The state covariance will now be a matrix

$$ \mathsf{W} = \begin{bmatrix} W_{\tau} & 0 \\ 0 & W_{\mu} \end{bmatrix}$$

representing the variance of the underlying mean and trend respectively.

### Stock exchange (again)

Let's now simulate the stock price of the Foo company, but assuming there's a trend to the values, rather than a jump.
We will create a mean varying a bit ($W_{\mu}=0.5$) but a rather smooth trend ($W_{\tau}=0.05$) and with some noise in the observations ($V=2.0$).

```scala211
val W = DenseMatrix.eye[Double](2)
W(0,0) = 0.05
W(1,1) = 0.5

val stock_structure = UnivariateStructure.createLocallyLinear(W = W)
val states = StateGenerator.states(nobs = 365,
                                   structure = stock_structure,
                                   state0 = DenseVector[Double](100.0, -1.0))
val observations = UnivariateGenerator.gaussian(states = states,
                                                structure = stock_structure,
                                                V = 2.0)
```

    W: DenseMatrix[Double] = 0.05  0.0
    0.0   0.5
    stock_structure: UnivariateStructure = UnivariateStructure(
      1.0
    0.0  ,
      1.0  1.0
    0.0  1.0  ,
      0.05  0.0
    0.0   0.5
    )
    states: Vector[org.ruivieira.ssm.package.State[Double]] = Vector(
      DenseVector(99.0226297726744, -0.6572694131644385),
      DenseVector(98.37715741913001, -0.41377550236256466),
      DenseVector(97.50397897633596, -0.16450921918242228),
      DenseVector(97.65052765827431, 0.4472775698674409),
      DenseVector(97.62848396649082, 1.1824135392159318),
      DenseVector(99.14115324358593, 0.5569171856690674),
      DenseVector(99.39297860861338, -0.02877343767543672),
      DenseVector(99.49852406846871, -0.576117541985426),
      DenseVector(99.39312623630217, -0.6770356481524766),
      DenseVector(98.72471594967918, -0.39906553495557223),
      DenseVector(98.5626543641813, 0.023187887856900402),
    ...
    observations: Vector[Double] = Vector(
      101.16844288351956,
      97.4315584982937,
      97.37012503095563,
      99.97179139386726,
      99.22456286543147,
      95.83912443799845,
      102.75196149680521,
      99.83822897837995,
      98.10140929817888,
      97.69928490456554,
      99.52671205724207,
    ...

```scala211
Scatter((1 until 365), observations.toSeq).plot()
```

    res27: String = "plot-1095543504"

One of the advantages of this formulation, is that we can decompose easily the states into the mean and the trend.
For instance:

```scala211
val x = 1 until 365

val plot = Seq(
  Scatter(
    x, states.map(_(0)), name = "trend"
  ),
  Scatter(
    x, states.map(_(1)), name = "mean"
  )
)
plot.plot(title = "Locally linear")

```

    x: Range = Range(
      1,
      2,
      3,
      4,
      5,
      6,
      7,
      8,
      9,
      10,
      11,
    ...
    plot: Seq[Scatter] = List(
      Scatter(
        Some(
          Doubles(
            Vector(
              1.0,
              2.0,
              3.0,
              4.0,
              5.0,
              6.0,
              7.0,
    ...
    res30_2: String = "plot-1662312060"

```scala211

```


```python
import numpy as np
import pandas as pd
import scipy as sp
import math
import time
import sklearn as sl
import seaborn as sns; sns.set()
import matplotlib as mpl
from matplotlib import pyplot as plt
from sympy import integrate, init_printing
from sympy.abc import x
from mpl_toolkits.mplot3d import axes3d
from matplotlib import cm

%matplotlib inline
```

# Tarea 4

Con base a los métodos vistos en clase resuelva las siguientes dos preguntas

## (A) Integrales

* $\int_{0}^{1}x^{-1/2}\,\text{d}x$

-Metodo Cuadratica Gaussianas

```python
x = 0.01
x0 = 1
N = 10
```

```python
x1 = np.linspace(x,x0, N)

V = np.matrix([x1**i for i in range(len(x1))]  )
V_i = np.linalg.inv(V)
```

```python
I = np.matrix([(( x0**(N+1) - x**(N+1)) /(N+1)) for N in range(len(x1)) ]).T
```

```python
def f(x):
    return (x)**(-1/2)
```

```python
vectF= np.matrix([f(x1[i]) for i in range(len(x1))]).T
```

```python
Integral=((vectF.T*V_i*I)[0,0])
```

```python
Integral=round(Integral)
```

```python
print ("El valor de la integral es",Integral)
```

    El valor de la integral es 2

-Metdodo del Trapecio

```python
def f(x):
    return (x)**(-1/2)

a=0.001
b=1
n=100

x=np.linspace(a,b,n)
dx=(b-a)/n
```

```python
I=0
for i in range (n):
    area=(f(a)+f(a+dx))*dx/2
    I=I+ area
    a=a+dx
r=round(I)
print ("El valor de la integral es",r)
```

    El valor de la integral es 2

* $\int_{0}^{\infty}e^{-x}\ln{x}\,\text{d}x$

-Metodo Cuadratica Gaussianas

```python
x = 0.1
x0 = 1
N = 10
```

```python
x1 = np.linspace(x,x0, N)

V = np.matrix([x1**i for i in range(len(x1))]  )
V_i = np.linalg.inv(V)
```

```python
I = np.matrix([(( x0**(N+1) - x**(N+1)) /(N+1)) for N in range(len(x1)) ]).T
```

```python
def f(x):
    return np.exp(-x)*np.log(x)
```

```python
vectF= np.matrix([f(x1[i]) for i in range(len(x1))]).T
```

```python
print ("El valor de la integral es", (vectF.T*V_i*I)[0,0])
```

    El valor de la integral es -0.4800999189984445

-Metodo del trapecio

```python
def f(x):
    return np.exp(-x)*np.log(x)

a=0.1
b=1/2.5
n=10

x=np.linspace(a,b,n)
dx=(b-a)/n
```

```python
I=0
for i in range (n):
    area=(f(a)+f(a+dx))*dx/2
    I=I+ area
print ("El valor de la integral es",I)

```

    El valor de la integral es -0.5812460606926427

* $\int_{0}^{\infty}\frac{\sin{x}}{x}\,\text{d}x$

-Metodo Cuadratica Gaussianas

```python
x = 0.000000001
x0 = 10-2
N = 10
```

```python
x1 = np.linspace(x,x0, N)

V = np.matrix([x1**i for i in range(len(x1))]  )
V_i = np.linalg.inv(V)
```

```python
I = np.matrix([(( x0**(N+1) - x**(N+1)) /(N+1)) for N in range(len(x1)) ]).T
```

```python
def f(x):
    return np.sin(x)/x

```

```python
vectF= np.matrix([f(x1[i]) for i in range(len(x1))]).T
```

```python
print ("El valor de la integral es", (vectF.T*V_i*I)[0,0])
```

    El valor de la integral es 1.5743680900256365

-Metodo del trapecio

```python
def f(x):
    return np.sin(x)/x

a=0.00001
b=1.5
n=101

x=np.linspace(a,b,n)
dx=(b-a)/n

```

```python
I=0
for i in range (n):
    area=(f(a)+f(a+dx))*dx/2
    I=I+ area
print ("El valor de la integral es",I)

```

    El valor de la integral es 1.499962392876672

## (B) Fourier

Calcule la transformada rápida de Fourier para la función de la **Tarea 3 (D)** en el intervalo $[0,4]$ ($k$ máximo $2\pi n/L$ para $n=25$). Ajuste la transformada de Fourier para los datos de la **Tarea 3** usando el método de regresión exacto de la **Tarea 3 (C)** y compare con el anterior resultado. Para ambos ejercicios haga una interpolación y grafique para comparar.

La funcion es:
$$y=\frac{a}{\left[(x-b)^2+c\right]^\gamma}.$$
Halle $a$, $b$, $c$ y $\gamma$.

En la tarea 3 se encontraron los valores de $a$, $b$, $c$ y $\gamma$.
Los cuales son:
* $a$ =0.68740987
* $b$ =1.39239372
* $c$ =0.49566194
* $\gamma$ =0.93666128

```python
df = pd.read_pickle('ex1.gz')
sns.scatterplot(x='x',y='y',data=df)
plt.show()
df
```

```python
a=0.68740987
b=1.39239372
c=0.49566194
g=0.93666128
```

```python
X=df["x"]
y=df["y"]
def f(x):
    return (a)/((x-b)**2 + c)**g

x=f(X)
```

```python

```

```python
plt.plot(x,y)
```

```python
Nf = 25
a = np.min(x)
b = np.max(x)
def a_j(j):
    global a, b, x, y
    # integra la funcion entre 0 y 10
    L = b - a
    k_j = 2*np.pi*j/4
    new_y = y*np.cos(k_j*x)/L
    if j > 0:
        new_y = new_y * 2
    return sp.integrate.simpson(new_y, x)
def b_j(j):
    global a, b, x, y
    # integra la funcion entre 0 y 10
    L = b - a
    k_j = 2*np.pi*j/4
    new_y = y*np.sin(k_j*x)/L
    if j > 0:
        new_y = new_y * 2
    return sp.integrate.simpson(new_y, x)
J = np.array(range(Nf))
A_j = np.array([a_j(j) for j in range(Nf)])
B_j = np.array([b_j(j) for j in range(Nf)])
```

```python
x_tilde = np.linspace(a,b-a,20000)
k_j = np.array([2*np.pi*j/4 for j in range(Nf)])
y_tilde = np.sum([
    A_j[j] * np.cos(k_j[j]*x_tilde) + B_j[j] * np.sin(k_j[j]*x_tilde) for j in range(Nf)
], axis=0)
```

```python
plt.plot(x,y)
plt.plot(x_tilde, y_tilde)
```

```python

```


# Moving frame calculations
## General idea
Fundamental thing to start with
$$
  f(z) = \bar{f}(\bar{z})
$$

Then you need a general group transformation.
Which you should simplify as far as possible.
    First: translations can be removed. This is always true, since you have choice of cross-section. It is an extremely good idea since it means that you will now evaluate everything at (x,y) = (0,0).
    Second: Remove any other group parameters you can.

Now prolong the group action. Just write it out, don't think. The code below should help.

Turn the prolonged action into a matrix up to the appropriate number of derivatives. Remember that you are solving for the entries of the matrix, not for the vectors.

Now comes the art. You need to find the rest of the cross-section. Choose values for sets of the barred derivatives in order to get all the parameters. What is left over is an invariant.

## Mobius
Fundamental thing to start with
$$
  f(z) = \bar{f}(\bar{z})
$$
A general Mobius transformation is
$$
  \bar{z} = \frac{\alpha z + \beta}{\gamma z + \delta}
$$

Assuming $\delta \neq 0$, we can normalise it: $\delta = 1$. For our cross-section, we'll choose $\bar{z} = 0$. From any point $z$, this determines $\beta$, so wlog assume we start at $z = 0$, i.e. that $\beta = 0$. *So $z = 0$ from now on!!!*.

$$
  \bar{x} + i\bar{y} = \frac{(a + ib)(x + iy)}{1 + (c + id)(x + iy)}
$$

After the zeroth-order frame translates the general point $\bar{x}$ to $0$. So all derivative calculations will be evaluated at $x = y = 0$.

```python
from sympy import Function, Symbol, symbols, init_printing, expand, I, re, im
from IPython.display import Math, display

init_printing()
from transvectants import *

def disp(expr):
    display(Math(my_latex(expr)))
```

```python

```

```python
# p and q are \bar{x} \bar{y}
x, y = symbols('x y')
p, q = symbols('p q')
a, b, c, d = symbols('a b c d')
p = ((a*x - b*y)*(1 + c*x - d*y) + (b*x + a*y)*(d*x + c*y))/((1 + c*x - d*y)**2 + (d*x + c*y)**2)
q = ((b*x + a*y)*(1 + c*x - d*y) - (a*x - b*y)*(d*x + c*y))/((1 + c*x - d*y)**2 + (d*x + c*y)**2)

# Can we tidy this later - but this does work
g = Function('g')(p, q)
g

```

```python
# In the below, interpret fb_blah as the the f derivative
foo = diff(g, x).subs([(x, 0), (y, 0)])

```

```python
foo
```

```python
disp(diff(g, x).subs([(x, 0), (y, 0)]))
disp(diff(g, x).subs([(x, 0), (y, 0)]))
disp(diff(g, x, x).subs([(x, 0), (y, 0)]))
disp(diff(g, x, y).subs([(x, 0), (y, 0)]))
disp(diff(g, y, y).subs([(x, 0), (y, 0)]))
disp(diff(g, x, x, x).subs([(x, 0), (y, 0)]))
disp(diff(g, x, x, y).subs([(x, 0), (y, 0)]))
disp(diff(g, x, y, y).subs([(x, 0), (y, 0)]))
print('boo')
disp(diff(g, y, y, y).subs([(x, 0), (y, 0)]))
```

$\displaystyle a g_{\xi_{1}} + b g_{\xi_{2}}$

$\displaystyle a g_{\xi_{1}} + b g_{\xi_{2}}$

$\displaystyle a \left(a g_{\xi_{1}\xi_{1}} + b g_{\xi_{1}\xi_{2}}\right) + b \left(a g_{\xi_{1}\xi_{2}} + b g_{\xi_{2}\xi_{2}}\right) - 2 \left(a c - b d\right) g_{\xi_{1}} + 2 \left(- a d - b c\right) g_{\xi_{2}}$

$\displaystyle a \left(a g_{\xi_{1}\xi_{2}} - b g_{\xi_{1}\xi_{1}}\right) + b \left(a g_{\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}}\right) - 2 \left(a c - b d\right) g_{\xi_{2}} - 2 \left(- a d - b c\right) g_{\xi_{1}}$

$\displaystyle a \left(a g_{\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}}\right) - b \left(a g_{\xi_{1}\xi_{2}} - b g_{\xi_{1}\xi_{1}}\right) - 2 \left(- a c + b d\right) g_{\xi_{1}} - 2 \left(- a d - b c\right) g_{\xi_{2}}$

$\displaystyle - a \left(a \left(- a g_{\xi_{1}\xi_{1}\xi_{1}} - b g_{\xi_{1}\xi_{1}\xi_{2}}\right) - b^{2} g_{\xi_{1}\xi_{2}\xi_{2}} + 2 \left(a c - b d\right) g_{\xi_{1}\xi_{1}} + 2 \left(a d + b c\right) g_{\xi_{1}\xi_{2}}\right) + b \left(a^{2} g_{\xi_{1}\xi_{1}\xi_{2}} + b \left(a g_{\xi_{1}\xi_{2}\xi_{2}} + b g_{\xi_{2}\xi_{2}\xi_{2}}\right) - 2 \left(a c - b d\right) g_{\xi_{1}\xi_{2}} + 2 \left(- a d - b c\right) g_{\xi_{2}\xi_{2}}\right) - 4 \left(a c - b d\right) \left(a g_{\xi_{1}\xi_{1}} + b g_{\xi_{1}\xi_{2}}\right) + 4 \left(- a d - b c\right) \left(a g_{\xi_{1}\xi_{2}} + b g_{\xi_{2}\xi_{2}}\right) + 6 \left(4 a c^{2} - a \left(c^{2} + d^{2}\right) - 2 c \left(a c + b d\right)\right) g_{\xi_{1}} - 6 \left(- 4 b c^{2} + b \left(c^{2} + d^{2}\right) - 2 c \left(a d - b c\right)\right) g_{\xi_{2}}$

$\displaystyle - a \left(- a b g_{\xi_{1}\xi_{2}\xi_{2}} + a \left(- a g_{\xi_{1}\xi_{1}\xi_{2}} + b g_{\xi_{1}\xi_{1}\xi_{1}}\right) + 2 \left(a c - b d\right) g_{\xi_{1}\xi_{2}} + 2 \left(- a d - b c\right) g_{\xi_{1}\xi_{1}}\right) + b \left(- a b g_{\xi_{1}\xi_{1}\xi_{2}} + b \left(a g_{\xi_{2}\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}\xi_{2}}\right) - 2 \left(a c - b d\right) g_{\xi_{2}\xi_{2}} - 2 \left(- a d - b c\right) g_{\xi_{1}\xi_{2}}\right) - 2 \left(a c - b d\right) \left(a g_{\xi_{1}\xi_{2}} + b g_{\xi_{2}\xi_{2}}\right) - 2 \left(a c - b d\right) \left(a g_{\xi_{1}\xi_{2}} - b g_{\xi_{1}\xi_{1}}\right) + 2 \left(- a d - b c\right) \left(a g_{\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}}\right) - 2 \left(- a d - b c\right) \left(a g_{\xi_{1}\xi_{1}} + b g_{\xi_{1}\xi_{2}}\right) - 2 \left(- 4 a c^{2} + a \left(c^{2} + d^{2}\right) + 8 b c d + 2 d \left(a d - b c\right)\right) g_{\xi_{2}} + 2 \left(- 8 a c d - 4 b c^{2} + b \left(c^{2} + d^{2}\right) + 2 d \left(a c + b d\right)\right) g_{\xi_{1}}$

$\displaystyle - a \left(- a^{2} g_{\xi_{1}\xi_{2}\xi_{2}} - b \left(- a g_{\xi_{1}\xi_{1}\xi_{2}} + b g_{\xi_{1}\xi_{1}\xi_{1}}\right) + 2 \left(- a c + b d\right) g_{\xi_{1}\xi_{1}} + 2 \left(- a d - b c\right) g_{\xi_{1}\xi_{2}}\right) + b \left(a \left(a g_{\xi_{2}\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}\xi_{2}}\right) + b^{2} g_{\xi_{1}\xi_{1}\xi_{2}} - 2 \left(- a c + b d\right) g_{\xi_{1}\xi_{2}} - 2 \left(- a d - b c\right) g_{\xi_{2}\xi_{2}}\right) - 4 \left(a c - b d\right) \left(a g_{\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}}\right) - 4 \left(- a d - b c\right) \left(a g_{\xi_{1}\xi_{2}} - b g_{\xi_{1}\xi_{1}}\right) - 2 \left(- 4 a d^{2} + a \left(c^{2} + d^{2}\right) - 8 b c d + 2 c \left(a c + b d\right)\right) g_{\xi_{1}} + 2 \left(- 8 a c d + 4 b d^{2} - b \left(c^{2} + d^{2}\right) + 2 c \left(a d - b c\right)\right) g_{\xi_{2}}$

    boo

$\displaystyle a \left(a \left(a g_{\xi_{2}\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}\xi_{2}}\right) + b^{2} g_{\xi_{1}\xi_{1}\xi_{2}} - 2 \left(- a c + b d\right) g_{\xi_{1}\xi_{2}} - 2 \left(- a d - b c\right) g_{\xi_{2}\xi_{2}}\right) + b \left(- a^{2} g_{\xi_{1}\xi_{2}\xi_{2}} - b \left(- a g_{\xi_{1}\xi_{1}\xi_{2}} + b g_{\xi_{1}\xi_{1}\xi_{1}}\right) + 2 \left(- a c + b d\right) g_{\xi_{1}\xi_{1}} + 2 \left(- a d - b c\right) g_{\xi_{1}\xi_{2}}\right) - 4 \left(- a c + b d\right) \left(a g_{\xi_{1}\xi_{2}} - b g_{\xi_{1}\xi_{1}}\right) - 4 \left(- a d - b c\right) \left(a g_{\xi_{2}\xi_{2}} - b g_{\xi_{1}\xi_{2}}\right) - 6 \left(- 4 a d^{2} + a \left(c^{2} + d^{2}\right) + 2 d \left(a d - b c\right)\right) g_{\xi_{2}} + 6 \left(- 4 b d^{2} + b \left(c^{2} + d^{2}\right) + 2 d \left(a c + b d\right)\right) g_{\xi_{1}}$

### Write this out as a matrix

### Now pick the cross-section

# Conformal
OK, let's try again. This time we are gonna be awesome and do conformal. The Taylor expansion of a general conformal map up to third order is
$$
  \bar{z} = c_0 + c_1 z + c_2 z^2 + c_3 z^3
$$
Or in components,
$$
\begin{align*}
  \bar{x} &= a_0 + a_1 x + a_2 (x^2 - y^2) + a_3 (x^3 - 3 xy^2) - b_1 y - 2b_2xy - 3b_3x^2y + b_3y^3   \\
  3 & = 4
\end{align*}
$$

```python
x, y = symbols('x y', real=True)
a0, a1, a2, a3, b0, b1, b2, b3 = symbols('a_0 a_1 a_2 a_3 b_0 b_1 b_2 b_3', real=True)
z = x + I*y
# We have removed the a_0 + I*b_0 term to take out the translation
w = (a1 + I*b1)*z + (a2 + I*b2)*z**2 + (a3 + I*b3)*z**3
p = re(w)
q = im(w)
p
```

```python
fb = Function('g')(p, q)
disp(diff(fb, x).subs([(x, 0), (y, 0)]))
disp(diff(fb, y).subs([(x, 0), (y, 0)]))
disp(diff(fb, x, x).subs([(x, 0), (y, 0)]))
disp(diff(fb, x, y).subs([(x, 0), (y, 0)]))
disp(diff(fb, y, y).subs([(x, 0), (y, 0)]))
disp(diff(fb, x, x, x).subs([(x, 0), (y, 0)]))
disp(diff(fb, x, x, y).subs([(x, 0), (y, 0)]))
disp(diff(fb, x, y, y).subs([(x, 0), (y, 0)]))
print('boo')
disp(diff(fb, y, y, y).subs([(x, 0), (y, 0)]))
```

$\displaystyle a_{1} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}$

$\displaystyle a_{1} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}$

$\displaystyle a_{1} \left(a_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) + 2 a_{2} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}\right) + 2 b_{2} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}$

$\displaystyle a_{1} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) + 2 a_{2} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left(a_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) - 2 b_{2} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}$

$\displaystyle a_{1} \left(a_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) - 2 a_{2} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} - b_{1} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) - 2 b_{2} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}$

$\displaystyle a_{1} \left(a_{1} \left(a_{1} \left. g_{\xi_{1}\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) + 2 a_{2} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1}^{2} \left. \left. g_{\xi_{1}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} \right|_{\substack{ \xi_{1}=0 }} + 2 b_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) + 4 a_{2} \left(a_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) + 6 a_{3} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left(a_{1}^{2} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + 2 a_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left. g_{\xi_{2}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}\right) + 2 b_{2} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}\right) + 4 b_{2} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}\right) + 6 b_{3} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}$

$\displaystyle - a_{1} \left(- a_{1} b_{1} \left. \left. g_{\xi_{1}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} \right|_{\substack{ \xi_{1}=0 }} + a_{1} \left(- a_{1} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left. g_{\xi_{1}\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) - 2 a_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + 2 b_{2} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) + 2 a_{2} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}\right) + 2 a_{2} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) + 6 a_{3} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left(- a_{1} b_{1} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + 2 a_{2} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left(a_{1} \left. g_{\xi_{2}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} \right|_{\substack{ \xi_{1}=0 }}\right) - 2 b_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) + 2 b_{2} \left(a_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) - 2 b_{2} \left(a_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) - 6 b_{3} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}$

$\displaystyle - a_{1} \left(a_{1} b_{1} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + 2 a_{2} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} - b_{1} \left(- a_{1} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left. g_{\xi_{1}\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) + 2 b_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) + 4 a_{2} \left(a_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) - 6 a_{3} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} + b_{1} \left(a_{1} \left(a_{1} \left. g_{\xi_{2}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} \right|_{\substack{ \xi_{1}=0 }}\right) - 2 a_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1}^{2} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} - 2 b_{2} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}\right) - 4 b_{2} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) - 6 b_{3} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}$

    boo

$\displaystyle a_{1} \left(a_{1} \left(a_{1} \left. g_{\xi_{2}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} \right|_{\substack{ \xi_{1}=0 }}\right) - 2 a_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1}^{2} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} - 2 b_{2} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }}\right) - 4 a_{2} \left(a_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) - 6 a_{3} \left. g_{\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left(a_{1} b_{1} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + 2 a_{2} \left. g_{\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }} - b_{1} \left(- a_{1} \left. \left. g_{\xi_{1}\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }} + b_{1} \left. g_{\xi_{1}\xi_{1}\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}\right) + 2 b_{2} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) - 4 b_{2} \left(a_{1} \left. g_{\xi_{2}\xi_{2}} \right|_{\substack{ \xi_{2}=0 }} - b_{1} \left. \left. g_{\xi_{1}\xi_{2}} \right|_{\substack{ \xi_{1}=0 }} \right|_{\substack{ \xi_{2}=0 }}\right) + 6 b_{3} \left. g_{\xi_{1}} \right|_{\substack{ \xi_{1}=0 }}$

### Write this out as a matrix

### Now for the cross-section

```python

```

$\displaystyle 2 f_{xxxx} f_{yyyy} - 8 f_{xyyy} f_{xxxy} + 6 \left(f_{xxyy}\right)^{2}$

```python
disp(expand(partial_transvectDant((f, f, f), [[0, 1], [0, 1], [0, 2], [0, 2]])))
```

$\displaystyle \left(f_{xx}\right)^{2} f_{yyyy} + 2 f_{xx} f_{yy} f_{xxyy} - 4 f_{xx} f_{xy} f_{xyyy} + f_{xxxx} \left(f_{yy}\right)^{2} - 4 f_{yy} f_{xy} f_{xxxy} + 4 \left(f_{xy}\right)^{2} f_{xxyy}$

```python
disp(expand(partial_transvectant((f, f, f, f, f), [[0, 1], [0, 1], [2, 3], [2, 3], [2, 4]]) ) -2*(expand(partial_transvectant((f, f, f, f, f), [[0, 1], [1, 2], [2, 3], [3, 0], [0, 4]]) )))
```

$\displaystyle 0$

```python
disp(expand(partial_transvectant((f, f, f), [[0, 1], [0, 1], [0, 1], [0, 2]])))
```

$\displaystyle f_{x} f_{xxx} f_{yyyy} - f_{x} f_{yyy} f_{xxxy} + 3 f_{x} f_{xyy} f_{xxyy} - 3 f_{x} f_{xyyy} f_{xxy} - f_{xxx} f_{y} f_{xyyy} + f_{xxxx} f_{y} f_{yyy} - 3 f_{y} f_{xyy} f_{xxxy} + 3 f_{y} f_{xxy} f_{xxyy}$

```python
disp(expand(partial_transvectant((f, f), [[0, 1], [0, 1], [0, 1]])))
```

$\displaystyle 0$

```python
#C = transvectant(f, f, 2)
#D = -partial_transvectant((f, f, f), [[0, 1], [1, 2]])
# We are going to build these by weight, not degree.
# Hence order does not match dispaper
# Weight 4 (2 of 'em)

I4_1 = partial_transvectant((f,f),[[0,1],[0,1]]) # = C
I4_2 = partial_transvectant((f, f, f), [[0, 1], [1, 2]]) # = -D

# Weight 6 (2 of 'em)
print('weight 3:')
I6_1 = partial_transvectant((f,f,f),[[0,1],[0,1],[0,2]]) # = transvectant(f, C, 1)
I6_2 = partial_transvectant((f,f,f,f),[[0,1],[0,2],[0,3]])

# Weight 8 (7 of 'em??)
print('weight 4:')
I8_1 = expand(partial_transvectant((f,f,f),[[0,1],[0,1],[1,2],[0,2]]))
I8_2 = expand(partial_transvectant((f,f,f,f),[[0,1],[0,1],[1,2],[2,3]]))
I8_3 = expand(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[3,0]]))
I8_4 = expand(partial_transvectant((f,f,f,f),[[0,1],[1,2],[1,2],[2,3]]))

I8_5 = expand(partial_transvectant((f,f,f,f,f),[[0,1],[1,2],[2,3],[3,4]]))
I8_6 = expand(partial_transvectant((f,f,f,f,f),[[0,1],[0,2],[0,3],[3,4]]))
I8_7 = expand(partial_transvectant((f,f,f,f,f),[[0,1],[0,2],[0,3],[0,4]]))

print('weight 2')
disp(I4_1)
disp(I4_2)

print('weight 3')
disp(I6_1)
disp(expand(I6_2))

print('weight 4')
disp(I8_1)
print('')
disp(I8_2)
print('')
disp(I8_3)
print('')
disp(I8_4)
print('')
disp(I8_5)
print('')
disp(I8_6)
print('')
disp(I8_7)
```

    weight 3:
    weight 4:
    weight 2

$\displaystyle 2 f_{xx} f_{yy} - 2 \left(f_{xy}\right)^{2}$

$\displaystyle - \left(f_{x}\right)^{2} f_{yy} + 2 f_{x} f_{y} f_{xy} - f_{xx} \left(f_{y}\right)^{2}$

    weight 3

$\displaystyle - \left(f_{xx} f_{yyy} + f_{yy} f_{xxy} - 2 f_{xy} f_{xyy}\right) f_{x} + \left(f_{xx} f_{xyy} + f_{xxx} f_{yy} - 2 f_{xy} f_{xxy}\right) f_{y}$

$\displaystyle - \left(f_{x}\right)^{3} f_{yyy} + 3 \left(f_{x}\right)^{2} f_{y} f_{xyy} - 3 f_{x} \left(f_{y}\right)^{2} f_{xxy} + f_{xxx} \left(f_{y}\right)^{3}$

    weight 4

$\displaystyle 2 f_{xx} f_{yyy} f_{xxy} - 2 f_{xx} \left(f_{xyy}\right)^{2} + 2 f_{xxx} f_{yy} f_{xyy} - 2 f_{xxx} f_{yyy} f_{xy} - 2 f_{yy} \left(f_{xxy}\right)^{2} + 2 f_{xy} f_{xyy} f_{xxy}$

$\displaystyle - f_{x} f_{xx} f_{yy} f_{xyy} + f_{x} f_{xx} f_{yyy} f_{xy} - f_{x} f_{xxx} \left(f_{yy}\right)^{2} + 3 f_{x} f_{yy} f_{xy} f_{xxy} - 2 f_{x} \left(f_{xy}\right)^{2} f_{xyy} - \left(f_{xx}\right)^{2} f_{y} f_{yyy} - f_{xx} f_{y} f_{yy} f_{xxy} + 3 f_{xx} f_{y} f_{xy} f_{xyy} + f_{xxx} f_{y} f_{yy} f_{xy} - 2 f_{y} \left(f_{xy}\right)^{2} f_{xxy}$

$\displaystyle 2 \left(f_{xx}\right)^{2} \left(f_{yy}\right)^{2} - 4 f_{xx} f_{yy} \left(f_{xy}\right)^{2} + 2 \left(f_{xy}\right)^{4}$

$\displaystyle - 2 \left(f_{x}\right)^{2} f_{yyy} f_{xxy} + 2 \left(f_{x}\right)^{2} \left(f_{xyy}\right)^{2} + 2 f_{x} f_{xxx} f_{y} f_{yyy} - 2 f_{x} f_{y} f_{xyy} f_{xxy} - 2 f_{xxx} \left(f_{y}\right)^{2} f_{xyy} + 2 \left(f_{y}\right)^{2} \left(f_{xxy}\right)^{2}$

$\displaystyle \left(f_{x}\right)^{2} f_{xx} \left(f_{yy}\right)^{2} - \left(f_{x}\right)^{2} f_{yy} \left(f_{xy}\right)^{2} - 2 f_{x} f_{xx} f_{y} f_{yy} f_{xy} + 2 f_{x} f_{y} \left(f_{xy}\right)^{3} + \left(f_{xx}\right)^{2} \left(f_{y}\right)^{2} f_{yy} - f_{xx} \left(f_{y}\right)^{2} \left(f_{xy}\right)^{2}$

$\displaystyle - \left(f_{x}\right)^{3} f_{yy} f_{xyy} + \left(f_{x}\right)^{3} f_{yyy} f_{xy} - \left(f_{x}\right)^{2} f_{xx} f_{y} f_{yyy} + 2 \left(f_{x}\right)^{2} f_{y} f_{yy} f_{xxy} - \left(f_{x}\right)^{2} f_{y} f_{xy} f_{xyy} + 2 f_{x} f_{xx} \left(f_{y}\right)^{2} f_{xyy} - f_{x} f_{xxx} \left(f_{y}\right)^{2} f_{yy} - f_{x} \left(f_{y}\right)^{2} f_{xy} f_{xxy} - f_{xx} \left(f_{y}\right)^{3} f_{xxy} + f_{xxx} \left(f_{y}\right)^{3} f_{xy}$

$\displaystyle \left(f_{x}\right)^{4} f_{yyyy} - 4 \left(f_{x}\right)^{3} f_{y} f_{xyyy} + 6 \left(f_{x}\right)^{2} \left(f_{y}\right)^{2} f_{xxyy} - 4 f_{x} \left(f_{y}\right)^{3} f_{xxxy} + f_{xxxx} \left(f_{y}\right)^{4}$

```python
# Only 'weight 4' affine invariant
disp(I4_2/I4_1)

# Only 'weight 6' affine invariant
disp(I6_2/I6_1)

```

$$\frac{- \left(f_{x}\right)^{2} f_{yy} + 2 f_{x} f_{y} f_{xy} - \left(f_{y}\right)^{2} f_{xx}}{2 f_{xx} f_{yy} - 2 \left(f_{xy}\right)^{2}}$$

$$\frac{- \left(f_{x}\right)^{3} f_{yyy} + 3 \left(f_{x}\right)^{2} f_{y} f_{xyy} - 3 f_{x} \left(f_{y}\right)^{2} f_{xxy} + \left(f_{y}\right)^{3} f_{xxx}}{\left(f_{xx} f_{xyy} - 2 f_{xy} f_{xxy} + f_{yy} f_{xxx}\right) f_{y} - \left(f_{xx} f_{yyy} - 2 f_{xy} f_{xyy} + f_{yy} f_{xxy}\right) f_{x}}$$

```python
disp(partial_transvectant((f,f,f,f,f),[[0,2],[1,2],[2,3],[3,4]]))
```

$$- \left(f_{x}\right)^{3} f_{xy} f_{yyy} + \left(f_{x}\right)^{3} f_{yy} f_{xyy} + \left(f_{x}\right)^{2} f_{y} f_{xx} f_{yyy} + \left(f_{x}\right)^{2} f_{y} f_{xy} f_{xyy} - 2 \left(f_{x}\right)^{2} f_{y} f_{yy} f_{xxy} - 2 f_{x} \left(f_{y}\right)^{2} f_{xx} f_{xyy} + f_{x} \left(f_{y}\right)^{2} f_{xy} f_{xxy} + f_{x} \left(f_{y}\right)^{2} f_{yy} f_{xxx} + \left(f_{y}\right)^{3} f_{xx} f_{xxy} - \left(f_{y}\right)^{3} f_{xy} f_{xxx}$$

```python
disp(partial_transvectant((f,f,C),[[0,1],[1,2]]))
```

$$4 \left(\left(f_{x} f_{xy} - f_{y} f_{xx}\right) \left(f_{xx} f_{yyy} - 2 f_{xy} f_{xyy} + f_{yy} f_{xxy}\right) - \left(f_{x} f_{yy} - f_{y} f_{xy}\right) \left(f_{xx} f_{xyy} - 2 f_{xy} f_{xxy} + f_{yy} f_{xxx}\right)\right) f{\left (x,y \right )}$$

```python
#disp(transvectant(C, C, 2))
funcs = (C, f**2)
pairs = [[0, 1]]
disp(partial_transvectant(funcs, pairs))
```

$$4 \left(\left(f_{xx} f_{xyy} - 2 f_{xy} f_{xxy} + f_{yy} f_{xxx}\right) f_{y} - \left(f_{xx} f_{yyy} - 2 f_{xy} f_{xyy} + f_{yy} f_{xxy}\right) f_{x}\right) f{\left (x,y \right )}$$

```python
# Construct linear, quadratic, cubic forms
fx, fy, fxx, fxy, fyy, fxxx, fxxy, fxyy, fyyy = symbols('f_x, f_y, f_{xx}, f_{xy}, f_{yy}, f_{xxx}, f_{xxy}, f_{xyy}, f_{yyy}')
l = fx*x + fy*y
q = fxx*x*x + 2*fxy*x*y + fyy*y*y
c = fxxx*x*x*x + 3*fxxy*x*x*y + 3*fxyy*x*y*y + fyyy*y*y*y
```

```python
# I3 as a form (Robert's method to annoy us...)
disp(-expand(transvectant(q,transvectant(c,c,2),2)/288))
# I5
disp(expand(transvectant(transvectant(c,c,2),transvectant(c,c,2),2)/10368))
# I6
disp(transvectant(c,l**3,3)/36)
```

$$- f_{xxx} f_{xyy} f_{yy} + f_{xxx} f_{xy} f_{yyy} + f_{xxy}^{2} f_{yy} - f_{xxy} f_{xx} f_{yyy} - f_{xxy} f_{xyy} f_{xy} + f_{xx} f_{xyy}^{2}$$

```python

```

```python

```

$$- 2 f_{xx} f_{xxy} f_{yyy} + 2 f_{xx} \left(f_{xyy}\right)^{2} + 2 f_{xy} f_{xxx} f_{yyy} - 2 f_{xy} f_{xxy} f_{xyy} - 2 f_{yy} f_{xxx} f_{xyy} + 2 f_{yy} \left(f_{xxy}\right)^{2}$$

```python

```

$$2 f_{xx} f_{xxy} f_{yyy} - 2 f_{xx} \left(f_{xyy}\right)^{2} - 2 f_{xy} f_{xxx} f_{yyy} + 2 f_{xy} f_{xxy} f_{xyy} + 2 f_{yy} f_{xxx} f_{xyy} - 2 f_{yy} \left(f_{xxy}\right)^{2}$$

```python

```

$$0$$

```python
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[0,1]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[0,2]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[0,1],[0,2]])))

disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[3,0]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[3,0],[0,1]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[3,0],[0,2]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[0,1],[1,2],[1,2],[2,3]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[0,1],[1,2],[1,2],[2,3],[2,3]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[1,2],[2,3],[3,0],[0,1],[1,2]])))
```

$$0$$

$$f_{x} f_{xx} f_{xy} f_{yyy} - f_{x} f_{xx} f_{yy} f_{xyy} - 2 f_{x} \left(f_{xy}\right)^{2} f_{xyy} + 3 f_{x} f_{xy} f_{yy} f_{xxy} - f_{x} \left(f_{yy}\right)^{2} f_{xxx} - f_{y} \left(f_{xx}\right)^{2} f_{yyy} + 3 f_{y} f_{xx} f_{xy} f_{xyy} - f_{y} f_{xx} f_{yy} f_{xxy} - 2 f_{y} \left(f_{xy}\right)^{2} f_{xxy} + f_{y} f_{xy} f_{yy} f_{xxx}$$

$$0$$

$$0$$

$$2 \left(f_{xx}\right)^{2} \left(f_{yy}\right)^{2} - 4 f_{xx} \left(f_{xy}\right)^{2} f_{yy} + 2 \left(f_{xy}\right)^{4}$$

$$0$$

$$0$$

$$\left(\left(f_{xx} f_{xxyy} - 2 f_{xy} f_{xxxy} + f_{yy} f_{xxxx}\right) f_{xyy} - 2 \left(f_{xx} f_{xyyy} - 2 f_{xy} f_{xxyy} + f_{yy} f_{xxxy}\right) f_{xxy} + \left(f_{xx} f_{yyyy} - 2 f_{xy} f_{xyyy} + f_{yy} f_{xxyy}\right) f_{xxx}\right) f_{y} - \left(\left(f_{xx} f_{xxyy} - 2 f_{xy} f_{xxxy} + f_{yy} f_{xxxx}\right) f_{yyy} - 2 \left(f_{xx} f_{xyyy} - 2 f_{xy} f_{xxyy} + f_{yy} f_{xxxy}\right) f_{xyy} + \left(f_{xx} f_{yyyy} - 2 f_{xy} f_{xyyy} + f_{yy} f_{xxyy}\right) f_{xxy}\right) f_{x}$$

$$2 \left(f_{xx}\right)^{2} f_{xxyy} f_{yyyy} - 2 \left(f_{xx}\right)^{2} \left(f_{xyyy}\right)^{2} - 4 f_{xx} f_{xy} f_{xxxy} f_{yyyy} + 4 f_{xx} f_{xy} f_{xxyy} f_{xyyy} + 2 f_{xx} f_{yy} f_{xxxx} f_{yyyy} - 4 f_{xx} f_{yy} f_{xxxy} f_{xyyy} + 2 f_{xx} f_{yy} \left(f_{xxyy}\right)^{2} + 8 \left(f_{xy}\right)^{2} f_{xxxy} f_{xyyy} - 8 \left(f_{xy}\right)^{2} \left(f_{xxyy}\right)^{2} - 4 f_{xy} f_{yy} f_{xxxx} f_{xyyy} + 4 f_{xy} f_{yy} f_{xxxy} f_{xxyy} + 2 \left(f_{yy}\right)^{2} f_{xxxx} f_{xxyy} - 2 \left(f_{yy}\right)^{2} \left(f_{xxxy}\right)^{2}$$

$$- f_{xx} \left(f_{xxy}\right)^{2} f_{yyyy} + 4 f_{xx} f_{xxy} f_{xyy} f_{xyyy} - 2 f_{xx} f_{xxy} f_{yyy} f_{xxyy} - 4 f_{xx} \left(f_{xyy}\right)^{2} f_{xxyy} + 4 f_{xx} f_{xyy} f_{yyy} f_{xxxy} - f_{xx} \left(f_{yyy}\right)^{2} f_{xxxx} + 2 f_{xy} f_{xxx} f_{xxy} f_{yyyy} - 4 f_{xy} f_{xxx} f_{xyy} f_{xyyy} + 2 f_{xy} f_{xxx} f_{yyy} f_{xxyy} - 4 f_{xy} \left(f_{xxy}\right)^{2} f_{xyyy} + 10 f_{xy} f_{xxy} f_{xyy} f_{xxyy} - 4 f_{xy} f_{xxy} f_{yyy} f_{xxxy} - 4 f_{xy} \left(f_{xyy}\right)^{2} f_{xxxy} + 2 f_{xy} f_{xyy} f_{yyy} f_{xxxx} - f_{yy} \left(f_{xxx}\right)^{2} f_{yyyy} + 4 f_{yy} f_{xxx} f_{xxy} f_{xyyy} - 2 f_{yy} f_{xxx} f_{xyy} f_{xxyy} - 4 f_{yy} \left(f_{xxy}\right)^{2} f_{xxyy} + 4 f_{yy} f_{xxy} f_{xyy} f_{xxxy} - f_{yy} \left(f_{xyy}\right)^{2} f_{xxxx}$$

```python
disp(simplify(partial_transvectant((f,f,f),[[0,1],[1,2],[2,0]])))
disp(simplify(partial_transvectant((f,f,f),[[0,1],[1,2],[0,1]])))
disp(simplify(partial_transvectant((f,f,f),[[0,1],[1,2],[2,0],[0,1]])))
```

```python
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[0,1],[1,2],[1,2],[2,3]])))
disp(simplify(partial_transvectant((f,f,f,f),[[0,1],[0,1],[1,2],[1,2],[2,3],[2,3]])))
```

$$\left(\left(f_{xx} f_{xxyy} - 2 f_{xy} f_{xxxy} + f_{yy} f_{xxxx}\right) f_{xyy} - 2 \left(f_{xx} f_{xyyy} - 2 f_{xy} f_{xxyy} + f_{yy} f_{xxxy}\right) f_{xxy} + \left(f_{xx} f_{yyyy} - 2 f_{xy} f_{xyyy} + f_{yy} f_{xxyy}\right) f_{xxx}\right) f_{y} - \left(\left(f_{xx} f_{xxyy} - 2 f_{xy} f_{xxxy} + f_{yy} f_{xxxx}\right) f_{yyy} - 2 \left(f_{xx} f_{xyyy} - 2 f_{xy} f_{xxyy} + f_{yy} f_{xxxy}\right) f_{xyy} + \left(f_{xx} f_{yyyy} - 2 f_{xy} f_{xyyy} + f_{yy} f_{xxyy}\right) f_{xxy}\right) f_{x}$$

$$2 \left(f_{xx}\right)^{2} f_{xxyy} f_{yyyy} - 2 \left(f_{xx}\right)^{2} \left(f_{xyyy}\right)^{2} - 4 f_{xx} f_{xy} f_{xxxy} f_{yyyy} + 4 f_{xx} f_{xy} f_{xxyy} f_{xyyy} + 2 f_{xx} f_{yy} f_{xxxx} f_{yyyy} - 4 f_{xx} f_{yy} f_{xxxy} f_{xyyy} + 2 f_{xx} f_{yy} \left(f_{xxyy}\right)^{2} + 8 \left(f_{xy}\right)^{2} f_{xxxy} f_{xyyy} - 8 \left(f_{xy}\right)^{2} \left(f_{xxyy}\right)^{2} - 4 f_{xy} f_{yy} f_{xxxx} f_{xyyy} + 4 f_{xy} f_{yy} f_{xxxy} f_{xxyy} + 2 \left(f_{yy}\right)^{2} f_{xxxx} f_{xxyy} - 2 \left(f_{yy}\right)^{2} \left(f_{xxxy}\right)^{2}$$

```python
disp(expand(partial_transvectant((f,f,f,f,f),[[0,1],[1,2],[2,3],[3,4]])))
```

$$\left(f_{x}\right)^{2} f_{xx} \left(f_{yy}\right)^{2} - \left(f_{x}\right)^{2} \left(f_{xy}\right)^{2} f_{yy} - 2 f_{x} f_{y} f_{xx} f_{xy} f_{yy} + 2 f_{x} f_{y} \left(f_{xy}\right)^{3} + \left(f_{y}\right)^{2} \left(f_{xx}\right)^{2} f_{yy} - \left(f_{y}\right)^{2} f_{xx} \left(f_{xy}\right)^{2}$$

```python
disp(expand(partial_transvectant((f,f,f,f,f),[[0,1],[1,2],[2,3],[3,4]])))
disp(expand(partial_transvectant((f,f,f,f,f),[[0,1],[0,2],[0,3],[3,4]])))
disp(expand(partial_transvectant((f,f,f,f,f),[[0,1],[0,2],[0,3],[0,4]])))
```

$$\left(f_{x}\right)^{2} f_{xx} \left(f_{yy}\right)^{2} - \left(f_{x}\right)^{2} \left(f_{xy}\right)^{2} f_{yy} - 2 f_{x} f_{y} f_{xx} f_{xy} f_{yy} + 2 f_{x} f_{y} \left(f_{xy}\right)^{3} + \left(f_{y}\right)^{2} \left(f_{xx}\right)^{2} f_{yy} - \left(f_{y}\right)^{2} f_{xx} \left(f_{xy}\right)^{2}$$

$$- \left(f_{x}\right)^{2} f_{xx} \left(f_{yy}\right)^{2} + \left(f_{x}\right)^{2} \left(f_{xy}\right)^{2} f_{yy} + 2 f_{x} f_{y} f_{xx} f_{xy} f_{yy} - 2 f_{x} f_{y} \left(f_{xy}\right)^{3} - \left(f_{y}\right)^{2} \left(f_{xx}\right)^{2} f_{yy} + \left(f_{y}\right)^{2} f_{xx} \left(f_{xy}\right)^{2}$$

$$\left(f_{x}\right)^{3} f_{xy} f_{yyy} - \left(f_{x}\right)^{3} f_{yy} f_{xyy} - \left(f_{x}\right)^{2} f_{y} f_{xx} f_{yyy} - \left(f_{x}\right)^{2} f_{y} f_{xy} f_{xyy} + 2 \left(f_{x}\right)^{2} f_{y} f_{yy} f_{xxy} + 2 f_{x} \left(f_{y}\right)^{2} f_{xx} f_{xyy} - f_{x} \left(f_{y}\right)^{2} f_{xy} f_{xxy} - f_{x} \left(f_{y}\right)^{2} f_{yy} f_{xxx} - \left(f_{y}\right)^{3} f_{xx} f_{xxy} + \left(f_{y}\right)^{3} f_{xy} f_{xxx}$$

$$\left(f_{x}\right)^{4} f_{yyyy} - 4 \left(f_{x}\right)^{3} f_{y} f_{xyyy} + 6 \left(f_{x}\right)^{2} \left(f_{y}\right)^{2} f_{xxyy} - 4 f_{x} \left(f_{y}\right)^{3} f_{xxxy} + \left(f_{y}\right)^{4} f_{xxxx}$$

```python

```

Transvectants:

(1) We can use transvectants to create lots of invariants. More than Robert can.
(2) We understand the role of weight and degree, and have a nice graph-based picture for all the invariants up to weight 8
(3) There is no issue with SA(2), but for A(2) the weight 4 invariants have non-removable singularities. There are three possible solutions: (i) claim you only do this on parts of images, (ii) look at the weight 8 invariants instead, particularly I8_1 and I8_3 which do not have f_x in at all, or I4^2/I8, (iii) consider projection as Robert did. Our preference is (ii) but we haven't done it yet.

Moving frame:

(1) We understand it, but it is still ugly
(2) So we just write it as is

Hence:

We can build all the invariants we need now.
We still have a projection issue. And hence a high weight invariant issue. And hence a serious noise problem.

But we probably now have enough for a paper. So we should write it.


# Introduction to Logistic Regression in PyTorch

In this notebook, we're going to build a very simple neural network in PyTorch to do handwritten digit classification.
First, we'll start with some exploration of the MNIST dataset, explaining how we load and format the data.
We'll then jump into motivating and then implementing the logistic regression model, including the forward and backwards pass, loss functions, and optimizers.
After training the model, we'll evaluate how we did and visualize what we've learned.
Finally, we'll refactor our code in an object-oriented manner, using higher level APIs.

Before we get started, some imports for the packages we'll be using:

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
import torch
from tqdm.notebook import tqdm
```

### MNIST Dataset

The MNIST dataset is very popular machine learning dataset, consisting of 70000 grayscale images of handwritten digits, of dimensions 28x28.
We'll be using it as our example dataset for this section of the tutorial, with the goal being to predict which digit is in each image.

The first (and often most important) step in machine learning is preparing the data.
This can include downloading, organizing, formatting, shuffling, pre-processing, augmenting, and batching examples so that they can be fed to a model.
The `torchvision` package makes this easy by implementing many of these, allowing us to put these datasets into a usable form in only a few lines of code.
First, let's download the train and test sets of MNIST:

```python
from torchvision import datasets, transforms

mnist_train = datasets.MNIST(root="./datasets", train=True, transform=transforms.ToTensor(), download=True)
mnist_test = datasets.MNIST(root="./datasets", train=False, transform=transforms.ToTensor(), download=True)
```

```python
print("Number of MNIST training examples: {}".format(len(mnist_train)))
print("Number of MNIST test examples: {}".format(len(mnist_test)))
```

    Number of MNIST training examples: 60000
    Number of MNIST test examples: 10000

As we'd expect, 60000 of the MNIST examples are in the train set, and the rest are in the test set.
We added the transform `ToTensor()` when formatting the dataset, to convert the input data from a Pillow `Image` type into a PyTorch `Tensor`. Tensors will eventually be the input type that we feed into our model.

Let's look at an example image from the train set and its label.
Notice that the `image` tensor defaults to something 3-dimensional.
The "1" in the first dimension indicates that the image only has one channel (i.e. grayscale).
We need to get rid of this to visualize the image with `imshow`.

```python
# Pick out the 4th (0-indexed) example from the training set
image, label = mnist_train[3]

# Plot the image
print("Default image shape: {}".format(image.shape))
image = image.reshape([28,28])
print("Reshaped image shape: {}".format(image.shape))
plt.imshow(image, cmap="gray")

# Print the label
print("The label for this image: {}".format(label))
```

While we could work directly with the data as a `torchvision.dataset`, we'll find it useful to use a `DataLoader`, which will take care of shuffling and batching:

```python
train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)
test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)
```

An example of a minibatch drawn from a `DataLoader`:

```python
data_train_iter = iter(train_loader)
images, labels = data_train_iter.next()

print("Shape of the minibatch of images: {}".format(images.shape))
print("Shape of the minibatch of labels: {}".format(labels.shape))
```

    Shape of the minibatch of images: torch.Size([100, 1, 28, 28])
    Shape of the minibatch of labels: torch.Size([100])

### Logistic Regression Model

Now that we have a good feel for how to load our data, let's start putting together our model.
In this tutorial, we'll be building a logistic regression model, which is essentially a fully-connected neural network without any hidden layers.
While fairly basic, logistic regression can perform surprisingly well on many simple classification tasks.

#### The forward pass

While our data inputs (which we'll call `x`) are images (i.e. 2-dimensional), MNIST digits are pretty small, and the model we're using is very simple.
Thus, we're going to be treating the input as flat vectors.
To convert our inputs into row vectors (a.k.a. flattening), we can use `view()`, the equivalent of NumPy's `reshape()`.
Also like NumPy, we can replace one of the dimensions of the reshaping with a `-1`, which tells PyTorch to infer this dimension based on the original dimensions and the other specified dimensions.
Let's do try this flattening on the minibatch of 100 images we drew in the previous section.

```python
x = images.view(-1, 28*28)
print("The shape of input x: {}".format(x.shape))
```

    The shape of input x: torch.Size([100, 784])

To get our predicted probabilities of each digit, let's first start with the probability of a digit being a 1 like the image above.
For our simple model, we can start by applying a linear transformation.
That is, we multiply each pixel $x_i$ of the input row vector by a weight $w_{i,1}$, sum them all together, and then add a bias $b_1$.
This is equivalent to a dot product between the class "1" weights and the input:

\begin{align}
y_1 = \sum_i x_i w_{i,1} + b_1
\end{align}

The magnitude of this result $y_1$, we'll take as being correlated to our belief in how likely we think the input digit was a 1.
The higher the value of $y_1$, the more likely we think the input image $x$ was a 1 (i.e., we'd hope we'd get a relatively large value for $y_1$ for the above image).
Remember though, our original goal was to identify all 10 digits, so we actually have:

\begin{align*}
y_0 =& \sum_i x_i w_{i,0} + b_0 \\
y_1 =& \sum_i x_i w_{i,1} + b_1 \\
y_2 =& \sum_i x_i w_{i,2} + b_2 \\
y_3 =& \sum_i x_i w_{i,3} + b_3 \\
y_4 =& \sum_i x_i w_{i,4} + b_4 \\
y_5 =& \sum_i x_i w_{i,5} + b_5 \\
y_6 =& \sum_i x_i w_{i,6} + b_6 \\
y_7 =& \sum_i x_i w_{i,7} + b_7 \\
y_8 =& \sum_i x_i w_{i,8} + b_8 \\
y_9 =& \sum_i x_i w_{i,9} + b_9
\end{align*}

We can express this in matrix form as:

\begin{align}
y = x W + b
\end{align}

To take advantage of parallel computation, we commonly process multiple inputs $x$ at once, in a minibatch.
We can stack each input $x$ into a matrix $X$, giving us

\begin{align}
Y = X W + b
\end{align}

Visualizing the dimensions:

In our specific example, the minibatch size $m$ is $100$, the dimension of the data is $28 \times 28=784$, and the number of classes $c$ is $10$.
While $X$ and $Y$ are matrices due to the batching, conventionally, they are often given lowercase variable names, as if they were for a single example.
We will use `x` and `y` throughout.

The weight $W$ and bias $b$ make up the parameters of this model.
When we say that we want to "learn the model," what we're really trying to do is find good values for every element in $W$ and $b$.
Before we begin learning, we need to initialize our parameters to some value, as a starting point.
Here, we don't really know what the best values are, so we going to initialize $W$ randomly (using something called [Xavier initialization](http://proceedings.mlr.press/v9/glorot10a.html)), and set $b$ to a vector of zeros.

```python
# Randomly initialize weights W
W = torch.randn(784, 10)/np.sqrt(784)
W.requires_grad_()

# Initialize bias b as 0s
b = torch.zeros(10, requires_grad=True)
```

As both `W` and `b` are parameters we wish to learn, we set `requires_grad` to `True`.
This tells PyTorch's autograd to track the gradients for these two variables, and all the variables depending on `W` and `b`.

With these model parameters, we compute $y$:

```python
# Linear transformation with W and b
y = torch.matmul(x, W) + b
```

We can see for example what the predictions look like for the first example in our minibatch. Remember, the bigger the number, the more the model thinks the input $x$ is of that class.

```python
print(y[0,:])
```

    tensor([-0.0530, -0.9864,  0.2400, -0.1909, -0.3135, -0.2814, -0.1149, -0.2485,
             0.1927, -0.2939], grad_fn=<SliceBackward>)

We can interpret these values (aka logits) $y$ as probabilities if we normalize them to be positive and add up to 1. In logistic regression, we do this with a softmax:

\begin{align}
p(y_i) = \text{softmax}(y_i) = \frac{\text{exp}(y_i)}{\sum_j\text{exp}(y_j)}
\end{align}

Notice that because the range of the exponential function is always non-negative, and since we're normalizing by the sum, the softmax achieves the desired property of producing values between 0 and 1 that sum to 1. If we look at the case with only 2 classes, we see that the softmax is the multi-class extension of the binary sigmoid function:

We can compute the softmax ourselves using the above formula if we'd like, but PyTorch already has the softmax function in `torch.nn.functional`:

```python
# Option 1: Softmax to probabilities from equation
py_eq = torch.exp(y) / torch.sum(torch.exp(y), dim=1, keepdim=True)
print("py[0] from equation: {}".format(py_eq[0]))

# Option 2: Softmax to probabilities with torch.nn.functional
import torch.nn.functional as F
py = F.softmax(y, dim=1)
print("py[0] with torch.nn.functional.softmax: {}".format(py[0]))
```

    py[0] from equation: tensor([0.1111, 0.0437, 0.1490, 0.0968, 0.0856, 0.0884, 0.1045, 0.0914, 0.1421,
            0.0873], grad_fn=<SelectBackward>)
    py[0] with torch.nn.functional.softmax: tensor([0.1111, 0.0437, 0.1490, 0.0968, 0.0856, 0.0884, 0.1045, 0.0914, 0.1421,
            0.0873], grad_fn=<SelectBackward>)

We've now defined the forward pass of our model: given an input image, the graph returns the probabilities the model thinks the input is each of the 10 classes. Are we done?

#### The cross-entropy loss

This tutorial isn't done yet, so you can probably guess that the answer is not quite.
We don't know the values of $W$ and $b$ yet!
Remember how we initialized them randomly?
Before we adjust any of the weights, we need a way to measure how the model is doing.
Specifically, we're going to measure how badly the model is doing.
We do this with a *loss* function, which takes the model's prediction and returns a single number (i.e. a scalar) summarizing model performance.
This loss will inform how we adjust the parameters of the model.

The loss we commonly use in classification is cross-entropy, a concept from information theory.
Explaining exactly what the cross-entropy represents goes slightly beyond the scope of this course, but you can think of it as a way of quantifying how far apart one distribution $y'$ is from another $y$.

\begin{align}
H_{y'}(y)=-\sum_i y'_i \text{log}(y_i)
\end{align}

In our case, $y$ is the set of probabilities predicted by the model ($py$ above); $y'$ is the target distribution.
What is the target distribution?
It's the true label, which is what we wanted the model to predict.

Cross-entropy not only captures how *correct* (max probability corresponds to the right answer) the model's answers are, it also accounts for how *confident* (high confidence in correct answers) they are. This encourages the model to produce very high probabilities for correct answers while driving down the probabilities for the wrong answers, instead of merely being satisfied with it being the argmax.

We focus here on supervised learning, a setting in which we have the labels.
Our `DataLoader` automatically includes the corresponding labels for each of our inputs.
Here are the labels from the first time we retrieved a minibatch:

```python
print(labels.shape)
```

    torch.Size([100])

Like the softmax operation, we can implement the cross-entropy directly from the equation, using the softmax output.
However, as with the softmax, `torch.nn.functional` already has the cross-entropy loss implemented as well.

```python
# Cross-entropy loss from equation
cross_entropy_eq = torch.mean(-torch.log(py_eq)[range(labels.shape[0]),labels])
print("cross entropy from equation: {}".format(cross_entropy_eq))

# Option 2: cross-entropy loss with torch.nn.functional
cross_entropy = F.cross_entropy(y, labels)
print("cross entropy with torch.nn.functional.cross_entropy: {}".format(cross_entropy))
```

    cross entropy from equation: 2.3115861415863037
    cross entropy with torch.nn.functional.cross_entropy: 2.311586618423462

Note that PyTorch's cross-entropy loss combines the softmax operator and cross-entropy into a single operation, for numerical stability reasons.
Don't do the softmax twice!
Make sure to feed in the pre-softmax logits `y`, not the post-softmax probabilities `py`.

#### The backwards pass

Now that we have the loss as a way of quantifying how badly the model is doing, we can improve our model by changing the parameters in a way that minimizes the loss.
For neural networks, the common way of doing this is with backpropagation: we take the gradient of the loss with respect to $W$ and $b$ and take a step in the direction that reduces our loss.

If we were not using a deep learning framework like PyTorch, we would have to go through and derive all the gradients ourselves by hand, then code them into our program.
We certainly still could.
However, with modern auto-differentiation libraries, it's much faster and easier to let the computer do it.

First, we need to create an optimizer.
There are many choices, but since logistic regression is fairly simple, we'll use standard stochastic gradient descent (SGD), which makes the following update:

\begin{align}
\theta_{t+1} = \theta_t - \alpha \nabla_\theta \mathcal{L}
\end{align}

where $\theta$ is a parameter, $\alpha$ is our learning rate (step size), and $\nabla_\theta \mathcal{L}$ is the gradient of our loss with respect to $\theta$.

```python
# Optimizer
optimizer = torch.optim.SGD([W,b], lr=0.1)
```

When we created our parameters $W$ and $b$, we indicated that they require gradients.
To compute the gradients for $W$ and $b$, we call the `backward()` function on the cross-entropy loss.

```python
cross_entropy.backward()
```

Each of the variables that required gradients have now accumulated gradients.
We can see these for example on `b`:

```python
b.grad
```

    tensor([-0.0433, -0.0318,  0.0345, -0.0162,  0.0438,  0.0039,  0.0375, -0.0498,
             0.0175,  0.0038])

To apply the gradients, we could manually update $W$ and $b$ using the update rule $\theta_{t+1} = \theta_t - \alpha \nabla_\theta \mathcal{L}$, but since we have an optimizer, we can tell it to perform the update step for us:

```python
optimizer.step()
```

We set our learning rate to 0.1, so `b` has been updated by `-0.1*b.grad`:

```python
b
```

    tensor([ 0.0043,  0.0032, -0.0034,  0.0016, -0.0044, -0.0004, -0.0038,  0.0050,
            -0.0018, -0.0004], requires_grad=True)

We've now successfully trained on a minibatch!
However, one minibatch probably isn't enough.
At this point, we've trained the model on 100 examples out of the 60000 in the training set.
We're going to need to repeat this process, for more of the data.

One more thing to keep in mind though: gradients calculated by `backward()` don't override the old values; instead, they accumulate.
Therefore, you'll want to clear the gradient buffers before you compute gradients for the next minibatch.

```python
print("b.grad before zero_grad(): {}".format(b.grad))
optimizer.zero_grad()
print("b.grad after zero_grad(): {}".format(b.grad))
```

    b.grad before zero_grad(): tensor([-0.0433, -0.0318,  0.0345, -0.0162,  0.0438,  0.0039,  0.0375, -0.0498,
             0.0175,  0.0038])
    b.grad after zero_grad(): tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])

#### Model Training

To train the model, we just need repeat what we just did for more minibatches from the training set.
As a recap, the steps were:
1. Draw a minibatch
2. Zero the gradients in the buffers for `W` and `b`
3. Perform the forward pass (compute prediction, calculate loss)
4. Perform the backward pass (compute gradients, perform SGD step)

Going through the entire dataset once is referred to as an epoch.
In many cases, we train neural networks for multiple epochs, but here, a single epoch is enough.
We also wrap the train_loader with `tqdm`.
This isn't neccessary, but it adds a handy progress bar so we can track our training progress.

```python
# Iterate through train set minibatchs
for images, labels in tqdm(train_loader):
    # Zero out the gradients
    optimizer.zero_grad()

    # Forward pass
    x = images.view(-1, 28*28)
    y = torch.matmul(x, W) + b
    cross_entropy = F.cross_entropy(y, labels)
    # Backward pass
    cross_entropy.backward()
    optimizer.step()
```

    HBox(children=(FloatProgress(value=0.0, max=600.0), HTML(value='')))

#### Testing

Now let's see how we did!
For every image in our test set, we run the data through the model, and take the digit in which we have the highest confidence as our answer.
We then compute an accuracy by seeing how many we got correct.
We're going to wrap evaluation with `torch.no_grad()`, as we're not interested in computing gradients during evaluation.
By turning off the autograd engine, we can speed up evaluation.

```python
correct = 0
total = len(mnist_test)

with torch.no_grad():
    # Iterate through test set minibatchs
    for images, labels in tqdm(test_loader):
        # Forward pass
        x = images.view(-1, 28*28)
        y = torch.matmul(x, W) + b

        predictions = torch.argmax(y, dim=1)
        correct += torch.sum((predictions == labels).float())

print('Test accuracy: {}'.format(correct/total))
```

    HBox(children=(FloatProgress(value=0.0), HTML(value='')))

    Test accuracy: 0.9010999798774719

Not bad for a simple model and a few lines of code.
Before we conclude this example, there's one more interesting thing we can do.
Normally, it can be difficult to inspect exactly what the filters in a model are doing, but since this model is so simple, and the weights transform the data directly to their logits, we can actually visualize what the model's learning by simply plotting the weights.
The results look pretty reasonable:

```python
# Get weights
fig, ax = plt.subplots(1, 10, figsize=(20, 2))

for digit in range(10):
    ax[digit].imshow(W[:,digit].detach().view(28,28), cmap='gray')
```

As we can see, the model learned a template for each digit.
Remember that our model takes a dot product between the weights of each digit and input.
Therefore, the more the input matches the template for a digit, the higher the value of the dot product for that digit will be, which makes the model more likely to predict that digit.

#### The Full Code

The entire model, with the complete model definition, training, and evaluation (but minus the weights visualization) as independently runable code:

```python
import numpy as np
import torch
import torch.nn.functional as F
from torchvision import datasets, transforms
from tqdm.notebook import tqdm

# Load the data
mnist_train = datasets.MNIST(root="./datasets", train=True, transform=transforms.ToTensor(), download=True)
mnist_test = datasets.MNIST(root="./datasets", train=False, transform=transforms.ToTensor(), download=True)
train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)
test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)

## Training
# Initialize parameters
W = torch.randn(784, 10)/np.sqrt(784)
W.requires_grad_()
b = torch.zeros(10, requires_grad=True)

# Optimizer
optimizer = torch.optim.SGD([W,b], lr=0.1)

# Iterate through train set minibatchs
for images, labels in tqdm(train_loader):
    # Zero out the gradients
    optimizer.zero_grad()

    # Forward pass
    x = images.view(-1, 28*28)
    y = torch.matmul(x, W) + b
    cross_entropy = F.cross_entropy(y, labels)
    # Backward pass
    cross_entropy.backward()
    optimizer.step()

## Testing
correct = 0
total = len(mnist_test)

with torch.no_grad():
    # Iterate through test set minibatchs
    for images, labels in tqdm(test_loader):
        # Forward pass
        x = images.view(-1, 28*28)
        y = torch.matmul(x, W) + b

        predictions = torch.argmax(y, dim=1)
        correct += torch.sum((predictions == labels).float())

print('Test accuracy: {}'.format(correct/total))
```

    HBox(children=(FloatProgress(value=0.0, max=600.0), HTML(value='')))

Note: The accuracy from the full version directly above might return a slightly different test accuracy from the step-by-step version we first went through.
We trained our model with stochastic gradient descent (SGD), with the word "stochastic" highlighting that training is an inherently random process.

### Higher level APIs

So far, we've primarily been building neural networks with fairy basic PyTorch operations.
We did this to provide a clearer picture of how models actually work and what's going on under the hood.
This can be important as you're learning concepts and the various frameworks, and sometimes the low-level control is necessary if you're trying to build something novel.

However, most of the time, we do find ourselves repeating the same fairly standard lines of code, which can slow us down.
Worse, it clutters up our code unnecessarily and introduces room for bugs and typos.
And finally, as researchers or engineers, we would like to spend most of our time thinking on the highest levels of abstractions: I want to add a convolution layer here, then a fully-connected there, etc.
Having to code all the small details are distractions that can detract from our ability to translate ideas into code.
For this reason, PyTorch has higher level abstractions to help speed up implementation and improve model organization.
While there are many ways to organize PyTorch code, one common paradigm is with `torch.nn.Module`.

#### Object-oriented Refactorization

It often makes sense for us to code our models in an [object-oriented manner](https://realpython.com/python3-object-oriented-programming/).
To understand why, let's look back at the linear transformation $y = xW + b$ that we used for logistic regression.
We can see that while the operation consisted of a matrix multiplication and addition, also associated with this operation was the instantiation of two parameters `W` and `b`, and these two parameters conceptually *belong* to the transform.
As such, it would make sense to bundle up the instantiation of the two parameters with the actual transformation:

```python
# Note: illustrative example only; see below for torch.nn usage
class xW_plus_b:
    def __init__(self, dim_in, dim_out):
        self.W = torch.randn(dim_in, dim_out)/np.sqrt(dim_in)
        self.W.requires_grad_()
        self.b = torch.zeros(dim_out, requires_grad=True)

    def forward(self, x):
        return torch.matmul(x, self.W) + self.b
```

To use what we just wrote, we can create an `xW_plus_b` instance using its `__init__()` method (the constructor).
In this case, we're going to set the dimensions to be 784 and 10, as we did in our logisitic regression example above.
This creates an `xW_plus_b` instance with two parameters `W` and `b`.

```python
# Note: illustrative example only; see below for torch.nn usage
lin_custom = xW_plus_b(784, 10)
print("W: {}".format(lin_custom.W.shape))
print("b: {}".format(lin_custom.b.shape))
```

After instantiating the instance, we can perform the actual linear transform of our custom `xW_plus_b` class by calling the instance's `forward()` function:

```python
# Note: illustrative example only; see below for torch.nn usage
x_rand = torch.randn(1,784)
y = lin_custom.forward(x_rand)
print(y.shape)
```

#### Using  `torch.nn`

While we can certainly implement our own classes for the operations we'd like to use, we don't have to, as PyTorch already has them in the `torch.nn` sublibrary.

```python
import torch.nn as nn
```

For example, the linear transform example we just went through is called `torch.nn.Linear`:

```python
lin = nn.Linear(784, 10)
print("Linear parameters: {}".format([p.shape for p in lin.parameters()]))

y = lin(x_rand)
print(y.shape)
```

The implementation for `nn.Linear` has a few more things under the hood (notice for example that the `forward()` function is aliased with calling the instance itself), but in spirit, it operates in much the same way as our custom `xW_plus_b` class.
In the first line, we instantiate a `Linear` object, which automatically creates weight and bias variables of the specified dimensions.
The fourth line then calls the `forward()` function (aliased with the object call), which performs the linear transformation.

#### Using `torch.nn.Module`

The `torch.nn.Linear` class we just saw is a subclass of `torch.nn.Module`.
However, `Module`s do not have to just describe a single operation; they can also define a chain of operations, each of which may also be `Module`s.
As such, we can place our entire neural network within a `Module`.
In this case, the module can track all of its associated parameters, some of which may also be associated with a submodule (e.g. `nn.Linear`), while also defining the `forward()` function, in one place.

```python
class MNIST_Logistic_Regression(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(784, 10)

    def forward(self, x):
        return self.lin(x)
```

In this particular example, we didn't need to chain any operations, but we'll see this come in handy as we move on to more complex models.
Additionally, the `nn.Module` that we subclassed has a few other nice features.
For example:
- The `forward()` function of a `nn.Module` will call the `forward()` function of any child `nn.Module`s.
- `print()` will print out a formatted summary of our model, recursively summarizing any child `nn.Module`s as well.
- The `parameters()` function will return a generator that returns all parameters of a `nn.Module` (including those of any children).

```python
model = MNIST_Logistic_Regression()
y = model(x_rand)
print("The model: \n{}".format(model))
print("\nParameters: \n{}".format(list(model.parameters())))
print("\nOutput shape: \n{}".format(y.shape))
```

#### Full code with `nn.Module`

Refactoring our previous complete logistic regression code to use a `nn.Module`:

```python
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
from tqdm.notebook import tqdm

class MNIST_Logistic_Regression(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin = nn.Linear(784, 10)

    def forward(self, x):
        return self.lin(x)

# Load the data
mnist_train = datasets.MNIST(root="./datasets", train=True, transform=transforms.ToTensor(), download=True)
mnist_test = datasets.MNIST(root="./datasets", train=False, transform=transforms.ToTensor(), download=True)
train_loader = torch.utils.data.DataLoader(mnist_train, batch_size=100, shuffle=True)
test_loader = torch.utils.data.DataLoader(mnist_test, batch_size=100, shuffle=False)

## Training
# Instantiate model
model = MNIST_Logistic_Regression()

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

# Iterate through train set minibatchs
for images, labels in tqdm(train_loader):
    # Zero out the gradients
    optimizer.zero_grad()

    # Forward pass
    x = images.view(-1, 28*28)
    y = model(x)
    loss = criterion(y, labels)
    # Backward pass
    loss.backward()
    optimizer.step()

## Testing
correct = 0
total = len(mnist_test)

with torch.no_grad():
    # Iterate through test set minibatchs
    for images, labels in tqdm(test_loader):
        # Forward pass
        x = images.view(-1, 28*28)
        y = model(x)

        predictions = torch.argmax(y, dim=1)
        correct += torch.sum((predictions == labels).float())

print('Test accuracy: {}'.format(correct/total))
```

While the benefits of organizing a model as a `nn.Module` may not be as obvious for a simple logistic regression model, such a programming style allows for much quicker and cleaner implementations for more complex models, as we'll see in later notebooks.


#### _Speech Processing Labs 2020: Signals: Module 2_

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

from dspMisc import *
```

# 3 Impulse Source, IIR Filter

### Learning Outcomes
* Be able to explain the difference between an FIR and an IIR filter
* Understand that IIR filters can be applied to impulse trains to produce different spectral envelopes

### Need to know
* Topic Videos: Impulse Response, Source Filter Model
* [Building the source: impulse trains](sp-m2-1-impulse-as-source.ipynb)
* [FIR filters](sp-m2-2-fir-filters.ipynb)

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 3.0 Filters again

The previous notebook explored Finite Impulse Response (FIR) filters which took a
 an input sequence $x[n]$ to give us some desired output sequence $y[n]$.  We saw that these FIR filters basically applied a weighted average across the input sequence and that this could be characterised in terms of convolution.
Now, we'll look at Infinite Impulse Response (IIR) filters, which allow us to model more complex feedback behaviour that's closer to what we might expect from the vocal tract.

## 3.1 Definition of an Infinite Impulse Response filter

Unlike FIR filters, IIR filters use outputs from previous time steps to calculate the next output.

$$ y[n] = \sum_{k=0}^N b[k] x[n-k] + \sum_{k=0}^M a[k] y[n-k]$$

The 'infinite' in the name comes from the fact that you can define a filter in this way so that a single impulse results in an infinitely long non-zero output sequence.

For example

$$ y[n] =  x[n] + y[n-1] $$

if $x[n] = 1$ for $n=1$ and 0 otherwise, then

* $ y[1] = 1 + 0  = 1$
* $ y[2] = 0 + 1 = 1$
* $ y[3] = 0 + 1 = 1$

i.e. $y[n] = 1$ for all $n$ to infinity!

These can be much more efficient than using an FIR filter: we can get the same effect with fewer parameters. (e.g. in some cases you can define a 4th order FIR filter that does the same job as a 19th order FIR filter). The flip side is that they're harder to analyze.

Unfortunately, there's no direct way to compute the a(k)'s and b(k)'s from the impulse response for an IIR filter, but we can use the **z Transform** to do this (we won't go into it in this class).  Instead we'll focus on how we can use an IIR filter to create a speech shaped sound.

## 3.2 Using an IIR filter to shape the spectral envelope

### Let's start with an impulse train

```python
N=400
f_s = 8000
t_s = 1/f_s

## Set the impulse train frequency in Hz
frequency = 100

## Generate N samples of an impulse train with frequency=frequency given a sampling rate of f_s
x_imp, time_steps = make_impulse_train(sample_rate=f_s, frequency=frequency, n_samples=N)

## plot and label
fig, ax = plt.subplots(figsize=(16, 4))
ax.set(ylim=(-1, 2))
ax.scatter(time_steps, x_imp)
ax.plot(time_steps, x_imp)
ax.set_xlabel("Time (s)")
ax.set_ylabel("Amplitude")
ax.set_title("An impulse train")
```

### Now let's define a function to calculate the output of an IIR filter given feedback coefficients `a` and feedforward coefficients `b`.

```python
# iir_filter applies the IIR filter defined by feedforward coefficients, b, and feedback coefficients, a,
# where the coefficients are input as ordered lists
def iir_filter(x, a=[], b=[]):

    ## Get the size of the input
    ## Note the size of N will depend on how we deal with edge cases

    N = len(x)

    ## Initialize our output array, y, to all zeros
    y = np.zeros(N)

    ## step through each input index n
    for n in range(N):
        y[n] = 0

        ## get the contribution from the inputs
        for k in range(len(b)):
            if n-k >= 0:
                ## multiply the previous inputs with filter coefficients, b, and add them up
                ## just like an FIR filter
                y[n] = y[n] + (b[k]*x[n-k])

        ## get contribution from the existing outputs
        for l in range(len(a)):
            ## Note: We don't want to include the current output in this calculation!

            if n-l > 0:
                ## multiply the previous outputs with filter coefficients, a, and add them up
                y[n] = y[n] + (a[l]*y[n-l-1])

    return y

```

### Apply the IIR filter to our impulse train

```python
## An impulse train with amplified impulse to make things easier to see.
## It doesn't change the shape of the frequency response
x = x_imp * 10

## Note coefficients opposite sign to what Simon has in his slides.  Maybe a matlab thing?
## Or a notation thing to make transfer function look nicer?
a = [3.22666099, -4.3967485, 3.03596532, -0.88529281]
b = [1]

## You might like to try this Low pass filter from Lyon's DSP textbook
#a=[1.194, -0.436]
#b=[0.0605, 0.121, 0.0605]

y_iir = iir_filter(x, a=a, b=b)

```

### Plot the filtered output in the time domain

```python
## Plot the filtered output
fig, ax = plt.subplots(figsize=(16, 4))
ax.plot(range(N), y_iir)
print(N, np.max(y_iir))
ax.set_xlabel("Time (s)")
ax.set_ylabel("Amplitude")
ax.set_title("Impulse train transformed by an IIR filter")

## Starting to look more speech...ish
```

Now we're starting to make something that looks like a speech waveform!

### Now let's look at the DFT of the filter's output

```python
## Get the DFT outputs of the IIR filter's output
mags, phases = get_dft_mag_phase(y_iir, N)

## Plot the DFT frequencies rather than DFT output indices
dft_freqs = (f_s/N) * np.arange(N)

## Plot the magnitude spectrum
fig, ax = plt.subplots(figsize=(16, 4))
ax.plot(dft_freqs[0:int(N/2)], mags[0:int(N/2)])
ax.set_xlabel("Frequency (Hz))")
ax.set_ylabel("Magnitude")
ax.set_title("Frequency response of IIR filter output (sample rate: %d samples/sec)" % f_s)

```

### Exercise

* What happens to the filtered signal if we alter the filter coefficients:
    * Try out the filter coefficients in the cell below and describe what sort of effect they have.
    * What sort of frequencies do they boost and which do they dampen?
    * Do any look like low-pass or high-pass filters, for example?
    * Do all of them produce outputs that look like vocal pulses?

* If you play around the coefficients it's quite easy to end up with an output that doesn't 'fade out' over time.  What happens to the DFT in this case?

```python
## Try out these different sets of IIR filter coefficients
## ex 1
a = [0.8]
b = [1]

## ex 2
a = [-0.8]
b = [1,-1]

## ex 3:
a = [0.566, -0.16017]
b = [1, 1.414, 1]

## ex 4:
a = [0.566, -0.16017]
b = [1, 0.414, -0.414, -1]

## ex 5:
a = [1.272, -0.80899]
b = [1, 0.414, -0.414, -1]

## ex 6: unstable (unbounded) but still gives us a frequency response like the one previous
a = [1.554, -1.207]
b = [1, 0.414, -0.414, -1]

```

```python
## set which coeffients you want to use here.
## e.g. ex 1
a = [1.554, -1.207]
b = [1, 0.414, -0.414, -1]

## the impulse train
x = x_imp * 10
y_iir = iir_filter(x, a=a, b=b)

## Plot the filtered output
fig, ax = plt.subplots(figsize=(16, 4))
ax.plot(range(N), y_iir)
ax.set_xlabel("Time (s)")
ax.set_ylabel("Amplitude")
ax.set_title("Impulse train transformed by an IIR filter: a=%s, b=%s" % (repr(a), repr(b)))

#print(N, np.max(y_iir))

## Plot the DFT frequencies rather than DFT output indices
mags, phases = get_dft_mag_phase(y_iir, N)
dft_freqs = (f_s/N) * np.arange(N)

## Plot the magnitude spectrum
fig, ax = plt.subplots(figsize=(16, 4))
#ax.scatter(dft_freqs[0:int(N/2)],mags[0:int(N/2)])
ax.plot(dft_freqs[0:int(N/2)], mags[0:int(N/2)])
ax.set_xlabel("Frequency (Hz))")
ax.set_ylabel("Magnitude")
ax.set_title("Frequency response of IIR filter output: a=%s, b=%s" % (repr(a), repr(b)))

```

### Notes

```python
# a = [0.8], b = [1]
# lowpass filter, doens't look like speech

# a = [-0.8], b = [1,-1]
# highpass filter, kind of looks like speech

# a = [0.566, -0.16017], b = [1, 1.414, 1]
# lowpass filter, doens't look like speech

# a = [0.566, -0.16017], b = [1, 0.414, -0.414, -1]
# gentle peak near the middle, smooth increase and decrease, doesn't look like speech

# a = [1.272, -0.80899] b = [1, 0.414, -0.414, -1]\
# sharp peak near 1000, looks like speech

# a = [1.554, -1.207], b = [1, 0.414, -0.414, -1]
# sharp peak around 1000 but the curve is smooth, doesn't look like speech
```

---

## 3.3 (Optional Extras) Designing a filter using poles and zeros

This section is very much optional extension. The following gives a very, very terse overview of how you can design an IIR filter frequency response using the concept of poles and filters. This requires a fair grip on solving polynomials and complex numbers, but once you've got that it's actually quite nice!

If you're interested in learning more about this (or signal processing in general), it's probably time to pick up a Digital Signal Processing textbook. Rick Lyon's [Understanding Digital Signal Processing](https://www.dsprelated.com/books/6.php) is probably the most accessible I've come across. The example filters above were taken from that book. The end of this section basically gives a excessively detailed derivation going from poles and filters to coefficients for those examples.

Previous SLP students have also recommended [The Scientist and Engineer's Guide to
Digital Signal Processing](http://www.dspguide.com/pdfbook.htm), though the free online version renders a bit weirdly!.

IIR filters have the following form:

$$ y[n] = \sum_{k=0}^N b(k) x[n-k] + \sum_{k=0}^M a(k) y[n-k]$$

Unlike an FIR filter, we can't just put an impulse in and find out what the frequency response of the filter is.  Instead, to understand the properties of an IIR filter, we use the **z-transform**.  The z-transform is the discrete version of the continuous Laplace transform.  We won't get into the details here, but if you do read more about this you'll see that you can think of the Laplace transform as a generalization of the Fourier Transform. So the intuitions you've developed about how the Fourier Transform works (e.g. finding correlations between inputs and basis functions) will help understand the Laplace Transform too.

We use the z-transform to convert filter difference equations to an equivalent form in the z-domain (z being a complex number).  We rewrite our difference equation in terms of the z-transformed input $x[n]$ and output $y[n]$ functions, which we denote as $X(z)$ and $Y(z)$ respectively.  Some algebra then allows us to define a **transfer function** $H(z)$:

$$ Y(z) = H(z)X(z)$$

So, $H(z)$ is what our filter does to our input to get our output in the z-domain. You can think of it as the filter part of the source filter model (the source being our impulse train and/or white noise).

Unsurprisingly, we can actually define $H(z)$ in terms of our the feedforward coefficients $b(k)$ and the feedback coefficients $a(k)$. What's surprising is that we can define it in a relatively simple way:

$$ H(z) = \frac{Y(z)}{X(z)} = \frac{\sum_{k=0}^N b(k)z^{-k}}{1- \sum_{k=1}^M a(k)z^{-k}}\\ $$

You can think of  $z^{-k}$ as referring to a position in the sequence $k$ steps back in time.

We can analyze $H(z)$ to figure out how the filter is going to behave. To do this we need to factorize the numerator and denominator polynomials of $H(z)$.

$$
\begin{align}
H(z) &= \frac{\sum_{k=0}^N b(k)z^{-k}}{1- \sum_{k=1}^M a(k)z^{-k}}\\
&= G\frac{(z-z_0)(z-z_1)...(z-z_N)}{(z-p_0)(z-p_1)...(z-p_M)}
\end{align}
$$

In the equation above, $G$ is an arbitrary gain constant which doesn't affect the frequency response or filter stability. The polynomial roots, $z_n$ and $p_m$ are called *zeros* and *poles* respectively.  These are the (complex) values of $z$ which cause the numerator (zeros) or the denominator (poles) to equal 0.  We can interpret them as follows:

* **Zeros:** when the numerator goes to zero, the transfer function $H(z)$ goes to zero, and we see decrease (attenuation) in the frequency magnitude response.
* **Poles:** when the denominator goes to zero, the transfer function $H(z)$ goes to infinity, leads to an increase (gain) the frequency magnitude response (like vowel formants!)

It may not look like it right now, but it's way easier to analyze IIR filters in the z-domain than in the time domain! Once we know the poles and the zeros of the transfer function, we basically know what the frequency response will look like and whether the filter is stable or not.  The main things to note are:

* If a pole falls inside the unit-circle (of the complex z-plane), the filter output will be **stable**.
    * Given an impulse input, $y[n]$ it will eventually go to zero as $n$ increases.
* If a pole falls outside the unit circle, the output will be **unstable**
    * Given an impulse input, $y[n]$ will keep increasing in magnitude as more time passes (definitely not the behaviour of the vocal tract!).
* If a pole lies exactly on the unit circle, the output will oscillate between finite values determined by the pole locations forever.

An IIR filter is only as stable as it's weakest link - if any poles sit outside the unit circle, the filter as a whole will be unstable.  Zeros don't affect filter stability.

Another property of poles (and zeros) is that, if the denominator polynomial has real-valued coefficients (i.e. the $a(k)$ values), poles (and zeros) either come in pairs of complex conjugates (i.e. if $p=a+ib$ is a pole, so is $p^*=a-ib$), or it's a value on the real axis of the z-plane.  We can use these facts to design filters using the factorized form of the transfer function, which will then allow us to derived the filter coefficients $a(k)$ and $b(k)$. We'll see some examples this below.

The frequency response of the filter is $H(z)$ evaluated at $e^{j\omega}$, where $-\pi \lt \omega \lt \pi$ represents frequencies in terms of radians per sample ($\pi$ radians/sample $\mapsto$ a frequency matching half the sampling frequency (i.e. $f_s/2$).

$$ H(\omega) = H(e^{j\omega}) = \frac{\sum_{k=0}^N b(k)e^{-jk\omega}}{1 -\sum_{k=0}^M a(k)e^{-jk\omega}} $$

These example IIR filters from Richard J Lyon's book
[Understanding Digital Signal Processing](https://www.dsprelated.com/books/6.php), which is a very nice introduction to the topic.

Working through from the factorized form to the coefficients is not hard, but it is very annoying and it's very easy to make mistakes, so I've given all the steps expanding out the factorized forms of the filter examples he gives in his book below.

### Ex 1:

$$
\begin{aligned}
H_1(z) &= \frac{G}{z-0.8}\\
       &= G\frac{1}{1-(0.8z^{-1})}
\end{aligned}
$$

* One pole on the real axis, inside the unit circle, stable peak at $\omega=0$ rad/sample
* no zeros

```python
a = [0.8]
b = [1]
```

### Ex 2:

$$
\begin{aligned}
H_1(z) &= G\frac{(z-1)}{z-(-0.8)}\\
       &= G\frac{1 - z^{-1}}{1 -  (-0.8z^{-1})}
\end{aligned}
$$

* pole on real axis at -0.8, inside the unit circle, stable peak at $\omega=-\pi$ rad/sample, i.e. $f_s/2$ Hz which is 8000/2 = 4000 Hz in our setup above.
* one zero at z=1, this causes attenuation at $\omega = 0$ rads/sample (0 Hz)

```python
a = [-0.8]
b = [1,-1]
```

### Ex 3
$$
\begin{align}
H_3(z) &= G\frac{[z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.283+j0.283)][z-(0.283-j0.283)]} \\
&= G\frac{z^2 -z(-0.707 - j0.707) - z(-0.707 + j0.707) + (-0.707 + j0.707)(-0.707 - j0.707) }{z^2 - z(0.283-j0.283) -z(0.283+j0.283) + (0.283+j0.283)0.283-j0.283)} \\
&= G\frac{z^2 +z0.707 + j0.707z + z0.707 - j0.707z + 1 }{z^2 - 0.283z + j0.283z - 0.283z - j0.283z + 0.16017} \\
&= G\frac{z^2 +z[0.707  + 0.707] + 1 }{z^2 - 0.283z - 0.283z  + 0.16017} \\
&= G\frac{z^2 + 1.414z + 1 }{z^2 - 0.566z + 0.16017} \\
&= G\frac{1 + 1.414z^{-1} + 1z^{-2} }{1 - 0.566z^{-1} + 0.16017z^{-2}} \\
\end{align}
$$

* The poles are inside the unit circle, this produces a low pass filter  (pass band is 'bordered' by +/- $\omega = \pi/4$ rad/sample, 1000 Hz)
    * $(0.283+j0.283) \mapsto 0.400e^{j\pi/4}$
* zeros on the unit circle, attenuation at $3\pi/4$ rad/sample = 3000 Hz.
    * $(-0.707+j0.707) \mapsto 1e^{j3\pi/4} $

```python
a = [0.566, -0.16017]
b = [1, 1.414, 1]

```

### Ex 4:
$$
\begin{align}
H_4(z) &= G\frac{[z-1][z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.283+j0.283)][z-(0.283-j0.283)]} \\
&= G\frac{[z-1] [z^2 + 1.414z + 1] }{z^2 - 0.566z + 0.16017}\\
&= G\frac{z^3 + 1.414z^2 + 1z -z^2 - 1.414z - 1}{z^2 - 0.566z + 0.16017}\\
&= G\frac{z^3 + 0.414z^2 - 0.414z - 1}{z^2 - 0.566z + 0.16017}\\
&= G\frac{1 + 0.414z^{-1} - 0.414z^{-2} - z^{-3}}{1 - 0.566z^{-1} + 0.16017z^{-2}}\\
\end{align}
$$

* We've added an extra zero at $z=0$ compared to example 3.  This means that we get a valley (attenuation) in the frequency response at $\omega = 0$ (0 Hz).

```python
a = [0.566, -0.16017]
b = [1, 0.414, -0.414, -1]
```

### Ex 5:
$$
\begin{align}
H_5(z) &= G\frac{[z-1][z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.636+j0.636)][z-(0.636-j636)]} \\
&= G\frac{[z-1] [z^2 + 1.414z + 1] }{z^2 - 1.272z + 0.808992}\\
&= G\frac{1 + 0.414z^{-1} - 0.414z^{-2} - z^{-3}}{1 - 1.272z^{-1} +  0.808992z^{-2}}\\
\end{align}
$$

* poles are nearer to the unit circle (magnitude 0.80899), so the peaks associated with them (+/- $\omega = \pi/4$ rad/sec (1000 Hz with our sample rate) are sharper.

```python
a = [1.272, -0.80899]
b = [1, 0.414, -0.414, -1]
```

### Ex 6:
$$
\begin{align}
H_6(z) &= G \frac{[z-1][z-(-0.707 + j0.707)][z-(-0.707 - j0.707)]}{[z-(0.777+j0.777)][z-(0.777-777)]} \\
&= G \frac{1 + 0.414z^{-1} - 0.414z^{-2} - z^{-3}}{1 - 1.554z^{-1} +  1.207z^{-2}}\\
\end{align}
$$

* poles are now outside the unit circle (magnitude 1.207), so the filter is now unstable: as n increases, the value of $y[n]$ eventually goes to infinity.
* It still has the same shape frequency response though!

```python
a = [1.554, -1.207]
b = [1, 0.414, -0.414, -1]
```