# Flow driven by an oscillating solid cylinder and bounded by a stationary outer hollow cylinder
\begin{equation}\label{p1introhpa} \Psi _{\eta \eta \tau }^{\mathrm{Sym}}  = \frac{1}{{R_s }}\Psi^{\mathrm{Sym}}_{\eta \eta \eta \eta }  + \Psi^{\mathrm{Sym}}_{\eta \eta \eta } \Psi _\theta^{\mathrm{Sym}}   - \Psi^{\mathrm{Sym}}_{\eta \eta \theta } \Psi^{\mathrm{Sym}}_\eta, \\
\quad \Psi^{\mathrm{Sym}}(\eta= 0) = 0,\quad \Psi^{\mathrm{Sym}}_{\eta\eta}(\eta= 0) = 0, \qquad \Psi^{\mathrm{Sym}}(\eta= 1) = 0, \quad \Psi^{\mathrm{Sym}}_\eta(\eta= 1) = - \mu sin\theta, \qquad
0 \le \eta \le1, \quad 0 \le \theta \le \pi
\end{equation}

## Calculating $\Psi(\theta,\eta)$ the steady state solution $\theta$-marching
The governing equation is solved using:

* Half domain $\eta$-domain $0 \le \eta \le 1$
* $\theta$ marching with some of the nonlinear terms being explicit

## Summary of results
The numerical soltion is only smooth for  $0\leq\theta\leq\pi$ for very small reynolds numbers $-1< R_s< 2.0$, at higher Reynolds numbers the numerical errors accumulated as $\theta$ increases from zero and seem to be related to the choice of the meshsize $\Delta\theta$. When $\Delta\theta$ is *increased* the errors seem to vanish.

The solution generated by $\theta$-marching still matches the time marched solution near $\theta = 0$ upto  around $R_s = 45$ with very strong agreement with the 1st Order winding timemarched solution and less so with the second order winding scheme.

## Small $R_s$ series solution

```julia
using DifferentialEquations
using Plots
using BenchmarkTools
using DelimitedFiles
using LaTeXStrings
using SparseArrays
using LinearAlgebra
using StaticArrays
using Crayons.Box
using Printf
```

```julia
function ψ_ηηηη(n ::Int64, m:: Int64, ψ::Matrix{Float64}, meshsize::SVector{2, Int64}, delta_sizes::SVector{2, Float64}, paramtrs::Tuple{Float64, String})
     μ = paramtrs[1]
     order = paramtrs[2]
     N = meshsize[1]
    (Δθ, Δη) = delta_sizes

    if  n > 2 && n < N

        return( ψ[n-2,m]  -4*ψ[n-1,m] +6*ψ[n,m] -4*ψ[n+1,m] + ψ[n+2,m])

    elseif n == 2 #Second order only

        return( (-ψ[n,m]) -4*ψ[n-1,m] +6*ψ[n,m] -4*ψ[n+1,m] + ψ[n+2,m])

    elseif n == N #Second order only

        return( ψ[n-2,m]  -4*ψ[n-1,m] +6*ψ[n,m] -4*ψ[n+1,m] + (ψ[n,m] - 2.0*Δη*μ*sin(Δθ*(m-1))))
    else
        println("Error in ψ_ηηη n =$n")
        return 0.0
    end
end

function ψ_ηηη(n ::Int64, m:: Int64, ψ::Matrix{Float64}, meshsize::SVector{2, Int64}, delta_sizes::SVector{2, Float64}, paramtrs::Tuple{Float64, String})
     μ = paramtrs[1]
     order = paramtrs[2]
     N = meshsize[1]
    (Δθ, Δη) = delta_sizes

    if  n > 2 && n < N
        if n > 3 && n < N - 1 && order == "3rdOrder"
          return( ψ[n-3,m]/8 - ψ[n-2,m] + 13*ψ[n-1,m]/8 - 13*ψ[n+1,m]/8 + ψ[n+2,m] - ψ[n+3,m]/8)
        else
          return(-0.5*ψ[n-2,m] + ψ[n-1,m] - ψ[n+1,m] + 0.5*ψ[n+2,m])
        end

    elseif n == 2

        return(-0.5*(-ψ[n,m]) + ψ[n-1,m] - ψ[n+1,m] + 0.5*ψ[n+2,m])

    elseif n == N

        return(-0.5*ψ[n-2,m] + ψ[n-1,m] - ψ[n+1,m] + 0.5*(ψ[n,m] - 2.0*Δη*μ*sin(Δθ*(m-1))))
    else
        println("Error in ψ_ηηη n =$n")
        return 0.0
    end
end

function ψ_ηη(n ::Int64, m:: Int64, ψ::Matrix{Float64}, meshsize::SVector{2, Int64})
    N = meshsize[1]
    if  n == 2 || n == N

        return(  ψ[n-1,m] -2*ψ[n,m]+ ψ[n+1,m] )
    elseif n > 2  && n < N
        return ( -ψ[n-2,m]/12 +4*ψ[n-1,m]/3 -2.5*ψ[n,m]+ 4*ψ[n+1,m]/3 -ψ[n+2,m]/12)
    else
        println("Error in ψ_ηη n =$n")
        return 0.0
    end
end

function ψ_η(n ::Int64, m:: Int64, ψ::Matrix{Float64},  meshsize::SVector{2, Int64})
    N = meshsize[1]
    if  n >= 2 && n <= N

        return(  -0.5*ψ[n-1,m] + 0.5*ψ[n+1,m] )
    else
        println("Error in ψ_η n =$n")
        return 0.0
    end
end
```

    ψ_η (generic function with 1 method)

```julia
const N = 256;  # Total number of intervals (steps-1) in the η direction
const M = 256;  # Total number of intervals (steps-1) in the θ direction
ψ_halfsteps = zeros(N+1,2*M+1); # Initialise the matrix for the streamfunction at the current time step
ψ = zeros(N+1,M+1);
const Δη = 1.0/N;
const Δθ = pi/(2*M);

const μ = 1.0
const crank_nic = 1.0 #crank_nic = 1 means implicit and crank_nic =0 means explicit θ-scheme
matrix_condition_numb = zeros(M+1);

meshsize = SA{Int64}[N,2*M+1]
const c_half    =  Δη/Δθ
delta_sizes_normal= SA{Float64}[Δθ, Δη]
delta_sizes_double= SA{Float64}[2*Δθ, Δη]
paramtrs =  (μ, "3rdOrder")

rey_number = 1.0
```

    WARNING: redefinition of constant N. This may fail, cause incorrect answers, or produce other errors.
    WARNING: redefinition of constant Δη. This may fail, cause incorrect answers, or produce other errors.
    WARNING: redefinition of constant c_half. This may fail, cause incorrect answers, or produce other errors.

    1.0

```julia
d3  =  1.0*crank_nic/rey_number        #Diagonal entries for the matrix problem
d2  = -4.0*crank_nic/rey_number
d1  =  6.0*crank_nic/rey_number
#ψ_j = []

for m = 1:M
    for i = 1:3
        if i == 3
            ψ_j = ψ
            rhs = @view ψ[2:N,m+1]
            c    =  c_half/2.0
            k    =  m+1
            delta_sizes= delta_sizes_double

        else
            ψ_j = ψ_halfsteps
            rhs = @view ψ_halfsteps[2:N,2*(m-1)+i+1]
            c    =  c_half
            k    = 2*(m-1)+i+1
            delta_sizes= delta_sizes_normal

        end

    A =spdiagm(-2=>zeros(N-3), -1 => zeros(N-2), 0 => zeros(N-1), 1 => zeros(N-2), 2 => zeros(N-3))
    row = 1
    n = row + 1
    #A[row,row-2]  = nuemann boundary condition applied and add 0 to rhs
    #A[row,row-1] = 0 dirichlet boundary condition applied, this term is zero
    A[row,row]    = -d3 + d1 + ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row+1]  = d2 -ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row+2]  = d3

    rhs[row]      = ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, k-1]*c +
                        -ψ_η(n, k-1, ψ_j, meshsize)*ψ_ηη(n, k-1, ψ_j, meshsize)*c +
                    -ψ_ηηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*(1.0-crank_nic)/rey_number

    row = 2
    n = row + 1
    #A[row,row-2]  = d3 dirichlet boundary condition applied, this term is zero
    A[row,row-1]  = d2 - ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row]    = d1 + ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row+1]  = d2 -ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row+2]  = d3

    rhs[row]      = ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, k-1]*c +
                    -ψ_η(n, k-1, ψ_j, meshsize)*ψ_ηη(n, k-1, ψ_j, meshsize)*c +
                    -ψ_ηηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*(1.0-crank_nic)/rey_number

    for row = 3:N-3
        n = row + 1
        A[row,row-2]  = d3 + ψ_η(n, k-1, ψ_j, meshsize)*c/12.0
        A[row,row-1]  = d2 - 4*ψ_η(n, k-1, ψ_j, meshsize)*c/3.0
        A[row,row]    = d1 + ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2.5*ψ_η(n, k-1, ψ_j, meshsize)*c
        A[row,row+1]  = d2 - 4*ψ_η(n, k-1, ψ_j, meshsize)*c/3.0
        A[row,row+2]  = d3 + ψ_η(n, k-1, ψ_j, meshsize)*c/12.0

        rhs[row]      = ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, k-1]*c +
                        -ψ_η(n, k-1, ψ_j, meshsize)*ψ_ηη(n, k-1, ψ_j, meshsize)*c+
                        -ψ_ηηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*(1.0-crank_nic)/rey_number
    end

    row = N-2
    n = row + 1
    A[row,row-2]  = d3
    A[row,row-1]  = d2 - ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row]    = d1 + ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row+1]  = d2 -ψ_η(n, k-1, ψ_j, meshsize)*c
    #A[row,row+2]  = d3 dirichlet boundary condition applied, this term is zero

    rhs[row]      = ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, k-1]*c +
                        -ψ_η(n, k-1, ψ_j, meshsize)*ψ_ηη(n, k-1, ψ_j, meshsize)*c +
                    -ψ_ηηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*(1.0-crank_nic)/rey_number

    row = N-1
    n = row + 1
    A[row,row-2]  = d3
    A[row,row-1]  = d2 - ψ_η(n, k-1, ψ_j, meshsize)*c
    A[row,row]    = d3 + d1 + ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*c +2*ψ_η(n, k-1, ψ_j, meshsize)*c
    #A[row,row+1]  = d2 -ψ_η(n, m-1, ψ_j, meshsize)*c  dirichlet boundary condition applied, this term is zero
    #A[row,row+2]  = d3 nuemann boundary condition applied and add +2Δημsin(θ)*d3 to rhs

    rhs[row]      = ψ_ηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*ψ_j[n, k-1]*c +
                        -ψ_η(n, k-1, ψ_j, meshsize)*ψ_ηη(n, k-1, ψ_j, meshsize)*c +
                    -ψ_ηηηη(n, k-1, ψ_j, meshsize, delta_sizes, paramtrs)*(1.0-crank_nic)/rey_number +
                    + 2*Δη*μ*sin(delta_sizes[1]*(k-1))*d3

    #LUA = lu(A, check=true)
    #LUA2= factorize(A)
    rhs .= A\rhs
    #ldiv!(LUA2,rhs)
    A = nothing       #This frees up the memory on the next automated garbage collection GC.gc()
    end

    for p = 1:N+1
        ψ[p,m+1] = 2*ψ_halfsteps[p,2*(m-1)+3] -ψ[p,m+1]
        ψ_halfsteps[p,2*(m-1)+3] =  ψ[p,m+1]
    end
end

```

```julia
"""The symmetric (odd in η) asymptotic solution for the streamfunction for small \$R_s\$ with 5 terms"""
function Ψ_smallr(θ::Float64, η::Float64 ,r::Float64, number_of_terms::Int)
    terms = [
        -η*(η^2-1)*sin(θ)/2.0,

        r*η*(η^2-1)^2.0*(2.0+η^2)*sin(2*θ)/560.0,

        r^2.0*η*(η^2-1)^2*((-591 -2294*η^2 + 161*η^4 + 1428*η^6)*sin(θ) +
            +3.0*(423 + 166*η^2 - 553*η^4 - 84*η^6)sin(3*θ))/62092800.0,

        r^3.0*η*(η - 1)^2*(η + 1)^2*(2184*η^10*cos(θ)^2 + 1540*η^10 + 24213*η^8*cos(θ)^2 - 44205*η^8 +
            + 18942*η^6*cos(θ)^2 + 33446*η^6 - 101444*η^4*cos(θ)^2 + 66052*η^4 - 23086*η^2*cos(θ)^2 -
            6746*η^2 - 14889*cos(θ)^2 - 9383)*sin(θ)*cos(θ)/28252224000,

        r^4.0*η*(η - 1)^2*(η + 1)^2*(968647680*η^14*sin(θ)^5 + 14774464320*η^14*sin(θ) + 3510571680*η^14*sin(3*θ) -
            393513120*η^14*sin(5*θ) - 39671478000*η^12*sin(θ)^5 - 75537556710*η^12*sin(θ) -
            28276357725*η^12*sin(3*θ) - 3752563815*η^12*sin(5*θ) - 181021246560*η^10*sin(θ)^5 +
            357286638324*η^10*sin(θ) + 75142631718*η^10*sin(3*θ) - 1180776366*η^10*sin(5*θ) +
            677920708080*η^8*sin(θ)^5 - 645834258762*η^8*sin(θ) - 123813530379*η^8*sin(3*θ) -
            1711397457*η^8*sin(5*θ) + 193167076032*η^6*sin(θ)^5 - 82870720296*η^6*sin(θ) +
            215941919508*η^6*sin(3*θ) + 59820477948*η^6*sin(5*θ) + 153978403824*η^4*sin(θ)^5 -
            123221433370*η^4*sin(θ) + 181687022685*η^4*sin(3*θ) - 60662329497*η^4*sin(5*θ) -
            500785425504*η^2*sin(θ)^5 + 367229681332*η^2*sin(θ) - 134582502330*η^2*sin(3*θ) -
            33288605262*η^2*sin(5*θ) - 1507832445168*sin(θ)^5 + 969877856394*sin(θ) - 482955016725*sin(3*θ) -
            9092499471*sin(5*θ))/16189310893916160000
        ]
    if number_of_terms < 6
        return sum(terms[1:number_of_terms])
    else
        return term[1]
    end
end

function Ψ_bigr(θ::Float64, η::Float64)
    return(sin(pi*η)*sin(θ)/pi)
end
```

    Ψ_bigr (generic function with 1 method)

```julia
pyplot()
psi_sol = reshape(ψ,length(ψ));
η_mesh = [(-1 +(n-1)*Δη)  for m=1:M+1 for n=1:N+1];
θ_mesh = [((m-1)*Δθ) for m=1:M+1 for n=1:N+1 ];
η_vector = range(-1,stop = 1,length = N+1);
θ_vector = range(0,stop = pi,length = M+1);
m_at_fail = M+1
for m = 1:M+1
    if abs(findmax(abs.(ψ[:,m]))[1]) > 0.4
        print("Convergence fails at m = $m \ni.e near θ = $(m-1)π/$M\u1D9C radians $(round((m-1)*Δθ,digits=1))\u1D9C \nθ = $(round((m-1)*180/M,digits=1))\U000B0 degrees R_s = $rey_number")
        m_at_fail = m
        break
    elseif m == M+1
        println("Solution seems to be smooth")
    end

end
```

    Convergence fails at m = 98
    i.e near θ = 97π/256ᶜ radians 0.6ᶜ
    θ = 68.2° degrees R_s = 1.0

```julia
compare_with_timemarch = true
ψ_tm = 0
if compare_with_timemarch# Load file with matrix N by M
    try
    ψ_tm =readdlm("psi_solution_R_$(rey_number)_1stOrder_sym.txt",Float64,); #Check that N and M dimensions match
    if size(ψ_tm) == (N+1,M+1)
        print("Meshsize dimensions of the timemarched ψ match with the θ-marched solution \nthis is good!")
    else
        print("Dimensions of timemarched ψ DO NOT match with the θ-marched  which are $(size(ψ_tm))\n ERROR..")
        compare_with_timemarch = false;
    end
    catch e
        println("Could not open file don't worry code will continue regardless")
        compare_with_timemarch = false;
    end
end
```

    Could not open file don't worry code will continue regardless

    false

```julia

wireframe( range(-1.0,stop = 1.0,length = round(Int,N/4)+1),range(0.0,stop = pi,length = round(Int,M/4)+1), transpose(ψ[1:4:end,1:4:end]), colour=:greys, fmt = :svg)
yticks!(([0:π/6:π;], ["0", "\\pi/6", "\\pi/3", "\\pi/2", "2\\pi/3", "5\\pi/6","\\pi"]))
plot!(xlabel=L"\eta", ylabel=L"\theta", zlabel=L"\Psi", title=L"R_s = %$rey_number"*"\n"*L"N_η=%$N, M_θ=%$M", size=(1000,800),camera=(-60,30) )
plot!(xtickfont = font(12), ytickfont = font(12), ztickfont = font(12), titlefont = font(11))
#yticks!(([0:π/6:π;], ["0", "\\pi/6", "\\pi/3", "\\pi/2", "2\\pi/3", "5\\pi/6","\\pi"]))
#plot!(xlabel=L"η", ylabel=L"θ", zlabel=L"Ψ", title=L"R_s = %$rey_number"*"\n"*L"N_η=%$N, M_θ=%$M")
#plot!(xtickfont = font(12), ytickfont = font(12), ztickfont = font(12), titlefont=font(10))
```

```julia

m=m_at_fail - 8
m = 3
eta=range(0, stop=1,length= N+1)
plot(eta, ψ[:,m], label="θ-marched numerical \n steady solution", linecolor=:red,    linestyle=:dot,   linewidth =2, legend=:outertopright, size=(1100,500))
if compare_with_timemarch
    plot!(eta, ψ_tm[:,m], label="timemarched numerical \nsolution", linecolor=:cyan,    linestyle=:dot,   linewidth =2, legend=:outertopright, size=(1100,500))
end
if rey_number < 60
    plot!(x->Ψ_smallr(2*Δθ*(m-1),x,rey_number,3),0,1, label="Small "*L"R_s"*" asymptotic",linestyle=:dot,   linewidth =3, linecolor=:blue)
elseif rey_number > 200

end
xlabel!("η \n"*L"N_η=%$N, M_θ=%$M")
ylabel!(L"Ψ", yguidefontrotation=-90)
title!("η-profile  "*L"θ=\frac{%$(m-1)}{%$M}π \quad R_s = %$rey_number")
```

```julia
eta_profile =0.6
theta=range(0, stop=pi,length= 2*M+1)

plot((0:m_at_fail-1)*pi/(M+1),ψ[round(Int,(eta_profile)/Δη)+1,1:m_at_fail], linecolor=:red, label="θ-marched numerical \n steady solution", linestyle=:dot, linewidth=1,legend=:outertopright, size=(1100,500))
plot!(x->Ψ_smallr(x,eta_profile,rey_number, 3),0,pi, label="Small "*L"R_s"*" asymptotic", linestyle=:dot, linecolor=:blue, linewidth=3)
xticks!(([0:π/6:π;], ["0", "\\pi/6", "\\pi/3", "\\pi/2", "2\\pi/3", "5\\pi/6","\\pi"]))
title!(L"θ"*"-profile at "*L"η= %$eta_profile, \ R_s = %$rey_number")
xlabel!("θ \n"*L"N_η=%$N, M_θ=%$M")
ylabel!(L"Ψ", yguidefontrotation=-90)
```

```julia

```


```
from sympy import *
init_printing()
from galgebra.ga import Ga

def augment(ga,sym='n',sig=1):
    '''
    augments an existing Geometric Algebra by a single dimension of specified
    signature
    '''

    g = ga.g
    g_proj =  eye(len(ga.basis)+1)*sig
    for m in range(0,shape(g)[0]):
        for n in range(0,shape(g)[1]):
            g_proj[m,n] = g[m,n]
    basis = ' '.join([str(k) for k in ga.basis] +[sym])

    GA =  Ga(basis, g=g_proj)
    GA.ga_orginal = ga
    return GA
```

```
affine =  lambda ga: augment(ga, 'n',0)
projective = lambda ga: augment(ga, 'n',1)
conformal = lambda ga: augment(augment(ga, 'n_+',1),'n_-',-1)
```

```
# euclidean geometries
ge2 = Ga('e1 e2',g=[1,1])
ge3 = Ga('e1 e2 e3',g=[1,1,1])

# projective  geometries
gp2 = projective(ge2)
gp3 = projective(ge3)

# conformal  geometries
gc2 = conformal(ge2)
gc3 = conformal(ge3)
```

```
gc3.mv('x', 'vector')
```

\begin{equation*} x = x^{1} e_{1} + x^{2} e_{2} + x^{3} e_{3} + x^{4} n_{+} + x^{5} n_{-} \end{equation*}

```
gp2.mv('p','vector')
```

\begin{equation*} p = p^{1} e_{1} + p^{2} e_{2} + p^{3} n \end{equation*}

```
alpha=ge3.mv('alpha','scalar')
a=ge3.mv('a','vector')
B = ge3.mv('B','bivector')
```

```
R = alpha+B
R
```

\begin{equation*} \alpha   + B^{12} e_{1}\wedge e_{2} + B^{13} e_{1}\wedge e_{3} + B^{23} e_{2}\wedge e_{3} \end{equation*}

```
e1,e2,e3 = ge3.mv()
e12= e1*e2
e12*a*e12

```

\begin{equation*} a^{1} e_{1} + a^{2} e_{2} - a^{3} e_{3} \end{equation*}

```
-e3*a*e3
```

\begin{equation*} a^{1} e_{1} + a^{2} e_{2} - a^{3} e_{3} \end{equation*}

```
I = e1*e2*e3
I

```

\begin{equation*}  e_{1}\wedge e_{2}\wedge e_{3} \end{equation*}

```
I*e3
```

\begin{equation*}  e_{1}\wedge e_{2} \end{equation*}

```
b = R*a*R.rev()
b
```

\begin{equation*} \left ( - {\left ( B^{12} \right )}^{2} a^{1} + 2 B^{12} B^{23} a^{3} + 2 B^{12} a^{2} \alpha  - {\left ( B^{13} \right )}^{2} a^{1} - 2 B^{13} B^{23} a^{2} + 2 B^{13} a^{3} \alpha  + {\left ( B^{23} \right )}^{2} a^{1} + a^{1} \alpha ^{2}\right ) e_{1} + \left ( - {\left ( B^{12} \right )}^{2} a^{2} - 2 B^{12} B^{13} a^{3} - 2 B^{12} a^{1} \alpha  + {\left ( B^{13} \right )}^{2} a^{2} - 2 B^{13} B^{23} a^{1} - {\left ( B^{23} \right )}^{2} a^{2} + 2 B^{23} a^{3} \alpha  + a^{2} \alpha ^{2}\right ) e_{2} + \left ( {\left ( B^{12} \right )}^{2} a^{3} - 2 B^{12} B^{13} a^{2} + 2 B^{12} B^{23} a^{1} - {\left ( B^{13} \right )}^{2} a^{3} - 2 B^{13} a^{1} \alpha  - {\left ( B^{23} \right )}^{2} a^{3} - 2 B^{23} a^{2} \alpha  + a^{3} \alpha ^{2}\right ) e_{3} \end{equation*}

## asfeas

```
ge3,e1,e2,e3 = Ga.build('e1 e2 e3',g=[1,1,1])
```

```
a= ge3.mv('a','vector')
b= ge3.mv('b','vector')

```

\begin{equation*} \left ( - a^{2} b^{3} + a^{3} b^{2}\right ) e_{1} + \left ( a^{1} b^{3} - a^{3} b^{1}\right ) e_{2} + \left ( - a^{1} b^{2} + a^{2} b^{1}\right ) e_{3} \end{equation*}

```
e12 = e1*e2
e13 = e1*e3
e13.dual()
```

\begin{equation*}  e_{2} \end{equation*}

```
import galge
```

```

```


# Método dos Mínimos Quadrados (MMQ)

## License

All content can be freely used and adapted under the terms of the
[Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).

## Imports

Coloque **todos** os `import` na célula abaixo. Não se esqueça do `%matplotlib inline` para que os gráficos apareçam no notebook.

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
```

## IMPORTANTE

Agora que vocês sabem técnicas de programação defensiva, eu espero que todo o código que vocês fizerem abaixo utilizem essas técnicas. Crie docstrings para suas funções, cheque as entradas (quando for possível) e cheque as saídas. **Não esqueçam dos comentários**.

## Fabricando dados para teste

Para saber se nosso código está funcionando, precisamos fabricar alguns dados com parâmetros conhecidos. Vamos gerar dados que seguem a equação da reta:

$$
d_i = a x_i + b
$$

**IMPORTANTE**: Eu vou usar a biblioteca numpy as gerar os dados abaixo.
Vocês **não podem** utilizar o numpy para calcular a sua solução.
Uso do numpy deve ser conquistado com ~~sangue~~esforço.
O código abaixo serve como exemplo do que vocês poderão fazer ao utilizar o Python no seu trabalho (fora da aula).

```python
a = 10
b = 50
N = 50
# Vou utilizar a função linspace do numpy para facilitar a vida
# Essa função cria N valores igualmente espaçados entre dois números (5 e 50)
x = np.linspace(5, 50, N)
# Agora podemos usar os valores de x, a e b acima para simular dados observados
dados_obs = a*x + b
# Vamos adicionar erro aleatório aos dados para ficar mais interessante
# O erro seguirá uma distribuição normal com os seguintes parâmetros
media_erro = 0
std_erro = 20
# A linha abaixo faz com que os valores aleatórios não sejam verdadeiramente aleatórios
# veja https://en.wikipedia.org/wiki/Pseudorandom_number_generator
np.random.seed(42)
# Gera a lista de numéros aleatórios
erro = np.random.normal(loc=media_erro, scale=std_erro, size=len(dados_obs))
# Agora podemos adicionar o erro aos dados observados
dados_obs += erro
```

Utilize a célula abaixo para gerar um gráfico de círculos pretos (`ok`) de seus dados.

```python
plt.figure () # Iremos gerar um gráfico valores x dados observados
plt.plot(x, dados_obs, "ok") # Iremos fazer com que as bolinhas geradas no gráfico sejam pretas

plt.title("gráfico valores x dados observados") # O titutlo do grafico será " valores x"

plt.xlabel("valores") # O eixo x recebe o nome de "valores"
plt.ylabel("dados observados") # O eixo y recebe o nome de "dados observados"
```

## Forma matricial da equação da reta e a matriz Jacobiana

Temos uma equação da reta para cada valor de $x_i$:

$$
\begin{align}
d_1 &= ax_1 + b \\
d_2 &= ax_2 + b \\
\vdots \\
d_N &= ax_N + b \\
\end{align}
$$

Esse sistema pode ser escrito de forma matricial com os parâmetros sendo $a$ e $b$:

$$
\begin{bmatrix}
d_1 \\ d_2 \\ \vdots \\ d_N
\end{bmatrix} =
\begin{bmatrix}
x_1 & 1 \\
x_2 & 1 \\
\vdots & \vdots \\
x_N & 1
\end{bmatrix}
\begin{bmatrix}
a \\ b
\end{bmatrix}
$$

$$
\bar{d} = \bar{\bar{A}}\bar{p}
$$

## Tarefa

Faça uma função chamada `jacobiana` que calcule e retorne a matrix Jacobiana ($\bar{\bar{A}}$).

**Para pensar**: o que essa função deve receber como argumento? (**Dica**: ela só precisa de 1)

```python
def jacobiana(x):
    """
    Cria uma matriz Jacobiana

    """
    jacobiana = [] # Essa lista vazia foi feita para criação da matriz jacobiana
    for i in range(N): # Esse loop faz a leitura de cada linha da matriz
        jacobiana.append([x[i], 1]) # adicionamos linhas a matriz (Xn , 1)
    return jacobiana
```

### Resultado esperado

A célula abaixo testa a sua Jacobiana contra uma produzida pelo numpy.

```python
assert np.allclose(jacobiana(x), np.transpose([x, np.ones_like(x)]))
```

```python
# Atribui a uma variavél a matriz jacobiana
jac = jacobiana(x)
```

## Tarefa

Calcule dados preditos para o vetor de parâmetros definido abaixo **utilizando a forma matricial da equação**. Guarde o resultado em uma variável chamada `preditos`.

Faça um gráfico dos dados observados (gerados acima) como pontos pretos e os dados preditos que você calculou como uma linha vermelha.

**Dica**: utilize as funções que você criou na aula passada.

```python
p = [5, 15]
```

```python
def vmult(m, v): # Essa função realiza o produto matriz x vetor
    """
    Multiplica uma matriz por um vetor
    """
    assert len(m[0]) == len(v), 'Número de colunas da matriz diferente do número de linhas do vetor.'

    U = [] # Cria uma lista vazia.

    for i in range(len(m)): # O loop anda sobre as linhas da matriz

        soma = 0 # Atriubuimos o valor zero a variável

        for k in range(len(m[0])): # Esse loop que anda sobre as colunas da matriz

            soma = soma + (m[i][k] * v[k]) # Realiza o produto matriz por vetor

        U.append(soma) # Adiciona os resultados a lista nova gerada
    return U

```

```python
preditos = vmult(jac,p) # Atribui a uma variável ao produto matriz pelo vetor
```

```python
plt.figure() # Iremos gerar um gráfico
plt.plot(x, dados_obs, "ok") # plota bolinhas pretas para representar o gráfico dados preditos por dados obs

plt.plot(x, preditos, "-r") # Plota os valores de x, y em uma linha vermelha
plt.title("dados preditos x dados obs") # Atribui título ao gráfico
```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
assert np.allclose(preditos, np.dot(jacobiana(x), p))
```

O gráfico deve ser parecido com o abaixo:

## Sistema de equações normais

A solução de mínimos quadrados é o vetor $\bar{p}$ que resolve o sistema linear abaixo (chamado de sistema de equações normais):

$$
\bar{\bar{A}}^T\bar{\bar{A}}\bar{p} = \bar{\bar{A}}^T\bar{d}^o
$$

Para resolver esse sistema, precisamos primeiramente calcular a matriz do sistema $\bar{\bar{A}}^T\bar{\bar{A}}$ e o vetor do lado direito $\bar{\bar{A}}^T\bar{d}^o$.

## Tarefa

Faça uma função chamada `eqnormais_sistema` que calcule e retorne a matriz $\bar{\bar{A}}^T\bar{\bar{A}}$ dada a matriz Jacobiana.

Utilize as funções criadas na aula anterior.

**Dica**: É possível saber quantas linhas e colunas o sistema deve conter. Cheque se o seu resulatdo possui esse número.

```python

```

```python

```

```python

```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
assert np.allclose(eqnormais_sistema(jacobiana(x)), np.transpose(jacobiana(x)).dot(jacobiana(x)))
```

```python

```

```python

```

## Tarefa

Faça uma função chamada `eqnormais_lado_direito` que calcule e retorne o vetor do lado direito do sistema de equações normais.

**Dicas**:

* Essa função deve receber 2 argumentos.
* O que essa função deve retornar é um vetor ou matriz?
* É possível saber o número de elementos que o resultado deve conter. Cheque esse número.

```python

```

```python

```

```python

```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
assert np.allclose(eqnormais_lado_direito(jacobiana(x), dados_obs), np.transpose(jacobiana(x)).dot(dados_obs))
```

```python

```

```python

```

## Solução de mínimos quadrados

Agora que temos o sistema de equações normais, podemos resolvê-lo numericamente para encontrar os valores de $a$ e $b$ que produzem a reta que melhor ajusta nossos dados.

## Tarefa

Faça uma função chamada `elim_gauss` que resolve um sistema de equações utilizando a eliminação de Gauss. Essa função deve receber como argumento a matriz do sistema e o vetor lado direito e retornar o vetor de solução.

**Dicas**:

* Cheque o número de elementos na matriz e no vetor.
* A matriz deve ser quadrada.

```python

```

```python

```

```python

```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
np.random.seed(42)
A_teste = np.random.uniform(10, 50, size=(21, 21))
x_teste = np.random.uniform(5, 20, size=21)
y_teste = A_teste.dot(x_teste)
assert np.allclose(elim_gauss(A_teste, y_teste), x_teste)
```

```python

```

```python

```

## Tarefa

Faça uma função `ajuste_reta` que recebe um vetor de valores de x e um de dados observados e retorna a solução de minimos quadrados $\bar{p}$ (vetor com os valores de $a$ e $b$ estimados).

Aplique essa função nos dados observados simulados acima. Cheque se a solução bate com o valor esperado (você pode fazer isso usando um `assert`).

Faça um gráfico dos dados observados (pontos pretos) pelos dados preditos pela solução que você obteve agora (linha vermelha). O gráfico deve conter uma legenda. A legenda para os dados preditos deve ser da forma "y = 234x + 244" (trocando os números pelos valores que você estimou).

**Dica**:r

* Quantos elementos deve ter o vetor retornado?
* Para inserir números em uma string (texto): `"y = {}".format(123.1)"` $\to$ `"y = 123.1"`
* Para formatar os números que você quer inserir numa string: `"y = {:.4f}".format(123.242524536362446353436335)` $\to$  `"y = 123.2425"`

```python

```

```python

```

```python

```

### Resultado esperado

Os valores estimados para $\bar{p}$ devem ser aproximadamente:

    [9.742296, 52.57738183]

O gráfico deve ser parecido com o abaixo:

## Tarefa Bônus

Podemos utilizar o  método dos mínimos quadrados para ajustar qualquer equação que seja linear com relação as parâmetros ($a$ e $b$ no caso da reta). Isso quer dizer que podemos ajustar uma parábola:

$$
d_i = ax_i^2 + bx + c
$$

Dessa vez, os parâmetros que queremos estimar são $a$, $b$ e $c$. Note que agora temos 3 parâmetros, não 2. Por isso, a Jacobiana terá 3 colunas ao invés de 2.

Faça ao menos as seguintes funções:

* `jacobiana_parabola`: calcula e retorna a matriz Jacobiana para o caso da parábola. Deve receber como argumento somente o vetor de coordenadas x.
* `ajuste_parabola`: calcula a solução de mínimos quadrados para o caso de uma parábola. Deve receber como argumento o vetor de coordenadas x e o vetor de dados. Deve retornar o vetor de parâmetros $\bar{p}$ estimado (contem os valores de $a$, $b$ e $c$)

Teste suas funções com os dados gerados abaixo. Note que estamos usando o mesmo vetor x. Gere gráficos dos dados fabricados e também dos dados preditos pela estimativa (como os que foram feitos acima).

O que acontece se você tentar ajustar uma reta aos dados da parábola? E se tentar ajustar uma parábola aos dados da reta?

**Dicas**:

* Você precisa criar outras funções para montar o sistema de equações normais e calcular a solução do sistema?

```python
a_par, b_par, c_par = 2, 20, 200
dados_parabola = a_par*x**2 + b_par*x + c_par + erro
```

```python

```

```python

```

```python

```

```python

```

### Resultado esperado

Os gráficos que você deve gerar deverão ser parecidos com os abaixo:


# Monte Carlo Markov Chain

Prerequisites: Monte Carlo Calculation of Pi

Author: Christina Lee

### Intro

If you didn't check it out already, take a look at the post that introduces using random numbers in calculations.  Any such simulation is a Monte Carlo simulation.  The most used kind of Monte Carlo simulation is a Markov Chain, also known as a random walk, or drunkard's walk.  A Markov Chain is a series of steps where
* each new state is chosen probabilitically
* the probabilities only depend on the current state (no memory)

Imagine a drunkard trying to walk.  At any one point, they could progress either left or right rather randomly.  Also, just because they had been traveling in a straight line so far does not guaruntee they will continue to do.  They've just had extremely good luck.

We use Markov Chains to approximate probability distributions.

To create a good Markov Chain, we need
*  Ergodicity: All states can be reached
*  Global Balance: A condition that ensures the proper equilibrium distribution

### The Balances

Let $\pi_i$ be the probability that a particle is at site $i$, and $p_{ij}$ be the probability that a particle moves from $i$ to $j$.  Then Global Balance can be written as,

\begin{equation}
\sum\limits_j \pi_i p_{i j} = \sum\limits_j \pi_j p_{j i} \;\;\;\;\; \forall i.
\end{equation}

In non-equation terms, this says the amount of "chain" leaving site $i$ is the same as the amount of "chain" entering site $i$ for every site in equilibrium.  There is no flow.

Usually though, we actually want to work with a stricter rule than Global Balance,  Detailed Balance , written as

\begin{equation}
\pi_i p_{i j} = \pi_j p_{j i}.
\end{equation}

Detailed Balance further constricts the transition probabilities we can assign and makes it easier to design an algorithm. Almost all MCMC algorithms out there use detailed balance, and only lately have certain applied mathematicians begun looking and breaking detailed balance to increase efficiency in certain classes of problems.

### Today's Test Problem

I will let you know now; this might be one of the most powerful numerical methods you could ever learn.  I was going to put down a list of applications, but the only limit to such a list is your imagination.

Today though, we will not be trying to predict stock market crashes, calculate the PageRank of a webpage, or calculate the entropy a quantum spin liquid at zero temperature.  We just want to calculate an uniform probability distribution, and look at how Monte Carlo Markov Chains behave.

* We will start with an $l\times l$ grid

* Our chain starts somewhere in that grid

* We can then move up, down, left or right equally

* If we hit an edge, we come out the opposite side (toroidal boundary conditions)

First question! Is this ergodic?

Yes! Nothing stops us from reaching any location.

Second question! Does this obey detailed balance?

Yes!  In equilibrium, each block has a probability of $\pi_i = \frac{1}{l^2}$, and can travel to any of its 4 neighbors with probability of $p_{ij} = \frac{1}{4}$.  For any two neighbors
\begin{equation}
\frac{1}{l^2}\frac{1}{4} = \frac{1}{l^2}\frac{1}{4},
\end{equation}
and if they are not neighbors,
\begin{equation}
0 = 0.
\end{equation}

```julia
using Statistics
using Plots
gr()
```

    Plots.GRBackend()

```julia
# This is just the equivalent of `mod`
# for using in an array that indexes from 1.
function armod(i,j)
    return (mod(i-1+j,j)+1)
end
```

    armod (generic function with 1 method)

```julia
# input the size of the grid
l=5;
n=l^2;
```

```julia
function Transition(i)
    #randomly chose up, down, left or right
    d=rand(1:4);
    if d==1 #if down
        return armod(i-l,n);

    elseif d==2 #if left
        row=convert(Int,floor((i-1)/l));
        return armod(i-1,l)+l*row;

    elseif d==3 #if right
        row=convert(Int,floor((i-1)/l));
        return armod(i+1,l)+l*row;

    else  # otherwise up
        return armod(i+l,n);
    end
end
```

    Transition (generic function with 1 method)

```julia
# The centers of blocks.
# Will be using for pictoral purposes

pos=zeros(Float64,2,n);
pos[1,:]=[floor((i-1)/l) for i in 1:n].+0.5;
pos[2,:]=[mod(i-1,l) for i in 1:n].+0.5;
```

```julia
# How many timesteps
tn=2000;

# Array of timesteps
ti=Array{Int64,1}()
# Array of errors
err=Array{Float64,1}()

# Stores current location, initialized randomly
current=rand(1:n);
# Stores last location, used for pictoral purposes
last=current;

#Keeps track of where chain went
Naccumulated=zeros(Int64,l,l);

# put in our first point
# can index 2d array as 1d
Naccumulated[current]+=1;
```

```julia
for ii in 1:tn

    last=current;
    # Determine the new point
    current=Transition(current);
    Naccumulated[current]+=1;

    # add new time steps and error points
    push!(ti,ii)
    push!(err,std(Naccumulated/ii))

end
```

When I was using an old version and pyplot I created this video of the state at each time point. https://www.youtube.com/watch?v=gxX3Fu1uuCs

```julia
heatmap(Naccumulated/tn .- 1/l^2)
```

```julia
scatter(ti,log10.(err), markerstrokewidth=0)

plot!(xlabel="Step"
    ,ylabel="Log10 std"
    ,title="Error for a $l x$l")
```

So, after running the above code and trying to figure out how it works (mostly plotting stuff), go back and study some properties of the system.

* How long does it take to forget it's initial position?

* How does the behaviour change with system size?

* How long would you have to go to get a certain accuracy? (especially if you didn't know what distribution you where looking for)

So hopefully you enjoyed this tiny introduction to an incredibly rich subject.  Feel free to explore all the nooks and crannies to really understand the basics of this kind of simulation, so you can gain more control over the more complex simulations.

Monte Carlo simulations are as much of an art as a science.  You need to live them, love them, and breathe them till you find out exactly why they are behaving like little kittens that can finally jump on top of your countertops, or open your bedroom door at 1am.

For all their mishaving, you love the kittens anyway.

@ARTICLE{1970Bimka..57...97H,
    title = "{Monte Carlo Sampling Methods using Markov Chains and their Applications}",
  journal = {Biometrika, Vol.~57, No.~1, p.~97-109, 1970},
     year = 1970,
    month = apr,
   volume = 57,
    pages = {97-109},
      doi = {10.1093/biomet/57.1.97},
   adsurl = {http://adsabs.harvard.edu/abs/1970Bimka..57...97H},
  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

```julia

```


#### _Speech Processing Labs 2020: Signals: Module 2_

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

import cmath
from math import floor
from matplotlib.animation import FuncAnimation
from IPython.display import HTML
plt.style.use('ggplot')

from dspMisc import *
```

# 2 Filtering the Source: Finite Impulse Response Filters

### Learning Outcomes
* Be able to describe what an FIR filter is
* Be able to explain what the impulse response of an FIR filter is
* See how an FIR filter can be used as a lowpass frequency filter.
* See how changing the coefficients of an FIR filter can change it's frequency response

### Need to know
* Topic Videos: Spectral Envelope, Filter, Impulse Train
* [Interpreting the DFT](./sp-m1-5-interpreting-the-dft.ipynb)
* [Building the source: impulse trains](sp-m2-1-impulse-as-source.ipynb)

Equation alert: If you're viewing this on github, please note that the equation rendering is not always perfect. You should view the notebooks through a jupyter notebook server for an accurate view.

## 2.0 Filters

We've seen in the past notebooks that sometimes our input signal isn't exactly what we want. There is a vast literature in signal processing about designing filters to transform one signal into another. In speech processing, our signals often includes some sort of noise that we'd like to get rid of.  However, we can also use filters to shape a simple input, like an impulse train, into something much more complicated, like a speech waveform.

In class you've seen two types of filters:

* Finite Impulse Response (FIR)
* Infinite Impulse Response (IIR)

Both perform a transform on an input sequence $x[n]$ to give us some desired output sequence $y[n]$.  The difference between the two types of filters is basically whether we only use the inputs to derive each output $y[n]$ (FIR), or whether we also use previous outputs (IIR).

In the following we'll illustrate some of the properties of FIR filters.

## 2.1  Finite Impulse Response Filters

Finite Impulse Response (FIR) filters have the following form:

$$
\begin{align}
y[n] &= b[0]x[n] + b[1]x[n-1] + \dots + b[K]x[n-K] \\
 &= \sum_{k=0}^K b(k) x[n-k]
\end{align}
$$

Here, we have:
* an input sequence $x[n]$ of length $N$
* a set of $K$ filter coefficients.

We can read the equation as saying that $n$th ouput of the filter, $y[n]$, is a weighted sum of the previous K inputs $x[n],...,x[n-K]$.

### Example
Let's plot the $b[k]\cdot x[n-k]$ terms where $x$ is sinusoid of 4 Hz and the filter is `b=[1.1, 1.2, 1.5, 1.2, 1.1]`

```python
## Plotting a filter input window

## Set the number of samples N, sampling rate f_s
N=64
f_s = 64
t_s = 1/f_s
print("sampling rate: f_s = %f\nsampling time: t_s: %f" % (f_s, t_s))

x, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

## Plot the sinusoid we've just created:
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.plot(time_steps, x, 'o', color='grey')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')

## Filter coefficients (you could try changing these)
b = [1.1, 1.2, 1.5, 1.2, 1.1]

## K is the number of filter coefficients
K=len(b)

## Let's see what happens when n=19 (or try changing this variable!)
n=19
print("filter using b at n=%d, K=%d\n" % (n, K))

## Plot the values that are input to the filter
## +1's because python indexing/slicing doesn't include the end point
timedom.plot(time_steps[n+1-K:n+1], x[n+1-K:n+1], 'o', color='red')

## Calculate the b[k]*x[n-k] terms and add them to a list
filtered_n = []
for k in range(len(b)):
    ## print out the variables here
    print("%d:, b[%d]=%f, x[%d-%d]=%f, b[%d]*x[%d-%d]=%f" % (n-k, k, b[k], n, k,  x[n-k], k, n, k, b[k]*x[n-k]))
    filtered_n.append(b[k]*x[n-k])

## reverse the list so that they're in time order
filtered_n.reverse()

## Plot the b[k]*x[n-k] terms
timedom.plot(time_steps[n+1-K:n+1], filtered_n, 'o', color='blue')

## Calculate the filter output (add up the product terms)
print("\ny[%d] = %f" % (n, sum(filtered_n)))
```

In the plot above, you should see:
* A cosine wave with frequency 4 Hz in grey.
* The inputs the the filter x[19],...,x[15] in red.
    * 5 input values for 5 filter coefficients
* The product $b[k]*x[n-k]$ for $n=19$, and $k=0,...,4$

So with `b = [1.1, 1.2, 1.5, 1.2, 1.1]` all the input values get scaled up, with the the middle of the filter window get biggest relative increase.

When we add all the product terms together, we get y[19] = 4.87 - a lot bigger than any of the input values!

## 2.2 An FIR moving average filter

A useful special case of an FIR filter is where each of the filter coefficients is just $1/K$. In this case our FIR equation looks like this:

 $$
 \begin{align}
     y[n] &= \sum_{k=0}^{K-1} \frac{1}{K} x[n-k] \\
          &= \frac{1}{K} \sum_{k=0}^{K-1}  x[n-k]
 \end{align}
 $$

This equation says that when we apply the filter, we step through the input. At each step, we output the average of the previous $K$ inputs. You might know this by another more intuitive name: a _moving average_.  You might also have seen this as method to 'smooth' an input.

Let's play around with this idea a bit and see how it relates to our notion of frequency response.

### Example:

Let's look at this 5-point moving average filter.  In this case all the filter coefficients $b[k] = 1/5$ for $k=0,..,K-1=4$

$$
\begin{align}
y[n] &= \frac{1}{5} \sum_{k=0}^4 x[n-k] \\
&= \frac{1}{5}x[n] + \frac{1}{5}x[n-1] + \frac{1}{5}x[n-2] + \frac{1}{5}x[n-3] + \frac{1}{5}x[n-3]
\end{align}
$$

Now, let's code this specific filter up and apply it to some sequences!

```python
# Apply a moving average filter of size K to input sequence x
def moving_average(x, K=5):
    ## There are quicker ways to do this in numpy, but let's just do it this way for now for transparency

    ## We know that we'll have as many outputs as inputs, so we can initialize y to all zeros
    N = len(x)
    y = np.zeros(N)

    ## Go through the input one step at a time
    for n in range(N):

        ## Add up the last K inputs (including the current one)
        for k in range(K):

            ## Exercise: why do we have to put this conditional in here?
            if n-k >= 0:
                y[n] = y[n] + x[n-k]

        ## Divide by the size of the input window to get an average
        y[n] = (1/K) * y[n]

    return y

```

### Generate a 'noisy' sinusoid

Let's generate a compound sinusoid with one low frequency and one high(er) frequency component.  We can take the higher frequency component as representing some sort of periodic noise in the signal.

```python
## Set the number of samples N, sampling rate f_s
N=64
f_s = 64
t_s = 1/f_s

print("sampling rate: f_s = %f\nsampling time: t_s: %f" % (f_s, t_s))
```

    sampling rate: f_s = 64.000000
    sampling time: t_s: 0.015625

```python
## make some sinusoids:
## Since the sample rate and sequence length is the same, the generated time steps will match for
## x1 and x2
x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
#x2, time_steps = gen_sinusoid(frequency=6, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=24, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
# add them up!
x_noisy = x1 + x2
```

```python
## Plot the compound sinusoid we've just created:
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.plot(time_steps, x_noisy, color='magenta')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
```

You should be able to see that the 4Hz cosine wave is perturbed by a 24 Hz cosine. So, this could represent a situation where there is high frequency noise in our signal that we'd like to get rid of.  Let's see if we can use our filter to smooth out this high frequency noise.

### Apply the moving average filter

```python
## Apply our moving average filter
K=5
y_mov = moving_average(x_noisy,K=K)
```

```python
## Plot the results
fig, timedom = plt.subplots(figsize=(16, 4))
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')

## The input signal
timedom.plot(time_steps, x_noisy, color='magenta', label='Noisy input')

## The underlying 4Hz signal
timedom.plot(time_steps, x1, color='grey', label='low freq input component')

## After the moving average has been applied
timedom.plot(time_steps, y_mov, color='blue', label='filter output')

timedom.set_title('%d-point moving average filter applied to a noisy sinsuoid' % K)
timedom.legend()
```

You should see:
* the original signal in magenta
* the low frequency cosine component in grey
* the output of the filter in blue

### Exercise:

* From the plot, does it appear that the moving average recovered the original 4Hz signal in terms of:
    * frequency?
    * amplitude?
    * phase?
* Are there still high frequency components in the filter output?

Can you see some issues with applying this filter? What happens at the end points? Are they valid?

### Notes

The phase is slightly off (positive direction).
The amplitude is slightly less, and the frequency appears to be recovered.

### Get the DFT of the filtered signal

We can apply the DFT to our output to check our observations from the time-domain output.

```python
## DFT of the original input
mags_x, phases_x = get_dft_mag_phase(x_noisy, N)

## DFT of the filter output
mags_y, phases_y = get_dft_mag_phase(y_mov, N)
```

```python
dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

fig, mags = plt.subplots(figsize=(16, 4))
mags.set(xlim=(-1, N/2), ylim=(-1, N))
mags.scatter(dft_freqs, mags_x, color='magenta', label='noisy input')
mags.scatter(dft_freqs, mags_y, color='blue', label='filtered output')
mags.set_xlabel("Frequency (Hz)")
mags.set_ylabel("Magnitude")
mags.set_title("DFT Magnitude before and after moving average filter")
mags.legend()

#Let's not worry about phase right now, but feel free to uncomment and have a look!
#fig, phases = plt.subplots(figsize=(16, 4))
#phases.set(xlim=(-1, N/2), ylim=(-10, 10))
#phases.scatter(dft_freqs, phases_x, color='magenta', label='noisy input')
#phases.scatter(dft_freqs, phases_y, color='blue', label='filtered output')
#phases.set_xlabel("Frequency (Hz)")
#phases.set_ylabel("Phase (rad)")
#phases.set_title("DFT Phase before and after moving average filter")
#phases.legend()

```

### Exercise

* Based on the magnitude spectrum:
    * Did the filter get rid of the 24Hz component?
    * Do you see any signs of leakage?

* What happens if you change the frequency of the second sinusoid to something lower (e.g. 6Hz)?

### Notes

```python

```

## 2.3 FIR as convolution

An FIR Filter that takes $K$ previous elements of $x[n]$ as input has the following general form:

$$ y[n] = \sum_{k=0}^{K-1} b[k] x[n-k] $$

You might recognize this as a **convolution** of the two sequences $b$ and $x$  (i.e. $b * x$).

So, we can theoretically set our filter to coefficients to whatever we want.  Here's a function that generalizes our moving average filter to allow for this:

```python
def fir_filter(x, filter_coeffs):
    N = len(x)
    K = len(filter_coeffs)

    y = np.zeros(N)
    for n in range(N):
        for k in range(K):
            if n-k >= 0:
                #print("y[%d]=%f, b[%d]=%f, x[%d]=%f" % (n, y[n], k, filter_coeffs[k], n-k, x[n-k]))
                y[n] = y[n] + (filter_coeffs[k]*x[n-k])
        #print("y[%d]=%f" % (n, y[n]))
    return y

```

### Changing the filter coefficients

Let's try it out with different coefficient values, comparing our unweighted average `h_avg` filter with a weighted average `h_wavg` filter.

```python

## The 5-point moving average from before
h_avg = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
y_avg = fir_filter(x_noisy, h_avg)

## A 5-point symmetrically weighted average
h_wavg = np.array([1/5, 1/3, 1, 1/3, 1/5])
y_wavg = fir_filter(x_noisy, h_wavg)
```

### Filter effects in the time domain

```python
## Plot the filter outputs
fig, timedom = plt.subplots(figsize=(16, 4))

## The original "noisy" input
timedom.plot(time_steps, x_noisy, color='magenta', label='input x_noisy')
timedom.scatter(time_steps, x_noisy, color='magenta')

## The 5-point moving average
timedom.plot(time_steps, y_avg, color='blue', label='unweighted average: y_avg')
timedom.scatter(time_steps, y_avg, color='blue')

## The 5-point weighted average
timedom.plot(time_steps, y_wavg, color='orange', label='weighted average: y_wavg')
timedom.scatter(time_steps, y_wavg, color='orange')
timedom.legend()
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
```

In this time vs amplitude graph, you should see:
* the 'noisy input' in magenta
* the output of the unweighted average filter in blue (`y_avg`)
* the output of the weighted average filter in orange (`y_wavg`)

### Exercise

Q: Why is the output of `y_wavg` more spikey than that of `y_avg`?

### Notes

## 2.4 FIR Filters in the Frequency Domain

### The DFT of the filtered outputs

We can look at the effect of the two FIR filters defined above in the frequency domain by performing a DFT on the filter outputs.

```python
## DFT of the original input
mags_x, phases_x = get_dft_mag_phase(x_noisy, N)

## DFT after weighted average filter: h_wavg = np.array([1/5, 1/3, 1, 1/3, 1/5])
mags_wavg, phases_wavg = get_dft_mag_phase(y_wavg, N)

## DFT after unweighted average filter: h_avg = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
mags_avg, phases_avg = get_dft_mag_phase(y_avg, N)

dft_freqs = get_dft_freqs_all(sample_rate=f_s, seq_len=N)

## Plot magnitude spectrums
fig, mags = plt.subplots(figsize=(16, 4))
mags.set(xlim=(-1, N/2), ylim=(-1, N))
mags.scatter(dft_freqs, mags_x, color='magenta', label='input')
mags.scatter(dft_freqs, mags_avg, color='blue', label='unweighted average')
mags.scatter(dft_freqs, mags_wavg, color='orange', label='weighted average')
mags.legend()

## Plot phase spectrums
fig, phases = plt.subplots(figsize=(16, 4))
phases.set(xlim=(-1, N/2), ylim=(-10, 10))
phases.scatter(dft_freqs, phases_x, color='magenta', label='input')
phases.scatter(dft_freqs, phases_avg, color='blue', label='unweighted average')
phases.scatter(dft_freqs, phases_wavg, color='orange', label='weighted average')
phases.legend()

```

### Exercise:

* Describe the difference between the different FIR filters based on the frequency magnitude and phases responses plotted above.

* Does the weighted average filter do as good a job at filtering out the higher frequency signals?

### Notes

```python

```

## 2.5 Convolution in Time,  Multiplication in Frequency

Now we get to the really cool bit.  We know that the DFT allows us to go from the time domain to the frequency domain (and the Inverse DFT let's us go back).  But it also has this very important property:

$$ h[k] * x[n] \mapsto \text{ DFT} \mapsto H(m) \cdot X(m) $$

That is, convolving an input sequence $x$ with a set of filter coefficients, $h$ in the time domain  ($h*x$) is the same as (pointwise) multiplication of the DFT of $h$ with the DFT of $x$.  So, if we know what type of frequency response $h$ has, we can treat this as apply a sort of mask to the DFT of $x$.  This property is known as the **convolution theorem**.

Another way to think about it is that if the DFT outputs of $h$ has zero magnitude, then applying that filter to some input will also zero out those frequencies in the filter output!

We'll see some visualizations of this shortly, but first, we can also note that you can go back the other way using the Inverse DFT:

$$ H(m) \cdot X(m) \mapsto \text{ IDFT} \mapsto h[k] * x[n] $$

And also it works for multiplication in time domain too:

$$ h[k] \cdot x[n] \mapsto \text{DFT} \mapsto H(m) * X(m) $$

This is useful for understanding why the leakage graph around each DFT bin looks the way it does (though we won't go into it here!).

In fact, we can use this to not only show that the moving average acts as a low pass filter, but also to be able to calculate exactly the type of filter it will be.

###  Plotting different FIR filter frequency responses

In order to multiply together the filter and the input frequency responses, we need to make sure the filter and the input have the same number of samples.  We can do this by just padding out the filter with zeros (you can see why in the optional extra material at the end of this notebook).  We can then look how changing the 'shape' of a filter changes it's frequency response. The following exercise shows some examples.

### Exercise
* Run the function plot_filter_freq_responses (defined in the next cell) to plot the frequency responses of filters with different shapes (e.g. `h_plat`, `h_tri`, `h_rect` in the cell below the next).
    * What's the difference in frequency response of the triangular filter and the rectangular filter?
    * What's the difference between the moving average of size 5 and one of size 9 (h_rect9)?
    * Try some other FIR filters!

```python
# Given a list of filters, the sample rate and a specific sequence length
# plot the DFT frequency response of each filter.
# Each filter should be defined as a list of coeffients (b[k])
def plot_filter_freq_responses(filters, sample_rate, seq_length):

    ## Get the set of DFT output frequencies given the sample rate and desired sequence length
    dft_freqs_filter = (sample_rate/seq_length) * np.arange(seq_length)

    ## Calculate the time steps for each filter value given the sample rate and sequence length
    time_steps = (1/sample_rate) * np.arange(seq_length)

    ## Set up some plots:
    # the filter itself (time v amplitude)
    fig_time, sinusoid = plt.subplots(figsize=(16, 4))

    # the frequency response (freq v magnitude)
    fig_freq, fresponse = plt.subplots(figsize=(16, 4))

    x_filters = {}
    ## For each filter:
    for i, h in enumerate(filters):

        # pad the filter coefficients with zeros until we get the desired sequence length
        x_zeros = np.zeros(seq_length - len(h))
        x = np.concatenate([h, x_zeros])

        # Get the DFT outputs
        mags, phases = get_dft_mag_phase(x, seq_length)

        # Plot the filter
        sinusoid.scatter(time_steps, x)
        sinusoid.plot(time_steps, x, label=repr(h))

        # plot the magnitude response
        fresponse.scatter(dft_freqs_filter, mags)
        fresponse.plot(dft_freqs_filter, mags, label=repr(h))
        fresponse.set(xlim=(-1,seq_length/2))

        # return the filters and the DFT responses just in case
        x_filters = {'x':x, 'mags':mags, 'phases':phases, 'coeffs':h}

    sinusoid.set_xlabel('Time(s)')
    sinusoid.set_ylabel('Amplitude')
    sinusoid.set_title('Zero padded filters of different shapes')
    sinusoid.legend()

    fresponse.set_xlabel('Frequency (Hz)')
    fresponse.set_ylabel('Magnitude')
    fresponse.set_title('DFT magnitude response of ero padded filters of different shapes')
    fresponse.legend()

    return x_filters

```

```python
#h_plat = np.array([0.1, 0.2, 0.2, 0.2, 0.1])
h_tri = np.array([0.04, 0.12, 0.15, 0.12, 0.01])
h_sqr = h_tri ** 2
h_half = h_tri * .5
#h_rect = np.array([1/5, 1/5, 1/5, 1/5, 1/5])
#h_rect9 = np.array([1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9, 1/9])
h_sqrt = np.sqrt(h_tri)
h_sqrt_sqrt = np.sqrt(h_sqrt)
## Try some others if you like!

N=64
f_s=64

filter_dict = plot_filter_freq_responses(filters=[h_tri, h_sqr, h_half, h_sqrt, h_sqrt_sqrt], sample_rate=f_s, seq_length=N)
```

### Notes

```python

```

## 2.7 Applying an FIR filter to an impulse train

Now, finally we can look at the effect of applying an FIR filter to an impulse train.  Remember, we're using impulse trains to represent a sound source (i.e. vocal pulses at a specific frequency).  Eventually, we want to be able to define filters that capture the effects of the vocal tract.

First, here's a function that produces a sequence of pulses at a given frequency and sample rate. We'll then apply a triangular filter and look at what the frequency response is of the output is.

```python
## Let's make an impulse train
N=200
f_s = 8000
t_s = 1/f_s

frequency = 200

x_imp, time_steps = make_impulse_train(sample_rate=f_s, frequency=frequency, n_samples=N)

```

```python
## Plot the impulse train made in the cell before
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, x_imp)
td.plot(time_steps, x_imp)
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.set_title('Impulse train')
```

You should see an impulse train, with an impulse frequency of 200 Hz.  Let's see what happens when we apply the filter.

```python
## define our FIR filter coefficients
h_mov = np.array([0.1, 0.3, 0.5, 0.3, 0.1])

#try this later!
h_mov = np.array([1])

## Apply the filter to our impulse train
y_mov = fir_filter(x_imp, h_mov)

## plot the impulse train and the filtered version of it
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, x_imp)
td.plot(time_steps, x_imp, label='impulse train')
td.plot(time_steps, y_mov, label='filtered impulse train')
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.legend()
td.set_title('Impulse train and filtered impulse train')
```

Here you should see our impulse train (red) and the filtered impulse train (blue).  The filtered version has smaller amplitude and a broader peak (more samples have non-zero value).  Also, the filtered peaks are slightly after the original impulses.

Now let's look at the DFT output:

```python
## Get the DFT the filter output
mags, phases = get_dft_mag_phase(y_mov, N)

## Plot the DFT frequencies rather than DFT output indices
dft_freqs = get_dft_freqs_all(f_s, N)

## Plot the magnitude spectrum
fig, ax = plt.subplots(figsize=(16, 4))
ax.scatter(dft_freqs[0:int(N/2)],mags[0:int(N/2)])
ax.plot(dft_freqs[0:int(N/2)], mags[0:int(N/2)])

```

### Exercises

* What do the spikes in the DFT outputs represent?
* What does the first spike after the 0 frequency one represent?
* What does this filter appear to do?
    * Remember applying the FIR filter in the time domain (via convolution) is the same as multiplying the DFT of the filter to the DFT of the input signal

### Notes

### Exercises

* What happens when the impulse train frequency doesn't fall on one of the bins? e.g. `frequency = 130`, for 200 samples, with sampling rate 8000 samples/second
* Does the magnitude spectrum have the harmonic structure you'd expect?

### Notes

```python

```

## 2.6 (Extra Extension) The moving average filter as a rectangular function

This section (2.6) illustrates how we can use the convolution theorem understand why the moving average type filters act like low pass filters, and also the connection with leakage in the DFT magnitude response we saw previously.  This is optional extra material.

If you want to see an example of the convolution theorem working in the frequency domain, you can just run the code and have a look at the graph titled 'Frequency response after applying 5-point weighted average filter'

The convolution theorem tells us that, if we know the frequency response of an FIR filter, we know how it will affect the frequency response of it's input (we just multiply the individual frequency response together).

To understand what filter frequency response will look like, it's helpful to first observe that our unweighted moving average filter is pretty much a rectangular window function.  It's easy to see what this means when we plot it. The following function allows us to generate rectangular functions:

```python
def gen_rect_window(start_index, end_index, sample_rate=64, seq_length=64):
    nsteps = np.array(range(seq_length))
    t_s = 1/sample_rate
    time_steps = t_s * nsteps

    ## Let's make a rectangular window
    x_rect = np.zeros(seq_length)
    x_rect[start_index:end_index] = 1

    return x_rect, time_steps

```

###  Now, we make a rectangular window

```python
## Make rectangular window
N=64
K=16
f_s=64
start_index=24
end_index=start_index+K

x_rect, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N)

fig, timedom = plt.subplots(figsize=(16, 4))
timedom.scatter(time_steps, x_rect, color='magenta')
timedom.plot(time_steps, x_rect, color='magenta')
timedom.set_xlabel('Time (s)')
timedom.set_ylabel('Amplitude')
timedom.set_title('a rectangular window')
```

You should see a sequence with 64 point where the mmiddle 16 points have value 1 and the rest have value 0  (i.e., it looks like a rectangle in the middle).

### Now, let's  look at the frequency response of the rectangular window

```python
## Now we do the DFT on the rectangular function:

## get the magnitudes and phases
mags_rect, phases_rect = get_dft_mag_phase(x_rect, N)

## the DFT output frequencies
dft_freqs_rect = get_dft_freqs_all(f_s, N)

## let's just look at the magnitudes
fig, fdom = plt.subplots(figsize=(16, 4))
fdom.set(xlim=(-1, N/2))
fdom.scatter(dft_freqs_rect, mags_rect)
fdom.set_xlabel("Frequency (Hz)")
fdom.set_ylabel('Magnitude')
fdom.set_title('Frequency response of rectangular window')
## Looks leaky!
```

### Leaky windows?

The plot of the frequency magnitude response of our rectangular window has the hallmarks of leakiness. That is, the frequency response looks scalloped, with the biggest peak occuring around 0Hz. That is, it looks like a low pass filter!

With a bit of algebra we can derive the frequency for any $m$ (not just the DFT output bins indices) to be the following:

If $x[n]$ is a rectangular function of N samples with $K$ continugous samples of value 1 (starting at index $n_0$), we can figure out what the DFT output will be:

$$X[m] = e^{i(2\pi m/N)(n_0-(K-1)/2)} . \frac{\sin(2\pi mK/2N)}{\sin(2\pi m /2N)}$$

This is called the **Dirichlet kernel**. It has the **sinc** shape we saw when we looked at spectral leakage.

How is this useful?  Since we know what the frequency response of a rectangular window is, we know what convolving this with different input sequences will look like in the frequency domain.  We just multiply the frequency magnitude responses together.

On a more general note, this sort of convolution with a (short) window is how we do frequency analysis of speech: we taking windows of speech (aka frames) through time and and apply the DFT to get a frequency response.  A rectangular window is the simplest type of window we can take.  The equation above tells us that the sinc shaped response is an inherent part of using this sort of window.  In fact, we can use other window types (e.g. Hanning) to make the main lobes shaper and the sidelobes flatter, but we never really get away from this sinc shape in real world applications.  This is a key component of this soft of short term analysis.

Let's write this up in a function:

```python
def gen_rect_response(n_0, K, N, stepsize=0.01, polar=True, amplitude=1):
    ms = np.arange(0.01, N, stepsize)
    qs = 2*np.pi*ms/N

    ## Infact, we can work the frequency response to be the Dirichlet Kernel:
    response = (np.exp(-1j*qs*(n_0-(K-1)/2)) * np.sin(qs*K/2))/np.sin(qs/2)

    if polar:
        response_polar = [cmath.polar(z) for z in response]
        mags = np.array([m for m, _ in response_polar]) * amplitude
        phases = np.array([ph if round(mag) > 0 else 0 for mag, ph in response_polar])
        return (mags, phases, ms)

    return response, ms

```

Now we can plot the dirichlet kernel with the leaky looking DFT magnitudes we calculated earlier for our rectangular window.

```python
## Overlay the dirichlet kernel onto the DFT magnitudes we calculated earlier
## You should be able to see that the DFT magnitudes appear as discrete samples of the Dirichlet Kernel

mags_rect, phases_rect = get_dft_mag_phase(x_rect, N)
mags_rect_sinc , _ , ms = response = gen_rect_response(start_index, K, N)

fig, ax = plt.subplots(figsize=(16, 4))
ax.scatter(dft_freqs_rect, mags_rect, label='rectangular window')
ax.plot((f_s/N)*ms, mags_rect_sinc, color='C2', label='dirichlet')
ax.set(xlim=(-1,N/2))
ax.set_xlabel('Frequency (Hz)')
ax.set_ylabel('Magnitude')
ax.set_title('Frequency response of a rectangular sequence, %d samples with %d contiguous ones' % (N, K))
```

You should be able to see that the DFT magnitudes appear as discrete samples of the sinc shaped Dirichlet Kernel

### The unweighted average filter as a rectangular function

We can think of our 5-point unweighted average filter as a 5-point input sequence with all values set to 1/5.  We can then deduce that the frequency response of the filter will have the same shape as the frequency response of a rectangular window of all ones, but scaled down by 1/5.

Now let's check:

```python
N_h=5
f_s=64
start_index=0
end_index=N_h - start_index

## A 5 point rectangular window of all ones
h_avg, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N_h)
h_avg = h_avg/N_h

fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, h_avg, color='magenta')
td.plot(time_steps, h_avg, color='magenta')
td.set_xlabel('Time (s)')
td.set_ylabel('Amplitude')
td.set_title('5 point unweighted average as a rectangular function')

## Not very exciting looking!
print("h_avg:", h_avg)
```

You should just see 5 point in a row, all with value 1/5.  Now, we can plot the DFT magnitude response, as well as it's idealized continuous version:

```python
## Get the frequency magnitude response for our rectangular function
mags_h_avg, phases_h_avg = get_dft_mag_phase(h_avg, N_h)

## Get the continuous
rect_mags_h_avg, _ , ms = gen_rect_response(start_index, N_h, N_h, amplitude=np.max(h_avg))

## x-axis as frequencies rather than indices
ms_freqs_h_avg = (f_s/N_h) * ms
dft_freqs_h_avg = (f_s/N_h) * np.arange(N_h)

## Plot the frequency magnitude response
fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1, N/2))
fd.scatter(dft_freqs_h_avg, mags_h_avg)
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response of 5-point unweighter average filter')
#fd.scatter(dft_freqs_rect, mags_rect)

fd.plot(ms_freqs_h_avg, rect_mags_h_avg, color="C2")
```

You should see $floor(N/2) = 2$ points, with a main lobe peaking at 0 Hz, and side lobes peaking between each of the DFT output frequencies.
So,  DFT frequencies sit exactly at the zeros of this function when the windown size K is the same as the number of samples.

### Matching the filter and input size with zero padding

The theorem we saw above told us that we could calculate the frequency response of applying the FIR filter to an input sequence (via convolution), but multiply the DFT outputs of the filter and the input sequence.

Now, the x-axis range matches our that of our noisy input sequence because that is determined by the sampling rate. However, the filter frequency response we have above only has 5 outputs, while our input sample size was 64 because the number of DFT outputs is determined by the number of samples we put into the DFT.

To get things in the right form, we need to do some **zero padding** of the filter. We'll see that this basically gives us more samples of the Dirichlet Kernel corresponding to the filter frequency response.

```python
N=64
K=5
f_s=64
start_index=0
end_index=K

## Make a rectangular filter: K ones at the start
h_avg_pad, time_steps = gen_rect_window(start_index=start_index, end_index=end_index, sample_rate=f_s, seq_length=N)

## Divide by K to make it an average
h_avg_pad = h_avg_pad/K

## Plot the filter
fig, td = plt.subplots(figsize=(16, 4))
td.scatter(time_steps, h_avg_pad, color='magenta')
td.plot(time_steps, h_avg_pad, color='magenta')
td.set_xlabel('Time (s)')
td.set_title('5 point unweighted average FIR filter padded with zeros')

#print("N=%d, K=%d, start=%d, end=%d" % (N, K, start_index, end_index))

```

```python
## Get the frequency magnitude response for our rectangular function
mags_havg, phases_havg = get_dft_mag_phase(h_avg_pad, N)

## Plot the frequency magnitude response
## x-axis as actual frequencies rather that DFT indices
dft_freqs_havg = (f_s/N) * np.arange(N)

fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1,N/2))
fd.scatter(dft_freqs_havg, mags_havg)
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Magnitude response of 5-point unweighter average filter zero padded to 64 samples')
```

You should be able to see more clearly in the frequency response graph that the zero padding doesn't change doesnt change the basic shape of the filter's frequency response, we just get a finer grained representation in terms of samples (red dots).

### Calculate the input and filter frequency responses

```python
## Now let's calculate frequency responses of the original input
mags, phases = get_dft_mag_phase(x_noisy, N)

## ...  the filter
mags_filter, phases_filter = get_dft_mag_phase(h_avg_pad, N)

## ... and the filtered output that we calculated above
mags_avg, phases_avg = get_dft_mag_phase(y_avg, N)

## Plot with actual frequencies on the x-axis
dft_freqs = get_dft_freqs_all(f_s, N)
```

```python
## plot frequency responses

fig, fd = plt.subplots(figsize=(16, 4))
fd.set(xlim=(-1,N/2), ylim=(-1, N))

# DFT(input)
fd.scatter(dft_freqs, mags, color='magenta', label='DFT(input)')

# DFT(filter) * DFT(input)
fd.scatter(dft_freqs, mags_filter*mags, color='blue', label='DFT(filter).DFT(input)')

# DFT(filtered input)
fd.scatter(dft_freqs, mags_avg, color='red', label='DFT(filter*input)')
fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response after applying 5-point weighted average filter')
fd.legend()
```

You should see that the results from multiplying the DFT magnitudes from the input and the filter (blue) is (more or less) the same as the DFT of applying the filter in th time domain via convolution (red)

* Notice that there are some differences between the results from the time domain application of the filter (red) and the frequency domain multiplication (blue).  In particular there appears to be some leakage in the time-domain convolution case, possibly due to floating point errors.

### Exercise

* Try changing the frequency of the second cosine component of our compound wave in the code below.
    * Does the amount of attenuation of the high frequency component change as suggested by the DFT of the filter?
    * e.g. try 26 Hz vs 19 Hz
* What does this tell you about how well this low pass filter get's rid of high frequency noise?

```python
## Change the frequency of x2

x1, time_steps = gen_sinusoid(frequency=4, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)
x2, time_steps = gen_sinusoid(frequency=19, phase=0, amplitude=1, sample_rate=f_s, seq_length=N, gen_function=np.cos)

# add them up!
x_noisy = x1 + x2

## Now let's calculate frequency responses of the original input
mags, phases = get_dft_mag_phase(x_noisy, N)

## ...  the filter
mags_filter, phases_filter = get_dft_mag_phase(h_avg_pad, N)

## Plot with actual frequencies on the x-axis
dft_freqs = get_dft_freqs_all(f_s, N)

## plot frequency responses

fig, fd = plt.subplots(figsize=(16, 4))

fd.set(xlim=(-1,N/2), ylim=(-1, N))

# DFT(input)
fd.scatter(dft_freqs, mags, color='magenta', label='DFT(input)')

# DFT(filter) * DFT(input)
fd.scatter(dft_freqs, mags_filter*mags, color='blue', label='DFT(filter)*DFT(input)')

fd.set_xlabel('Frequency (Hz)')
fd.set_ylabel('Magnitude')
fd.set_title('Frequency response after applying 5-point weighted average filter')
fd.legend()
```

### Notes

```python

```

```python

```


# SINDy-PI Feature Overview

This notebook provides a simple overview of the basic functionality of SINDy-PI implemented in PySINDy, and is written by Alan Kaptanoglu and Kadierdan Kaheman. SINDy-PI is used to handle generic implicit ODEs of the form,
$$ \frac{d}{dt}\mathbf{x} = \mathbf{f}(\mathbf{x}, \dot{\mathbf{x}})$$
This is based off of the paper "Kaheman, K., Kutz, J. N., & Brunton, S. L. (2020). SINDy-PI: a robust algorithm for parallel implicit sparse identification of nonlinear dynamics. Proceedings of the Royal Society A, 476(2242), 20200279."

An interactive version of this notebook is available on binder
[](https://mybinder.org/v2/gh/dynamicslab/pysindy/v1.5.1?filepath=examples/9_sindypi_with_sympy.ipynb)

```python
# Import libraries.
import numpy as np
from matplotlib import pyplot as plt
import pysindy as ps
from pysindy.differentiation import FiniteDifference
from pysindy.utils import enzyme, bacterial
import sympy as sp
from scipy.integrate import solve_ivp

# ignore user warnings
import warnings
warnings.filterwarnings("ignore", category=UserWarning)

# For reproducibility
np.random.seed(100)

integrator_keywords = {}
integrator_keywords['rtol'] = 1e-12
integrator_keywords['method'] = 'LSODA'
integrator_keywords['atol'] = 1e-12
```

# Example to show library functionality
Below, we define custom libraries for x and x_dot, showing that the resulting full library and library names are a tensor product of the two independent libraries. Note that to generate the constant function 1 without generating repeated entries, we need to use include_bias=True.

```python
t = np.linspace(0, 1, 5)
x = np.ones((5, 2))

# Functions to be applied to the data x
functions = [lambda x: np.exp(x), lambda x, y: np.sin(x + y)]

# Functions to be applied to the data x_dot
x_dot_functions = [lambda x: x]

lib = ps.SINDyPILibrary(
    library_functions=functions,
    x_dot_library_functions=x_dot_functions,
    t=t,
).fit(x)
lib.transform(x)
print("Without function names: ")
print(lib.get_feature_names(), "\n")

# Function names includes both the x and x_dot functions
function_names = [
    lambda x: "exp(" + x + ")",
    lambda x, y: "sin(" + x + y + ")",
    lambda x: x,
]
lib = ps.SINDyPILibrary(
    library_functions=functions,
    x_dot_library_functions=x_dot_functions,
    function_names=function_names,
    t=t,
).fit(x)
lib.transform(x)
print("With function names: ")
print(lib.get_feature_names(), "\n")

# Now include a bias term (constant term)
lib = ps.SINDyPILibrary(
    library_functions=functions,
    x_dot_library_functions=x_dot_functions,
    function_names=function_names,
    t=t,
    include_bias=True,
).fit(x)
lib.transform(x)
print("With function names and bias term: ")
print(lib.get_feature_names())

```

    Without function names:
    ['f0(x0)', 'f0(x1)', 'f1(x0,x1)', 'f_dot0(x0_dot)', 'f_dot0(x1_dot)', 'f0(x0)f_dot0(x0_dot)', 'f0(x1)f_dot0(x0_dot)', 'f1(x0,x1)f_dot0(x0_dot)', 'f0(x0)f_dot0(x1_dot)', 'f0(x1)f_dot0(x1_dot)', 'f1(x0,x1)f_dot0(x1_dot)']

    With function names:
    ['exp(x0)', 'exp(x1)', 'sin(x0x1)', 'x0_dot', 'x1_dot', 'exp(x0)x0_dot', 'exp(x1)x0_dot', 'sin(x0x1)x0_dot', 'exp(x0)x1_dot', 'exp(x1)x1_dot', 'sin(x0x1)x1_dot']

    With function names and bias term:
    ['1', 'exp(x0)', 'exp(x1)', 'sin(x0x1)', 'x0_dot', 'x1_dot', 'exp(x0)x0_dot', 'exp(x1)x0_dot', 'sin(x0x1)x0_dot', 'exp(x0)x1_dot', 'exp(x1)x1_dot', 'sin(x0x1)x1_dot']

### Solve 1D Michelson-Menten enzyme model as first demonstration
The Michelson-Menten enzyme model is,
$$\dot{x} = 0.6 - \frac{1.5 x}{0.3 + x}.$$
Note that, instead of dealing with fractional terms, it can be rearranged into
$$\dot{x} = \frac{3}{5} -3 x - \frac{10}{3} x\dot{x},$$
which is an implicit ODE, since the right-hand-side now depends on $\dot{x}$! In order to handle this scenario, SINDy-PI builds a candidate library with terms depending on both $x$ and $\dot{x}$, and solves the optimization problem,
$$\|\Theta(x, \dot{x}) - \Theta(x, \dot{x})\Xi\|_2 + \lambda\|\Xi\|_1, \quad s.t. \text{diag}(\Xi) = 0.$$
This reason this modified optimization is required is that linear terms in $\dot{x}$ are no longer guaranteed, so every candidate library term is fit with a different model to find the best combination of implicit terms. Note $\text{diag}(\Xi) = 0$ is required because without this, the model would just fit every candidate library term with itself, $\dot{x} = \dot{x}$, $x\dot{x} = x\dot{x}$, $x^2\dot{x} = x^2\dot{x}$, and so on.

```python
# define parameters
r = 1
dt = 0.001
T = 4
t = np.arange(0, T + dt, dt)
t_span = (t[0], t[-1])
x0_train = [0.55]
x_train = solve_ivp(enzyme, t_span, x0_train, t_eval=t, **integrator_keywords).y.T

# Initialize custom SINDy library so that we can have x_dot inside it. Using the library
# Found in Equation A1 in the SINDy-PI paper
x_library_functions = [
    lambda x: x,
    lambda x, y: x * y,
    lambda x: x ** 2,
    lambda x, y, z: x * y * z,
    lambda x, y: x * y ** 2,
    lambda x: x ** 3,
    lambda x, y, z, w: x * y * z * w,
    lambda x, y, z: x * y * z ** 2,
    lambda x, y: x * y ** 3,
    lambda x: x ** 4,
]
x_dot_library_functions = [lambda x: x]

# library function names includes both the x_library_functions and x_dot_library_functions names
library_function_names = [
    lambda x: x,
    lambda x, y: x + y,
    lambda x: x + x,
    lambda x, y, z: x + y + z,
    lambda x, y: x + y + y,
    lambda x: x + x + x,
    lambda x, y, z, w: x + y + z + w,
    lambda x, y, z: x + y + z + z,
    lambda x, y: x + y + y + y,
    lambda x: x + x + x + x,
    lambda x: x,
]

# Need to pass time base to the library so can build the x_dot library from x
sindy_library = ps.SINDyPILibrary(
    library_functions=x_library_functions,
    x_dot_library_functions=x_dot_library_functions,
    t=t[1:-1],
    function_names=library_function_names,
    include_bias=True,
)

# Use the SINDy-PI optimizer, which relies on CVXPY.
# Note that if LHS of the equation fits the data poorly,
# CVXPY often returns failure.
sindy_opt = ps.SINDyPI(
    threshold=1e-6,
    tol=1e-8,
    thresholder="l1",
    max_iter=20000,
)
model = ps.SINDy(
    optimizer=sindy_opt,
    feature_library=sindy_library,
    differentiation_method=ps.FiniteDifference(drop_endpoints=True),
)
model.fit(x_train, t=t)
model.print()
```

    Solver failed on model  3 , setting coefs to zeros
    1 = 5.000 x0 + 1.667 x0_dot + 5.556 x0x0_dot
    x0 = 0.200 1 + -0.333 x0_dot + -1.111 x0x0_dot
    x0x0 = 0.198 x0 + 0.007 x0x0x0 + -0.338 x0x0_dot + -1.099 x0x0x0_dot
    x0x0x0 = 0.000
    x0x0x0x0 = -0.001 1 + 0.363 x0x0x0 + 0.041 x0x0_dot + -1.205 x0x0x0x0x0_dot
    x0_dot = 0.600 1 + -3.000 x0 + -3.333 x0x0_dot
    x0x0_dot = 0.180 1 + -0.900 x0 + -0.300 x0_dot
    x0x0x0_dot = -0.004 1 + 0.136 x0 + -0.508 x0x0 + -0.344 x0x0x0 + -0.102 x0x0_dot + -0.219 x0x0x0x0x0_dot
    x0x0x0x0_dot = 0.003 1 + 0.001 x0 + -0.391 x0x0x0 + -0.247 x0x0x0x0 + -0.108 x0x0_dot
    x0x0x0x0x0_dot = 0.001 1 + -0.670 x0x0x0x0 + -0.005 x0_dot + 0.029 x0x0_dot + -0.271 x0x0x0_dot

### Okay, so how to choose which model to use?
So SINDy-PI has generated a different dynamical model for each of the candidate library terms, but how do we choose which one to use? This can be decided in many ways, but an easy choice is picking the model that minimizes the mean-squared error of the $\dot{x}$ testing data. There are many other ways to choose, and the SINDy-PI paper noted that bad models tend to also be less sparse, so sparsity is another metric for selection. Below, we simply integrate forward all the models and on this testing trajectory, they basically all perform fine.

```python
# Plot the fit of the derivative
x_dot_train = model.differentiate(x_train, t)
x_dot_train_pred = sindy_opt.Theta_ @ sindy_opt.coef_.T
plt.figure()
for i in range(r):
    plt.subplot(1, r, i + 1)
    plt.plot(t, x_dot_train[:, i], 'k')
    plt.plot(t[1:-1], x_dot_train_pred[:, 5], 'r--')
```

### Now we need to reformat all the feature strings for conversion into symbolic variables
So we have all these nice models but these are implicit ODEs, and the default Python solvers (solve_ivp and odeint) cannot solve implicit ODEs. There are two options although we only illustrate the first option below; we must convert the equations into symbolic variables and then either (1) use SymPy to symbolically solve the equation for $\dot{x}$, after which the equation is no longer implicit and can be solved with the default Python solves, or (2) convert the equations into a symbolic differential algebraic equation (DAE), which can be numerically solved with the diffeqpy Python package (but requires Julia to be installed).

```python
features = model.get_feature_names()
features[0] = "1"
features_copy = list(np.copy(features))
nfeatures = len(features)
features_formatted = []
# Need to put multiplication between terms for sympy
for i in range(nfeatures):
    for j in range(r):
        # Overkill to make sure all the x0, x1, etc. get replaced
        temp_string = features[i].replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        features[i] = temp_string
    features_formatted.append(temp_string)
features = features_copy
print(features)
print(features_formatted)
coefs = model.coefficients()
sym_features = [sp.symbols(feature) for feature in features]
sym_theta = [sp.symbols(feature) for feature in features]
```

    ['1', 'x0', 'x0x0', 'x0x0x0', 'x0x0x0x0', 'x0_dot', 'x0x0_dot', 'x0x0x0_dot', 'x0x0x0x0_dot', 'x0x0x0x0x0_dot']
    ['1', 'x0', 'x0 * x0', 'x0 * x0 * x0', 'x0 * x0 * x0 * x0', 'x0_dot', 'x0 * x0_dot', 'x0 * x0 * x0_dot', 'x0 * x0 * x0 * x0_dot', 'x0 * x0 * x0 * x0 * x0_dot']

### Check all the model fits produce sensible models

```python
sym_equations = []
sym_equations_rounded = []
for i in range(nfeatures):
    sym_equations.append(
        sp.solve(
            sp.Eq(sym_theta[i], sym_theta @ np.around(coefs[i], 10)), sym_features[i]
        )
    )
    sym_equations_rounded.append(
        sp.solve(
            sp.Eq(sym_theta[i], sym_theta @ np.around(coefs[i], 2)), sym_features[i]
        )
    )
    print(sym_theta[i], " = ", sym_equations_rounded[i][0])

```

    1  =  5.0*x0 + 1.67*x0_dot + 5.56*x0x0_dot
    x0  =  0.2*1 - 0.33*x0_dot - 1.11*x0x0_dot
    x0x0  =  0.2*x0 - 0.34*x0x0_dot + 0.01*x0x0x0 - 1.1*x0x0x0_dot
    x0x0x0  =  0
    x0x0x0x0  =  0.04*x0x0_dot + 0.36*x0x0x0 - 1.21*x0x0x0x0x0_dot
    x0_dot  =  0.6*1 - 3.0*x0 - 3.33*x0x0_dot
    x0x0_dot  =  0.18*1 - 0.9*x0 - 0.3*x0_dot
    x0x0x0_dot  =  0.14*x0 - 0.51*x0x0 - 0.1*x0x0_dot - 0.34*x0x0x0 - 0.22*x0x0x0x0x0_dot
    x0x0x0x0_dot  =  -0.11*x0x0_dot - 0.39*x0x0x0 - 0.25*x0x0x0x0
    x0x0x0x0x0_dot  =  -0.01*x0_dot + 0.03*x0x0_dot - 0.27*x0x0x0_dot - 0.67*x0x0x0x0

### Now we will:
1. reformat all the equations
2. solve symbolically for x_dot
3. integrate and plot the new equations for x_dot

```python
# Define the ODE symbol variables
t_sym = sp.symbols("t_sym")
x_sym = sp.symbols("x:%d" % r)
x_dot_sym = sp.symbols("x:%d_dot" % r)

# Need to format the above equations so that there are space between x0 * x0, x0 * x_dot0, and so on.
sym_equations_formatted = []
sym_equations_rounded_formatted = []
for i in range(nfeatures):
    for j in range(r):
        # Overkill to make sure all the x0, x1, etc. get replaced
        temp_string = str(sym_equations[i]).replace(
            "x" + str(j) + "x", "x" + str(j) + " * x"
        )
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_rounded_string = str(sym_equations_rounded[i]).replace(
            "x" + str(j) + "x", "x" + str(j) + " * x"
        )
        temp_rounded_string = temp_rounded_string.replace(
            "x" + str(j) + "x", "x" + str(j) + " * x"
        )
        temp_rounded_string = temp_rounded_string.replace(
            "x" + str(j) + "x", "x" + str(j) + " * x"
        )
        temp_rounded_string = temp_rounded_string.replace(
            "x" + str(j) + "x", "x" + str(j) + " * x"
        )
        sym_equations_formatted.append(temp_string)
        sym_equations_rounded_formatted.append(temp_rounded_string)

# Now that the equations are mathematically formatted, solve for x_dot0 in the algebraic equation
sym_equations_simplified = []
sym_equations_rounded_simplified = []
for i in range(nfeatures):
    sym_equations_simplified.append(
        sp.solve(
            sp.Add(
                sp.sympify(sym_equations_formatted)[i][0],
                -sp.sympify(features_formatted[i]),
            ),
            x_dot_sym,
        )
    )
    rounded = sp.solve(
        sp.Add(
            sp.sympify(sym_equations_rounded_formatted)[i][0],
            -sp.sympify(features_formatted[i]),
        ),
        x_dot_sym,
    )
    if len(rounded) != 0:
        rounded_temp = rounded[0]
        for a in sp.preorder_traversal(rounded):
            if isinstance(a, sp.Float):
                rounded_temp = rounded_temp.subs(a, round(a, 2))
        sym_equations_rounded_simplified.append(rounded_temp)
    else:
        sym_equations_rounded_simplified.append([])

# Plot the results for each of the models
plt.figure(figsize=(16, 3))
x0_test = np.random.rand(1)
x_test = solve_ivp(enzyme, t_span, x0_test, t_eval=t).y.T
plt.plot(t, x_test, color="k", linestyle="solid", label="True")
for i in range(nfeatures):
    ax = plt.gca()
    if i != nfeatures - 1:
        ax.set_xticklabels([])
    if len(sym_equations_simplified[i]) != 0:
        ODE_Func = sp.lambdify([t_sym, x_sym], sym_equations_simplified[i][0])

        # Now simulate the system we identified
        x_test_sim = solve_ivp(ODE_Func, t_span, x0_test, t_eval=t).y.T
        if (
            np.linalg.norm(x_test_sim) < 1e3
            and sym_equations_rounded_simplified[i] != 0
        ):
            plt.plot(
                t,
                x_test_sim,
                linestyle="dashed",
                label=str(sp.sympify(features_formatted[i]))
                + " = "
                + str(sym_equations_rounded_simplified[i]),
            )
plt.grid(True)
plt.legend(fontsize=8)
```

### We will now try a 2D example, the bacterial competence system (Mangan et al. 2016)
$$ \begin{aligned} \dot{x} &= a_1 + \frac{a_2x^2}{a_3 + x^2} - \frac{x}{1 + x + y} \\  \dot{y} &= \frac{b_1}{1 + b_2 x^5} - \frac{y}{1 + x + y} \end{aligned} $$
with $a_1=0.004$, $a_2=0.07$, $a_3=0.04$, $b_1=0.82$, $b_2=1854.5$. Note that multi-variable SymPy will require some changes, and multi-variable SINDy-PI typically requires many trajectories to correctly identify models.

```python
# define parameters
r = 2
dt = 0.01
T = 20
t_train = np.arange(0, T + dt, dt)
t_train_span = (t_train[0], t_train[-1])

# Typically need at least 20-40 trajectories for a decent model.
# Note that this can be fairly time-consuming.
n_trajectories = 10
x_train_multi = []
t_train_multi = []
x0 = np.random.rand(n_trajectories, 2)
for i in range(n_trajectories):
    x_train_multi.append(solve_ivp(bacterial, t_train_span,
                                   x0[i, :], t_eval=t_train, **integrator_keywords).y.T)
    t_train_multi.append(t_train[1:-1])

# Initialize custom SINDy library up to sixth order polynomials
x_library_functions = [
    lambda x: x,
    lambda x, y: x * y,
    lambda x: x ** 2,
    lambda x, y, z: x * y * z,
    lambda x, y: x * y ** 2,
    lambda x: x ** 3,
    lambda x, y, z, w: x * y * z * w,
    lambda x, y, z: x * y * z ** 2,
    lambda x, y: x * y ** 3,
    lambda x: x ** 4,
    lambda x: x ** 5,
    lambda x: x ** 6,
]
x_dot_library_functions = [lambda x: x]

library_function_names = [
    lambda x: x,
    lambda x, y: x + y,
    lambda x: x + x,
    lambda x, y, z: x + y + z,
    lambda x, y: x + y + y,
    lambda x: x + x + x,
    lambda x, y, z, w: x + y + z + w,
    lambda x, y, z: x + y + z + z,
    lambda x, y: x + y + y + y,
    lambda x: x + x + x + x,
    lambda x: x + x + x + x + x,
    lambda x: x + x + x + x + x + x,
    lambda x: x,
]

# Get the indices of the equations for x0_dot and x1_dot
lib = ps.SINDyPILibrary(
    library_functions=x_library_functions,
    x_dot_library_functions=x_dot_library_functions,
    t=t_train,
    function_names=library_function_names,
    include_bias=True,
).fit(np.asarray(x_train_multi)[0, :, :])
lib.transform(np.asarray(x_train_multi)[0, :, :])
features = np.asarray(lib.get_feature_names())
ind1 = int(np.ravel(np.where(features == "x0_dot"))[0] - 1)  # - 1 because bias term
ind2 = int(np.ravel(np.where(features == "x1_dot"))[0] - 1)

# Need to flatten t_train_multi before passing time base to SINDyPILibrary
sindy_library = ps.SINDyPILibrary(
    library_functions=x_library_functions,
    x_dot_library_functions=x_dot_library_functions,
    t=np.ravel(np.asarray(t_train_multi)),
    function_names=library_function_names,
)

# Note, using model_subset (list of indices) to only compute two of the fits
sindy_opt = ps.SINDyPI(
    threshold=10,
    tol=1e-5,
    thresholder="l1",
    max_iter=20000,
    model_subset=[ind1, ind2],
)
model = ps.SINDy(
    optimizer=sindy_opt,
    feature_library=sindy_library,
    differentiation_method=ps.FiniteDifference(drop_endpoints=True),
)
model.fit(x_train_multi, t=dt, multiple_trajectories=True)
model.print()
```

    x0 = 0.000
    x1 = 0.000
    x0x1 = 0.000
    x0x0 = 0.000
    x1x1 = 0.000
    x0x1x1 = 0.000
    x0x0x0 = 0.000
    x1x1x1 = 0.000
    x0x1x1x1 = 0.000
    x0x0x0x0 = 0.000
    x1x1x1x1 = 0.000
    x0x0x0x0x0 = 0.000
    x1x1x1x1x1 = 0.000
    x0x0x0x0x0x0 = 0.000
    x1x1x1x1x1x1 = 0.000
    x0_dot = -0.407 x0 + 0.007 x1 + 0.022 x0x1
    x1_dot = 0.009 x0 + 0.244 x1 + -0.331 x1x1 + 0.084 x1x1x1x1 + -0.025 x1x1x1x1x1 + 0.002 x1x1x1x1x1x1 + 0.768 x0x1_dot + 1.045 x1x1_dot + -0.002 x1x1x1x1x1x1x1_dot
    x0x0_dot = 0.000
    x1x0_dot = 0.000
    x0x1x0_dot = 0.000
    x0x0x0_dot = 0.000
    x1x1x0_dot = 0.000
    x0x1x1x0_dot = 0.000
    x0x0x0x0_dot = 0.000
    x1x1x1x0_dot = 0.000
    x0x1x1x1x0_dot = 0.000
    x0x0x0x0x0_dot = 0.000
    x1x1x1x1x0_dot = 0.000
    x0x0x0x0x0x0_dot = 0.000
    x1x1x1x1x1x0_dot = 0.000
    x0x0x0x0x0x0x0_dot = 0.000
    x1x1x1x1x1x1x0_dot = 0.000
    x0x1_dot = 0.000
    x1x1_dot = 0.000
    x0x1x1_dot = 0.000
    x0x0x1_dot = 0.000
    x1x1x1_dot = 0.000
    x0x1x1x1_dot = 0.000
    x0x0x0x1_dot = 0.000
    x1x1x1x1_dot = 0.000
    x0x1x1x1x1_dot = 0.000
    x0x0x0x0x1_dot = 0.000
    x1x1x1x1x1_dot = 0.000
    x0x0x0x0x0x1_dot = 0.000
    x1x1x1x1x1x1_dot = 0.000
    x0x0x0x0x0x0x1_dot = 0.000
    x1x1x1x1x1x1x1_dot = 0.000

```python
# Plot the fit of the derivative
x_train = np.asarray(x_train_multi)[0, :, :]
x_dot_train = model.differentiate(x_train, t_train)
x_dot_train_pred = sindy_opt.Theta_[:len(t_train) - 2, :] @ sindy_opt.coef_.T
plt.figure()
inds = [ind1, ind2]
for i in range(r):
    plt.subplot(1, r, i + 1)
    plt.plot(t_train, x_dot_train[:, i], 'k')
    plt.plot(t_train[1:-1], x_dot_train_pred[:, inds[i]], 'r--')
```

#### SINDy-PI model is not good here, but we proceed anyways to illustrate how to do the symbolic solve + integration in more than 1 dimensions.

```python
# Repeat formatting the feature names and building the symbolic equations
features = model.get_feature_names()
nfeatures = len(features)
features[0] = "1"
features_copy = list(np.copy(features))
nfeatures = len(features)
features_formatted = []
# Need to put multiplication between terms for sympy
for i in range(nfeatures):
    for j in range(r):
        # Overkill to make sure all the x0, x1, etc. get replaced
        temp_string = features[i].replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        temp_string = temp_string.replace("x" + str(j) + "x", "x" + str(j) + " * x")
        features[i] = temp_string
    features_formatted.append(temp_string)
features = features_copy

coefs = model.coefficients()
sym_features = [sp.symbols(feature) for feature in features]
sym_theta = [sp.symbols(feature) for feature in features]
sym_equations = []
sym_equations_rounded = []
for i in range(nfeatures):
    if np.any(np.around(coefs[i], 2) != 0.0):
        sym_equation = sp.solve(
            sp.Eq(sym_theta[i], sym_theta @ np.around(coefs[i], 6)), sym_features[i]
        )
        sym_equations.append(sym_equation)
        sym_equation_rounded = sp.solve(
            sp.Eq(sym_theta[i], sym_theta @ np.around(coefs[i], 2)), sym_features[i]
        )
        sym_equations_rounded.append(sym_equation_rounded)

```

```python
# Define the ODE symbol variables
t_sym = sp.symbols('t_sym')
x_sym = sp.symbols('x:%d' % r)
x_dot_sym = sp.symbols('x:%d_dot' % r)

# Need to format the above equations so that there are space between x0 * x0, x0 * x_dot0, and so on.
sym_equations_formatted = []
for i in range(len(sym_equations)):
    for j in range(r):
        # Overkill to make sure all the x0, x1, etc. get replaced
        temp_string = str(sym_equations[i]).replace('x' + str(j) + 'x', 'x' + str(j) + ' * x')
        temp_string = temp_string.replace('x' + str(j) + 'x', 'x' + str(j) + ' * x')
        temp_string = temp_string.replace('x' + str(j) + 'x', 'x' + str(j) + ' * x')
        temp_string = temp_string.replace('x' + str(j) + 'x', 'x' + str(j) + ' * x')
        sym_equations[i] = temp_string
    sym_equations_formatted.append(temp_string)

# IMPORTANT change for multi-variable implicit equations:
# Now that the equations are mathematically formatted,
# solve for x0_dot and x1_dot in the algebraic equation.
# Have two unknowns, (x0_dot, x1_dot),
# so need to solve using two of ANY of the equations,
# but for simplicity we choose the equations for x0_dot and x1_dot
sym_equations_simplified = sp.solve(
    (sp.Add(
        sp.sympify(sym_equations_formatted)[0][0],
        - sp.sympify(features_formatted[ind1])),
    sp.Add(
        sp.sympify(sym_equations_formatted)[1][0],
        - sp.sympify(features_formatted[ind2]))),
    x_dot_sym)
print('Solved for xdot equations = ', sym_equations_simplified)

# Not the cleanest notation for multiple variables, but it works
ODE_Func = sp.lambdify([x_sym, t_sym],
                       [sym_equations_simplified[x_dot_sym[0]],
                        sym_equations_simplified[x_dot_sym[1]]])

```

    Solved for xdot equations =  {x0_dot: (16830573616.0*x0**2*x1 + 210.0*x0*x1**9 - 42428048.0*x0*x1**7 + 18441096.0*x0*x1**6 - 191326371.0*x0*x1**4 + 22894783632.0*x0*x1**2 - 16702074170.0*x0*x1 - 312825531996.0*x0 + 394302.0*x1**12 - 5298930.0*x1**11 + 18117693.0*x1**10 - 69594420.0*x1**8 + 63181862.0*x1**7 + 766455168.0*x1**6 - 260263017.0*x1**5 + 249000000.0*x1**4 + 7076227410.0*x1**2 - 432309439892.0*x1 + 407124000000.0)/(768379000000.0*x0 + 4830.0*x1**12 + 1470.0*x1**11 + 420.0*x1**10 + 210.0*x1**9 - 1937000000.0*x1**6 + 1045233000000.0*x1 - 1000000000000.0), x1_dot: (503792.0*x0*x1**7 + 153328.0*x0*x1**6 + 43808.0*x0*x1**5 + 21904.0*x0*x1**4 - 1000000.0*x0*x1**3 + 552.0*x1**12 + 168.0*x1**11 - 5679.0*x1**10 - 1719.0*x1**9 - 498.0*x1**8 + 155461.0*x1**7 - 2108316462.0*x1**6 + 25230163672.0*x1**5 - 83978807478.0*x1**4 - 407124.0*x1**3 + 331402000000.0*x1**2 - 243856000000.0*x1 - 8838000000.0)/(768379000000.0*x0 + 4830.0*x1**12 + 1470.0*x1**11 + 420.0*x1**10 + 210.0*x1**9 - 1937000000.0*x1**6 + 1045233000000.0*x1 - 1000000000000.0)}

#### We can simulate the system we identified but tends to be unstable -- need more memory to train on additional trajectories.


```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## Regressão Linear

A Regressão Linear pode ser entendida como uma equação da reta, onde existe uma variável Y (dependente) cujo valor está relacionado à X (independente). O objetivo dessa abordagem é chegar ao valor de Y a partir de X.

De maneira geral, o objetivo do algoritmo é gerar uma reta que faça a divisão entre os pontos do gráfico abaixo.

```python
np.random.seed(42)
x = np.linspace(-10,10, 20)
y = x * .1 + 3 + np.random.normal(0,2,size=20)
plt.scatter(x, y)
plt.xlabel('$X$')
plt.ylabel('$y$')
plt.show()
```

Em um gráfico 2D, é possível traçar uma reta de maneira até simples. Entretanto, se trabalharmos com valores mais altos, ou até mesmo urealizar essa operação de forma manual

```python
plt.scatter(x, y)
plt.scatter(1, 1 * .5 + 2, c='C4', s=100) # x=1
plt.plot(x, x * .5 + 2, c='C1')
plt.xlabel('$X$')
plt.ylabel('$y$')
plt.show()
```

\begin{equation}
\label{eq:linear_regression}
y = \alpha + \beta X + \epsilon
\end{equation}

* $\alpha$    =  Interceptação da reta no eixo Y
* $\beta$     = Coeficiente angular
* $\epsilon$  = Erro aleatório
* $X$  = Variável independente

### Gradiente descente

## Referências

[Regressão Linear Simples](https://www.ime.usp.br/~fmachado/MAE229/AULA10.pdf)

[Regressão Linear Simples II](https://edisciplinas.usp.br/pluginfile.php/1479289/mod_resource/content/0/regr_lin.pdf)

[Análise de Regressão](https://www.ime.unicamp.br/~nancy/Cursos/me104/regressao.pdf)

```python

```