# Muestreo de señales

[](https://colab.research.google.com/github/ivan-jgr/computacion-cientifica/blob/main/Laboratorios/Laboratorio-6.ipynb)

Actualmente la mayor parte del procesamiento de señales se realiza de forma digital. Esto significa que las señales se transforman en valores discretos y se manipulan mediante un ordenador.

Cuando la señal original no es un conjunto de valores discretos sino una variación continua a lo largo del tiempo, es necesaria una transformación para obtener la señal en forma digital. Esta transformación se conoce como conversión analógico - digital (A/D), que implica dos etapas: el muestreo y la cuantificación de la señal. El muestreo consiste en tomar valores de la señal cada cierto tiempo, es decir, no considerar todo el tiempo de forma continua sino solamente unas muestras equiespaciadas, con lo que el resultado es un conjunto finito de valores. La cuantificación se refiere a asignar a la amplitud de estos instantes de tiempo escogidos (es decir, al valor que toma la señal en estos instantes) un valor concreto de entre un conjunto finito de posibles valores, que son los que vendrán determinados por el número de bits N considerados en este proceso.

En este laboratorio nos enfocaremos solamente en el muestreo de señales.

Para simplificar el problema, consideremos que la señal a muestrear es una señal sinusoidal de la forma $f(t) = \sin(2 \pi f t)$ donde $f$ es la frecuencia de la señal ( en Hz) y $t$ es la variable temporal. El proceso de muestreo consiste en tomar valores de $f(t)$ en instantes concretos de $t$ esto es en instantes de la forma $t = nT_s$ donde $n$ es un entero que indica el número de muestra:

\begin{equation}
  \left. f[n] = f(t) \right\vert_{t = n T_s}.
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import rcParams
from numpy import pi

# Configuración de las gráficas
from matplotlib import rcParams
rcParams['figure.figsize'] = 16, 6
rcParams['axes.grid'] = True
```

```python
# intervalo de muestreo
t0, t1 = 0, 1
t = np.linspace(t0, t1, 300)

# señal de ejemplo que queremos muestrear
def f(u):
    return np.sin(2*pi*5*u)

plt.plot(t, f(t), label='f(t)')
plt.legend()
plt.show()
```

```python
# muestreamos a una frecuencia fs
fs = 15 # Hz
T = 1/fs # espaciado entre las muestras

# Note que al punto final le sumamos T para poder incluirlo en las muestras
tn = np.arange(t0, t1 + T, T)

print(f'Se tomaron %d muestras a intervalos regulares de duración %.2f s. cada uno' % (len(tn), T))
print(f'Muestras: {f(tn)}')
```

    Se tomaron 16 muestras a intervalos regulares de duración 0.07 s. cada uno
    Muestras: [ 0.00000000e+00 -4.32425679e-01 -9.07591137e-01  4.39201132e+00
     -1.38348220e+00 -8.66025404e-01 -3.30219753e+00  1.49537639e+00
      2.30884968e+00 -1.40008449e+00  8.66025404e-01 -9.67658814e-01
      3.21644081e+00 -2.89663493e+00 -1.91871533e+00  9.79717439e-16]

```python
# Graficamos las muestras y la función
markerline, stemlines, baseline = plt.stem(tn, f(tn), use_line_collection=True)
plt.plot(t, f(t), linestyle='--', color='gray', label='Señal muestreada')

plt.setp(baseline, visible=True, color='k',linewidth=1)
plt.setp(markerline, markersize = 10, label='Muestras')
plt.setp(markerline, markeredgecolor = 'r')

plt.legend()
plt.show()
```

### Frecuencia de Muestreo

La frecuencia con la que debemos tomar las muestras va depende de cómo varía la señal a muestrear: una señal con variaciones lentas precisará de una frecuencia de muestreo menor que una señal de variaciones rápidas.

En términos formales, la frecuencia de muestreo $f_s$ depende de los componentes de frecuencia de la señal que se está muestreando. La mínima frecuencia (teórica) de muestreo $f_s$ necesaria para conseguir recuperar la señal original se conoce como frecuencia de Nyquist ($f_N$).

En clase demostramos que $$f_N = 2 f_{max},$$

donde $f_{max}$ es la frecuencia máxima que contiene la señal a muestrear.

```python
# Definamos una función más compleja para el ejemplo
t0, t1 = 0, 1
t = np.linspace(t0, t1, 300)

# señal de ejemplo que queremos muestrear
def f(u):
    return np.sin(2*pi*2*u)+ 4*np.cos(2*pi*10*u)*np.sin(3*pi*u)

plt.plot(t, f(t), label='f(t)')
plt.legend()
plt.show()
```

```python
# muestreamos a una frecuencia fs
fs = 5 # Hz
T = 1/fs # espaciado entre las muestras

# Note que al punto final le sumamos T para poder incluirlo en las muestras
tn = np.arange(t0, t1 + T, T)

print(f'Se tomaron %d muestras a intervalos regulares de duración %.2f s. cada uno' % (len(tn), T))
print(f'Muestras: {f(tn)}')
```

    Se tomaron 6 muestras a intervalos regulares de duración 0.20 s. cada uno
    Muestras: [ 0.00000000e+00  4.39201132e+00 -3.30219753e+00 -1.40008449e+00
      3.21644081e+00  9.79717439e-16]

La fórmula de interpolación de Whittaker–Shannon nos permite reconstruir una señal a partir de sus muestras.

$$x(t) = \sum_{n = -\infty}^{\infty} x\left[\frac{n}{f_s} \right] \operatorname{sinc} \left(f_s t - n \right)$$

```python
# Aplicamos la formula de interpolación
x_reconstruida = lambda t: sum([f(tk)*np.sinc(fs*(t- tk)) for tk in tn ])
```

```python
plt.plot(t, x_reconstruida(t), color='r', label='Señal recuperada')
plt.plot(t, f(t), linestyle='--', color='gray', label = 'Señal original')
plt.stem(tn, f(tn), use_line_collection=True, label='muestras')
plt.legend()
plt.show()
```

Note que la señal recuperada y la señal original no coinciden puesto que la frecuencia de muestreo utilizada $f_s = 15$ Hz es menor a la frecuencia de Nyquist $f_N = 20$ Hz. A este fenómeno se le conoce como **Aliasing**.

```python
# muestreamos a una frecuencia fs mayor a la frecuencia de Nyquist
fs = 35 # Hz
T = 1/fs # espaciado entre las muestras

# Note que al punto final le sumamos T para poder incluirlo en las muestras
tn = np.arange(t0, t1 + T, T)

print(f'Se tomaron %d muestras a intervalos regulares de duración %.2f s. cada uno' % (len(tn), T))
print(f'Muestras: {f(tn)}')
```

    Se tomaron 36 muestras a intervalos regulares de duración 0.03 s. cada uno
    Muestras: [ 0.00000000e+00  1.14579755e-01 -1.19048640e+00  2.68321644e+00
      3.18711910e+00 -2.53859088e+00 -5.46142270e-02  4.39201132e+00
     -4.76803234e-01 -2.46076855e+00  6.48204607e-01 -2.77481252e-01
     -5.96478866e-01 -6.86240050e-01 -3.30219753e+00 -8.59359958e-02
      2.80096431e+00 -2.64994518e+00 -2.29283139e+00  3.82676286e+00
      1.47772697e+00 -1.40008449e+00  1.31174608e+00  1.24257668e+00
      1.16810848e+00  1.51597209e+00 -2.28148993e+00 -1.00887693e+00
      3.21644081e+00 -1.72376073e+00 -4.48844670e+00  1.20521957e+00
      9.22025374e-01 -2.50636385e+00 -5.88169893e-01  9.79717439e-16]

```python
# Aplicamos la formula de interpolación
x_reconstruida = lambda t: sum([f(tk)*np.sinc(fs*(t- tk)) for tk in tn ])

plt.plot(t, x_reconstruida(t), color='r', label='Señal recuperada')
plt.plot(t, f(t), linestyle='--', color='gray', label = 'Señal original')
plt.stem(tn, f(tn), use_line_collection=True, label='muestras')
plt.legend()
plt.show()
```

### Problema 1

Implemente las siguientes funciones que permiten muestrear una función dada y reconstruirla usando la formula de interpolación de Whittaker-Shannon.

```python
def muestreo(fs, t0, t1):
    """
    Obtienen los puntos en donde se debe muestrear la señal
    a una frecuencia fs (en Hz) en el intervalo t0 <= t < t1

    Params
    fs (float): frecuencia de muestreo.
    t0 (float): float, valor inicial de t.
    t1 (float): float, valor final de t.

    Returns
    numpy.ndarray: los puntos en el dominio en donde se debe muestrear la señal
    """
    pass
```

```python
def interpolacion(x, tn, fs):
    """
    Aplica la formula de interpolacion de Shannon para generar x(t) a partir de las muestras xn

    Params:
    x (function): funcion a muestrear.
    tn (numpy.ndarray): ountos en donde se debe muestrear x(t)
    fs (float):  frecuencia de muestreo.

    Returns:
    function: funcion generada al aplicar la formula de interpolacion.
    """
    pass
```

```python
def plot(x, x_interp, t0, t1):
    """
    Grafica las funciones x y x_interp en el intervalo t0 <= t <= t1

    Params:
    x (function): es una funcion cualquiera
    x_inter (function): es la funcion obtenida al muestrear x y aplicar la formula de interpolacion

    Returns:

    """
    pass
```

###  Problema 2

Para cada una de las siguientes funciones identifique su ancho de banda y elija una frecuencia de muestreo adecuada.

1. $f(t) = \sin(12 \pi t - \frac{\pi}{2}) + \cos(2 \pi t)$

```python
## Seleccione una frecuencia de muestreo adecuada
fs =
```

```python
# Muestreamos en el intervalo 0 <= t <= 12
t0, t1 = 0, 12
# definimos la funcion a muestrear
f1 = lambda t: np.sin(12*pi*t - np.pi/2)+ np.cos(2*pi*t)
# obtenemos los puntos en donde se debe muestrear la funcion
muestras = muestreo(fs, t0, t1)
# aplicamos la formula de Shannon
f1_int = interpolacion(f1, muestras, fs)
# Graficamos, identifique si existe aliasing o no
plot(x, f1_int, t0, t1)
```

2. $f(t) = \cos(3 \pi t)\sin^2(8 \pi t)$

```python
# Muestreamos en el intervalo -5 <= t <= 5
t0, t1 = -5, 5
# definimos la funcion a muestrear
f1 = lambda t: np.cos(3*np.pi*t)*np.sin(8*np.pi*t)**2
# obtenemos los puntos en donde se debe muestrear la funcion
muestras = muestreo(fs, t0, t1)
# aplicamos la formula de Shannon
f1_int = interpolacion(f1, muestras, fs)
# Graficamos, identifique si existe aliasing o no
plot(x, f1_int, t0, t1)
```

3. $f(t) = 2\operatorname{sinc}(20 \pi t) + 3\sin(10 \pi t ) - 0.5 \cos(2 \pi (t-  \frac{\pi}{3}))$

```python
## Seleccione una frecuencia de muestreo adecuada
fs =
```

```python
# Muestreamos en el intervalo -12 <= t <= 12
t0, t1 = -12, 12
# definimos la funcion a muestrear
f1 = lambda t: 2*np.sinc(20*t) + 3*np.sin(10*np.pi*t) - 0.5*np.cos(2*np.pi*(t - np.pi/3))
# obtenemos los puntos en donde se debe muestrear la funcion
muestras = muestreo(fs, t0, t1)
# aplicamos la formula de Shannon
f1_int = interpolacion(f1, muestras, fs)
# Graficamos, identifique si existe aliasing o no
plot(x, f1_int, t0, t1)
```


```julia
using Rocket
using ReactiveMP
using GraphPPL
using Distributions
using BenchmarkTools
using Random
using Plots
```

    ┌ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
    └ @ Base loading.jl:1342

```julia
# All benchmarks have been peroformed on a regular MacbookPro 2018 with Julia 1.6.2
import Pkg; Pkg.status(); versioninfo()
```

         Project ReactiveMPBenchmarkCI v0.1.0
          Status `~/.julia/dev/ReactiveMP/benchmark/Project.toml`
      [20533458] BenchmarkCI v0.1.7
      [6e4b80f9] BenchmarkTools v0.5.0
      [13f3f980] CairoMakie v0.6.3
      [31c24e10] Distributions v0.25.11
      [b3f8163a] GraphPPL v1.0.3 `https://github.com/biaslab/GraphPPL.jl#master`
      [32113eaa] PkgBenchmark v0.2.10
      [91a5bcdd] Plots v1.19.4
      [a194aa59] ReactiveMP v1.0.3 `..`
      [df971d30] Rocket v1.3.14
      [2a0f44e3] Base64
      [37e2e46d] LinearAlgebra
      [9a3f8284] Random
    Julia Version 1.6.2
    Commit 1b93d53fc4 (2021-07-14 15:36 UTC)
    Platform Info:
      OS: macOS (x86_64-apple-darwin18.7.0)
      CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
      WORD_SIZE: 64
      LIBM: libopenlibm
      LLVM: libLLVM-11.0.1 (ORCJIT, skylake)

## State Estimation in a Linear Multivariate Gaussian State-space Model

In this demo, the goal is to smooth noisy observations by performing full inference with a state-space model (SSM).

We wil use the following model:

\begin{equation} \label{eq:linear_gaussian_ssm}
  \begin{aligned}
    \mathbf{x}_k & = \, \mathbf{A}\mathbf{x}_{k - 1} + \mathcal{P} \\
    \mathbf{y}_k & = \, \mathbf{B}\mathbf{x}_{k} + \mathcal{Q} \\
  \end{aligned}
\end{equation}

In this model, we denote by $\mathbf{x}_k$ the current state of the system (at time step $k$), by $\mathbf{x}_{k - 1}$ the previous state at time $k-1$, $\mathbf{A}$ and $\mathbf{B}$ are a constant system inputs and $\mathbf{y}_k$ is a noisy observation of $\mathbf{x}_k$. We further assume that the states and the observations are corrupted by i.i.d. Gaussian noise with variances $\mathcal{P}$ and $\mathcal{Q}$ respectively.

The SSM can be represented by the following factor graph, where the pictured section is chained over time:

For smoothing (forward-backward) estimation in ReactiveMP.jl, it is possible to build a full graph of the model.

```julia
# We create model using GraphPPL.jl package interface with @model macro
# For simplicity of the example we consider all matrices to be known and constant
@model function linear_gaussian_ssm_smoothing(n, A, B, P, Q)

    x = randomvar(n) # Create a sequence of random variables
    y = datavar(Vector{Float64}, n) # Create a sequence of observed multivariate data points

    cA = constvar(A)
    cB = constvar(B)
    cP = constvar(P)
    cQ = constvar(Q)

    # Set a prior distribution for x[1]
    x[1] ~ MvGaussianMeanCovariance([ 0.0, 0.0 ], [ 100.0 0.0; 0.0 100.0 ])
    y[1] ~ MvGaussianMeanCovariance(B * x[1], Q)

    for t in 2:n
        x[t] ~ MvGaussianMeanCovariance(A * x[t - 1], P)
        y[t] ~ MvGaussianMeanCovariance(B * x[t], Q)
    end

    return x, y
end

# It is also possible to create a single time step of the graph
# for filtering algorithm
@model function linear_gaussian_ssm_filtering(A, B, P, Q)

    # Priors for the previous x_t-1 step
    x_min_t_mean = datavar(Vector{Float64})
    x_min_t_cov  = datavar(Matrix{Float64})

    x_min_t ~ MvGaussianMeanCovariance(x_min_t_mean, x_min_t_cov)
    x_t     ~ MvGaussianMeanCovariance(A * x_min_t, P)

    y_t = datavar(Vector{Float64})
    y_t ~ MvGaussianMeanCovariance(B * x_t, Q)

    return x_min_t_mean, x_min_t_cov, x_t, y_t
end
```

    linear_gaussian_ssm_filtering (generic function with 1 method)

```julia
function generate_data(n, A, B, P, Q; seed = 1234)
    Random.seed!(seed)

    x_prev = zeros(2)
    x      = Vector{Vector{Float64}}(undef, n)
    y      = Vector{Vector{Float64}}(undef, n)

    for i in 1:n
        x[i]   = rand(MvNormal(A * x_prev, P))
        y[i]   = rand(MvNormal(B * x[i], Q))
        x_prev = x[i]
    end

    return x, y
end
```

    generate_data (generic function with 1 method)

```julia
n = 100
θ = π / 15
A = [ cos(θ) -sin(θ); sin(θ) cos(θ) ]
B = [ 1.3 0.0; 0.0 0.7 ]
P = [ 1.0 0.0; 0.0 1.0 ]
Q = [ 1.0 0.0; 0.0 1.0 ]
θ = π / 15

real_x, real_y = generate_data(n, A, B, P, Q);
```

```julia
# Inference procedure for full graph
function reactivemp_inference_smoothing(observations, A, B, P, Q)
    n = length(observations)

    model, (x, y) = linear_gaussian_ssm_smoothing(n, A, B, P, Q, options = (limit_stack_depth = 500, ))

    xbuffer       = buffer(Marginal, n)
    xsubscription = subscribe!(getmarginals(x), xbuffer)

    update!(y, observations)
    unsubscribe!(xsubscription)

    return getvalues(xbuffer)
end

# Inference procedure for single time step graph and filtering
function reactivemp_inference_filtering(observations, A, B, P, Q)
    n = length(observations)

    model, (x_min_t_mean, x_min_t_cov, x_t, y_t) = linear_gaussian_ssm_filtering(A, B, P, Q)

    xbuffer = keep(Marginal)

    redirect_to_prior_subscription = subscribe!(getmarginal(x_t), (x_t_posterior) -> begin
        update!(x_min_t_mean, mean(x_t_posterior))
        update!(x_min_t_cov, cov(x_t_posterior))
    end)

    xsubscription = subscribe!(getmarginal(x_t), xbuffer)

    update!(x_min_t_mean, [ 0.0, 0.0 ])
    update!(x_min_t_cov, [ 100.0 0.0; 0.0 100.0 ])

    for observation in observations
        update!(y_t, observation)
    end

    unsubscribe!(xsubscription)
    unsubscribe!(redirect_to_prior_subscription)

    return getvalues(xbuffer)
end
```

    reactivemp_inference_filtering (generic function with 1 method)

```julia
x_reactivemp_smoothing_inferred = reactivemp_inference_smoothing(real_y, A, B, P, Q);
```

```julia
x_reactivemp_filtering_inferred = reactivemp_inference_filtering(real_y, A, B, P, Q);
```

```julia
# Results verification

reshape_data(data) = transpose(reduce(hcat, data))

ylimit = (-20, 20)

generated_data = plot(1:n, real_x |> reshape_data, label = [ "x[:, 1]" "x[:, 2]" ])
generated_data = plot!(1:n, real_y |> reshape_data, seriestype = :scatter, ms = 3, alpha = 0.5, label = [ "observations[:, 1]" "observations[:, 2]" ])
generated_data = plot!(generated_data, legend = :bottomleft, ylimit = ylimit)

rmp_smoothing_results_plot = plot(1:n, real_x |> reshape_data, label = [ "x[:, 1]" "x[:, 2]" ])
rmp_smoothing_results_plot = plot!(rmp_smoothing_results_plot, 1:n, mean.(x_reactivemp_smoothing_inferred) |> reshape_data, ribbon = var.(x_reactivemp_smoothing_inferred) |> reshape_data, label = [ "inferred[:, 1]" "inferred[:, 2]" ])
rmp_smoothing_results_plot = plot!(rmp_smoothing_results_plot, legend = :bottomleft, ylimit = ylimit)

rmp_filtering_results_plot = plot(1:n, real_x |> reshape_data, label = [ "x[:, 1]" "x[:, 2]" ])
rmp_filtering_results_plot = plot!(rmp_filtering_results_plot, 1:n, mean.(x_reactivemp_filtering_inferred) |> reshape_data, ribbon = var.(x_reactivemp_filtering_inferred) |> reshape_data, label = [ "inferred[:, 1]" "inferred[:, 2]" ])
rmp_filtering_results_plot = plot!(rmp_filtering_results_plot, legend = :bottomleft, ylimit = ylimit)

plot(generated_data, rmp_smoothing_results_plot, rmp_filtering_results_plot, layout = @layout([ a b c ]), size = (1200, 300))
```

```julia
benchmark_rmp_sizes = [ 50, 100, 250, 500, 1_000, 2_000, 5_000, 10_000, 15_000, 20_000, 25_000, 50_000 ];
```

```julia
reactivemp_benchmark_results = map(benchmark_rmp_sizes) do size
    states, observations = generate_data(size, A, B, P, Q);
    benchmark_smoothing = @benchmark reactivemp_inference_smoothing($observations, $A, $B, $P, $Q)
    benchmark_fitlering = @benchmark reactivemp_inference_filtering($observations, $A, $B, $P, $Q)
    println("Finished $size")
    return (size, benchmark_smoothing, benchmark_fitlering)
end
```

    Finished 50
    Finished 100
    Finished 250
    Finished 500
    Finished 1000
    Finished 2000
    Finished 5000
    Finished 10000
    Finished 15000
    Finished 20000
    Finished 25000
    Finished 50000

    12-element Vector{Tuple{Int64, BenchmarkTools.Trial, BenchmarkTools.Trial}}:
     (50, 3.495 ms, 411.618 μs)
     (100, 8.382 ms, 742.607 μs)
     (250, 23.451 ms, 1.755 ms)
     (500, 48.582 ms, 3.453 ms)
     (1000, 103.082 ms, 6.994 ms)
     (2000, 246.209 ms, 14.194 ms)
     (5000, 740.280 ms, 38.176 ms)
     (10000, 1.450 s, 98.812 ms)
     (15000, 2.429 s, 144.916 ms)
     (20000, 3.660 s, 199.967 ms)
     (25000, 4.385 s, 245.062 ms)
     (50000, 10.652 s, 519.485 ms)

```julia
benchmark_time_ms(trial) = minimum(trial).time / 1_000_000
```

    benchmark_time_ms (generic function with 1 method)

```julia
lgssm_scaling = plot(xscale = :log10, yscale = :log10, xlabel = "Number of observations", ylabel = "Minimum execution time (in ms)", title = "ReactiveMP.jl Linear Gaussian State Space Model Benchmark", legend = :bottomright, size = (650, 400))
lgssm_scaling = plot!(lgssm_scaling, benchmark_rmp_sizes, map(i -> benchmark_time_ms(reactivemp_benchmark_results[i][2]), 1:length(benchmark_rmp_sizes)), markershape = :diamond, label = "Smoothing")
lgssm_scaling = plot!(lgssm_scaling, benchmark_rmp_sizes, map(i -> benchmark_time_ms(reactivemp_benchmark_results[i][3]), 1:length(benchmark_rmp_sizes)), markershape = :utriangle, label = "Filtering")

display(lgssm_scaling)

# savefig(lgssm_scaling, "./plots/lgssm_scaling.svg")
```

```julia
import Turing
```

```julia
Turing.@model LinearGaussianSSM(y, A, B, P, Q) = begin
    n = length(y)

    # State sequence.
    x = Vector(undef, n)

    # Observe each point of the input.
    x[1] ~ MvNormal([ 0.0, 0.0 ], 1e1)
    y[1] ~ MvNormal(B * x[1], Q)

    for t in 2:n
        x[t] ~ MvNormal(A * x[t - 1], P)
        y[t] ~ MvNormal(B * x[t], Q)
    end
end;
```

```julia
function turing_inference_smoothing(observations, A, B, P, Q; nsamples = 500)
    return Turing.sample(LinearGaussianSSM(observations, A, B, P, Q), Turing.HMC(0.1, 5), nsamples)
end
```

    turing_inference_smoothing (generic function with 1 method)

```julia
x_turing_500_smoothing_inferred  = turing_inference_smoothing(real_y, A, B, P, Q, nsamples = 500);
x_turing_1000_smoothing_inferred = turing_inference_smoothing(real_y, A, B, P, Q, nsamples = 1000);
```

    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:32
    Sampling: 100%|█████████████████████████████████████████| Time: 0:01:06

```julia
samples_500  = get(x_turing_500_smoothing_inferred, :x);
samples_1000 = get(x_turing_1000_smoothing_inferred, :x);

mx_500 = [ mean(samples_500.x[i].data) for i in 1:2n ]
vx_500 = [ std(samples_500.x[i].data)  for i in 1:2n ];

mx_1000 = [ mean(samples_1000.x[i].data) for i in 1:2n ]
vx_1000 = [ std(samples_1000.x[i].data)  for i in 1:2n ];
```

```julia
reshape_turing_data(data) = transpose(reshape(data, (2, Int(length(data) / 2))))
```

    reshape_turing_data (generic function with 1 method)

```julia
generated_data = plot(1:n, real_x |> reshape_data, label = [ "x[:, 1]" "x[:, 2]" ])
generated_data = plot!(1:n, real_y |> reshape_data, seriestype = :scatter, ms = 3, alpha = 0.5, label = [ "observations[:, 1]" "observations[:, 2]" ])
generated_data = plot!(generated_data, legend = :bottomleft)

turing_500_results_plot = plot(1:n, real_x |> reshape_data, label = [ "x[:, 1]" "x[:, 2]" ])
turing_500_results_plot = plot!(turing_500_results_plot, 1:n, mx_500 |> reshape_turing_data, ribbon = vx_500 |> reshape_turing_data, label = [ "inferred[:, 1]" "inferred[:, 2]" ])
turing_500_results_plot = plot!(turing_500_results_plot, legend = :bottomleft)

turing_1000_results_plot = plot(1:n, real_x |> reshape_data, label = [ "x[:, 1]" "x[:, 2]" ])
turing_1000_results_plot = plot!(turing_1000_results_plot, 1:n, mx_1000 |> reshape_turing_data, ribbon = vx_1000 |> reshape_turing_data, label = [ "inferred[:, 1]" "inferred[:, 2]" ])
turing_1000_results_plot = plot!(turing_1000_results_plot, legend = :bottomleft)

plot(generated_data, turing_500_results_plot, turing_1000_results_plot, layout = @layout([ a b c ]), size = (1200, 300))
```

```julia
reactivemp_filtering_n_100_benchmark = @benchmark reactivemp_inference_filtering($real_y, $A, $B, $P, $Q);
reactivemp_smoothing_n_100_benchmark = @benchmark reactivemp_inference_smoothing($real_y, $A, $B, $P, $Q);
turing_500_n_100_benchmark  = @benchmark turing_inference_smoothing($real_y, $A, $B, $P, $Q, nsamples = 500);
turing_1000_n_100_benchmark = @benchmark turing_inference_smoothing($real_y, $A, $B, $P, $Q, nsamples = 1000);
```

    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:29
    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:30
    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:25
    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:25
    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:52
    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:53
    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:50
    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:51

```julia
using ColorSchemes, Colors ;

c = palette(:rainbow)

c1 = get(ColorSchemes.rainbow, 0.5) ;
c2 = get(ColorSchemes.rainbow, 0.2) ;
c3 = get(ColorSchemes.rainbow, 0.3) ;
c4 = get(ColorSchemes.rainbow, 0.8) ;
c5 = get(ColorSchemes.rainbow, 0.9) ;

colors = [ c1, c2, c3, c4 ];
```

```julia
import CairoMakie
```

```julia
comparison_fig = begin

    # Local vars
    local bar_names  = [ "ReactiveMP Filtering", "ReactiveMP Smoothing", "Turing HMC(500)", "Turing HMC(1000)" ]
    local bar_values = [
        benchmark_time_ms(reactivemp_filtering_n_100_benchmark),
        benchmark_time_ms(reactivemp_smoothing_n_100_benchmark),
        benchmark_time_ms(turing_500_n_100_benchmark),
        benchmark_time_ms(turing_1000_n_100_benchmark)
    ]

    local range = 1:length(bar_names)

    # Figure

    local fig = CairoMakie.Figure(resolution = (650, 500))

    # Axis

    local ax  = CairoMakie.Axis(fig)
    ax.ylabel = "Inference time (log scale, in ms)"
    ax.title  = "Linear Gaussian State Space Model for 100 observations"
    ax.yscale = CairoMakie.Makie.pseudolog10
    ax.yticks = ([ 1, 10, 100, 1000, 10_000, 50_000 ], [ "1", "10", "1e2", "1e3", "1e4", "5e4" ])
    ax.xticklabelrotation = -π / 16
    ax.xticklabelalign = (0.5, 1.5)

    ax.xticks = (range, bar_names)
    ax.xticklabelsize = 14

    # Barplot 1

    local bp = CairoMakie.barplot!(ax, range, bar_values)

    bp.strokewidth = 1
    bp.color    = range
    bp.colormap = (:Accent_4, 0.9)
    bp.strokecolor = :black
    bp.x_gap = 0.12

    local text_values = map(bar_values) do value
        return string(round(value, digits = 2), " ms")
    end

    text_values[2] = string(text_values[2], "\n", string("ratio = ", Int(round(bar_values[2] / bar_values[2])), "\n(baseline)"))
    text_values[3] = string(text_values[3], "\n\n", string("ratio = ", Int(round(bar_values[3] / bar_values[2])), ""))
    text_values[4] = string(text_values[4], "\n\n", string("ratio = ", Int(round(bar_values[4] / bar_values[2])), ""))

    local tx = CairoMakie.text!(text_values, position = CairoMakie.Point.(range, bar_values .+ [ 1.55, 0.25, 0, 0 ]))

    tx.align  = (:center, :center)
    tx.offset = (0, -30)
    tx.color  = [ :black, :black, :black, :black ]
    tx.textsize = 13

    # Layout

    fig[1, 1] = ax

    # Return

    CairoMakie.save("./plots/lgssm_comparison.svg", fig)
    CairoMakie.save("./plots/lgssm_comparison.png", fig)

    fig
end
```

```julia
scaling_fig = begin
    local fig = CairoMakie.Figure(resolution = (650, 500))

    local ax = CairoMakie.Axis(fig[1, 1])

    ax.title = "ReactiveMP.jl Linear Gaussian State Space Model Benchmark"

    ax.xlabel = "Number of observations (log scale)"
    ax.xscale = CairoMakie.Makie.pseudolog10
    ax.xticks = (
        [ 50, 100, 250, 500, 1_000, 2_000, 5_000, 10_000, 20_000, 50_000 ],
        [ "50", "100", "250", "500", "1e3", "2e3", "5e3", "1e4", "2e4", "5e4" ]
    )

    ax.ylabel = "Minimum execution time (log scale, in ms)"
    ax.yscale = CairoMakie.Makie.pseudolog10
    ax.yticks = (
        [ 1, 10, 100, 1_000, 10_000, 50_000 ],
        [ "1", "10", "100", "1e3", "1e4", "5e4" ]
    )

    local range = benchmark_rmp_sizes
    local smoothing = map(i -> benchmark_time_ms(reactivemp_benchmark_results[i][2]), 1:length(benchmark_rmp_sizes))
    local filtering = map(i -> benchmark_time_ms(reactivemp_benchmark_results[i][3]), 1:length(benchmark_rmp_sizes))

    CairoMakie.lines!(ax, range, smoothing)
    CairoMakie.scatter!(ax, range, smoothing, marker = :diamond, markersize = 15, label = "Smoothing")

    CairoMakie.lines!(ax, range, filtering)
    CairoMakie.scatter!(ax, range, filtering, marker = :circle, markersize = 15, label = "Filtering")

    CairoMakie.axislegend(position = :rb)

    CairoMakie.save("./plots/lgssm_scaling.svg", fig)
    CairoMakie.save("./plots/lgssm_scaling.png", fig)

    fig
end
```

```julia

```

benchmark/notebooks/Linear Multivariate Gaussian State Space Model Benchmark.ipynb


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2015 L.A. Barba, Pi-Yueh Chuang.

# Joukowski Transformation

This is your [_AeroPython_](https://github.com/barbagroup/AeroPython) assignment for the second course module, titled **"Potential vortices and lift."** The first course module, "Building blocks of potential flow" (first three IPython Notebooks of the series) ended with the representation of potential flow around a 2D cylinder by means of superposing a doublet singularity and a free stream. And in the second module, you learned that adding a vortex singularity, you can get lift around the cylinder. You may ask: *is all this useful?*

Here's how it all starts getting useful. Using some simple techniques from complex-variable calculus, we can generate the flow around some airfoils starting with the flow around 2D cylinders. The trick is to use a [*conformal map*](http://en.wikipedia.org/wiki/Conformal_map) (a complex function that preserves angles) to move from the cylinder plane to the airfoil plane.

Let's explore this classic method!

## Introduction

You learned how to construct potential flow over a cylinder via superposition of a free stream and a [doublet](http://nbviewer.ipython.org/github/barbagroup/AeroPython/blob/master/lessons/03_Lesson03_doublet.ipynb) singularity. You also learned that adding a vortex can generate [lift on the cylinder](http://nbviewer.ipython.org/github/barbagroup/AeroPython/blob/master/lessons/06_Lesson06_vortexLift.ipynb) But what's the big deal? Why is this important to us? What can we do with this potential cylinder flow?

Back in the years when computers were not available, fluid dynamicists and mathematicians used a powerful tool—complex analysis—to study potential flows without directly solving partial-differential governing equations. With the magic of complex analysis and the known solution of potential flow over a cylinder, they could easily obtain many kinds of external potential flows, including flows over several types of airfoils.

Nowadays, though, we no longer use these magical tools. But, not just out of nostalgia, it's still interesting to know the basic concept behind the magic: conformal mappings. In this assignment, we will guide you step by step to obtain potential flow over an airfoil starting with the flow over a cylinder and the famous conformal mapping called the **Joukowski transformation**. You will realize how important potential cylinder flow is in the history of aerodynamics!

Don't worry. We won't talk too much about the mathematics. And you don't have to calculate by hand as the pioneers of aerodynamics did. Just follow the steps and let Python do the heavy lifting.

## 1. Complex numbers in Python

Let's start with two complex planes, one defined by points $z = x + iy$, and the other defined by points $\xi = \xi_x+i\xi_y$. The Joukowski transformation takes a point in the $z$-plane and "maps" it to the $\xi$-plane by:

\begin{equation}
\xi = z + \frac{c^2}{z}
\end{equation}

where $c$ is a constant parameter. Before we discuss the Joukowski transformation, let's gain some practice with complex numbers in Python.

Using complex numbers, your function for the Joukowski transformation will look
very simple, and you won't have to calculate the real and imaginary parts separately.

Python (and therefore, NumPy) can deal with complex numbers right out of the box. But the imaginary number, $i=\sqrt{-1}$, is represented by `j`, *not* `i`, to avoid clashes with the common use of `i` in iterations.

If you haven't used them before, try some simple manipulations of complex numbers now. For example, enter the following in a cell block:

```Python
3 + 2j
```

Now try:

```Python
a = 3
b = 3
z = a + b * 1j
print('z = ', z)
print('The type of the variable is ', type(z))
```

### Exercises:

Get familiar with complex-number operations in Python and obtain the answer to the following:

1. $(2.75+3.69i)\times(8.55-6.13i)=$

2. $1.4\times e^{i5.32}=$

3. $\frac{7.51-9.15i}{4.43+9.64i}=$

In the auto-graded homework in our [Open edX course](http://openedx.seas.gwu.edu/courses/GW/MAE6226/2014_spring/about), you need to enter the results with two digits after the decimal point.

## 2. Shapes created by the Joukowski transformation

Start by writing a Python function that takes `z` and `c` as parameters, and returns  the Joukowski transformation of `z`.

We can generate several patterns using Joukowski transforms. Use your Python function to perform the calculations described below and answer the questions.

Just take $c=1$ here for simplicity.

1. On the $z$-plane, place a circle with radius $R$ larger than $c=1$, say $R=1.5$, centered at the origin. What does the shape look like when mapped on the $\xi$-plane?
    1. circle
    2. ellipse
    3. symmetric airfoil
    4. cambered airfoil
2. Now, place a circle on the $z$-plane with radius such that $c \lt R \lt 2c$ (e.g., $c=1$; $R=1.2$), centered at $(x_c,y_c)=(c-R, 0)$. What does the shape look like on the $\xi$-plane?
    1. circle
    2. ellipse
    3. symmetric airfoil
    4. cambered airfoil
3. Put a circle at $(x_c, y_c)=(-\Delta x, \Delta y)$, where $\Delta x$ and $\Delta y$ are small positive numbers, e.g. $\Delta x=0.1$ and $\Delta y=0.1$. The radius $R$ of the circle is $R = \sqrt{(c - x_c)^2 + y_c^2}$. What does the shape on $\xi$-plane look like?
    1. circle
    2. ellipse
    3. symmetric airfoil
    4. cambered airfoil
4. Consider the case when you obtain a symmetric airfoil. In polar coordinates, $(\theta, r=R)$, which point on the circle corresponds to the trailing edge of the airfoil?
    * $\theta=$?

## 3. Grid points on the $z$-plane using polar coordinates

By the Joukowski transformation, a point on the $z$-plane corresponds to a point on the $\xi$-plane. As you saw in the previous section, this transformation sometimes gives a shape that looks quite a lot like an airfoil. _What's the use of this?_

It turns out that in complex analysis, when you have a solution to Laplace's equation on the complex plane and apply a [conformal mapping](http://en.wikipedia.org/wiki/Conformal_map), the transformed function is still a solution of the Laplace equation.

This means that we can map the potential function and the streamlines of flow over a cylinder on the $z$-plane to the $\xi$-plane, and obtain the streamlines of flow over an airfoil. The stream function around the airfoil will be given by:

$$
\psi(\xi_x, \xi_y) = \psi(\xi_x(x, y), \xi_y(x, y))
$$

where the complex coordinates of $\xi$, $\xi_x$ and $\xi_y$, are obtained from the Joukowski transformation of $z=x+iy$.

In this exercise, you will obtain the flow over a symmetric Joukowski airfoil with zero and non-zero angle of attack. We obtain the shape of our target airfoil by placing a cylinder centered at $(x_c, y_c)=(-0.15, 0)$ with radius $R = 1.15$ and parameter $c=1$. You'll reach the goal step-by-step with problems (3) to (6).

First, build a set of grid points on the $z$-plane and see how these points look like on the $\xi$-plane. Use polar coordinates to build your grid on the $z$-plane. If you place grid points inside the cylinder, they will end up outside the airfoil after a Joukowski transformation (try it for yourself!). This is a problem. The streamlines inside the cylinder are not physical anyway, so we'll just ignore this region.

### Exercises:

Build a grid on the $z$-plane using polar coordinates and map the grid to the $\xi$-plane using a Joukowski transformation.

Discretize the radial direction into $N_r = 100$ grid points in the range $R \le r \le 5$ and the axial direction into $N_{\theta}=145$ grid points. You want to end up with  two grid points at $\theta=0^\circ$. If you plot the grid points using the function `pyplot.scatter()`, it should look something like this:

## 4. Flow over a symmetric Joukowski airfoil with zero angle of attack

### Stream function and streamlines

Now, you will evaluate potential flow over the cylinder on the $z$-plane. As mentioned above, $\psi(\xi) = \psi(\xi(z))$. This means that, after evaluating the stream function at a certain point on the $z$-plane, the corresponding point on the $\xi$-plane has the same value of the stream function. We can plot the streamlines on both planes using the function `pyplot.contour()`, because the stream function is a scalar function.

Use $1$ as the free stream velocity, i.e., $U_{\infty}=1$. You have to calculate the strength of the doublet first in order to have a cylinder with radius $R=1.15$.

You should obtain streamline patterns similar to those shown in the following figures.

### Velocity vectors and pressure coefficients

In order to obtain the pressure coefficients, we also need the velocity fields. We can simply obtain the velocity field on the $z$-plane using the coordinates of the grid points on the $z$-plane. But can we just say that the velocity at the corresponding points on the $\xi$-plane is the same as that on the $z$-plane, just like what we did for the stream function? _The answer is no._

The values of stream function remain the same on the original and mapped points because the stream function is a scalar solution of Laplace's equation!

However, velocity is a vector and it is not a solution of Laplace's equation. When the coordinate system changes by the conformal map, the coordinate values of the vector in the new system are different.

Now, let's go back to our problem: the $z$-plane and the $\xi$-plane are two different coordinate systems. The velocity at a specific point on the $z$-plane is not the same as that at the corresponding point on the $\xi$-plane. Some manipulation must be done.

The velocities on the $z$-plane and $\xi$-plane are:

\begin{equation}
\left\{
\begin{array}{l}
u_z = \frac{\partial \psi}{\partial y} \\
v_z = -\frac{\partial \psi}{\partial x}
\end{array}
\right.
\text{  and  }
\left\{
\begin{array}{l}
u_\xi = \frac{\partial \psi}{\partial \xi_y} \\
v_\xi = - \frac{\partial \psi}{\partial \xi_x}
\end{array}
\right.
\end{equation}

Once we have $u_z$ and $v_z$, how can we obtain $u_\xi$ and $v_\xi$? This can be done through the chain rule. Recall the potential function $\phi$, which is a solution of Laplace's equation. Therefore, the potential and stream function values remain the same at a point under the conformal transformation. The same applies to the complex potential, $F(\xi)  = F(\xi(z))= \phi + i\psi$.

By the chain rule, $dF/d\xi=dF/dz\times dz/d\xi$. So,

\begin{equation}
W_\xi = u_\xi - iv_\xi = \frac{d F}{d \xi} = \frac{d F}{d z}\times\frac{d z}{d \xi} = \frac{d F}{d z}/\frac{d \xi}{d z} = (u_z-iv_z) / \frac{d \xi}{d z}
\end{equation}

And

\begin{equation}
\frac{d \xi}{d z} = \frac{d (z + c^2/z)}{dz} = 1 - \left(\frac{c}{z}\right)^2
\end{equation}

Now, we can obtain the velocity on the $\xi$-plane, $u_\xi$ and $v_\xi$, using the two equations above.

Using the function `pyplot.quiver()` to plot vectors, the velocity field should look like this:

And the pressure coefficients in the fluid domain should look like this:

### Exercises:

* Write Python code to obtain the plots as shown in the above figures: streamlines, velocity vectors, and pressure coefficient on the domain.
* Answer the following questions:
    1. What is the strength of the doublet?
    2. What is the velocity at 62nd point on the airfoil surface? Assume we index the trailing edge as the 1st point and index the points counterclockwise.
    3. What is the minimum value of the pressure coefficient on the airfoil surface?

In the auto-graded homework in our [Open edX course](http://openedx.seas.gwu.edu/courses/GW/MAE6226/2014_spring/about), you need to enter the results with two digits after decimal point.

## 5. Flow over a symmetric Joukowski airfoil with non-zero angle of attack, without circulation

Now, we want to place the airfoil at an angle of attack (AoA) with respect to the free stream. Of course we can use flow over a cylinder and the Joukowski tranformation to do this. *But how?* Can we superpose a free stream with non-zero inlet angle and a doublet to obtain what we want? Actually, we can't. If we do so, we won't be able to obtain a closed streamline, like we did in the previous case (in which a closed circular streamline can be taken as a cylinder surface).

A way to achieve a uniform flow with an inlet angle is simply rotating the coordinate system. We first create a new coordinate system (or we can call it a complex plane here) $z'$, where the $x'$-axis (i.e., real part of $z'$) is parallel to our free stream, and its origin is located at the center of the cylinder, as shown in the picture.

The relationship between the $z'$- and $z$-plane is:

\begin{equation}
z'=\left[ z-(x_c+iy_c) \right]e^{-i\times AoA}
\end{equation}

Explicitly expressed in $x$, $y$, $x'$, and $y'$:

\begin{equation}
\left\{
\begin{array}{l}
x' = (x-x_c)\cos(AoA) + (y-y_c)\sin(AoA) \\
y' = - (x-x_c)\sin(AoA) + (y-y_c)\cos(AoA)
\end{array}
\right.
\end{equation}

where $(x_c, y_c)$ is the location of the center of the cylinder, and $AoA$ is the angle of attack.

Now, we can obtain flow over a cylinder on the new plane $z'$ by adding a free stream with **zero inlet angle** to a doublet **centered at the origin**. Then, we can obtain the flow on the $z$-plane and the $\xi$-plane. Again, the stream function remains the same at the same point under three different coordinate systems ($z'$, $z$, and $\xi$). You should get streamlines on $z$- and $\xi$-planes that look like this:

The velocity vectors need to be rotated back from the $z'$-plane to $z$-plane.

\begin{equation}
u-iv=\frac{d F}{d z}=\frac{d F}{d z'}\times\frac{d z'}{d z}=(u'-iv')e^{-i\times AoA}
\end{equation}

Of course you can use an explicit form in $x$, $y$, $x'$, and $y'$. Derive it yourself, if you feel more comfortable using an explicit form. After obtaining the velocity on the $z$-plane, you can use what you learned in the last exercise to obtain the velocity on $\xi$-plane. You should get velocity field and pressure coefficient plots that look like this:

### Exercises:

* Write Python code to obtain the figures as shown above. Set the angle of attack as $AoA=20^\circ$.
* Answer the following questions:
    1. Do you think the flow over the airfoil we obtained here is reasonable? Why or why not?
    2. Where are the stagnation points on the airfoil? Assume we index the trailing edge as the 1st point and index the points counterclockwise.
    3. How much is the lift?
    4. How much is the drag?
    5. What is the velocity at the 50th point on the airfoil surface?
    6. What is the pressure coefficient at the 75th point on the airfoil surface?

Numerical answers in the auto-graded homework in our [Open edX course](http://openedx.seas.gwu.edu/courses/GW/MAE6226/2014_spring/about) need to be entered with two digits after the decimal point.

## 6. Flow over a symmetric Joukowski airfoil with non-zero angle of attack, with circulation

The result you obtained in the last exercise is not physical. What we need is a **vortex**. As you saw in [Lesson 6: Vortex Lift](http://nbviewer.ipython.org/urls/github.com/barbagroup/AeroPython/blob/master/lessons/06_Lesson06_vortexLift.ipynb), adding a vortex (i.e., circulation) to potential flow over a cylinder can modify the position of the stagnation points and also generate lift.

We need to satisfy the [Kutta condition](http://en.wikipedia.org/wiki/Kutta_condition),
>"A body with a sharp trailing edge which is moving through a fluid will create about itself a circulation of sufficient strength to hold the rear stagnation point at the trailing edge."

This tells us that the strength of the vortex needs to move the rear stagnation point on the cylinder from the $\theta=$AoA to $\theta=0^\circ$ on the $z$-plane. You should be able to calculate the strength through what you learned in Lesson 6.

The results look like this:

### Exercises

* Write a Python code to obtain the figures shown above.
* Answer the following questions:
    1. What is the strength of the vortex?
    2. What is the lift? (Hint: the direction of lift we learned in Lesson 6 is toward the normal direction of free stream, i.e., the $y'$-direction in our case. Don't forget that the direction of lift now is in the $y$-direction.)
    3. Try to calculate the lift and drag directly by $L=-\oint p \times \sin{\theta} dA$ and $D=\oint p \times \cos{\theta} dA$. Does the lift agree with the Kutta-Joukowski theorem? How much is the drag?
    4. Where are the stagnation points on the airfoil? Assume we index the trailing edge as the 1st point and index the points counterclockwise.
    5. What is the velocity at the 92th point on the airfoil surface?
    6. What is the pressure coefficient at the 111th point on the airfoil surface?
    7. What's going on with the pressure coefficient at the trailing edge of the airfoil?

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('../styles/custom.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


# Gaussian Process Distribution of Relaxation Times

## In this tutorial we will reproduce Figure 7 of the article https://doi.org/10.1016/j.electacta.2019.135316

GP-DRT is our newly developed approach that can be used to obtain both the mean and covariance of the DRT from the EIS data by assuming that the DRT is a Gaussian process (GP). The GP-DRP can  predict the DRT and the imaginary part of the impedance at frequencies that were not previously measured.

To obtain the DRT from the impedance we take that $\gamma(\xi)$ is a GP where $f$ is the frequency and $\xi=\log f$. Under the DRT model and considering that GPs are closed linear transformations, it follows that $Z^{\rm DRT}_{\rm im}\left(\xi\right)$ is also a GP.

More precisely we can write

$$\begin{pmatrix}
\gamma(\xi) \\
Z^{\rm DRT}_{\rm im}\left(\xi\right)
\end{pmatrix}\sim \mathcal{GP}\left(\mathbf 0, \begin{pmatrix}
k(\xi, \xi^\prime) & \mathcal L^{\rm im}_{\xi^\prime} \left(k(\xi, \xi^\prime)\right)\\
\mathcal L^{\rm im}_{\xi} k(\xi, \xi^\prime) & \mathcal L^{\rm im}_{\xi^\prime}\left(\mathcal L^{\rm im}_{\xi} \left(k(\xi, \xi^\prime)\right)\right)
\end{pmatrix}\right)$$

where

$$\mathcal L^{\rm im}_\xi \left(\cdot\right) = -\displaystyle \int_{-\infty}^\infty \frac{2\pi \displaystyle e^{\xi-\hat \xi}}{1+\left(2\pi \displaystyle e^{\xi-\hat \xi}\right)^2}  \left(\cdot\right) d \hat \xi$$

is a linear functional. The latter functional, transforms the DRT to the imaginary part of the impedance.

Assuming we have $N$ observations, we can set $\left(\mathbf Z^{\rm exp}_{\rm im}\right)_n = Z^{\rm exp}_{\rm im}(\xi_n)$ with $\xi_n =\log f_n$ and $n =1, 2, \ldots N $. The corresponding multivariate Gaussian random variable can be written as

$$\begin{pmatrix}
\boldsymbol{\gamma} \\
\mathbf Z^{\rm exp}_{\rm im}
\end{pmatrix}\sim \mathcal{N}\left(\mathbf 0, \begin{pmatrix}
\mathbf K & \mathcal L_{\rm im} \mathbf K\\
\mathcal L_{\rm im}^\sharp \mathbf K & \mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I
\end{pmatrix}\right)$$

where

$$\begin{align}
(\mathbf K)_{nm} &= k(\xi_n, \xi_m)\\
(\mathcal L_{\rm im} \mathbf K)_{nm} &= \left. \mathcal L^{\rm im}_{\xi^\prime} \left(k(\xi, \xi^\prime)\right) \right |_{\xi_n, \xi_m}\\
(\mathcal L_{\rm im}^\sharp \mathbf K)_{nm} &= \left.\mathcal L^{\rm im}_{\xi} \left(k(\xi, \xi^\prime)\right) \right|_{\xi_n, \xi_m}\\
(\mathcal L^2_{\rm im} \mathbf K)_{nm} &= \left.\mathcal L^{\rm im}_{\xi^\prime}\left(\mathcal L^{\rm im}_{\xi} \left(k(\xi, \xi^\prime)\right)\right) \right|_{\xi_n, \xi_m}
\end{align}$$

and $\mathcal L_{\rm im} \mathbf K^\top = \mathcal L_{\rm im}^\sharp \mathbf K$.

To obtain the DRT from impedance, the distribution of $\mathbf{\gamma}$ conditioned on $\mathbf Z^{\rm exp}_{\rm im}$ can be written as

$$\boldsymbol{\gamma}|\mathbf Z^{\rm exp}_{\rm im}\sim \mathcal N\left( \mathbf \mu_{\gamma|Z^{\rm exp}_{\rm im}}, \mathbf\Sigma_{\gamma| Z^{\rm exp}_{\rm im}}\right)$$

with

$$\begin{align}
\mathbf \mu_{\gamma|Z^{\rm exp}_{\rm im}} &= \mathcal L_{\rm im} \mathbf K \left(\mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I \right)^{-1} \mathbf Z^{\rm exp}_{\rm im} \\
\mathbf \Sigma_{\gamma| Z^{\rm exp}_{\rm im}} &= \mathbf K- \mathcal L_{\rm im} \mathbf K \left(\mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I \right)^{-1}\mathcal L_{\rm im} \mathbf K^\top
\end{align}$$

```python
import numpy as np
import matplotlib.pyplot as plt
from math import sin, cos, pi
import GP_DRT
from scipy.optimize import minimize
%matplotlib inline
```

## 1) Define parameters of the ZARC circuit which will be used for the synthetic experiment generation

The impedance of a ZARC can be written as
$$
Z^{\rm exact}(f) = R_\infty + \displaystyle \frac{1}{\displaystyle \frac{1}{R_{\rm ct}}+C \left(i 2\pi f\right)^\phi}
$$

where $\displaystyle C = \frac{\tau_0^\phi}{R_{\rm ct}}$.

The analytical DRT can be computed analytically as

$$
\gamma(\log \tau) =  \displaystyle \frac{\displaystyle R_{\rm ct}}{\displaystyle 2\pi} \displaystyle \frac{\displaystyle \sin\left((1-\phi)\pi\right)}{\displaystyle \cosh(\phi \log(\tau/\tau_0))-\cos(\pi(1-\phi))}
$$

```python
# number of frequencies
N_freqs = 81

# define the frequency range
freq_vec = np.logspace(-4., 4., num=N_freqs, endpoint=True)
xi_vec = np.log(freq_vec)
tau  = 1/freq_vec

# define the frequency range used for prediction
# note: we could have used other values
freq_vec_star = np.logspace(-4., 4., num=N_freqs, endpoint=True)
xi_vec_star = np.log(freq_vec_star)

# parameters for ZARC model, the impedance and analytical DRT are calculated as the above equations
R_inf = 10
R_ct = 50
phi = 0.8
tau_0 = 1.

C = tau_0**phi/R_ct
Z_exact = R_inf+1./(1./R_ct+C*(1j*2.*pi*freq_vec)**phi)
gamma_fct = (R_ct)/(2.*pi)*sin((1.-phi)*pi)/(np.cosh(phi*np.log(tau/tau_0))-cos((1.-phi)*pi))

# we will use a finer mesh for plotting the results
freq_vec_plot  = np.logspace(-4., 4., num=10*(N_freqs-1), endpoint=True)
tau_plot  = 1/freq_vec_plot
gamma_fct_plot = (R_ct)/(2.*pi)*sin((1.-phi)*pi)/(np.cosh(phi*np.log(tau_plot/tau_0))-cos((1.-phi)*pi)) # for plotting only

# we will add noise to the impedance computed analytically
rng = np.random.seed(214975)
sigma_n_exp = 1.
Z_exp = Z_exact + (sigma_n_exp**2)*(np.random.normal(0, 1, N_freqs)+1j*np.random.normal(0, 1, N_freqs))
```

## 2) Show the synthetic impedance in the Nyquist plot - this is similar to Figure 7 (a)

```python
# Nyquist plot of the impedance
plt.plot(np.real(Z_exact), -np.imag(Z_exact), linewidth=4, color="black", label="exact")
plt.plot(np.real(Z_exp), -np.imag(Z_exp), "o", markersize=10, color="red", label="synthetic experiment")
plt.plot(np.real(Z_exp[20:60:10]), -np.imag(Z_exp[20:60:10]), 's', markersize=10, color="black")

plt.rc('text', usetex=True)
plt.rc('font', family='serif', size=15)
plt.rc('xtick', labelsize=15)
plt.rc('ytick', labelsize=15)
plt.legend(frameon=False, fontsize = 15)
plt.axis('scaled')

plt.xticks(range(10, 70, 10))
plt.yticks(range(0, 60, 10))
plt.gca().set_aspect('equal', adjustable='box')
plt.xlabel(r'$Z_{\rm re}/\Omega$', fontsize = 20)
plt.ylabel(r'$-Z_{\rm im}/\Omega$', fontsize = 20)
# label the frequency points
plt.annotate(r'$10^{-2}$', xy=(np.real(Z_exp[20]), -np.imag(Z_exp[20])),
             xytext=(np.real(Z_exp[20])-2, 10-np.imag(Z_exp[20])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.annotate(r'$10^{-1}$', xy=(np.real(Z_exp[30]), -np.imag(Z_exp[30])),
             xytext=(np.real(Z_exp[30])-2, 6-np.imag(Z_exp[30])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.annotate(r'$1$', xy=(np.real(Z_exp[40]), -np.imag(Z_exp[40])),
             xytext=(np.real(Z_exp[40]), 10-np.imag(Z_exp[40])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.annotate(r'$10$', xy=(np.real(Z_exp[50]), -np.imag(Z_exp[50])),
             xytext=(np.real(Z_exp[50])-1, 10-np.imag(Z_exp[50])),
             arrowprops=dict(arrowstyle="-",connectionstyle="arc"))
plt.show()
```

## 3) Obtain the optimal hyperparameters of the GP-DRT model by minimizing the negative marginal log likelihood (NMLL)

More needed, JP, please add

```python
# initialize the parameter for global 3D optimization to maximize the marginal log-likelihood as shown in eq (31)
sigma_n = sigma_n_exp
sigma_f = 5.
ell = 1.

theta_0 = np.array([sigma_n, sigma_f, ell])
seq_theta = np.copy(theta_0)
def print_results(theta):
    global seq_theta
    seq_theta = np.vstack((seq_theta, theta))
    print('%f, %f, %f' %(theta[0], theta[1], theta[2]))

print('sigma_n,  sigma_f,  ell')

# minimize the NMLL L(\theta) w.r.t sigma_n, sigma_f, ell using the Newton-CG method as implemented in scipy
res = minimize(GP_DRT.NMLL_fct, theta_0, args=(Z_exp, xi_vec), method='Newton-CG', \
               jac=GP_DRT.grad_NMLL_fct,  callback=print_results, options={'disp': True})

# collect the optimized parameters
sigma_n, sigma_f, ell = res.x
```

    sigma_n,  sigma_f,  ell
    0.890360, 5.001415, 1.012087
    0.813635, 5.003534, 1.029191
    0.829186, 5.035787, 1.258867
    0.830393, 5.083237, 1.211778
    0.830445, 5.206077, 1.228366
    0.830528, 5.387435, 1.252416
    0.830527, 5.406889, 1.254664
    0.830527, 5.407084, 1.254687
    0.830526, 5.407086, 1.254686
    Optimization terminated successfully.
             Current function value: 53.657989
             Iterations: 9
             Function evaluations: 11
             Gradient evaluations: 83
             Hessian evaluations: 0

## 4) Core of the GP-DRT

### 4a) Compute matrices
Once we have identified the optimized parameters we can compute $\mathbf K$, $\mathcal L_{\rm im} \mathbf K$, and $\mathcal L^2_{\rm im} \mathbf K_{nm}$, which are given in equation (18) in the article

```python
K = GP_DRT.matrix_K(xi_vec, xi_vec, sigma_f, ell)
L_im_K = GP_DRT.matrix_L_im_K(xi_vec, xi_vec, sigma_f, ell)
L2_im_K = GP_DRT.matrix_L2_im_K(xi_vec, xi_vec, sigma_f, ell)
Sigma = (sigma_n**2)*np.eye(N_freqs)

# in the next step we will need the inverse of the matrix $\mathcal L^2_{\rm im} \mathbf K + \sigma_n^2 \mathbf I$
K_im_full = L2_im_K + Sigma
```

### 4b) Factorize the matrices and solve the linear equations

```python
# Cholesky factorization, L is a lower-triangular matrix
L = np.linalg.cholesky(K_im_full)

# solve the following
alpha = np.linalg.solve(L, Z_exp.imag)
alpha = np.linalg.solve(L.T, alpha)

# estimate the gamma of eq (21a), the minus sign, which is not included in L_im_K, refers to eq (65)
gamma_fct_est = -np.dot(L_im_K.T, alpha)

# covariance matrix
inv_L = np.linalg.inv(L)
inv_K_im_full = np.dot(inv_L.T, inv_L)
np.diag(np.dot(inv_K_im_full, K_im_full))

# estimate the sigma of gamma, from eq (21b)
cov_gamma_fct_est = K - np.dot(L_im_K.T, np.dot(inv_K_im_full, L_im_K))
sigma_gamma_fct_est = np.sqrt(np.diag(cov_gamma_fct_est))
```

### 4c) Plot the obtained DRT against the analytical DRT

```python
# plot the DRT and its confidence region
plt.semilogx(freq_vec_plot, gamma_fct_plot, linewidth=4, color="black", label="exact")
plt.semilogx(freq_vec, gamma_fct_est, linewidth=4, color="red", label="GP-DRT")
plt.fill_between(freq_vec, gamma_fct_est-3*sigma_gamma_fct_est, gamma_fct_est+3*sigma_gamma_fct_est, color="0.4", alpha=0.3)
plt.rc('text', usetex=True)
plt.rc('font', family='serif', size=15)
plt.rc('xtick', labelsize=15)
plt.rc('ytick', labelsize=15)
plt.axis([1E-4,1E4,-5,25])
plt.legend(frameon=False, fontsize = 15)
plt.xlabel(r'$f/{\rm Hz}$', fontsize = 20)
plt.ylabel(r'$\gamma/\Omega$', fontsize = 20)
plt.show()
```

### 4d) Calculate the imaginary part of the GP-DRT impedance

```python
# initialize the imaginary part of impedance vector
Z_im_vec_star = np.empty_like(xi_vec_star)
Sigma_Z_im_vec_star = np.empty_like(xi_vec_star)

gamma_vec_star = np.empty_like(xi_vec_star)
Sigma_gamma_vec_star = np.empty_like(xi_vec_star)

# calculate the imaginary part of impedance at each $\xi$ point for the plot
for index, val in enumerate(xi_vec_star):
    xi_star = np.array([val])

    # compute matrices shown in eq (18), k_star corresponds to a new point
    k_star = GP_DRT.matrix_K(xi_vec, xi_star, sigma_f, ell)
    L_im_k_star = GP_DRT.matrix_L_im_K(xi_vec, xi_star, sigma_f, ell)
    L2_im_k_star = GP_DRT.matrix_L2_im_K(xi_vec, xi_star, sigma_f, ell)
    k_star_star = GP_DRT.matrix_K(xi_star, xi_star, sigma_f, ell)
    L_im_k_star_star = GP_DRT.matrix_L_im_K(xi_star, xi_star, sigma_f, ell)
    L2_im_k_star_star = GP_DRT.matrix_L2_im_K(xi_star, xi_star, sigma_f, ell)

    # compute Z_im_star mean and standard deviation using eq (26)
    Z_im_vec_star[index] = np.dot(L2_im_k_star.T, np.dot(inv_K_im_full, Z_exp.imag))
    Sigma_Z_im_vec_star[index] = L2_im_k_star_star-np.dot(L2_im_k_star.T, np.dot(inv_K_im_full, L2_im_k_star))

    # compute Z_im_star mean and standard deviation
    gamma_vec_star[index] = -np.dot(L_im_k_star.T, np.dot(inv_K_im_full, Z_exp.imag))
    Sigma_gamma_vec_star[index] = k_star_star-np.dot(L_im_k_star.T, np.dot(inv_K_im_full, L_im_k_star))
```

### 4e) Plot the imaginary part of the GP-DRT impedance together with the exact one and the synthetic experiment

```python
plt.semilogx(freq_vec_star, -np.imag(Z_exact), ":", linewidth=4, color="blue", label="exact")
plt.semilogx(freq_vec, -Z_exp.imag, "o", markersize=10, color="black", label="synth exp")
plt.semilogx(freq_vec_star, -Z_im_vec_star, linewidth=4, color="red", label="GP-DRT")
plt.fill_between(freq_vec_star, -Z_im_vec_star-3*np.sqrt(abs(Sigma_Z_im_vec_star)), -Z_im_vec_star+3*np.sqrt(abs(Sigma_Z_im_vec_star)), alpha=0.3)
plt.rc('text', usetex=True)
plt.rc('font', family='serif', size=15)
plt.rc('xtick', labelsize=15)
plt.rc('ytick', labelsize=15)
plt.axis([1E-4,1E4,-5,25])
plt.legend(frameon=False, fontsize = 15)
plt.xlabel(r'$f/{\rm Hz}$', fontsize = 20)
plt.ylabel(r'$-Z_{\rm im}/\Omega$', fontsize = 20)
plt.show()
```


```python
import sympy as sp
from sympy import conjugate as con
```

```python
lam, kappa, mu, nu  = sp.var('\\lambda, \\kappa, \\mu, \\nu', real=True)
z, w = sp.var('z, w')
tw = sp.var('t_w')
zx, zy = sp.var("z_x, z_y", real=True)
U = sp.Function('U')
nu_expr = sp.solve(3 - 4 * nu - kappa, nu)[0]
lam_expr = (2 * mu * nu / (1 - 2 * nu))
kappa_expr = sp.solve(
    lam_expr.subs(nu, nu_expr).simplify() - lam, kappa
)[0]
G = sp.Function('G')
Gp = sp.Function('G_1')
g1, g2 = sp.var('g_1, g_2')
C = sp.var('C', real=True)
```

```python

Ux = sp.Rational(1,2) * (U(zx, zy,w) + con(U(zx, zy,w)))
Uy = -sp.I * sp.Rational(1,2) * (U(zx, zy,w) - con(U(zx,zy,w)))
uxx = sp.diff(Ux, zx)
uxy = sp.diff(Ux, zy)
uyx = sp.diff(Uy, zx)
uyy = sp.diff(Uy, zy)
sxx = lam * (uxx + uyy) + 2 * mu * uxx
sxy = mu * (uxy + uyx)
syx = sxy
syy = lam * (uxx + uyy) + 2 * mu * uyy
```

```python
(sp.Rational(1,2) * (sp.diff(Ux, zx) + sp.diff(Ux, zy))).simplify()
```

$\displaystyle \frac{\overline{\frac{\partial}{\partial z_{x}} U{\left(z_{x},z_{y},w \right)}}}{4} + \frac{\overline{\frac{\partial}{\partial z_{y}} U{\left(z_{x},z_{y},w \right)}}}{4} + \frac{\frac{\partial}{\partial z_{x}} U{\left(z_{x},z_{y},w \right)}}{4} + \frac{\frac{\partial}{\partial z_{y}} U{\left(z_{x},z_{y},w \right)}}{4}$

```python
Usubs = C * ((kappa * (G(zx, zy,z,w) + con(G(zx, zy,z,w)))) * tw - (w - (zx + sp.I * zy)) * con(Gp(zx, zy,z,w) * tw))
expr = (sxy
    .subs(U(zx,zy,w), Usubs)
    .simplify()
    .subs(sp.diff(G(zx, zy, z, w), zx), g1 + con(g1))
    .subs(sp.diff(G(zx, zy, z, w), zy), sp.I * (g1 - con(g1)))
    .subs(sp.diff(Gp(zx, zy, z, w), zx), g2 + con(g2))
    .subs(sp.diff(Gp(zx, zy, z, w), zy), sp.I * (g2 - con(g2)))
    .subs(lam, lam_expr)
    .subs(nu,nu_expr)
    .simplify()
    .factor()
    .collect((kappa, g2, con(g2), tw, con(tw)))
    .collect(tw)
    .subs(zx - sp.I * zy, con(z))
    .collect(con(tw))
    .subs(zx + sp.I * zy, z)
)
expr
```

$\displaystyle i C \mu \left(\kappa \left(2 g_{1} \overline{t_{w}} - 2 t_{w} \overline{g_{1}}\right) + g_{2} t_{w} \left(- \overline{w} + \overline{z}\right) + \left(w - z\right) \overline{g_{2}} \overline{t_{w}}\right)$

```python
sp.exp(-2 * sp.I * sp.pi / 2)
```

$\displaystyle -1$

```python
k1dz = sp.diff(sp.log((z-w)/(con(z)-con(w))), w).subs(sp.diff(con(w), w), 1).simplify()
k2dz = (sp.diff((z-w)/(con(z)-con(w)), w)).subs(sp.diff(con(w), w), 1).simplify()
-kappa * k1dz + k2dz
```

$\displaystyle - \frac{\kappa \left(- w + z + \overline{w} - \overline{z}\right)}{\left(w - z\right) \left(\overline{w} - \overline{z}\right)} + \frac{- w + z + \overline{w} - \overline{z}}{\left(\overline{w} - \overline{z}\right)^{2}}$

```python

```

```python

ux = sp.Function("u_x", real=True)
uy = sp.Function("u_y", real=True)
zx, zy = sp.var("z_x, z_y", real=True)
u = ux(zx, zy) + uy(zx, zy) * sp.I
d_fnc = lambda f: sp.Rational(1, 2) * (sp.diff(f, zx) - sp.I * sp.diff(f, zy))
d_con_fnc = lambda f: sp.Rational(1, 2) * (sp.diff(f, zx) + sp.I * sp.diff(f, zy))

# t1 = (2 * mu) * ((d_fnc(u) + d_con_fnc(con(u))) / (kappa_expr - 1))
# t2 = (2 * mu) * d_con_fnc(u)
t1 = (2 * mu) * ((d_fnc(u) + con(d_fnc(u))) / (kappa_expr - 1))
# t2 = (2 * mu) * con(d_fnc(con(u)))
t2 = (2 * mu) * -sp.I * (con(d_fnc(con(u))) - d_fnc(con(u)))

uxx, uxy, uyx, uyy = sp.var("u_{xx}, u_{xy}, u_{yx}, u_{yy}", real=True)
t_sxx_sxy = (
    (t1 + t2).simplify()
    .subs(sp.diff(ux(zx, zy), zx), uxx)
    .subs(sp.diff(ux(zx, zy), zy), uxy)
    .subs(sp.diff(uy(zx, zy), zx), uyx)
    .subs(sp.diff(uy(zx, zy), zy), uyy)
)
sxx = sp.re(t_sxx_sxy).expand()
sxy = sp.im(t_sxx_sxy).expand()

t_syy_syx = (
    (t1 - t2).simplify()
    .subs(sp.diff(ux(zx, zy), zx), uxx)
    .subs(sp.diff(ux(zx, zy), zy), uxy)
    .subs(sp.diff(uy(zx, zy), zx), uyx)
    .subs(sp.diff(uy(zx, zy), zy), uyy)
)
syy = sp.re(t_syy_syx).expand()
syx = -sp.im(t_syy_syx).expand()

from IPython.display import display
display(sxx)
display(sxy)
display(syy)
display(syx)

```

$\displaystyle \lambda u_{xx} + \lambda u_{yy} + \mu u_{xx} + 2 \mu u_{xy} + 2 \mu u_{yx} + \mu u_{yy}$

$\displaystyle 0$

$\displaystyle \lambda u_{xx} + \lambda u_{yy} + \mu u_{xx} - 2 \mu u_{xy} - 2 \mu u_{yx} + \mu u_{yy}$

$\displaystyle 0$

```python
zv = sp.var('z')
t1 = (2 * mu) * ((d_fnc(u) + con(d_fnc(u))) / (kappa - 1))
t2 = (2 * mu) * con(d_fnc(con(u)))
z = zx + zy * sp.I
G = sp.Function('G')
Gp = sp.Function('G_1')
g1, g2 = sp.var('g_1, g_2')
g2x, g2y = sp.var('g_{2x}, g_{2y}')
C = sp.var('C', real=True)
U = C * ((kappa * (G(z,w) + con(G(z,w)))) * tw + (w - z) * con(Gp(z,w) * tw))
xi1 = sp.var('_xi_1')
def proc(expr):
    return (
        expr
        .subs(ux(zx, zy), sp.Rational(1,2) * (U + con(U)))
        .subs(uy(zx, zy), -sp.I * sp.Rational(1,2) * (U - con(U)))
        .simplify()
        .replace(sp.Subs(sp.Derivative(G(xi1, w), xi1), xi1, z), g1)
        .replace(sp.Subs(sp.Derivative(Gp(xi1, w), xi1), xi1, z), g2)
        .subs(Gp(z, w), g1)
        .collect(con(g2 * tw)).subs(z, zv)
        .simplify()
        .collect(con(g1) * con(tw))
        .collect(g1 * tw)
    )
t1_proc = proc(t1)
t2_proc = proc(t2)
display(t1_proc)
display(t2_proc)
```

$\displaystyle 2 C \mu \left(g_{1} t_{w} + \overline{g_{1}} \overline{t_{w}}\right)$

$\displaystyle 2 C \mu \left(\kappa t_{w} \overline{g_{1}} + \left(w - z\right) \overline{g_{2}} \overline{t_{w}}\right)$

```python
t2
```

$\displaystyle 2 \mu \left(\frac{i \left(\overline{\frac{\partial}{\partial z_{y}} \operatorname{u_{x}}{\left(z_{x},z_{y} \right)}} + i \overline{\frac{\partial}{\partial z_{y}} \operatorname{u_{y}}{\left(z_{x},z_{y} \right)}}\right)}{2} + \frac{\overline{\frac{\partial}{\partial z_{x}} \operatorname{u_{x}}{\left(z_{x},z_{y} \right)}}}{2} + \frac{i \overline{\frac{\partial}{\partial z_{x}} \operatorname{u_{y}}{\left(z_{x},z_{y} \right)}}}{2}\right)$

```python
sxx_sxy = (t1_proc + t2_proc).simplify()
```

```python
syy_sxy = sp.I * (t1_proc - t2_proc).simplify()
```

```python
sp.re(sxx_sxy, evaluate=False)
```

$\displaystyle \operatorname{re}{\left(2 C \mu \left(\kappa t_{w} \overline{g_{1}} + g_{1} t_{w} + \left(w - z\right) \overline{g_{2}} \overline{t_{w}} + \overline{g_{1}} \overline{t_{w}}\right)\right)}$

```python
q=sp.var('q')
t1_proc.subs(kappa,q-1).expand().collect(q).subs(q, kappa+1)
```

$\displaystyle \left(\kappa + 1\right) \left(\frac{2 C \mu g_{1} tw}{\kappa - 1} + \frac{2 C \mu \overline{g_{1}} \overline{tw}}{\kappa - 1}\right)$

```python
((d_con_fnc(con(u)) - con(d_fnc(u))).simplify()
    .subs(sp.diff(ux(zx, zy), zx), uxx)
    .subs(sp.diff(ux(zx, zy), zy), uxy)
    .subs(sp.diff(uy(zx, zy), zx), uyx)
    .subs(sp.diff(uy(zx, zy), zy), uyy)
)
```

$\displaystyle 0$

```python
((con(d_fnc(con(u))) - d_con_fnc(u)).simplify()
    .subs(sp.diff(ux(zx, zy), zx), uxx)
    .subs(sp.diff(ux(zx, zy), zy), uxy)
    .subs(sp.diff(uy(zx, zy), zx), uyx)
    .subs(sp.diff(uy(zx, zy), zy), uyy)
)
```

$\displaystyle 0$

```python
C = sp.var('C', real=True)
G = sp.Function('G')
Gp = sp.Function('G_1')
Gpp = sp.Function('G_2')
tw, w, z, z0 = sp.var('t_w, w, z, z_0')
```

```python
U = C * (kappa * (G(z,w) + con(G(z,w))) * tw - (z - w) * con(Gp(z,w) * tw))
U
```

$\displaystyle C \left(\kappa t_{w} \left(G{\left(z,w \right)} + \overline{G{\left(z,w \right)}}\right) - \left(- w + z\right) \overline{t_{w}} \overline{\operatorname{G_{1}}{\left(z,w \right)}}\right)$

```python
t1a = 2 * mu * sp.diff(U, z) / (kappa - 1)
t1b = 2 * mu * sp.diff(con(U), con(z)) / (kappa - 1)
t2 = 2 * mu * sp.diff(con(U), con(z))
g1, g2, g1c, g2c = sp.var('g_1, g_2, g_{1c}, g_{2c}')
```

```python
t2
```

$\displaystyle 2 C \mu \left(\kappa t_{w} \left(\overline{\frac{\partial}{\partial z} G{\left(z,w \right)}} + \overline{\frac{\partial}{\partial z} \overline{G{\left(z,w \right)}}}\right) - \left(- w + z\right) \overline{t_{w}} \overline{\frac{\partial}{\partial z} \operatorname{G_{1}}{\left(z,w \right)}} - \overline{t_{w}} \overline{\operatorname{G_{1}}{\left(z,w \right)}} \overline{\frac{d}{d z} \overline{z}}\right)$

```python
def proc(expr):
    return (expr
        .subs(sp.diff(G(z,w), z), g1)
        .subs(sp.diff(con(G(z,w)), z), con(g1))
        .subs(sp.diff(Gp(z,w), z), g2)
        .subs(sp.diff(con(Gp(z,w)), z), con(g2))
        .subs(sp.diff(con(z), z), 1)
        .subs(Gp(z,w), g1)
    ).simplify()
```

```python
t1a_p = proc(t1a)
t1a_p
```

$\displaystyle \frac{2 C \mu \left(\kappa t_{w} \left(g_{1} + \overline{g_{1}}\right) + \left(w - z\right) \overline{g_{2}} \overline{t_{w}} - \overline{g_{1}} \overline{t_{w}}\right)}{\kappa - 1}$

```python
t1b_p = proc(t1b)
t1b_p
```

$\displaystyle \frac{2 C \mu \left(\kappa \left(g_{1} + \overline{g_{1}}\right) \overline{t_{w}} - g_{1} t_{w} + g_{2} t_{w} \left(\overline{w} - \overline{z}\right)\right)}{\kappa - 1}$

```python
t2_p = proc(t2)
t2_p
```

$\displaystyle 2 C \mu \left(\kappa t_{w} \left(g_{1} + \overline{g_{1}}\right) + \left(w - z\right) \overline{g_{2}} \overline{t_{w}} - \overline{g_{1}} \overline{t_{w}}\right)$

```python
sp.diff(con(sp.log(w-z)), z)
```

$\displaystyle \frac{\partial}{\partial z} \overline{\log{\left(w - z \right)}}$

```python
-con(sp.log(w - z)).series(z, x0=z0, n=3)
```

$\displaystyle - \overline{\log{\left(w - z_{0} \right)}} - \left(z - z_{0}\right) \left. \frac{\partial}{\partial \xi} \overline{\log{\left(w - \xi - z_{0} \right)}} \right|_{\substack{ \xi=0 }} - \frac{\left(z - z_{0}\right)^{2} \left. \frac{\partial^{2}}{\partial \xi^{2}} \overline{\log{\left(w - \xi - z_{0} \right)}} \right|_{\substack{ \xi=0 }}}{2} + O\left(\left(z - z_{0}\right)^{3}; z\rightarrow z_{0}\right)$

```python
-sp.log(w - z).series(z, x0=z0, n=3)
```

$\displaystyle \frac{\left(z - z_{0}\right)^{2}}{2 \left(w - z_{0}\right)^{2}} + \frac{z - z_{0}}{w - z_{0}} - \log{\left(w - z_{0} \right)} + O\left(\left(z - z_{0}\right)^{3}; z\rightarrow z_{0}\right)$

```python
R = sp.var('R')
ratio = (z-z0)/(w-z0)
sp.diff(-sp.log(w - z), z).series(z, x0=z0, n=3).subs(ratio, R)
```

$\displaystyle \frac{1}{w - z_{0}} + \frac{R}{w - z_{0}} + \frac{R^{2}}{w - z_{0}} + O\left(\left(z - z_{0}\right)^{3}; z\rightarrow z_{0}\right)$

```python
sp.diff(sp.diff(sp.log(w - z), z), z).series(z, x0=z0, n=3)
```

$\displaystyle - \frac{3 \left(z - z_{0}\right)^{2}}{\left(w - z_{0}\right)^{4}} - \frac{2 \left(z - z_{0}\right)}{\left(w - z_{0}\right)^{3}} - \frac{1}{\left(w - z_{0}\right)^{2}} + O\left(\left(z - z_{0}\right)^{3}; z\rightarrow z_{0}\right)$


# MCPC 2018 Problem J: Need for Speed

This problem is same as Problem-E of 2017 ICPC Final. It is a variation of Candie Problem(ICPC training).
This problem can be solved using **simulation** and **binary search**.

### Input

- 1 Line, 1 Integer: Number of Test Case
   + 1 Line 2 Integers: Number of sections(n), Total Time(t)
      * n Lines: Distance(d), SpeedMeter Reading(s)

### Output

- 1 Line per Test case: Offset of SpeedMeter(c)

### Sample Input

```
2
3 5
4 -1
4 0
10 3
4 10
5 3
2 2
3 6
3 1
```

### Sample Output

```
3.00
-0.51
```

## 1st Step: Simulation

In case of 1st test case of above sample input, following fomula needs to be solved. The bigger the offset(c), the smaller the total time(t).

$(4/(-1+c)+(4/(0+c))+(10/(3+c))=5$

```python
import sys

def simulate(c):
    '''
    c: offset of speed meter
    return total time
    '''

    ret = ((4/(-1+c)+(4/(0+c))+(10/(3+c))))
    # print('simulate:', c, ret, file=sys.stderr)
    return ret
```

```python
### test
print('c=1.1', simulate(1.1))
print('c=2', simulate(2))
print('c=3', simulate(3))
print('c=5', simulate(5))
print('c=100', simulate(100))
```

## 2md step: Finding Upper and Lower boudary

In order to perform binary search, Lower and Upper boundary need to be decided. In this problem, Lower boundary can be -(minimum SpeedMeter Read(s) value), because $s+c>0$,
As for Upper boundary, we need to search it.

- Set temporary value, which is bigger than lower value
- Simulate with the temporary value, if the result is bigger than total time, use bigger value until big enough

```python
TOTAL_TIME=10
low_bound = 1  # Minimum SpeedMeter Read is -1, -(-1)=1
high_bound = abs(low_bound)+1
while True:
    if simulate(high_bound) < TOTAL_TIME:
        break

    low_bound = high_bound  # we can update low_bound here
    high_bound *= 2

print('Low Boundary:', low_bound, 'High Boundary:', high_bound)
```

## Binary Search

Now we know that the answer is between low and high boundary.

```python
DELTA = 0.001

while high_bound-low_bound > DELTA:
    mid = (high_bound+low_bound) / 2
    calc_time = simulate(mid)
    print('Trying:', mid, 'calc_time:', calc_time, file=sys.stderr)
    if calc_time == TOTAL_TIME:  # Bingo
        high_bound = low_bound = mid
    elif calc_time < TOTAL_TIME:  # mid is too big
        high_bound = mid
    elif calc_time > TOTAL_TIME:  # mid is too small
        low_bound = mid
    else:                         # should not occur, raise error
        raise

print(high_bound, low_bound)
```

### Test

Please change some parameters and retry.

## Java sample code

```
// MCPC 2018 Problem J: Need for Speed

import java.util.Scanner ;

class prob_j {

    static double simulate(int num_sec, int[][] section, double offset) {
        double total = 0.0 ;
        for (int i=0; i<num_sec; i++) {
            total += section[i][0]/(section[i][1]+offset) ;
        }
        return total ;
    }

    static void solve(int num_sec, int [][] section, int total_time) {
        int min_dist= section[0][1] ;
        for (int i=0; i<num_sec; i++) {
            if (min_dist > section[i][1]) min_dist = section[i][1] ;
        }
        double low_bound = (double)(-min_dist) ;
        double high_bound = Math.abs(low_bound) + 1.0 ; // > low_bound, >0

        // find high_bound
        while (true) {
            double total = simulate(num_sec, section, high_bound) ;
            if (total < (double)total_time) break ;
            high_bound += high_bound ;
        }
        //System.out.printf("low: %.5f high: %.5f\n", low_bound, high_bound) ;

        // binary search
        final double max_err=0.001 ;
        while (high_bound-low_bound > max_err) {
            double mid = (high_bound+low_bound) / 2 ;
            double total = simulate(num_sec, section, mid) ;
            //System.out.printf("mid: %.5f total: %.5f\n", mid, total) ;
            if (total == (double)total_time) high_bound = low_bound = mid ;
            else if (total < (double)total_time) { // too big
                high_bound = mid ;
            }
            else low_bound = mid ;
        }

        System.out.printf("%.2f\n", (high_bound+low_bound)/2) ;
    }

    public static void main(String[] args) {
        Scanner sc = new Scanner(System.in) ;
        int num_tc = sc.nextInt() ;

        for (int tc=0; tc<num_tc; tc++) {
            int num_sec = sc.nextInt() ;
            int total_time = sc.nextInt() ;
            int section[][] = new int[num_sec][2] ;
            for (int i=0; i<num_sec; i++) {
                section[i][0] = sc.nextInt() ;  // Distance
                section[i][1] = sc.nextInt() ;  // SpeedMeter read
            }
            solve(num_sec, section, total_time) ;
        }
    }
}

```

## Sympy and Scipy

In case of Python, Algebra can be solved using **symbpy** or **scipy**, unfotunately **sympy** and **scipy** are not standard module and can not be used in programming contest such as ICPC.

```python
from sympy import Symbol, solve

x = Symbol('x', real=True)  ## calc only real number
f = 4/(x-1)+4/x+10/(x+3)-5
print('Test case 1:', solve(f))

g = 5/(x+3)+2/(x+2)+3/(x+6)+3/(x+1)-10
print('Test case 2:', solve(g))
```

    Test case 1: [3, -7/10 + sqrt(129)/10, -sqrt(129)/10 - 7/10]
    Test case 2: [-107/40 - sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))/2 - sqrt(2009/200 - 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3) + 49923/(4000*sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))) - 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)))/2, -107/40 - sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))/2 + sqrt(2009/200 - 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3) + 49923/(4000*sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))) - 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)))/2, -107/40 + sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))/2 - sqrt(2009/200 - 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3) - 49923/(4000*sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))) - 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)))/2, -107/40 + sqrt(2009/200 - 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3) - 49923/(4000*sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))) - 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)))/2 + sqrt(2009/400 + 1817/(600*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3)) + 2*(1753/8000 + sqrt(17747502810)*I/72000)**(1/3))/2]

```python
from scipy import optimize

def h(x):
    return 5/(x+3)+2/(x+2)+3/(x+6)+3/(x+1)-10

print('fsolve', optimize.fsolve(h, 0))   ## 2nd parameter is the value near the result, low_bound+1 here
print('newton', optimize.newton(h, 0))
```

    fsolve [-0.50865338]
    newton -0.5086533767953939

```python
from sympy import simplify
simplify(g, ratio=3)
```

    -(10*x**4 + 107*x**3 + 354*x**2 + 425*x + 138)/(x**4 + 12*x**3 + 47*x**2 + 72*x + 36)

```python
from numpy.polynomial.polynomial  import Polynomial

p = Polynomial([138, 425, 354, 107, 10])
print(p.roots())
```

    [-5.76862142 -2.64381466 -1.77891055 -0.50865338]

```python
# Verify

def eval_x(x):
    return -(10*x**4 + 107*x**3 + 354*x**2 + 425*x + 138)/(x**4 + 12*x**3 + 47*x**2 + 72*x + 36)

vals = [-5.76862142, -2.64381466, -1.77891055, -0.50865338]
for v in vals:
    print(v, eval_x(v))
```

    -5.76862142 1.0807564552297866e-07
    -2.64381466 1.852137141040639e-07
    -1.77891055 4.135766121592931e-08
    -0.50865338 4.56037450908601e-08

```python
# Verify using sympy

print('verify with sympy subs')
for v in vals:
    print(v, g.subs([(x,v)]))
```

    verify with sympy subs
    -5.76862142 1.08075778992855e-7
    -2.64381466 1.85213512082782e-7
    -1.77891055 4.13577927460551e-8
    -0.50865338 4.56037447804647e-8

```python

```


# Support Vector Machine

A *Support Vector machine* (SVM) is a very powerful and versatile supervised machine learning model, capable of performing linear or nonlinear classification and regression tasks. To separate the two classes of data points, there are many possible hyperplanes that could be chosen. The key idea behind SVM is that the algorithm finds an optimal hyperplane that has the **maximum margin**, i.e the maximum distance between data points of both classes.

This fundamental idea is explained with the plots above. The plot on the left shows many hyperplanes that can separate the circles from the squares. These hyperplanes work perfectly on this training dataset, but their decision boundaries come so close to the instances that they will probably not perform as well on new instances.  However, the right plot shows the optimal hyperplane that maximize the margin between these two classes. Maximizing the margin distance provides some reinforcement so that future data points can be classified with more confidence.

In this lab assignment, you will learn to use and fine-tune linear SVM, the Polynomial kernel SVM and the RBF kernel SVM, and understand their pros and cons by comparing their prediction performance on a synthetic dataset.

```python
# import packages
%matplotlib inline
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import seaborn as sns
import pandas as pd
from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.svm import LinearSVR
from sklearn.svm import SVC
from sklearn.svm import SVR
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV

# make this notebook's output stable across runs
np.random.seed(0)
```

```python
scores = pd.DataFrame(
    columns=['Accuracy','ROC AUC']
)
```

```python
# helper functions used in this lab
def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    """
    Plot the decision boundary of a learnt classifier
    """
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=1)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)

def evaluate_model(model, X, y, scores, model_name=''):
    # make prediction and evaluate the model performance on test data
    z = model.predict(X)
    z_prob = model.predict_proba(X)[:,1]
    print("model accuracy: {}".format(accuracy_score(y, z)))
    print("model ROC AUC: {}".format(roc_auc_score(y, z_prob)))
    df = pd.DataFrame(
        index=[model_name],
        columns = ['Accuracy', 'ROC AUC'],
        data = [[round(accuracy_score(y, z),4),
                round(roc_auc_score(y, z_prob),4)]]
    )
    summary = pd.concat([scores,df])
    print(summary)
    return summary
```

### Synthetic Moon dataset
In this lab, we will classify a moon shaped synthetic dataset with two features (_x1_ and _x2_) and two classes (0 or 1) with some added noise. We use this synthetic dataset because it's easier to help demonstrate the pros and cons of different SVM models covered in this lab.

```python
# load the moon train and test data from CSV files
train = pd.read_csv("../data/moon_train.csv")
test = pd.read_csv("../data/moon_test.csv")

train_x = train.iloc[:,0:2]
train_y = train.iloc[:,2]

test_x = test.iloc[:,0:2]
test_y = test.iloc[:,2]

print("Number of train data: {}".format(len(train_y)))
print("Number of test data: {}".format(len(test_y)))
```

    Number of train data: 3000
    Number of test data: 1000

```python
# plot the train dataset
plt.scatter(train_x.x1, train_x.x2, s=40, c=train_y, cmap=plt.cm.Spectral)
plt.title('Scatter plot of the train dataset')
plt.xlabel('x1')
plt.ylabel('x2')
```

```python
# plot the test dataset
plt.scatter(test_x.x1, test_x.x2, s=40, c=test_y, cmap=plt.cm.Spectral)
plt.title('Scatter plot of the test dataset')
plt.xlabel('x1')
plt.ylabel('x2')
```

## Linear SVM

**Coding Part 1: Learn to use and fine-tune the Linear Support Vector Machine.**

### Pipeline

Unlike tree-based models (e.g. CART, Random Forest and Gradient Boosted Trees), SVMs are sensitive to the feature scales. It is best practice to apply feature scaling with [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) function in scikit-learn.

[Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) in scikit-learn allows us to sequentially apply a list of transformations and a final estimator. For example, in the following code we first apply the feature scaling before feeding the data to train a linear SVM model. Note that you need to set the argument _probability_ to be True so that the SVC model outputs probabilities as well as the predictions.

```python
# define the pipeline
linear_svc_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("svc", SVC(kernel="linear", probability=True, random_state=0))
    )
)
linear_svc_pipe.fit(train_x, train_y)
```

    Pipeline(steps=[('scaler', StandardScaler()),
                    ('svc',
                     SVC(kernel='linear', probability=True, random_state=0))])

```python
# plot the decision boundary
plot_decision_boundary(linear_svc_pipe, train_x.values, train_y.values)
```

As we can see from the plot above, a linear SVM can produce only a linear hyperplane as the decision boundary and is not capable of modeling this moon-shaped dataset and generating the decision boundaries close to the ground truth.

```python
# make prediction and evaluate the model performance on test data
scores = evaluate_model(linear_svc_pipe, test_x, test_y, scores, model_name='linear_svc_pipe')
```

    model accuracy: 0.839
    model ROC AUC: 0.9256840000000002
                     Accuracy  ROC AUC
    linear_svc_pipe     0.839   0.9257

#### Penalty parameter C

SVM's objective function consists of two terms: the first term measures how large the margin is (we want to maximize the margin, which corresponds to minimizing $W^T W$ in the follow equation), and the second term measures the violations of the hyperplane ($\sum_i^m \xi^i$). This is where the hyper-parameter $C$ comes in. It is used to tradeoff between these two objectives.

\begin{align}
\min_{W, b} \frac{1}{2}W^T W & + C \sum_i^m \xi^i \\
S.T. y^i (W^T X^i + b) & \geqslant 1 - \xi^i \\
\xi^i & \geqslant 0
\end{align}

```python
# TODO: Use grid search with cross-validation on penalty parameter C to fine-tune the SVC model.
#       Note that you need to set probability to be True so that the trained model outputs both predictions as well as probabilities.
parameters = {
    "C": [0.01, 0.1, 1, 10, 100]
}
# TODO: Use grid search with cross-validation on penalty parameter C to fine-tune the SVC model.
linear_svc_grid = GridSearchCV(SVC(kernel="linear",probability=True), parameters)
#linear_svc_grid = GridSearchCV(SVC(probability=True), parameters)
# define the pipeline
# TODO: build the pipeline for training the model and fit the model to the training dataset.
linear_svc_grid_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("svc", linear_svc_grid),
    )
)
linear_svc_grid_pipe.fit(train_x, train_y)
# summarize the results of the grid search
print("The best score is {}".format(linear_svc_grid.best_score_))
print("The best hyper parameter setting is {}".format(linear_svc_grid.best_params_))
```

    The best score is 0.8453333333333333
    The best hyper parameter setting is {'C': 1}

```python
# plot the decision boundary
plot_decision_boundary(linear_svc_grid_pipe, train_x.values, train_y.values)
```

After fine-tuning the model, we should see the model performance improve slightly as compared to the SVC with the default setting.

```python
scores = evaluate_model(linear_svc_grid_pipe, test_x, test_y, scores, 'linear_svc_grid_pipe')
```

    model accuracy: 0.839
    model ROC AUC: 0.9256900000000001
                          Accuracy  ROC AUC
    linear_svc_pipe          0.839   0.9257
    linear_svc_grid_pipe     0.839   0.9257

---

## Polynomial Kernel SVM

**Coding Part 2: Understand the limitation of linear SVM and learn how to use kernel trick to extend linear SVM to handle non-linear scenarios.**

### Polynomial Features

Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets are not linearly separable. One approach to handling nonlinear datasets is to add more non-linear features, such as polynomial features, and hope the dataset is linear separable in the higher dimensional space with the help of non-linear features.

In the following example, we apply polynomial functions with degree of 3 to the original features before feeding the data to the linear SVC.

```python
poly_feature_svm_pipe = Pipeline(
    (
        ("poly_features", PolynomialFeatures(degree=3)),
        ("scaler", StandardScaler()),
        ("svm_clf", SVC(kernel="linear", probability=True))
    )
)
poly_feature_svm_pipe.fit(train_x, train_y)
```

    Pipeline(steps=[('poly_features', PolynomialFeatures(degree=3)),
                    ('scaler', StandardScaler()),
                    ('svm_clf', SVC(kernel='linear', probability=True))])

```python
# plot the decision boundary
plot_decision_boundary(poly_feature_svm_pipe, train_x.values, train_y.values)
```

```python
# make prediction and evaluate the model performance on test data
scores = evaluate_model(poly_feature_svm_pipe, test_x, test_y, scores, 'poly_feature_svm_pipe')
```

    model accuracy: 0.903
    model ROC AUC: 0.964172
                           Accuracy  ROC AUC
    linear_svc_pipe           0.839   0.9257
    linear_svc_grid_pipe      0.839   0.9257
    poly_feature_svm_pipe     0.903   0.9642

### Polynomial SVM
Instead of explicitly applying the polynomial functions to each data point, kernel trick allows us to get the same result as if you added many polynomial features (even with very high-degree polynomials) without actually having to add them to each data point. So the number of features won't grow combinatorially.

To train the SVC with polynomial kernel, we need to specify argument _kernel_ to be "poly". Like linear SVC, we need to fine tune the penalty parameter *C* to tradeoff minimizing margin and minimizing violations. Also, we need to specify the degree of the polynomial functions as well as a hyper-parameter _coef0_ which controls how much the model is influenced by high-degree polynomials vs. low-degree polynomials. Please refer to the [SVC](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) API doc for more details.

```python
# TODO: Use grid search with cross-validation on penalty parameter C, polynomial degree, and coef0 to
#       fine-tune the model.
parameters = {
    "C": [0.1, 1, 10],
    "degree": [3, 5],
    "coef0": [0.5, 1.0]
}
# TODO: Use grid search with cross-validation on hyper-parameters to fine-tune the SVC model.
poly_svc_grid = GridSearchCV(SVC(kernel='poly',probability=True), parameters)
# define the pipeline
# TODO: build the pipeline for training the model and fit the model to the training dataset.
poly_svc_grid_pipe = Pipeline(
    (
        ("poly_features", PolynomialFeatures(degree=3)),
        ("scaler", StandardScaler()),
        ("svm_clf", poly_svc_grid),
    )
)
poly_svc_grid_pipe.fit(train_x, train_y)
# summarize the results of the grid search
print("The best score is {}".format(poly_svc_grid.best_score_))
print("The best hyper parameter setting is {}".format(poly_svc_grid.best_params_))
```

    The best score is 0.9073333333333334
    The best hyper parameter setting is {'C': 1, 'coef0': 1.0, 'degree': 5}

```python
# plot the decision boundary
plot_decision_boundary(poly_svc_grid_pipe, train_x.values, train_y.values)
```

```python
# make prediction and evaluate the model performance on test data
scores = evaluate_model(poly_svc_grid_pipe, test_x, test_y, scores, 'poly_svc_grid_pipe')
```

    model accuracy: 0.902
    model ROC AUC: 0.958508
                           Accuracy  ROC AUC
    linear_svc_pipe           0.839   0.9257
    linear_svc_grid_pipe      0.839   0.9257
    poly_feature_svm_pipe     0.903   0.9642
    poly_svc_grid_pipe        0.902   0.9585

Both the polynomial SVC and adding polynomial features in a linear SVC result in very similar performance and significantly improve the model performance on the test dataset. The decision boundaries are no longer linear in the original feature space and better capture the ground truth of this moon-shaped dataset.

## Gaussian Radial Basis Function (RBF) SVM
**Coding Part 3: Learn to use Gaussian Radial Basis Function (RBF) SVM.**

Another very popular kernel used in SVM is called the Gaussian Radial Basis Function (RBF) kernel. The idea of RBF kernel is to allow each training instance to resemble a particular landmark. The RBF kernel on two samples x and x', represented as feature vectors in some input space, is defined as

\begin{align}
K(x, x') = \exp(-\gamma||x - x'||^2)
\end{align}

RBF Kernel has a hyper-parameter $\gamma$, which controls the shape of the Gaussian function. The parameter $\gamma$ is the inverse of the standard deviation of the RBF kernel (Gaussian function), which is used as similarity measure between two points. Intuitively, a small gamma value define a Gaussian function with a large variance. In this case, two points can be considered similar even if are far from each other. In the other hand, a large gamma value means define a Gaussian function with a small variance and in this case, two points are considered similar just if they are close to each other.

To demonstrate this effect of hyper-parameter $\gamma$, we train two separate SVCs, one with a large value of $\gamma$ and one with a small value of $\gamma$.

```python
# train RBF SVC with a small gamma and underfit the training set
rbf_svc_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("svc", SVC(kernel="rbf", gamma=0.01, probability=True))
    )
)
rbf_svc_pipe.fit(train_x, train_y)
plot_decision_boundary(rbf_svc_pipe, train_x.values, train_y.values)
```

```python
# train RBF SVC with a large gamma and overfit the training set
rbf_svc_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("svc", SVC(kernel="rbf", gamma=100, probability=True))
    )
)
rbf_svc_pipe.fit(train_x, train_y)
plot_decision_boundary(rbf_svc_pipe, train_x.values, train_y.values)
```

As we can see from the previous two plots, $\gamma$ acts like a regularization hyper-parameter. A small value of $\gamma$ leads to model underfitting (the upper figure) and a large value of $\gamma$ leads to model overfitting (the lower figure). Like other hyper-parameters, the value of $\gamma$ needs to be fine-tuned with a validation set.

```python
# TODO: use grid search with cross-validation on penalty parameter C to fine-tune the model
parameters = {
    "C": [0.01, 0.1, 1, 10, 100],
    "gamma": [0.1, 1, 10, 100]
}
# TODO: Use grid search with cross-validation on hyper-parameters to fine-tune the SVC model.
rbf_svc_grid = GridSearchCV(SVC(kernel='rbf',probability=True), parameters)
# define the pipeline
# TODO: build the pipeline for training the model and fit the model to the training dataset.
rbf_svc_grid_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("svc", rbf_svc_grid),
    )
)
rbf_svc_grid_pipe.fit(train_x, train_y)
# summarize the results of the grid search
print("The best score is {}".format(rbf_svc_grid.best_score_))
print("The best hyper parameter setting is {}".format(rbf_svc_grid.best_params_))
```

    The best score is 0.9109999999999999
    The best hyper parameter setting is {'C': 10, 'gamma': 10}

```python
# plot the decision boundary
plot_decision_boundary(rbf_svc_grid_pipe, train_x.values, train_y.values)
```

```python
# make prediction and evaluate the model performance on test data
scores = evaluate_model(rbf_svc_grid_pipe, test_x, test_y, scores, 'rbf_svc_grid_pipe')
```

    model accuracy: 0.893
    model ROC AUC: 0.9243999999999999
                           Accuracy  ROC AUC
    linear_svc_pipe           0.839   0.9257
    linear_svc_grid_pipe      0.839   0.9257
    poly_feature_svm_pipe     0.903   0.9642
    poly_svc_grid_pipe        0.902   0.9585
    rbf_svc_grid_pipe         0.893   0.9244

### End of ML 310 Lab Assignment 4
---

Comparing the different estimators, the poly feature SVM had the best test results. This past week I gained a better understanding for predicting nonlinear decision boundaries. In practice, the most important hyper parameters to tune are kernel, C (cost), gamma (model sensitivity), and degree. Lastly, I learned that kernels map an input into a high dimensional space without the full cost of mapping each point individually. For a quick example, review: https://towardsdatascience.com/kernel-function-6f1d2be6091.

```python

```