---
## 02. Diferencias Finitas. Diferenciación

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

### Resumen

En este cuaderno se presentará el esquema de las diferencias finitas para calcular derivadas.

---

## Discretización de una Función Suave

Considere una función suave $f(x)$ en un intervalo finito $[a,b]$. Para repesentar numéricamente esta función, debemos dividir el intervalo mediante una malla con $N-1$ intervalos igualmente espaciados o equivalentemente, utilizando $N$ nodos y evaluar allí la función. Este proceso se denomina *discretizar* la función.

Existen diversos métodos de discretización con mallas (y también existen métodos de discretizaciónen los cuales las funciones continuas se representan sin una malla y en su lugar, utilizando bases de funciones tales como los senos o los cosenos. Estos se denominan *métodos espectrales*).

Alguno de los métodos basados en mallas utilizan las denominadas *mallas estructuradas*, en las cuales se utilizan coordenadas cartesianas (i.e. mallas rectangulares) y dentro de ellos podemos nombrar

- Diferencias-Finitas
- Volumenes-Finitos
- Elementos-Finitos

Por otra parte, los métodos con *mallas no-estructuradas* utilizan otro tipo de coordenadas o estan basados en celdas con formas triangulares o con tetraedros.  Este tipo de celdas no-rectangulares permite representar dominios con formas irregulares pero su manejo es mas complejo.

### Método de Diferencias Finitas

El primer método que utilizaremos es el **Diferencias Finitas** en el cual se considerará una malla rectangular basada en las coordenadas cartesianas. De esta forma, el dominio se discretizará con una malla con $N-1$ intervalos igualmente espaciados (la distancia entre cualquier par consecutivo de puntos es el mismo) o equivalentemente mediante $N$ puntos (nodos). Los datos discretos que representarán la función se obtienen al evaluar en algunos puntos específicos. En la siguiente figura se puede observar dos elecciones posibles. En una de ellas, la función se evalua en cada nodo mientras que en la otra, la función se evalua en los puntos medios de cada intervalo.

<center>
</center>

---
Por ejemplo, conisdere la función $\sin x$ en el intervalo $[0,2\pi]$. Crearemos una malla en este dominio y obtendremos los puntos para evaluar la función y poder graficarla. Es claro que dependiendo el tamaño de la malla, se obtendra una mejor o peor representación gráfica de la función.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def f(x):
    return np.sin(x)

# Interval
a = 0.
b = 2*np.pi

# Number of points in the grid
N1 = 5
N2 = 10
N3 = 100

# Grid size
dx1 = (b-a)/N1
dx2 = (b-a)/N2
dx3 = (b-a)/N3

x_grid1= np.linspace(a, b, N1)
x_grid2= np.linspace(a, b, N2)
x_grid3= np.linspace(a, b, N3)

plt.figure(figsize=(7,5))
plt.plot(x_grid1, f(x_grid1),color='red', label=f'N={N1:.0f}, $\Delta x$={dx1:.2f}')
plt.plot(x_grid2, f(x_grid2),color='blue', label=f'N={N2:.0f}, $\Delta x$={dx2:.2f}')
plt.plot(x_grid3, f(x_grid3),color='grey', label=f'N={N3:.0f}, $\Delta x$={dx3:.2f}')

plt.xlabel(r'$x$')
plt.ylabel(r'$\sin x$')
plt.grid()
plt.legend()
plt.show()
```

---
## Derivada Numérica de una Función Analítica con Diferencias Finitas

Dada una función analítica $f(x)$, podemos escribir su expansión en serie de Taylor alrededor de un punto $x=x_0$ como

$f(x_0 + \Delta x) = \sum_{n=0}^\infty \frac{f^{(n)} (x_0)}{n!} \Delta x ^n = f(x_0) +
 f'(x_0) \Delta x +f''(x_0) \frac{\Delta x^2}{2}  + f'''(x_0) \frac{\Delta x^3}{6} + ...$

Truncando la serie a segundo orden tenemos

$f(x_0 + \Delta x) = f(x_0) +  f'(x_0) h+ \mathcal{O}(\Delta x^2)$,

donde se obtiene una aproximación a primer orden hacia adelante de la primera derivada de la función:

$f'(x_0) = \frac{f(x_0+\Delta x) - f(x_0)}{\Delta x} + \mathcal{O}(\Delta x)$   : (*Forward Difference Derivative*)

De forma similar, es posible obtener una aproximación a primer orden hacia atras de la primera derivada de la función:

$f'(x_0) = \frac{f(x_0) -f(x_0-\Delta x)}{\Delta x} + \mathcal{O}(\Delta x)$   : (*Backward Difference Derivative*)

Combinando estas dos relaciones, se obtiene una aproximación a segundo orden centrada de la primera derivada de la función:

$f'(x_0) = \frac{f(x_0 + \Delta x) -f(x_0-\Delta x)}{2\Delta x} + \mathcal{O}(\Delta x^2)$   : (*Central Difference Derivative*)

En la siguiente figura se representan las tres posibilidades de derivada numérica:

```python
import numpy as np
import matplotlib.pyplot as plt

def f(x):
    return np.sin(x)

def forwardDerivative(f, x, dx):
    return (f(x + dx) - f(x))/dx

def backwardDerivative(f, x, dx):
    return (f(x) - f(x - dx))/dx

def centeredDerivative(f, x, dx):
    return (f(x + dx) - f(x - dx))/(2*dx)

# To draw a staright line defined by the numerical derivatives
def straight_line(x_0, y_0, m, x):
    y = y_0 + m*(x - x_0)
    return y

# Point to calculate the derivative
x_0 = np.pi/2 +0.1
y_0 = f(x_0)

# Step size in the derivative
dx = 0.1

# Points involved in the numerical derivatives
points_x = [x_0-dx, x_0, x_0+dx]
points_y = [f(x_0-dx), y_0, f(x_0+dx)]

# Numerical derivatives
f_dfdx = forwardDerivative(f, x_0, dx)
b_dfdx = backwardDerivative(f, x_0, dx)
c_dfdx = centeredDerivative(f, x_0, dx)

# x-Ranges to plot function and straight lines
x = np.linspace(0, np.pi, 100)
xline = np.linspace(x_0-0.3, x_0+0.3, 50)

plt.figure(figsize=(7,5))
plt.plot(x, f(x), 'k')
plt.scatter(points_x, points_y, color='black')
plt.plot(xline, straight_line(x_0, y_0, f_dfdx, xline), 'r', label='Forward derivative')
plt.plot(xline, straight_line(x_0, y_0, b_dfdx, xline), 'b', label='Backward derivative')
plt.plot(xline, straight_line(x_0, y_0, c_dfdx, xline), 'g', label='Centered derivative')

plt.xlim(1.,2.2)
plt.ylim(0.9,1.06)
plt.xlabel(r'$x$')
plt.ylabel(r'$\sin x$')
plt.legend()
plt.show()

```

Una de las preguntas más frecuente es, cómo se escoge el valor de $\Delta x$? La teoría del cálculo no indica que deberiamos tomar $\Delta x \rightarrow 0$. Sin embargo, debido al error de redondeo y a la precisión de la máquina tomar este límite no es una buena idea para los calculos numéricos.

Como un ejemplo, considere la función $\sin x$ y su derivada en el punto $x=\frac{\pi}{2}$. A continuación calcularemos la derivada utilizando diferentes valores de $h$ y compararemos los resultados con la derivada analítica,

```python
import numpy as np
import matplotlib.pyplot as plt

def f(x):
    return np.sin(x)

def dfdx(x):
    return np.cos(x)

def forwardDerivative(f, x, dx):
    return (f(x + dx) - f(x))/dx

def backwardDerivative(f, x, dx):
    return (f(x) - f(x - dx))/dx

def centeredDerivative(f, x, dx):
    return (f(x + dx) - f(x - dx))/(2*dx)

x = 1.

dx_max = 1
dx_min = 1E-12
dx = np.linspace(dx_min, dx_max, 100)

f_dfdx = forwardDerivative(f, x, dx)
b_dfdx = backwardDerivative(f, x, dx)
c_dfdx = centeredDerivative(f, x, dx)

abs_error_f = np.abs(dfdx(x) - f_dfdx)
abs_error_b = np.abs(dfdx(x) - b_dfdx)
abs_error_c = np.abs(dfdx(x) - c_dfdx)

plt.figure()
plt.plot(dx, abs_error_f, 'r', label='Absolute error Forward derivative')
plt.plot(dx, abs_error_b, 'b', label='Absolute error Backward derivative')
plt.plot(dx, abs_error_c, 'g', label='Absolute error Centered derivative')

plt.xlabel(r'$\Delta x$')
plt.ylabel(r'Absolute error')
plt.xlim(dx_max, dx_min)
plt.legend()
plt.show()

```

Como se puede observar, al parecer un valor muy pequeño de $\Delta x$ hace que las tres derivadas den el valor correcto (el error absoluto tiende a cero). Sin embargo, un zoom en la región de $\Delta x$ muy pequeños muestra un comportamiento interesante,

```python
x = 1.

dx_max = 1E-1
dx_min = 1E-15
dx = np.linspace(dx_min, dx_max, 100000)

f_dfdx = forwardDerivative(f, x, dx)
b_dfdx = backwardDerivative(f, x, dx)
c_dfdx = centeredDerivative(f, x, dx)

abs_error_f = np.abs(dfdx(x) - f_dfdx)
abs_error_b = np.abs(dfdx(x) - b_dfdx)
abs_error_c = np.abs(dfdx(x) - c_dfdx)

plt.figure(figsize=(8,6))
plt.loglog(dx, abs_error_f, 'r' ,label='Absolute error Forward derivative')
plt.loglog(dx, abs_error_b, 'b', label='Absolute error Backward derivative')
plt.loglog(dx, abs_error_c, 'g', label='Absolute error Centered derivative')

plt.xlabel(r'$\Delta x$')
plt.ylabel(r'Absolute error')
plt.xlim(dx_max, dx_min)
plt.legend()
plt.show()
```

El error absoluto asociado con los tres métodos numéricos presentan un mínimo y luego vuelven a aumentar. Otro acercamiento permite localizar mejor la ubicación del mínimo,

```python
x = 1.

dx_max = 1E-4
dx_min = 1E-10
dx = np.linspace(dx_min, dx_max, 100000)

f_dfdx = forwardDerivative(f, x, dx)
b_dfdx = backwardDerivative(f, x, dx)
c_dfdx = centeredDerivative(f, x, dx)

abs_error_f = np.abs(dfdx(x) - f_dfdx)
abs_error_b = np.abs(dfdx(x) - b_dfdx)
abs_error_c = np.abs(dfdx(x) - c_dfdx)

plt.figure(figsize=(10,15))
plt.subplot(3,1,1)
plt.loglog(dx, abs_error_f, color='red', label='Absolute error Forward derivative')
plt.xlim(dx_max, dx_min)
plt.legend()

plt.subplot(3,1,2)
plt.loglog(dx, abs_error_b, color='blue',label='Absolute error Backward derivative')
plt.ylabel(r'Absolute error')
plt.xlim(dx_max, dx_min)
plt.legend()

plt.subplot(3,1,3)
plt.loglog(dx, abs_error_c, color='green', label='Absolute error Centered derivative')
plt.xlim(dx_max, dx_min)
plt.legend()

plt.xlabel(r'$h$')
plt.xlim(dx_max, dx_min)
plt.show()
```

### Selección del parámetro $\Delta x$

En las graficas anteriores, se ilustra una regla común para la selección del parámetro $\Delta x$, en la cual se sugiere tomar la raiz cuadrada del epsilon de la máquina,

$\Delta x \sim \sqrt{\epsilon}$

Ya que determinamos antes que $\epsilon \sim 10^{-16}$, tomaremos $\Delta x \sim 10^{-8}$.

##### Ejemplo
En este ejemplo se calcula la derivada numérica de la función $\sin x$ en un rango de la variable $x$ y se compara con el valor de la derivada analítica.

```python
import numpy as np
import matplotlib.pyplot as plt

def f(x):
    return np.sin(x)

def dfdx(x):
    return np.cos(x)

def forwardDerivative(f, x, dx=1e-8):
    return (f(x + dx) - f(x))/dx

def backwardDerivative(f, x, dx=1e-8):
    return (f(x) - f(x - dx))/dx

def centeredDerivative(f, x, dx=1e-8):
    return (f(x + dx) - f(x - dx))/(2*dx)

x = np.linspace(0, np.pi/2, 100)
function = f(x)
f_dfdx = forwardDerivative(f, x)
b_dfdx = backwardDerivative(f, x)
c_dfdx = centeredDerivative(f, x)

abs_error_f = np.abs(f_dfdx - dfdx(x))
abs_error_b = np.abs(b_dfdx - dfdx(x))
abs_error_c = np.abs(c_dfdx - dfdx(x))

plt.figure(figsize=(7,10))
plt.subplot(2,1,1)
plt.plot(x, function, color='darkviolet', label=r'$\sin x$')
plt.plot(x, f_dfdx, 'r', label=r'Forward derivative')
plt.plot(x, b_dfdx, 'b', label=r'Backward derivative')
plt.plot(x, c_dfdx, 'g', label=r'Centered derivative')
plt.ylabel(r'$f(x)$ , $\frac{df}{dx}$' )
plt.legend()

plt.subplot(2,1,2)
plt.plot(x, abs_error_f, 'r', label=r'Forward derivative')
plt.plot(x, abs_error_b, 'b', label=r'Backward derivative')
plt.plot(x, abs_error_c, 'g', label=r'Centered derivative')
plt.yscale('log')
plt.ylabel(r'Absolute error')

plt.xlabel(r'$x$')
plt.show()

```

###### Note que el error absoluto en la derivada numérica nunca es mayor que $10^{-8}$

---
## Diferenciación en un intervalo.

Considere una malla igualmente espaciada definiendo un dominio $[x_0, x_1, ..., x_j, ...,x_{n-1}] $. En cualquiera de los puntos internos de este intervalo, se puede calcular la primera derivada de una función mediante las diferencias finitas hacia adelante o hacia atras,

$f'(x_i) = \frac{f_{i+1} - f_{i}}{\Delta x} + \mathcal{O}(\Delta x) $

$f'(x_i) = \frac{f_i - f_{i-1}}{\Delta x} + \mathcal{O}(\Delta x) $

y también con la aproximación a segundo orden centrada,

$f'(x_i) = \frac{f_{i+1} - f_{i-1}}{2\Delta x} + \mathcal{O}(\Delta x^2) .$

Sin embargo, debe notarse que ne el punto $x_{n-1}$ no es posible utilizar la derivada hacia adelante mientras que en el punto $x_0$ no es posible utilizar la derivada hacia atrás. De forma similar, la derivada centrada no puede utilizarse en ninguno de estos dos puntos frontera del dominio.

---
## Diferencias Finitas en una Malla con Intervalos Desiguales

En algunas aplicaciones de la astrofísica es adecuado utilizar una malla con intervalos desiguales para describir con mayor precisión el sistema.

En este caso, las derivadas unlaterales (hacia adelante o hacia atras) no se modifican debido a que solamente incluyen un intervalo (dos puntos consecuticos) y por lo tanto un solo valor de $\Delta x$.

Por otra parte, la derivada centrada involucra tres puntos, es decir dos intervalos y por lo tanto dos valores de $h$. Para obtener una expresión para esta derivada de segundo orden se consideran las siguientes expansiones en serie de la función,

$f(x_i + \Delta x_2) = f(x_i) + f'(x_i)  \Delta x_2 + f''(x_i) \frac{\Delta x_2^2}{2}  +
\mathcal{O}(\Delta x_2^3)$

$f(x_i - \Delta x_1) = f(x_i) - f'(x_i)\Delta x_1  + f''(x_i) \frac{\Delta x_1^2}{2}  +
\mathcal{O}(\Delta x_1^3)$,

donde $\Delta x_1 = x_i - x_{i-1}$ y $\Delta x_2 = x_{i+1} - x_{i}$. Restando estas ecuaciones y eliminando el término de segunda derivada se puede despejar el valor de $f'(x_i)$ en la forma

$f'(x_i) = \frac{\Delta x_1}{\Delta x_2(\Delta x_1+\Delta x_2)} f(x_{i+1}) - \frac{\Delta x_1 - \Delta x_2}{\Delta x_2 \Delta x_1} f(x_i)
- \frac{\Delta x_2}{\Delta x_1(\Delta x_1 + \Delta x_2)} f(x_{i-1})$.

Es claro que esta expresión se reduce a la derivada centrada en una malla igualmente espaciada cuando $\Delta x_1 = \Delta x_2 = \Delta x$.

---
## Diferenciación Numérica de una Función Discreta

Ahora se implemenará un algoritmo para calcular la derivada numérica de una función que se proporciona mediante un conjunto de puntos discretos en un archivo .txt.

```python
import numpy as np
import matplotlib.pyplot as plt

def Derivative(x, f):
    '''
    ------------------------------------------
    This function returns the numerical
    derivative of a discretely-sample function
    using one-side derivatives in the extreme
    points of the interval and second order
    accurate derivative in the middle points.
    The data points may be evenly or unevenly
    spaced.
    ------------------------------------------
    '''
    # Number of points
    N = len(x)
    dfdx = np.zeros([N, 2])
    dfdx[:,0] = x

    # Derivative at the extreme points
    dfdx[0,1] = (f[1] - f[0])/(x[1] - x[0])
    dfdx[N-1,1] = (f[N-1] - f[N-2])/(x[N-1] - x[N-2])

    #Derivative at the middle points
    for i in range(1,N-1):
        h1 = x[i] - x[i-1]
        h2 = x[i+1] - x[i]
        dfdx[i,1] = h1*f[i+1]/(h2*(h1+h2)) - (h1-h2)*f[i]/(h1*h2) -\
                    h2*f[i-1]/(h1*(h1+h2))

    return dfdx

# Loading the data
data = np.loadtxt('function_data.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

# Calling the derivative function
dfdx_data = Derivative(x,f)

plt.plot(x, f, 'b', label=r'$f(x)$')
plt.plot(dfdx_data[:,0], dfdx_data[:,1], 'r', label=r'$\frac{df}{dx}$')
plt.xlabel(r'$x$')
plt.legend()
plt.show()

```

---
## Convergencia del Método Numérico

### 1. Cuando la solución analítica es conocida

Si un problema astrofísico posee una solución analítica $Y(x)$ y un cierto algoritmo da como resultado una solución numérica $y(x;\Delta x)$, se dice que el método numérico es **convergente** si

\begin{equation}
\lim_{\Delta x \rightarrow 0} y(x;\Delta x) = Y(x).
\end{equation}

Para medir la convergencia del método numérico, se introducirá el **factor de convergencia**, $C$, como

$$
C = \frac{\left| y(x:\Delta x_2) - Y(x) \right|}{\left| y(x:\Delta x_1) - Y(x) \right|} \equiv \left(\frac{\Delta x_2}{\Delta x_1} \right)^n
$$

donde se han escogido dos tamaños de discretización $\Delta x_2 < \Delta x_1$. Cuando $C<1$, el método numérico es convergente y, debido a la ultima identificación, el número $n$ corresponde al orden de precisión del método.

Por ejemplo, si se toman los valores de discretización tales que $\frac{\Delta x_2}{\Delta x_1} = 10^{-1}$, se obtiene

$$
C \equiv \left(\frac{\Delta x_2}{\Delta x_1} \right)^n = (10^{-1})^n = 10^{-n}
$$

y por lo tanto, el orden de precisión del método será $n=-\log_{10} C$.

##### Ejemplo

```python
import numpy as np
import matplotlib.pyplot as plt

def f(x):
    return np.sin(x)

def dfdx(x):
    return np.cos(x)

def forwardDerivative(f, x, dx=1e-8):
    return (f(x + dx) - f(x))/dx

def centeredDerivative(f, x, dx=1e-8):
    return (f(x + dx) - f(x - dx))/(2*dx)

x_0 = 1.
dx1 = 1E-3
dx2 = 1E-4

# Forward Derivative Convergence Test
y_1 = forwardDerivative(f, x_0, dx1)
y_2 = forwardDerivative(f, x_0, dx2)

C_forward = np.abs(y_2 - dfdx(x_0))/np.abs(y_1 - dfdx(x_0))

print('Para la derivada hacia adelante:')
print(f'Para dx_1 = {dx1:.1e} y dx_2 = {dx2:.1e}' )
print(f'el factor de convergencia es C = {C_forward:.2f}')
print(f'que corresponde a n = {-np.log10(C_forward):.1f}')

# Centered Derivative Convergence Test
y_1 = centeredDerivative(f, x_0, dx1)
y_2 = centeredDerivative(f, x_0, dx2)

C_centered = np.abs(y_2 - dfdx(x_0))/np.abs(y_1 - dfdx(x_0))

print('')
print('Para la derivada centrada:')
print(f'Para dx_1 = {dx1:.1e} y dx_2 = {dx2:.1e}' )
print(f'el factor de convergencia es C = {C_centered:.2f}')
print(f'que corresponde a n = {-np.log10(C_centered):.1f}')
```

    Para la derivada hacia adelante:
    Para dx_1 = 1.0e-03 y dx_2 = 1.0e-04
    el factor de convergencia es C = 0.10
    que corresponde a n = 1.0

    Para la derivada centrada:
    Para dx_1 = 1.0e-03 y dx_2 = 1.0e-04
    el factor de convergencia es C = 0.01
    que corresponde a n = 2.0

#### 2. Cuando la solución analítica NO se concoce.

En muchos casos la solución analítica  $Y(x)$ NO es conocida. Entonces, como se puede realizar la prueba de convergencia del método numérico?

En este caso se introducirán tres tamaños de discretización $\Delta x_3 < \Delta x_2 < \Delta x_1$, y se define el **factor de auto-convergencia**, $C_{self}$,

$C_{self} = \frac{\left| y(x:\Delta x_3) - y(x:\Delta x_2) \right|}{\left| y(x:\Delta x_2) - y(x:\Delta x_1) \right|} \equiv \frac{\Delta x_3^n - \Delta x_2^n}{\Delta x_2^n - \Delta x_1^n}. $

Por ejemplo, si se toman valores tales que $\frac{\Delta x_3}{\Delta x_2} = \frac{\Delta x_2}{\Delta x_1} = 10^{-1}$ , se obtendrá

$C_{self} \equiv \frac{10^{-n} \Delta x_2^n - \Delta x_2^n}{10^{-n}\Delta x_1^n - \Delta x_1^n} = \frac{(10^{-n}-1) \Delta x_2^n }{(10^{-n}-1)\Delta x_1^n } = \left( \frac{\Delta x_2}{\Delta x_1} \right)^n = (10^{-1})^n = 10^{-n}$

y por lo tanto el orden de precisión es $n=-\log_{10} C$.

##### Example

```python
import numpy as np
import matplotlib.pyplot as plt

def f(x):
    return np.sin(x)

def forwardDerivative(f, x, dx=1e-8):
    return (f(x + dx) - f(x))/dx

def centeredDerivative(f, x, dx=1e-8):
    return (f(x + dx) - f(x - dx))/(2*dx)

x_0 = 1.
dx1 = 1E-2
dx2 = 1E-3
dx3 = 1E-4

# Forward Derivative Convergence Test
y_1 = forwardDerivative(f, x_0, dx1)
y_2 = forwardDerivative(f, x_0, dx2)
y_3 = forwardDerivative(f, x_0, dx3)

C_self = np.abs(y_3 - y_2)/np.abs(y_2 - y_1)

print('Para la derivada hacia adelante:')
print(f'Para dx_1 = {dx1:.1e}, dx_2 = {dx2:.1e} y dx_3 = {dx3:.1e}' )
print(f'el factor de convergencia es C = {C_self:.2f}')
print(f'que corresponde a n = {-np.log10(C_self):.1f}')

# Centered Derivative Convergence Test
y_1 = centeredDerivative(f, x_0, dx1)
y_2 = centeredDerivative(f, x_0, dx2)
y_3 = centeredDerivative(f, x_0, dx3)

C_self = np.abs(y_3 - y_2)/np.abs(y_2 - y_1)

print('')
print('Para la derivada centrada:')
print(f'Para dx_1 = {dx1:.1e}, dx_2 = {dx2:.1e} y dx_3 = {dx3:.1e}' )
print(f'el factor de convergencia es C = {C_self:.2f}')
print(f'que corresponde a n = {-np.log10(C_self):.1f}')
```

    Para la derivada hacia adelante:
    Para dx_1 = 1.0e-02, dx_2 = 1.0e-03 y dx_3 = 1.0e-04
    el factor de convergencia es C = 0.10
    que corresponde a n = 1.0

    Para la derivada centrada:
    Para dx_1 = 1.0e-02, dx_2 = 1.0e-03 y dx_3 = 1.0e-04
    el factor de convergencia es C = 0.01
    que corresponde a n = 2.0

```python

```

01. Errores y Differenciacion/02. Diferenciacion.ipynb


# TPHLCT 3階の導関数を相殺する

```python
import numpy as np
import scipy.misc
from scipy.fftpack import dct, idct
import sys
from PIL import Image
import matplotlib
import matplotlib.pyplot as plt
import random
from tqdm._tqdm_notebook import tqdm_notebook
from scipy.fftpack import dct, idct
import seaborn as sns
from skimage.metrics import structural_similarity as ssim

%matplotlib inline

class ImageLoader:
    def __init__(self, FILE_PATH):
        self.img = np.array(Image.open(FILE_PATH))
        # 行数
        self.row_blocks_count = self.img.shape[0] // 8
        # 列数
        self.col_blocks_count = self.img.shape[1] // 8

    def get_points(self, POINT):
        Row = random.randint(0, len(self.img) - POINT - 1)
        Col = random.randint(0, len(self.img) - 1)
        return self.img[Row : Row + POINT, Col]

    def get_block(self, col, row):
        return self.img[col * 8 : (col + 1) * 8, row * 8 : (row + 1) * 8]

# plt.rcParams['font.family'] ='sans-serif'#使用するフォント
# plt.rcParams["font.sans-serif"] = "Source Han Sans"
plt.rcParams["font.family"] = "Source Han Sans JP"  # 使用するフォント
plt.rcParams["xtick.direction"] = "in"  # x軸の目盛線が内向き('in')か外向き('out')か双方向か('inout')
plt.rcParams["ytick.direction"] = "in"  # y軸の目盛線が内向き('in')か外向き('out')か双方向か('inout')
plt.rcParams["xtick.major.width"] = 1.0  # x軸主目盛り線の線幅
plt.rcParams["ytick.major.width"] = 1.0  # y軸主目盛り線の線幅
plt.rcParams["font.size"] = 12  # フォントの大きさ
plt.rcParams["axes.linewidth"] = 1.0  # 軸の線幅edge linewidth。囲みの太さ
matplotlib.font_manager._rebuild()

MONO_DIR_PATH = "../../Mono/"
AIRPLANE = ImageLoader(MONO_DIR_PATH + "airplane512.bmp")
BARBARA = ImageLoader(MONO_DIR_PATH + "barbara512.bmp")
BOAT = ImageLoader(MONO_DIR_PATH + "boat512.bmp")
GOLDHILL = ImageLoader(MONO_DIR_PATH + "goldhill512.bmp")
LENNA = ImageLoader(MONO_DIR_PATH + "lenna512.bmp")
MANDRILL = ImageLoader(MONO_DIR_PATH + "mandrill512.bmp")
MILKDROP = ImageLoader(MONO_DIR_PATH + "milkdrop512.bmp")
SAILBOAT = ImageLoader(MONO_DIR_PATH + "sailboat512.bmp")

N = 16
```

# DCT

## 基底関数

$$
\phi_k[i] = \begin{cases}
\cfrac{1}{\sqrt{N}}  \quad \quad \quad (k=0) \\
\sqrt{\cfrac{2}{N}} \cos \left({\cfrac{\pi}{2N}(2i+1)k}\right) \quad  (k=1,2,...,N-1)
\end{cases}
$$

```python
class DCT:
    def __init__(self, N):
        self.N = N  # データ数
        # 1次元DCTの基底ベクトルの生成
        self.phi_1d = np.array([self.phi(i) for i in range(self.N)])
        # 2次元DCTの基底ベクトルの格納
        self.phi_2d = np.zeros((N, N))

    def phi(self, k):
        """ 離散コサイン変換(DCT)の基底関数 """
        # DCT-II
        if k == 0:
            return np.ones(self.N) / np.sqrt(self.N)
        else:
            return np.sqrt(2.0 / self.N) * np.cos(
                (k * np.pi / (2 * self.N)) * (np.arange(self.N) * 2 + 1)
            )

    def dct(self, data):
        """ 1次元離散コサイン変換を行う """
        return self.phi_1d.dot(data)

    def idct(self, c):
        """ 1次元離散コサイン逆変換を行う """
        return np.sum(self.phi_1d.T * c, axis=1)

    def get_dct2_phi(self, y, x):
        """ 2次元離散コサイン変換の基底を返す """
        phi_x, phi_y = np.meshgrid(self.phi_1d[x], self.phi_1d[y])
        return phi_x * phi_y

    def get_dct2(self, y, x, data):
        """ i,jの2次元DCT係数を返す """
        phi_2d_phi = np.zeros((self.N, self.N))
        phi_2d_phi = self.get_dct2_phi(y, x)
        return np.sum(np.sum(phi_2d_phi * data))

    def dct2(self, data):
        """ 2次元離散コサイン変換を行う """
        for y in range(self.N):
            for x in range(self.N):
                self.phi_2d[y, x] = self.get_dct2(y, x, data)

        return self.phi_2d

    def idct2(self, c):
        """ 2次元離散コサイン逆変換を行う """
        idct2_data = np.zeros((self.N, self.N))
        phi_2d_phi = np.zeros((self.N, self.N))
        for y in range(self.N):
            for x in range(self.N):
                phi_2d_phi = self.get_dct2_phi(y, x)
                idct2_data += c[y,x] * phi_2d_phi

        return idct2_data
```

# MSDS

```python
def msds(N,arr):
    w_e = 0
    e_e = 0
    n_e = 0
    s_e = 0
    nw_e = 0
    ne_e = 0
    sw_e = 0
    se_e = 0
    for row in range(arr.shape[0] // N):
        for col in range(arr.shape[1] // N):
            f_block = arr[row * N : (row + 1) * N, col * N : (col + 1) * N]
            # w
            if col == 0:
                w_block = np.fliplr(f_block)
            else:
                w_block = arr[row * N : (row + 1) * N, (col - 1) * N : col * N]
            # e
            if col == arr.shape[1] // N - 1:
                e_block = np.fliplr(f_block)
            else:
                e_block = arr[row * N : (row + 1) * N, (col + 1) * N : (col + 2) * N]
            # n
            if row == 0:
                n_block = np.flipud(f_block)
            else:
                n_block = arr[(row - 1) * N : row * N, col * N : (col + 1) * N]
            # s
            if row == arr.shape[0] // N - 1:
                s_block = np.flipud(f_block)
            else:
                s_block = arr[(row + 1) * N : (row + 2) * N, col * N : (col + 1) * N]

            w_d1 = f_block[:, 0] - w_block[:, N-1]
            e_d1 = f_block[:, N-1] - e_block[:, 0]
            n_d1 = f_block[0, :] - n_block[N-1, :]
            s_d1 = f_block[N-1, :] - s_block[0, :]

            w_d2 = (w_block[:, N-1] - w_block[:, N-2] + f_block[:, 1] - f_block[:, 0]) / 2
            e_d2 = (e_block[:, 1] - e_block[:, 0] + f_block[:, N-1] - f_block[:, N-2]) / 2
            n_d2 = (n_block[N-1, :] - n_block[N-2, :] + f_block[1, :] - f_block[0, :]) / 2
            s_d2 = (s_block[1, :] - s_block[0, :] + f_block[N-1, :] - f_block[N-2, :]) / 2

            w_e += np.sum((w_d1 - w_d2) ** 2 )
            e_e += np.sum((e_d1 - e_d2) ** 2 )
            n_e += np.sum((n_d1 - n_d2) ** 2)
            s_e += np.sum((s_d1 - s_d2) ** 2)

            # nw
            if row == 0 or col == 0:
                nw_block = np.flipud(np.fliplr(f_block))
            else:
                nw_block = arr[(row - 1) * N : row * N, (col - 1) * N : col * N]
            # ne
            if row == 0 or col == arr.shape[1] // N - 1:
                ne_block = np.flipud(np.fliplr(f_block))
            else:
                ne_block = arr[(row-1) * N : row * N, (col + 1) * N : (col + 2) * N]
            # sw
            if row == arr.shape[0] // N -1 or col == 0:
                sw_block = np.flipud(np.fliplr(f_block))
            else:
                sw_block = arr[row * N : (row+1) * N, (col-1) * N : col * N]
            # se
            if  row == arr.shape[0]//N-1 or col == arr.shape[0] // N -1:
                se_block = np.flipud(np.fliplr(f_block))
            else:
                se_block = arr[(row + 1) * N : (row + 2) * N, (col+1) * N : (col + 2) * N]

            nw_g1 = f_block[0, 0] - nw_block[N-1, N-1]
            ne_g1 = f_block[0, N-1] - ne_block[N-1, 0]
            sw_g1 = f_block[N-1, 0] - sw_block[0, N-1]
            se_g1 = f_block[N-1, N-1] - se_block[0, 0]

            nw_g2 = (nw_block[N-1,N-1] - nw_block[N-2,N-2] + f_block[1,1] - f_block[0,0])/2
            ne_g2 = (ne_block[N-1,0] - ne_block[N-2,1] + f_block[1,N-2] - f_block[0,N-1])/2
            sw_g2 = (sw_block[0,N-1] - nw_block[1,N-2] + f_block[N-2,1] - f_block[N-1,0])/2
            se_g2 = (nw_block[0,0] - nw_block[1,1] + f_block[N-2,N-2] - f_block[N-1,N-1])/2

            nw_e += (nw_g1 - nw_g2) ** 2
            ne_e += (ne_g1 - ne_g2) ** 2
            sw_e += (sw_g1 - sw_g2) ** 2
            se_e += (se_g1 - se_g2) ** 2

    MSDSt = (w_e + e_e + n_e + s_e + nw_e + ne_e + sw_e + se_e)/ ((arr.shape[0]/N)**2)
    MSDS1 = (w_e + e_e + n_e + s_e)/ ((arr.shape[0]/N)**2)
    MSDS2 = (nw_e + ne_e + sw_e + se_e)/ ((arr.shape[0]/N)**2)
    return MSDSt, MSDS1, MSDS2
```

# 係数の計算

$\alpha_k, \beta_k$を計算する。
1階導関数を相殺するような予測関数は
$$u(x)=-\frac{f_x(0)}{2}(1-x^2)+\frac{f_x(1)}{2}x^2$$
で与えられる。

そのとき、
$$\alpha_k = -\sqrt{\frac{2}{N}}\sum_{\ell=0}^{N-1}\frac{(1-x_\ell)^2}{2}\cos(\pi k x_\ell)$$
$$\beta_k = \sqrt{\frac{2}{N}}\sum_{\ell=0}^{N-1}\frac{x_\ell^2}{2}\cos(\pi k x_\ell)$$

$a_k, b_k, c_k, d_k$を計算する。
3階導関数を相殺するような予測関数は
$$
\begin{align}
u(x) & =( -\frac{1}{2}f_x(0) + \frac{1}{12}f^{(3)}_x(0) )(1-x^2)\\
     & -\frac{1}{24}f^{(3)}_x(0)(1-x)^4\\
     & +( \frac{1}{2}f_x(1)-\frac{1}{12}f^{(3)}_x(1) )x^2\\
     & + \frac{1}{24}f^{(3)}_x(1)x^4
\end{align}
$$
で与えられる。

そのとき、
$$a_k = -\sqrt{\frac{2}{N}}\sum_{\ell=0}^{N-1}\frac{(1-x_\ell)^2}{2}\cos(\pi k x_\ell)$$

$$b_k= \sqrt{\frac{2}{N}}\sum_{\ell=0}^{N-1}( \frac{1}{24}-\frac{x_\ell^2}{6}+\frac{x_\ell^3}{6}-\frac{x_\ell^4}{24} )\cos(\pi k x_\ell)$$

$$c_k = \sqrt{\frac{2}{N}}\sum_{\ell=0}^{N-1}\frac{x_\ell^2}{2}\cos(\pi k x_\ell)$$

$$d_k = \sqrt{\frac{2}{N}}\sum_{\ell=0}^{N-1}( -\frac{x_\ell^2}{12}+\frac{x_\ell^4}{24} )\cos(\pi k x_\ell)$$

```python
sampling_x = (0.5 + np.arange(N)) / N
u_1 = (1 - sampling_x) ** 2 / 2
u_2 = 1 / 24 - sampling_x ** 2 / 6 + sampling_x ** 3 / 6 - sampling_x ** 4 / 24
u_3 = sampling_x ** 2 / 2
u_4 = -sampling_x ** 2 / 12 + sampling_x ** 4 / 24
plt.plot(u_1, label="u_1")
plt.plot(u_2, label="u_2")
plt.plot(u_3, label="u_3")
plt.plot(u_4, label="u_4")
plt.legend()
```

```python
ak = - scipy.fftpack.dct(u_1,norm="ortho")
ak
```

    array([-6.66015625e-01, -5.72235345e-01, -1.42356572e-01, -6.27352552e-02,
           -3.48506640e-02, -2.19246012e-02, -1.48813893e-02, -1.06106807e-02,
           -7.81250000e-03, -5.86493057e-03, -4.43936892e-03, -3.34812741e-03,
           -2.47676043e-03, -1.75117449e-03, -1.12037363e-03, -5.46725910e-04])

```python
bk = scipy.fftpack.dct(u_2,norm="ortho")
bk
```

    array([7.77777036e-02, 5.80730594e-02, 3.62946361e-03, 7.16837860e-04,
           2.26725367e-04, 9.27830699e-05, 4.46610568e-05, 2.40199717e-05,
           1.39872233e-05, 8.63042295e-06, 5.54832686e-06, 3.65886073e-06,
           2.43057806e-06, 1.58247439e-06, 9.55052288e-07, 4.50105928e-07])

```python
ck = scipy.fftpack.dct(u_3,norm="ortho")
ck
```

    array([ 6.66015625e-01, -5.72235345e-01,  1.42356572e-01, -6.27352552e-02,
            3.48506640e-02, -2.19246012e-02,  1.48813893e-02, -1.06106807e-02,
            7.81250000e-03, -5.86493057e-03,  4.43936892e-03, -3.34812741e-03,
            2.47676043e-03, -1.75117449e-03,  1.12037363e-03, -5.46725910e-04])

```python
dk = scipy.fftpack.dct(u_4,norm="ortho")
dk
```

    array([-7.77777036e-02,  5.80730594e-02, -3.62946361e-03,  7.16837860e-04,
           -2.26725367e-04,  9.27830699e-05, -4.46610568e-05,  2.40199717e-05,
           -1.39872233e-05,  8.63042295e-06, -5.54832686e-06,  3.65886073e-06,
           -2.43057806e-06,  1.58247439e-06, -9.55052288e-07,  4.50105928e-07])

```python
alpha = ak
alpha
```

    array([-6.66015625e-01, -5.72235345e-01, -1.42356572e-01, -6.27352552e-02,
           -3.48506640e-02, -2.19246012e-02, -1.48813893e-02, -1.06106807e-02,
           -7.81250000e-03, -5.86493057e-03, -4.43936892e-03, -3.34812741e-03,
           -2.47676043e-03, -1.75117449e-03, -1.12037363e-03, -5.46725910e-04])

```python
beta = ck
beta
```

    array([ 6.66015625e-01, -5.72235345e-01,  1.42356572e-01, -6.27352552e-02,
            3.48506640e-02, -2.19246012e-02,  1.48813893e-02, -1.06106807e-02,
            7.81250000e-03, -5.86493057e-03,  4.43936892e-03, -3.34812741e-03,
            2.47676043e-03, -1.75117449e-03,  1.12037363e-03, -5.46725910e-04])

```python
Ak = (2 * ak - 16 * bk) / np.sqrt(N)
Ak
```

    array([-6.44118627e-01, -5.18409910e-01, -8.56961403e-02, -3.42349791e-02,
           -1.83322335e-02, -1.13334329e-02, -7.61933886e-03, -5.40142024e-03,
           -3.96219889e-03, -2.96698698e-03, -2.24187777e-03, -1.68869915e-03,
           -1.24810253e-03, -8.81917144e-04, -5.64007025e-04, -2.75163379e-04])

```python
Bk = (2 * ck - 16 * dk) / np.sqrt(N)
Bk
```

    array([ 6.44118627e-01, -5.18409910e-01,  8.56961403e-02, -3.42349791e-02,
            1.83322335e-02, -1.13334329e-02,  7.61933886e-03, -5.40142024e-03,
            3.96219889e-03, -2.96698698e-03,  2.24187777e-03, -1.68869915e-03,
            1.24810253e-03, -8.81917144e-04,  5.64007025e-04, -2.75163379e-04])

```python
Ck = (2 * ak - 32 * bk) / np.sqrt(N)
Ck
```

    array([-9.55229441e-01, -7.50702148e-01, -1.00213995e-01, -3.71023305e-02,
           -1.92391349e-02, -1.17045652e-02, -7.79798309e-03, -5.49750012e-03,
           -4.01814779e-03, -3.00150867e-03, -2.26407108e-03, -1.70333459e-03,
           -1.25782484e-03, -8.88247042e-04, -5.67827235e-04, -2.76963802e-04])

```python
Dk = (2 * ck - 32 * dk) / np.sqrt(N)
Dk
```

    array([ 9.55229441e-01, -7.50702148e-01,  1.00213995e-01, -3.71023305e-02,
            1.92391349e-02, -1.17045652e-02,  7.79798309e-03, -5.49750012e-03,
            4.01814779e-03, -3.00150867e-03,  2.26407108e-03, -1.70333459e-03,
            1.25782484e-03, -8.88247042e-04,  5.67827235e-04, -2.76963802e-04])

# DCTして残差を計算

$
V_k = F_k - U_k\\
V_k = F_k - A_k(F_0-F_0^L) -B_k(F_0^R-F_0)-C_k(F_1+F_1^L)-D_k(F_1^R+F_1)
$

```python
IMG = LENNA
```

```python
Fk = np.zeros(IMG.img.shape)
```

## DCT 縦方向

2次元入力信号を垂直方向の一次元信号が水平方向に並列に並んでいるものとみなし、各列において8画素単位の1次元DCTを適用する

```python
for row in range(IMG.img.shape[0] // 16):
    for col in range(IMG.img.shape[1]):
        eight_points = IMG.img[16 * row : 16 * (row + 1), col]
        c = scipy.fftpack.dct(eight_points,norm="ortho")
        Fk[16 * row : 16 * (row + 1), col] = c

```

## 縦方向の残差

### 3階の導関数

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * (row + 1), col]

        F_0_r = 0
        F_1_r = 0
        if row is not Fk.shape[0] // 16 - 1:
            F_0_r = Fk[16 * (row + 1), col]
            F_1_r = Fk[16 * (row + 1) + 1, col]

        F_0_l = 0
        F_1_l = 1
        if row is not 0:
            F_0_l = Fk[16 * (row - 1), col]
            F_1_l = Fk[16 * (row - 1) + 1, col]

        # 残差
        F_0 = F[0]
        F_1 = F[1]

        F = (
            F
            - Ak * (F_0 - F_0_l)
            - Bk * (F_0_r - F_0)
            - Ck * (F_1 + F_1_l)
            - Dk * (F_1_r + F_1)
        )
        # F_0, F_1は残す
        F[0] = F_0
        F[1] = F_1
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[16 * row : 16 * (row + 1), col] = F

```

### 1階の導関数

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * (row + 1), col]

        F_0_r = 0
        if row is not Fk.shape[0] // 16 - 1:
            F_0_r = Fk[16 * (row + 1), col]

        F_0_l = 0
        if row is not 0:
            F_0_l = Fk[16 * (row - 1), col]

        # 残差
        F_0 = F[0]
        F_temp = F - alpha * (F_0_r - F_0) / np.sqrt(N) - beta * (F_0 - F_0_l) / np.sqrt(N)
        # F_0は残す
        F[1] = F_temp[1]
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[16 * row : 16 * (row + 1), col] = F

```

## DCT 横方向

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        eight_points = Fk[row, 16 * col : 16 * (col + 1)]
        c = scipy.fftpack.dct(eight_points,norm="ortho")
        Fk[row, 16 * col : 16 * (col + 1)] = c

```

## 横方向の残差

### 3階の導関数

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * (col + 1)]

        F_0_r = 0
        F_1_r = 0
        if col is not Fk.shape[1] // 16 - 1:
            F_0_r = Fk[row, 16 * (col + 1)]
            F_1_r = Fk[row, 16 * (col + 1) + 1]

        F_0_l = 0
        F_1_l = 0
        if col is not 0:
            F_0_l = Fk[row, 16 * (col - 1)]
            F_1_l = Fk[row, 16 * (col - 1) + 1]

        # 残差
        F_0 = F[0]
        F_1 = F[1]

        F = (
            F
            - Ak * (F_0 - F_0_l)
            - Bk * (F_0_r - F_0)
            - Ck * (F_1 + F_1_l)
            - Dk * (F_1_r + F_1)
        )
        # F_0は残す
        F[0] = F_0
        F[1] = F_1
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[row, 16 * col : 16 * (col + 1)] = F

```

### 1階の導関数

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * (col + 1)]

        F_0_r = 0
        if col is not Fk.shape[1] // 16 - 1:
            F_0_r = Fk[row, 16 * (col + 1)]

        F_0_l = 0
        if col is not 0:
            F_0_l = Fk[row, 16 * (col - 1)]

        # 残差
        F_0 = F[0]
        F_temp = F - alpha * (F_0_r - F_0) / np.sqrt(N) - beta * (F_0 - F_0_l) / np.sqrt(N)
        # F_0は残す
        F[1] = F_temp[1]
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[row, 16 * col : 16 * (col + 1)] = F

```

# 係数の確保

```python
Fk_Ori = np.copy(Fk)
```

# 低域3成分 (0,1)(1,0)(1,1)の絶対値の和

```python
low_3_value = 0
others_value = 0
```

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1] // 16):
        block = Fk[row * 16 : (row + 1) * 16, col * 16 : (col + 1) * 16]
        low_3_value += np.abs(block[0, 1]) + np.abs(block[1, 0]) + np.abs(block[1, 1])
        others_value += (
            np.sum(np.sum(np.abs(block)))
            - np.abs(block[0, 0])
            - np.abs(block[0, 1])
            - np.abs(block[1, 0])
            - np.abs(block[1, 1])
        )

```

```python
low_3_value
```

    238373.41544100267

```python
others_value
```

    1336691.007132512

# 逆変換

$
F_k = F_k + U_k\\
F_k = V_k + A_k(F_0-F_0^L) +B_k(F_0^R-F_0)+C_k(F_1+F_1^L)+D_k(F_1^R+F_1)
$

```python
# recover = np.zeros(IMG.img.shape).astype("uint8")
recover = np.zeros(IMG.img.shape)
```

## 横方向の残差

### 1階の導関数

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * col + 16]

        F_0_r = 0
        if col is not Fk.shape[1] // 16 - 1:
            F_0_r = Fk[row, 16 * (col + 1)]

        F_0_l = 0
        if col is not 0:
            F_0_l = Fk[row, 16 * (col - 1)]

        # 残差
        F_0 = F[0]
        F_temp = F + alpha * (F_0_r - F_0) / np.sqrt(N) + beta * (F_0 - F_0_l) / np.sqrt(N)
        # F_0は残す
        F[1] = F_temp[1]
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[row, 16 * col : 16 * col + 16] = F

```

### 3階の導関数

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * (col + 1)]

        F_0_r = 0
        F_1_r = 0
        if col is not Fk.shape[1] // 16 - 1:
            F_0_r = Fk[row, 16 * (col + 1)]
            F_1_r = Fk[row, 16 * (col + 1) + 1]

        F_0_l = 0
        F_1_l = 0
        if col is not 0:
            F_0_l = Fk[row, 16 * (col - 1)]
            F_1_l = Fk[row, 16 * (col - 1) + 1]

        # 残差
        F_0 = F[0]
        F_1 = F[1]

        F = (
            F
            + Ak * (F_0 - F_0_l)
            + Bk * (F_0_r - F_0)
            + Ck * (F_1 + F_1_l)
            + Dk * (F_1_r + F_1)
        )
        # F_0は残す
        F[0] = F_0
        F[1] = F_1
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[row, 16 * col : 16 * (col + 1)] = F

```

### IDCT 横方向

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * col + 16]
        data = scipy.fftpack.idct(F,norm="ortho")
        # Fkに代入した後、縦方向に対して処理
        Fk[row, 16 * col : 16 * col + 16] = data
        # 復元画像
#         recover[row, 16 * col : 16 * col + 16] = data

```

## 縦方向

### 1階の導関数

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * row + 16, col]

        F_0_r = 0
        if row is not Fk.shape[0] // 16 - 1:
            F_0_r = Fk[16 * (row + 1), col]

        F_0_l = 0
        if row is not 0:
            F_0_l = Fk[16 * (row - 1), col]

        # 残差
        F_0 = F[0]
        F_temp = F + alpha * (F_0_r - F_0) / np.sqrt(N) + beta * (F_0 - F_0_l) / np.sqrt(N)
        # F_0は残す
        F[1] = F_temp[1]
        # F_0 F_1 F_2 F_3 F_4 F_5 F_6 F_7
        Fk[16 * row : 16 * row + 16, col] = F

```

### 3階の導関数

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * (row + 1), col]

        F_0_r = 0
        F_1_r = 0
        if row is not Fk.shape[0] // 16 - 1:
            F_0_r = Fk[16 * (row + 1), col]
            F_1_r = Fk[16 * (row + 1) + 1, col]

        F_0_l = 0
        F_1_l = 1
        if row is not 0:
            F_0_l = Fk[16 * (row - 1), col]
            F_1_l = Fk[16 * (row - 1) + 1, col]

        # 残差
        F_0 = F[0]
        F_1 = F[1]

        F = (
            F
            + Ak * (F_0 - F_0_l)
            + Bk * (F_0_r - F_0)
            + Ck * (F_1 + F_1_l)
            + Dk * (F_1_r + F_1)
        )
        # F_0, F_1は残す
        F[0] = F_0
        F[1] = F_1
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[16 * row : 16 * (row + 1), col] = F

```

### 縦方向IDCT

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * (row + 1), col]
        data = scipy.fftpack.idct(F,norm="ortho")
        # 復元画像
        recover[16 * row : 16 * (row + 1), col] = data
        # FKに代入した後、横方向に対して処理
#         Fk[16 * row : 16 * (row + 1), col] = data

```

```python
plt.imshow(np.round(recover), cmap="gray")
```

```python
plt.imshow(IMG.img, cmap="gray")
```

```python
recover[0, 0:10]
```

    array([162., 162., 162., 161., 162., 157., 163., 161., 165., 161.])

```python
IMG.img[0, 0:10]
```

    array([162, 162, 162, 161, 162, 157, 163, 161, 165, 161], dtype=uint8)

ちゃんと復元できた

# 量子化テーブル

```python
Q = 38.5
Q_Luminance = np.ones((16,16)) * Q
```

# 量子化

```python
Fk = np.copy(Fk_Ori)
```

```python
plt.imshow(Fk)
```

```python
Q_Fk = np.zeros(Fk.shape)
```

```python
for row in range(IMG.img.shape[0] // 16):
    for col in range(IMG.img.shape[1] // 16):
        block = Fk[row * 16 : (row + 1) * 16, col * 16 : (col + 1) * 16]
        # 量子化
        block = np.round(block / Q_Luminance)
        # 逆量子化
        block = block * Q_Luminance
        Q_Fk[row * 16 : (row+1)*16, col * 16 : (col+1)*16] = block

```

# 逆変換

```python
Fk = np.copy(Q_Fk)
# Fk = np.copy(Fk_Ori)
```

```python
Q_recover = np.zeros(Q_Fk.shape)
```

## 横方向の残差

### 1階の導関数

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * col + 16]

        F_0_r = 0
        if col is not Fk.shape[1] // 16 - 1:
            F_0_r = Fk[row, 16 * (col + 1)]

        F_0_l = 0
        if col is not 0:
            F_0_l = Fk[row, 16 * (col - 1)]

        # 残差
        F_0 = F[0]
        F_temp = F + alpha * (F_0_r - F_0) / np.sqrt(N) + beta * (F_0 - F_0_l) / np.sqrt(N)
        # F_0は残す
        F[1] = F_temp[1]
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[row, 16 * col : 16 * col + 16] = F

```

### 3階の導関数

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * (col + 1)]

        F_0_r = 0
        F_1_r = 0
        if col is not Fk.shape[1] // 16 - 1:
            F_0_r = Fk[row, 16 * (col + 1)]
            F_1_r = Fk[row, 16 * (col + 1) + 1]

        F_0_l = 0
        F_1_l = 0
        if col is not 0:
            F_0_l = Fk[row, 16 * (col - 1)]
            F_1_l = Fk[row, 16 * (col - 1) + 1]

        # 残差
        F_0 = F[0]
        F_1 = F[1]

        F = (
            F
            + Ak * (F_0 - F_0_l)
            + Bk * (F_0_r - F_0)
            + Ck * (F_1 + F_1_l)
            + Dk * (F_1_r + F_1)
        )
        # F_0は残す
        F[0] = F_0
        F[1] = F_1
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[row, 16 * col : 16 * (col + 1)] = F

```

### IDCT 横方向

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // 16):
        F = Fk[row, 16 * col : 16 * col + 16]
        data = scipy.fftpack.idct(F,norm="ortho")
        # Fkに代入した後、縦方向に対して処理
        Fk[row, 16 * col : 16 * col + 16] = data
        # 復元画像
#         recover[row, 16 * col : 16 * col + 16] = data

```

## 縦方向

### 1階の導関数

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * row + 16, col]

        F_0_r = 0
        if row is not Fk.shape[0] // 16 - 1:
            F_0_r = Fk[16 * (row + 1), col]

        F_0_l = 0
        if row is not 0:
            F_0_l = Fk[16 * (row - 1), col]

        # 残差
        F_0 = F[0]
        F_temp[1] = F + alpha * (F_0_r - F_0) / np.sqrt(N) + beta * (F_0 - F_0_l) / np.sqrt(N)
        # F_0は残す
        F[1] = F_temp[1]
        # F_0 F_1 F_2 F_3 F_4 F_5 F_6 F_7
        Fk[16 * row : 16 * row + 16, col] = F

```

### 3階の導関数

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * (row + 1), col]

        F_0_r = 0
        F_1_r = 0
        if row is not Fk.shape[0] // 16 - 1:
            F_0_r = Fk[16 * (row + 1), col]
            F_1_r = Fk[16 * (row + 1) + 1, col]

        F_0_l = 0
        F_1_l = 1
        if row is not 0:
            F_0_l = Fk[16 * (row - 1), col]
            F_1_l = Fk[16 * (row - 1) + 1, col]

        # 残差
        F_0 = F[0]
        F_1 = F[1]

        F = (
            F
            + Ak * (F_0 - F_0_l)
            + Bk * (F_0_r - F_0)
            + Ck * (F_1 + F_1_l)
            + Dk * (F_1_r + F_1)
        )
        # F_0, F_1は残す
        F[0] = F_0
        F[1] = F_1
        # F_0 V_1 V_2 V_3 V_4 V_5 V_6 V_7
        Fk[16 * row : 16 * (row + 1), col] = F

```

### IDCT 縦方向

```python
for row in range(Fk.shape[0] // 16):
    for col in range(Fk.shape[1]):
        F = Fk[16 * row : 16 * (row + 1), col]
        data = scipy.fftpack.idct(F,norm="ortho")
        # 復元画像
        Q_recover[16 * row : 16 * (row + 1), col] = data
        # FKに代入した後、横方向に対して処理
#         Fk[16 * row : 16 * (row + 1), col] = data

```

```python
Q_recover = np.round(Q_recover)
```

```python
plt.imshow(Q_recover, cmap="gray")
plt.imsave("TPHLCT3_16x16_LENNA.png",Q_recover,cmap="gray")
```

# 情報量

$$
S = - \sum ^{255}_{i=0} p_i log_2 p_i
$$

```python
import pandas as pd
qfk = pd.Series(Q_Fk.flatten())
pro = qfk.value_counts() / qfk.value_counts().sum()
pro.head()
```

     0.0     0.943512
     38.5    0.019016
    -38.5    0.018814
     77.0    0.003876
    -77.0    0.003597
    dtype: float64

```python
S = 0
for pi in pro:
    S -= pi * np.log2(pi)
S
```

    0.48797832637655647

# PSNR

$$
PSNR = 10 log_{10} \frac{MAX^2}{MSE}
$$

$$
{MSE = \frac{1}{m \, n} \sum^{m-1}_{i=0} \sum^{n-1}_{j=0} [ I(i,j) - K(i,j) ]^2}
$$
$I(i,j)$は原画像, $K(i,j)$は圧縮画像

```python
MSE = np.sum(np.sum(np.power((IMG.img - Q_recover),2)))/(Q_recover.shape[0] * Q_recover.shape[1])
```

```python
PSNR = 10 * np.log10(255 * 255 / MSE)
PSNR
```

    33.701731577930644

# MSSIM

```python
MSSIM = ssim(IMG.img,Q_recover.astype(IMG.img.dtype),gaussian_weights=True,sigma=1.5,K1=0.01,K2=0.03)
MSSIM
```

    0.8796161712890167

# MSDS

```python
MSDSt, MSDS1, MSDS2 = msds(16,Q_recover)
```

```python
MSDS1
```

    12180.48193359375

```python
MSDS2
```

    3242.2177734375


Authors: Timothy Hochberg <tim@skytruth.org>, Egil Möller <egil@skytruth.org>

Model Descriptions
------------------

## Heuristic Model

The first model developed is referred to as the
*heuristic model* and was derived by observing that there were
correlations between fishing behaviour and several of the
values present in AIS messages. In particular, the
likelihood that a vessel was fishing tends to increase with
the standard deviation of the speed and course, but to decrease
with mean speed. These features were used to develop the
*heursitic model*:

$$
fishing\_score = \frac{2}{3}\left(\sigma_{s_m} + \sigma_{c_m} + \overline{s_m}\right)
$$

Here $s_m$ and $c_m$ are simple features derived from the
speed and course respectively and $\sigma_x$ and $\overline{x}$
have their standard meanings

$$
\begin{align}
s_m & \equiv 1.0 - \min\left(1, speed\,/\,17\right) \\
c_m & \equiv course\,/\,360 \\
\sigma_x & \equiv \text{standard deviation of } x \\
\overline{x} & \equiv \text{mean of } x
\end{align}
$$

For the *heuristic model*, the means and standard deviations are computed
over a one hour window.

The heuristic model performs reasonably well trawlers and
longliners, but poorly for purse seiners.

  * Implementation: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.0/vessel_scoring/legacy_heuristic_model.py
  * Normalization and calculation of $\sigma_{s_m}$, $\sigma_{c_m}$, $\overline{s_m}$
    (note that $s_m$ and $c_m$ are referred to as `measure_speed` and `measure_course`
    in the code):
    https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.0/vessel_scoring/add_measures.py

## Generic Model

A series of logistic regression models were then developed
using the same three features found in the *heursitic
model*. In order to increase the expressiveness of the
logistic model, powers of the 3 base features are added to
the features. Thus, the full feature vector consists of:
$$
\sigma_{s_m}, \sigma_{s_m}^2,\ldots, \sigma_{s_m}^n, \sigma_{c_m}, \sigma_{c_m}^2,\ldots, \sigma_{c_m}^n,
\overline{s_m},\overline{s_m}^2, \ldots \overline{s_m}^n \\
$$
where $n$ is what we shall be refer to as the *feature order*.
Note that that despite the odd form of
$s_m$, from the point of view of the
logistic model, it is equivalent to the the speed
capped at 17 knots.

The first of the logistic models, referred to as the
*generic model*, is the model currently in use and
is a logistic model using a 12 hour time window
and a feature order of 6. One model is trained for all gear
types. This model generally performs bettter than the
heuristic model, but still performs rather poorly on purse
seiners. The 12-hour window was arrived at by plotting the model
accuracy versus window size. There is a different optimal
window size for each gear type, but 12 hours performed
well for a model trained and tested on all gear types.

  * Implementation: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.0/vessel_scoring/logistic_model.py
  * Parameters for the implementation: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.0/vessel_scoring/models.py#L17
  * Normalization and calculation of $\sigma_{s_m}$, $\sigma_{c_m}$, $\overline{s_m}$
    (note that $s_m$ and $c_m$ are referred to as `measure_speed` and `measure_course`
    in the code): https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.0/vessel_scoring/add_measures.py

## Multi-Window Model

The multi-window model
is a logistic model similar to the *generic model* except that
it uses
multiple time windows, ranging in duration from one-half to
twenty four hours. Using multiple window sizes both provides
a richer feature set and avoids the needs to optimize over
window size.

* Implementation: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.1/vessel_scoring/logistic_model.py
* Parameters for the implementation: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.1/vessel_scoring/models.py#L20
  * Windows used: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.1/vessel_scoring/colspec.py#L4

## Multi-Window, Gear-Types-Specific Models

The multi-window gear-type-specific model, which
is on the verge of being deployed, are a set of
models, each the same as the Multi-Window model,
but each trained on only vessels with a specific
gear type. We have currently trained the model
for longliners, trawlers and purse seiners.
We are also experimenting with adding other features.
In particular, whether it is currently daylight appears to
be a very useful feature for predicting purse seine fishing.
These changes, taken together, dramatically improve the
performance, particularly of purse seiners.

* Implementation: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.1/vessel_scoring/logistic_model.py
* Parameters for the implementation: https://github.com/GlobalFishingWatch/vessel-scoring/blob/release-1.1/vessel_scoring/models.py#L22-L38

## Future Models

It is straightforward to use the multi-window logistic
model features described above with a random forest or neural net
model. In early experiments, both of these model types offer
slightly improved performance relative to logistic model while at
the same eliminating the need to augment the feature vector
with powers of the base features.

We eventually plan to experiment with using convolutional or
recurrent neural networks to find features in the AIS data
directly rather than hand engineering the features.

General Notes
-------------

The precision of the models vary by gear type: Long liners are easiest to
predict, even for a model trained on all gear types,
followed by trawlers; purse seiners are the worst.

We have evaluated the models using a separate test set (and
for window size and feature order, optimization, using
separate train-, validation- and test-sets) plotting
precision/recall and ROC curves.

We have also evaluated the generic model on each gear type
separately as well as on the combined data set. In addition,
for longliners we have cross trained and validated between
two separately labelled datasets with slightly different
labeling methods (Kristinas' and Alex data).

```python
from __future__ import print_function, division
%matplotlib inline
import numpy as np
import vessel_scoring.models
from vessel_scoring.models import train_model_on_data
from vessel_scoring import data, utils
from vessel_scoring.evaluate_model import evaluate_model, compare_models
from IPython.core.display import display, HTML, Markdown
from sklearn import metrics
import pandas as pd
```

```python
# Load training and test data

# Data supplied by Kristina
_, train_lline,  valid_lline, test_lline = data.load_dataset_by_vessel(
        'datasets/kristina_longliner.measures.npz')
_, train_trawl,  valid_trawl, test_trawl = data.load_dataset_by_vessel(
        'datasets/kristina_trawl.measures.npz')
_, train_pseine, valid_pseine, test_pseine = data.load_dataset_by_vessel(
        'datasets/kristina_ps.measures.npz')

# Crowd sourced longliner data
test_lline_crowd, _, _, _ = data.load_dataset_by_vessel(
        "datasets/classified-filtered.measures.npz")

# Slow transits (used to train models to avoid classifying slow transits as fishing)
TRANSIT_WEIGHT = 10
x_tran, xtrain_tran, xcross_tran, xtest_tran = data.load_dataset_by_vessel(
                                    'datasets/slow-transits.measures.npz', even_split=False)
xtrain_tran = utils.clone_subset(xtrain_tran, test_lline.dtype)
xcross_tran = utils.clone_subset(xcross_tran, test_lline.dtype)
xtest_tran = utils.clone_subset(xtest_tran, test_lline.dtype)
train_tran = np.concatenate([xtrain_tran, xcross_tran] * TRANSIT_WEIGHT)

train = np.concatenate([train_trawl, train_lline, train_pseine,
                        valid_lline, valid_trawl, valid_pseine, train_tran])
```

    Warning, insufficient items to sample, returning all
    Warning, inufficient items to sample, returning 4430
    Warning, inufficient items to sample, returning 2084

## How much test data do we have

Our initial test and training data consisted of roughly a dozen different vessels of each type
classified over a multi-year period by Kristina Boerder of Dalhousie University. One-quarter of
those are used for testing, so there is a relatively small number of different vessels in the test
sets.

In addition, we are beginning to collect crowd sourced data for both testing and training. Some of the
early crowd sourced data, available for long liners only, is used as an additional test set in the examples
below.

```python
for name, test_data in [("trawlers", test_trawl),
                        ("purse seiners", test_pseine),
                        ("longliners", test_lline),
                        ("crowd sourced longliners", test_lline_crowd)]:
    mmsi_count = len(set(test_data['mmsi']))
    pt_count = len(test_data)
    print("For {0} we have {1} test vessels with {2} test points".format(name, mmsi_count, pt_count))
```

    For trawlers we have 3 test vessels with 5000 test points
    For purse seiners we have 3 test vessels with 5000 test points
    For longliners we have 2 test vessels with 5000 test points
    For crowd sourced longliners we have 118 test vessels with 324166 test points

```python
# Prepare the models

from vessel_scoring.legacy_heuristic_model import LegacyHeuristicModel
from vessel_scoring.logistic_model import LogisticModel

uniform_training_data = {'longliner': train,
                         'longliner crowd' : train,
                         'trawler': train,
                         'purse_seine': train}

gear_specific_training_data = {'longliner': np.concatenate([train_lline, valid_lline, train_tran]),
                               'longliner crowd' : np.concatenate([train_lline, valid_lline, train_tran]),
                               'trawler': np.concatenate([train_trawl, valid_trawl, train_tran]),
                               'purse_seine': np.concatenate([train_pseine, valid_pseine, train_tran])}

test_data = {'longliner': test_lline,
             'longliner crowd': test_lline_crowd,
             'trawler': test_trawl,
             'purse_seine': test_pseine}

untrained_models = [
    ("Heuristic Model", LegacyHeuristicModel(window=3600),
         uniform_training_data),
    ('Generic Model', LogisticModel(colspec=dict(windows=[43200]), order=6),
         uniform_training_data),
    ('Multi-Window, Gear-Type-Specific Models', LogisticModel(colspec=dict(
        windows=[1800, 3600, 10800, 21600, 43200, 86400],
        measures=['measure_daylight', 'measure_speed']), order=6),
         gear_specific_training_data),
]
```

## Model Comparisons

The models output a numbers between 0 and 1 that correspond to how
confident they are that there is fishing occuring. For
the first set of comparisons we treat predictions `>0.5`
as fishing and those `<=0.5` as nonfishing. This allows us to use
*precision*, *recall* and *f1-score* as metrics. We also show Receiver
Operator Characteristic (ROC) area under the curve (AUC) plots and
precision recall plots.

```python
for gear in ['purse_seine', 'trawler', 'longliner', 'longliner crowd']:
    X_test = test_data[gear]

    display(HTML("<h2>{}</h2>".format(gear.replace('_', ' ').title())))

    trained_models = [(name, train_model_on_data(mdl, X_train[gear])) for (name, mdl, X_train) in untrained_models]

    predictions = []
    for name, mdl in trained_models:
        predictions.append((name, (mdl.predict_proba(X_test)[:,1] > 0.5), X_test['classification'] > 0.5))

    lines = ["|Model|Recall|Precision|F1-Score|",
         "|-----|------|---------|--------|"]
    for name, pred, actual in predictions:
        lines.append("|{}|{:.2f}|{:.2f}|{:.2f}|".format(name,
                                            metrics.recall_score(actual, pred),
                                            metrics.precision_score(actual, pred),
                                            metrics.f1_score(actual, pred)))

    display(Markdown('\n'.join(lines)))

    compare_models(trained_models, X_test)

    display(HTML("<hr/>"))
```


########################################################
### This file is used to generate Table 4-6, Fig 2-3 ###
########################################################

- [Forward Problem](#Forward-Problem)
    - [Verify Assumption 1](#Verify-Assumption-1)
        - [Table 4](#Table-4)
        - [Table 5](#Table-5)
    - [Verify Lemma 1](#Verify-Lemma-1)
        - [Left plot in Figure 2](#Left-plot-in-Figure-2)
    - [Verify Theorem 3.1](#Verify-Theorem-3.1)
        - [Right plot in Figure 2](#Right-plot-in-Figure-2)
- [Inverse Problem](#Inverse-Problem)
    - [Verify Assumption 2](#Verify-Assumption-2)
        - [Table 6](#Table-6)
    - [Verify Theorem 4.2](#Verify-Theorem-4.2)
        - [Figure 3](#Figure-3)

```python
import os
import numpy as np
import numpy.polynomial.legendre as leg
from scipy.stats import beta
from scipy.stats import uniform
from scipy.integrate import odeint
from scipy.stats import gaussian_kde as kde
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

from matplotlib import pyplot as plt
%matplotlib inline
```

```python
####### Plot Formatting ######
plt.rc('lines', linewidth = 1.5)
plt.rc('xtick', labelsize = 14)
plt.rc('ytick', labelsize = 14)
plt.rc('legend',fontsize=14)
# plt.rcParams["font.family"] = "serif"
plt.rcParams['axes.labelsize'] = 20
plt.rcParams['axes.titlesize'] = 12
plt.rcParams['lines.markersize'] = 6
plt.rcParams['figure.figsize'] = (8.0, 6.0)
```

## Modified version of Example from Xiu2002

$$ \frac{dy(t)}{dt} = -\lambda y, \ \  y(0)=1 $$
$$ y(t) = e^{-\lambda t} $$
$$QoI = y(0.5)$$

$\lambda\sim U[-1,1]$, $t\in[0,1]$

$\Lambda = [-1,1]$, $\mathcal{D}=[e^{-0.5},e^{0.5}]$

```python
def Phi(n):
    '''Define L_n'''
    coeffs = [0]*(n+1)
    coeffs[n] = 1
    return coeffs

def inner2_leg(n):
    return 2/(2*n+1)

def product3_leg(i,j,l):
    #compute \Phi_i*\Phi_j*\Phi_l
    return lambda x: leg.legval(x, leg.legmul(leg.legmul(Phi(i),Phi(j)),Phi(l)))

def inner3_leg(i,j,l):
    '''
    compute <\Phi_i\Phi_j\Phi_l>
    Set up Gauss-Legendra quadrature
    '''

    x, w=leg.leggauss(20)
    inner=sum([product3_leg(i,j,l)(x[idx]) * w[idx] for idx in range(20)])

    return inner
```

```python
def ode_system_leg(y, t, P):
    '''P indicates highest order of Polynomial we use'''
    dydt = np.zeros(P+1)
    for l in range(len(dydt)):
        dydt[l] = -(sum(sum(inner3_leg(i,j,l)*ki_leg[i]*y[j] for j in range(P+1)) for i in range(P+1)))/inner2_leg(l)
    return dydt
```

```python
P=5
ki_leg = [0,1]+[0]*(P-1)
sol_leg = odeint(ode_system_leg, [1.0]+[0.0]*P, np.linspace(0,1,101), args=(P,))
```

```python
def a(i):
    return sol_leg[:,i][50]
coef = np.array([a(0), a(1), a(2), a(3), a(4), a(5)])   #fixed

def Q(i,x):
    return leg.legval(x,coef[:(i+1)])

def Qexact(x):
    return np.exp(-x*0.5)
```

```python
#### Use plot to show the difference between the exact and approximate map #####
fig = plt.figure()

def plot_Qn(n):
    fig.clear()
    x = np.linspace(-3,3,100)
    y = Qexact(x)
    yn = Q(n, x)
    plt.plot(x,y,linestyle='-.',linewidth=4,label="$Q(\lambda)$")
    plt.plot(x,yn,label='Q_'+str(n)+'$(\lambda)$')
    plt.xlabel('$\Lambda$')
    plt.legend();

interact(plot_Qn,
         n = widgets.IntSlider(value=1,min=1,max=5,step=1))
```

    <Figure size 576x432 with 0 Axes>

    interactive(children=(IntSlider(value=1, description='n', max=5, min=1), Output()), _dom_classes=('widget-inte…

    <function __main__.plot_Qn(n)>

## Forward Problem

$\lambda\sim U([-1,1])$, QOI is the value at $t=0.5$ ($y(0.5)$). $Q_n$ defines the Polynomial Chaos expansion with degree $n$.

$$
Q(\lambda)=y(0.5)=\sum\limits_{i=0}^{\infty} y_i(0.5)\Phi_i
$$
$$
Q_n(\lambda)=\sum\limits_{i=0}^n y_i(0.5)\Phi_i
$$

Verify Result of Lemma 2:

$Q_n(\lambda)\to Q(\lambda)$ in $L^p(\Lambda)$, if Assumptions 1 holds and $D_c\subset\mathcal{D}$ being compact, then
\begin{equation}
	\pi_{\mathcal{D}}^{Q_n}(q) \to \pi_{\mathcal{D}}^{Q}(q) \text{ almost in} {L^r(D_c)}
\end{equation}

Since $\mathcal{D}$ is compact in this problem, we choose $D_c=\mathcal{D}$.

Verify Result of Theorem 3.1:

$Q_n(\lambda)\to Q(\lambda)$ in $L^p(\Lambda)$, if Assumptions 1 holds, $\{\pi_{\mathcal{D}}^{Q_n}\}$ are uniformly integrable in $L^p(\mathcal{D})$, $\mathcal{D}$ is compact, then
\begin{equation}
        \pi_{\mathcal{D}}^{Q_n}(Q_n(\lambda)) \to \pi_{\mathcal{D}}^{Q}(Q(\lambda)) \text{ in } L^p(\Lambda)
\end{equation}

### Verify Assumption 1

```python
##### Generate data in Table 4 and 5 #####
def assumption1(n,J):
    np.random.seed(123456)
    initial_sample = np.random.uniform(-1,1,size = J)
    pfprior_sample_n = Q(n,initial_sample)
    pfprior_dens_n = kde(pfprior_sample_n)
    x = np.linspace(-1,3,1000)
    return np.round(np.max(np.abs(np.gradient(pfprior_dens_n(x), x))), 2), np.round(np.max(pfprior_dens_n(x)),2)

size_J = [int(1E3), int(1E4), int(1E5)]
degree_n = [1, 2, 3, 4, 5]
Bound_matrix, Lip_Bound_matrix = np.zeros((3,5)), np.zeros((3,5))
for i in range(3):
    for j in range(5):
        n, J = degree_n[j], size_J[i]
        Lip_Bound_matrix[i,j] = assumption1(n, J)[0]
        Bound_matrix[i,j] = assumption1(n, J)[1]
```

#### Table 4

```python
###########################################
################ Table 4 ##################
###########################################
print('Table 4')
print('Bound under certain n and J values')
print(Bound_matrix)
```

    Table 4
    Bound under certain n and J values
    [[1.06 1.28 1.24 1.24 1.24]
     [1.03 1.49 1.42 1.42 1.42]
     [1.   1.61 1.49 1.5  1.5 ]]

#### Table 5

```python
###########################################
################ Table 5 ##################
###########################################
print('Table 5')
print('Lipschitz bound under certain n and J values')
print(Lip_Bound_matrix)
```

    Table 5
    Lipschitz bound under certain n and J values
    [[ 5.58  8.18  7.73  7.76  7.76]
     [ 8.4  14.18 12.97 13.05 13.05]
     [13.45 23.43 21.   21.22 21.21]]

```python
#### Use plot to show the difference between the exact pushforward and approximate pushforward #####
fig=plt.figure()

def plot_pushforward(n,J):
    fig.clear()
    np.random.seed(123456)
    initial_sample = np.random.uniform(-1,1,size = J)

    pfprior_sample = Qexact(initial_sample)
    pfprior_dens = kde(pfprior_sample)

    pfprior_sample_n = Q(n,initial_sample)
    pfprior_dens_n = kde(pfprior_sample_n)

    fig.clear()
    x = np.linspace(-1,3,1000)
    y = pfprior_dens(x)
    yn = pfprior_dens_n(x)
    plt.plot(x,y,color='r', linestyle='-.', linewidth=4, label="$\pi_{\mathcal{D}}^Q$")
    plt.plot(x,yn,linewidth=2,label="$\pi_{\mathcal{D}}^{Q_{n}}$")
    plt.title('Lipschitz const. = %4.2f and Bound = %2.2f' %(np.max(np.abs(np.gradient(pfprior_dens_n(x), x))),
                                                                             np.max(pfprior_dens_n(x))))
    plt.xlabel("$\mathcal{D}$")
    plt.legend()

interact(plot_pushforward,
         n = widgets.IntSlider(value=1,min=1,max=5,step=1),
         J = widgets.IntSlider(value=int(1E3),min=int(1E3),max=int(1E5),step=int(1E3)))
```

    <Figure size 576x432 with 0 Axes>

    interactive(children=(IntSlider(value=1, description='n', max=5, min=1), IntSlider(value=1000, description='J'…

    <function __main__.plot_pushforward(n, J)>

### Verify Lemma 1
**Print out Monte Carlo Approximation of $	\|\pi_{\mathcal{D}}^Q(q)-\pi_{\mathcal{D}}^{Q_n}(q)\|_{L^r(\mathcal{D_c})} $ where $r>0$ and $D_c=\mathcal{D}$ because $\mathcal{D}$ is compact.**

```python
#Build $\pi_D^Q$ and $\pi_D^{Q,n}$, use 10,000 samples
N_kde = int(1E4)
N_mc = int(1E4)

np.random.seed(123456)
initial_sample = np.random.uniform(-1,1,size = N_kde)
pfprior_sample = Qexact(initial_sample)
pfprior_dens = kde(pfprior_sample)

def pfprior_dens_n(n,x):
    pfprior_sample_n = Q(n,initial_sample)
    pdf = kde(pfprior_sample_n)
    return pdf(x)
```

```python
error_r_D = np.zeros((5,5))
np.random.seed(123456)
qsample = np.random.uniform(np.exp(-0.5),np.exp(0.5),N_mc)

for i in range(5):
    for j in range(5):
        error_r_D[i,j] = (np.mean((np.abs(pfprior_dens(qsample) - pfprior_dens_n(j+1,qsample)))**(i+1)))**(1/(i+1))
```

```python
np.set_printoptions(linewidth=110)
print('L^r error on data space for Forward Problem',end='\n\n')
print(error_r_D)
```

    L^r error on data space for Forward Problem

    [[1.94238726e-01 2.05042717e-02 1.47356406e-03 7.87288538e-05 3.59183884e-06]
     [2.22506150e-01 2.49390997e-02 1.76945246e-03 9.32066906e-05 4.07528839e-06]
     [2.44870659e-01 2.96231360e-02 2.02466852e-03 1.04710022e-04 4.44179886e-06]
     [2.63194738e-01 3.41497229e-02 2.25659747e-03 1.14385798e-04 4.75561998e-06]
     [2.78426119e-01 3.81183280e-02 2.46559182e-03 1.22601045e-04 5.03864501e-06]]

```python
#### To make it cleaner, create Directory "images" to store all the figures ####
imagepath = os.path.join(os.getcwd(),"images")
os.makedirs(imagepath,exist_ok=True)
```

#### Left plot in Figure 2

```python
###########################################
######### The left plot of Fig 2 ##########
###########################################
fig = plt.figure()
plt.xlim([0,6])
marker = ['-D', '-o', '-v', '-s', '-.']
for i in range(5):
    plt.semilogy([1,2,3,4,5],error_r_D[i,:],marker[i],label='r = ' + np.str(i+1))
plt.xlabel('Order of PCE (n)')
plt.ylabel('$L^r$'+' Error in Push-Forward on '+'$\mathcal{D}$')
plt.legend();
# fig.savefig("images/1forward_D_uniform.png")
fig.savefig("images/Fig2(Left).png")
```

### Verify Theorem 3.1
**Print out Monte Carlo Approximation of $	\|\pi_{\mathcal{D}}^Q(Q(\lambda))-\pi_{\mathcal{D}}^{Q_n}(Q_n(\lambda))\|_{L^2(\Lambda)} $**

```python
##### Generate data for the right plot of Fig 2 #####
np.random.seed(123456)
lamsample = np.random.uniform(-1,1,size = N_mc)

error_2 = np.zeros(5)
for i in range(5):
    error_2[i] = (np.mean((np.abs(pfprior_dens(Qexact(lamsample)) - pfprior_dens_n(i+1,Q(i+1,lamsample))))**2))**(1/2)
```

```python
np.set_printoptions(linewidth=110)
print('L^2 error on parameter space for Forward Problem',end='\n\n')
print(error_2)
```

    L^2 error on parameter space for Forward Problem

    [2.42452472e-01 3.51921960e-02 2.34712105e-03 1.20430887e-04 4.89385277e-06]

#### Right plot in Figure 2

```python
############################################
######### The right plot of Fig 2 ##########
############################################
fig = plt.figure()
plt.xlim([0,6])
plt.semilogy([1,2,3,4,5],error_2,'-s')
plt.xlabel('Order of PCE (n)')
plt.ylabel('$L^2$'+' Error in Push-Forward on '+'$\Lambda$');
# fig.savefig("images/1forward_Lam_uniform.png")
fig.savefig("images/Fig2(Right).png")
```

## Inverse Problem

Initial guess is $\lambda\sim U([-1,1])$.

Observation is $\pi_{\mathcal{D}}\sim Beta(4,4)$ with location and scale parameters chosen to be on $[1,1.25]$.

Verify Result of Theorem 4.2:

$Q_n(\lambda)\to Q(\lambda)$ in $L^p(\Lambda)$, $\pi_{\Lambda}^i\in L^p(\mathcal{D})$. If Assumptions 1, 2 hold, $\{\pi_{\mathcal{D}}^{Q_n}\}$ are uniformly integrable in $L^p(\mathcal{D})$, then
\begin{equation}
        \pi_{\Lambda}^{u,n}(\lambda) \to \pi_{\Lambda}^{u}(\lambda) \text{ in } L^p(\Lambda)
\end{equation}

```python
def pdf_obs(x):
    return beta.pdf(x, a=4, b=4, loc=1, scale=0.25)
```

```python
#### Use plot to show the difference between the pushforward of the init and the observed #####
fig = plt.figure()
xx = np.linspace(-1,3,1000)
y = pdf_obs(xx)
y_pf = pfprior_dens(xx)
plt.plot(xx,y,label="$\pi_{\mathcal{D}}^{obs}$")
plt.plot(xx,y_pf, label="$\pi_{\mathcal{D}}^{Q(init)}$")
plt.xlabel("$\mathcal{D}$")
plt.legend();
```

### Verify Assumption 2

```python
def Meanr(n):
    pfprior_sample_n = Q(n,initial_sample)
    if n==0:
        r = pdf_obs(pfprior_sample)/pfprior_dens(pfprior_sample)
    else:
        r = pdf_obs(pfprior_sample_n)/pfprior_dens_n(n,pfprior_sample_n)
    return np.mean(r)

def pdf_update(n,x):
    if n==0:
        r = pdf_obs(pfprior_sample)/pfprior_dens(pfprior_sample)
        pdf = kde(initial_sample,weights=r)
    else:
        pfprior_sample_n = Q(n,initial_sample)
#         pfprior_dens_n = kde(pfprior_sample_n)
        r = pdf_obs(pfprior_sample_n)/pfprior_dens_n(n,pfprior_sample_n)
        pdf = kde(initial_sample,weights=r)
    return pdf(x)

Expect_r = np.zeros(6)
for i in range(6):
    Expect_r[i] = Meanr(i)
```

#### Table 6

```python
###########################################
################ Table 6 ##################
###########################################
print('Table 6')
print('Expected ratio for verifying Assumption 2')
print(Expect_r[1:])
```

    Table 6
    Expected ratio for verifying Assumption 2
    [1.00342642 0.97855489 0.97788712 0.97809176 0.97809595]

```python
#### Use plot to show the difference between the initial, updated, approximate updated #####
fig=plt.figure()

def plot_update(n):
    fig.clear()
    xx = np.linspace(-1.1,1.1,100)
    plt.plot(xx, uniform.pdf(xx, loc=-1, scale=2), label="Initial Density")
    plt.plot(xx, pdf_update(0,xx), label="$\pi_{\Lambda}^u$")
    plt.plot(xx, pdf_update(n,xx), label="$\pi_{\Lambda}^{u,n}$, n="+str(n))
    plt.legend()
    plt.xlabel("$\Lambda$")
    plt.title('$\mathbb{E}(r) =$ %3.2f' %(Expect_r[n]));

interact(plot_update,
         n = widgets.IntSlider(value=int(1),min=int(1),max=int(5),step=1))
```

    <Figure size 576x432 with 0 Axes>

    interactive(children=(IntSlider(value=1, description='n', max=5, min=1), Output()), _dom_classes=('widget-inte…

    <function __main__.plot_update(n)>

```python
#### Use plot to show the difference between the observed and the pushforward of the approximate updated pdf #####
def update_pushforward(n,x):
    pfprior_sample_n = Q(n,initial_sample)
    r = pdf_obs(pfprior_sample_n)/pfprior_dens_n(n,pfprior_sample_n)
    pdf = kde(pfprior_sample_n,weights=r)
    return pdf(x)

fig = plt.figure()
xx = np.linspace(-1,3,100)
y = pdf_obs(xx)
plt.plot(xx,y,label="$\pi_{\mathcal{D}}^{obs}$")
for i in range(1,6,1):
    y_pf = update_pushforward(i,xx)
    plt.plot(xx,y_pf, label="n="+str(i))
plt.xlabel("$\mathcal{D}$")
plt.legend();
```

### Verify Theorem 4.2

Print out Monte Carlo Approximation of $\|\pi_{\Lambda}^{u,n}(\lambda)-\pi_{\Lambda}^u(\lambda)\|_{L^2(\Lambda)} $

```python
##### Generate data for Fig 3 #####
np.random.seed(123456)
lamsample = np.random.uniform(-1,1,size = N_mc)

error_update = np.zeros(5)
for i in range(5):
    error_update[i] = (np.mean((np.abs(pdf_update(0,lamsample) - pdf_update(i+1,lamsample)))**2))**(1/2)
```

```python
np.set_printoptions(linewidth=110)
print('L^2 Error for Inverse Problem',end='\n\n')
print(error_update)
```

    L^2 Error for Inverse Problem

    [7.93989864e-01 6.16431006e-02 3.38614091e-03 2.52824991e-04 6.91016768e-06]

#### Figure 3

```python
###########################################
################ Figure 3 #################
###########################################
fig = plt.figure()
plt.xlim([0,6])
plt.semilogy([1,2,3,4,5],error_update,'-s')
plt.xlabel('Order of PCE (n)')
plt.ylabel('$L^2$'+' Error in Update');
# fig.savefig("images/1inverse_error_uniform.png")
fig.savefig("images/Fig3.png")
```


```python
from IPython.display import Image
Image("stats.png")
```

#### Time between request arrivals (officer 1):
$$ \begin{split}
X_1 \sim \mathcal{Exp}( \mu_1)
\end{split}$$
#### Number of request arrivals in unit time (officer 1):
$$ \begin{split}
Y_{1_0} \sim \mathcal{Pois}( \mu_1)
\end{split}$$

#### Time between request arrivals (officer 2):
$$ \begin{split}
X_2 \sim \mathcal{Exp}( \mu_2)
\end{split}$$
#### Number of request arrivals in unit time (officer 2):
$$ \begin{split}
Y_{2_0} \sim \mathcal{Pois}( \mu_2)
\end{split}$$

#### Number of arrivals, given probability $p_1$ of referring to the supervisor (officer 1): :
$$ \begin{split}
Y_1 \sim \mathcal{Pois}( \mu_1p_1)
\end{split}$$

#####  prove as below,

\begin{align}
\Pr(X=k)&=\sum_{m=k}^\infty e^{-\mu_1}{\mu_1^m\over m!}
{m\choose k}p_1^k(1-p_1)^{m-k}\\
&={e^{-\mu_1}\mu_1^kp_1^k\over k!}\sum_{m=k}^\infty {\mu_1^{m-k}(1-p_1)^{m-k}\over(m-k)!}\\
&={e^{-\mu_1}(\mu_1p_1)^k\over k!}\sum_{j=0}^\infty {(\mu_1(1-p_1))^j\over j!}\\
&={e^{-\mu_1}(\mu_1p_1)^k\over k!}e^{\mu_1-\mu_1p_1}\\
&={e^{-\mu_1p_1}(\mu_1p_1)^k\over k!}
\end{align}

#### Number of arrivals, given probability $p_2$ of referring to the supervisor (officer 2): :
$$ \begin{split}
Y_2 \sim \mathcal{Pois}( \mu_2p_2)
\end{split}$$

## $$ \begin{split} (Y_1 +Y_2) \sim \mathcal{Pois}( \mu_1p_1+\mu_2p_2) \end{split}$$
# $$ \begin{split}? \end{split}$$

#####  prove as below,
\begin{align*}
  P(Y_1+ Y_2 =k) &= \sum_{i = 0}^k P(Y_1+ Y_2 = k, Y_1 = i)= \sum_{i=0}^k P(Y_2 = k-i , Y_1 =i)\\
    &= \sum_{i=0}^k P(Y_2 = k-i)P(Y_1=i)= \sum_{i=0}^k e^{-\mu_2p_2}\frac{{(\mu_2p_2)}^{k-i}}{(k-i)!}e^{-\mu_1p_1}\frac{{(\mu_1p_1)}^i}{i!}\\
   &= e^{-(\mu_2p_2 + \mu_1p_1)}\frac 1{k!}\sum_{i=0}^k \frac{k!}{i!(k-i)!}{(\mu_2p_2)}^{k-i}{(\mu_1p_1)}^i\\
   &= e^{-(\mu_1p_1 + \mu_2p_2  )}\frac 1{k!}\sum_{i=0}^k \binom ki{(\mu_1p_1)}^i{(\mu_2p_2)}^{k-i}\\
   &= \frac{(\mu_1p_1 + \mu_2p_2)^k}{k!} \cdot e^{-(\mu_1p_1 + \mu_2p_2)}
\end{align*}

#### So number of arrivals is Poisson distribution with parameters   $\mu_1p_1+\mu_2p_2$

#### Time between request arrivals :
## $$ \begin{split}
X \sim \mathcal{Exp}( \mu_1p_1+\mu_2p_2)
\end{split}$$

### With mean = $ \frac{1}{\mu_1p_1+\mu_2p_2}$

```python

```

```python

```

```python

```

```python
!jt -l
```

    Available Themes:
       chesterish
       grade3
       gruvboxd
       gruvboxl
       monokai
       oceans16
       onedork
       solarizedd
       solarizedl

```python
!jt -r
```

    Reset css and font defaults in:
    C:\Users\Vin\.jupyter\custom &
    C:\Users\Vin\AppData\Roaming\jupyter\nbextensions

```python
!jt -t monokai
```

### Analyzing the first 10 million digits of pi: Randomness within structure
dataset - https://introcs.cs.princeton.edu/java/data/

```
Benford's Law
```

```python
import requests
from collections import Counter
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline
```

```python
url = b"https://introcs.cs.princeton.edu/java/data/pi-10million.txt"
response = requests.get(url)
data = response.text
cnts = Counter(data)
```

```python
explode = [0.1]*len(cnts.keys())
fig, ax = plt.subplots(figsize=(6,6))
digits_dist = ax.pie(list(cnts.values()), explode=explode,
                     labels=list(cnts.keys()), autopct='%1.3f%%',
                     shadow=True, startangle=90);
```

## Solving 24 game

```python
import operator
from itertools import product, permutations

syms = [operator.add, operator.sub, operator.mul, mydiv]

def mydiv(n, d):
    return n / d if d != 0 else 9999999

op = {sym: ch for sym, ch in zip(syms, '+-*/')}

def solve24(nums):
    for x, y, z in product(syms, repeat=3):
        for a, b, c, d in permutations(nums):
            if round(x(y(a,b),z(c,d)),5) == 24:
                return f"({a} {op[y]} {b}) {op[x]} ({c} {op[z]} {d})"
            elif round(x(a,y(b,z(c,d))),5) == 24:
                return f"{a} {op[x]} ({b} {op[y]} ({c} {op[z]} {d}))"
            elif round(x(y(z(c,d),b),a),5) == 24:
                return f"(({c} {op[z]} {d}) {op[y]} {b}) {op[x]} {a}"
            elif round(x(y(b,z(c,d)),a),5) == 24:
                return f"({b} {op[y]} ({c} {op[z]} {d})) {op[x]} {a}"
    return '--Not Found--'
```

```python
if __name__ == '__main__':
    #nums = eval(input('Four integers in the range 1:9 inclusive, separated by commas: '))
    for nums in [
        [1,1,1,1],
        [2,2,2,2],
        [3,3,3,3],
        [4,4,4,4],
        [5,5,5,5],
        [6,6,6,6],
        [7,7,7,7],
        [8,8,8,8],
        [9,9,9,9],
            ]:
        print(f"solve24({nums}) -> {solve24(nums)}")
```

    solve24([1, 1, 1, 1]) -> --Not Found--
    solve24([2, 2, 2, 2]) -> --Not Found--
    solve24([3, 3, 3, 3]) -> ((3 * 3) * 3) - 3
    solve24([4, 4, 4, 4]) -> (4 + 4) + (4 * 4)
    solve24([5, 5, 5, 5]) -> (5 * 5) - (5 / 5)
    solve24([6, 6, 6, 6]) -> (6 + 6) + (6 + 6)
    solve24([7, 7, 7, 7]) -> --Not Found--
    solve24([8, 8, 8, 8]) -> --Not Found--
    solve24([9, 9, 9, 9]) -> --Not Found--

# Interview questinos

```
Q1: We have two types of reviewers: careful reviewer (80% of  reviewers) and lazy reviewers (20% of reviewers). Careful reviewers give good reviews 60% of time and bad reviews 40% of time). Lazy reviewers however give good reviews 100% of time.
1)     Probability of a review being good
```
#### $$
\begin{split}
P(good) &= P(good|Lazy) \times P(Lazy) + P(good|Careful) \times P(Careful)\\
&=1\times 0.2 + 0.6\times 0.8\\
&=0.68
\end{split}
$$

```
2)     Expected number of good reviews in 100 reviews
```
### \\[P(good) \times 100 = 68\\]
```
3)     A reviewer gave 3 good reviews, probability of him/her being lazy reviewer
```
###### $$
\begin{split}
P(Lazy|3 good) &=\frac{P(3 good|Lazy)\times P(Lazy)}{P(3 good)} \\
&= \frac{P(3 good|Lazy)\times P(Lazy)}{P(3 good|Lazy)\times P(Lazy)+P(3 good|Careful)\times P(Careful)}\\
&= \frac{1^3\times 0.2 }{1^3\times 0.2 + 0.6^3\times 0.8}
\end{split}
$$
```
4)     What would the above probability change as number of good reviews (N) approach infinity
```
### $$ \frac{1^N\times 0.2 }{1^N\times 0.2 + 0.6^N\times 0.8}$$
```
5)     Based on the results, how would you design an approach to classify lazy and careful reviewer
```

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
%matplotlib inline
def goodreviews(N):
    prob_lazy = []
    for i in range(1,N+1):
        prob_lazy.append(0.2*1/(0.2*1+0.8*0.6**i))
    return prob_lazy
num=25
plt.plot(list(range(1,num+1)),goodreviews(num), 'r+')
plt.title('Probability of being lazy reviewer')
plt.xlabel('# of good reviews');
```

```python
# - # of good reviews vs. probability of being lazy reviewer
[(i+1,round(p,5)) for i,p in enumerate(goodreviews(num))]
```

    [(1, 0.29412),
     (2, 0.40984),
     (3, 0.53648),
     (4, 0.65859),
     (5, 0.76275),
     (6, 0.84273),
     (7, 0.8993),
     (8, 0.93704),
     (9, 0.96125),
     (10, 0.97638),
     (11, 0.9857),
     (12, 0.99137),
     (13, 0.9948),
     (14, 0.99688),
     (15, 0.99812),
     (16, 0.99887),
     (17, 0.99932),
     (18, 0.99959),
     (19, 0.99976),
     (20, 0.99985),
     (21, 0.99991),
     (22, 0.99995),
     (23, 0.99997),
     (24, 0.99998),
     (25, 0.99999)]

```python
==============================================
"""
Approach to classify lazy and careful reviewer
"""
prob_threshold = 0.5
if bad_review:
    reviewer = 'Careful'
elif not bad_review and good_review<3:
    reviewer = 'Careful'
elif not bad_review and good_review>=3:
    reviewer = 'Lazy'
==============================================

```

## Assume you have a fair coin.
### Q1: What's expected number of coin flips to get Head, Tail in a row?
### Q2: Expected number of coin flips to get two/three Heads consecutively.

- #### Method 1: Markov Chains Transition Matrices
- #### Method 2: Monte Carlo Simulation

```

```

### Head, Tail
_state 0:
state 1: Head
state 2: Head, Tail_

|                |     State 0    |     State 1    |     State 2    |
|----------------|----------------|----------------|----------------|
|     State 0    |     0.5        |     0.5        |     0          |
|     State 1    |     0          |     0.5        |     0.5        |
|     State 2    |     0          |     0          |     1          |

\begin{split}
\Phi(0) &= 0.5 \times (1+\Phi(0))+ 0.5 \times (1+\Phi(1))\\
\Phi(1) &= 0.5 \times (1+\Phi(1))+ 0.5 \times (1+\Phi(2))\\
\Phi(2) &=0
\end{split}

\begin{split}
\Phi(0) = 4
\end{split}

---

### Head, Head
_state 0:
state 1: Head
state 2: Head, Head_

|                |     State 0    |     State 1    |     State 2    |
|----------------|----------------|----------------|----------------|
|     State 0    |     0.5        |     0.5        |     0          |
|     State 1    |     0.5        |     0          |     0.5        |
|     State 2    |     0          |     0          |     1          |

\begin{split}
\Phi(0) &= 0.5 \times (1+\Phi(0))+ 0.5 \times (1+\Phi(1))\\
\Phi(1) &= 0.5 \times (1+\Phi(0))+ 0.5 \times (1+\Phi(2))\\
\Phi(2) &=0
\end{split}

\begin{split}
\Phi(0) = 6
\end{split}

```python
import numpy as np

def flip_coin(p=0.5):
    """ Simulate a coin flipping of a given probability p to get Head.
    Parameters:
        p: Parameter of the distribution, >= 0 and <=1.
    Returns:
        "H" or "T" (string)
    """
    return "H" if np.random.binomial(1, p, 1) == 1 else "T"

def flip_pattern(end_pattern=['H', 'T'], print_opt=False):
    """ One experiment of given ending pattern
    Parameters:
        end_pattern: Pattern to end simulation (list)
        print_opt: Option to print flipping sequence (boolean)
    Returns:
        current_index: number of flips before stop (int)
    """
    flip_list, current_index, current_condition= [], 0, None
    while current_condition != end_pattern:
        flip_list.append(flip_coin())
        if len(flip_list) >= len(end_pattern):
            current_condition = flip_list[-len(end_pattern):]
        else:
            pass
        current_index +=1
    if print_opt:
        print(flip_list)
    return current_index
```

```python
[flip_pattern(['H', 'H'], print_opt=True) for _ in range(10)]
```

    ['H', 'H']
    ['H', 'T', 'T', 'T', 'H', 'H']
    ['H', 'T', 'T', 'T', 'H', 'T', 'H', 'T', 'T', 'H', 'H']
    ['T', 'T', 'H', 'T', 'H', 'H']
    ['H', 'H']
    ['T', 'H', 'T', 'T', 'T', 'H', 'T', 'T', 'T', 'T', 'H', 'T', 'T', 'T', 'T', 'H', 'T', 'H', 'T', 'T', 'T', 'H', 'T', 'T', 'T', 'H', 'T', 'T', 'H', 'T', 'H', 'T', 'H', 'H']
    ['T', 'H', 'H']
    ['T', 'T', 'T', 'H', 'H']
    ['H', 'T', 'T', 'T', 'T', 'H', 'H']
    ['T', 'H', 'H']

    [2, 6, 11, 6, 2, 34, 3, 5, 7, 3]

```python
np.mean([flip_pattern(['H', 'T']) for _ in range(10000)])
```

    3.9706

```python
np.mean([flip_pattern(['H', 'H']) for _ in range(10000)])
```

    5.9878

```python
# import seaborn as sns
# sns.set()
# plt.figure(figsize=(16,4))
# sns.distplot([flip_pattern(['H', 'H']) for _ in range(10000)]);
```

```python
np.mean([flip_pattern(['H', 'H', 'H']) for _ in range(10000)])
```

    14.0748

```python
# plt.figure(figsize=(16,4))
# sns.distplot([flip_pattern(['H', 'H', 'H']) for _ in range(10000)]);
```

```python
np.mean([flip_pattern(['H', 'H', 'H', 'H', 'H']) for _ in range(10000)])
```

    61.3854

## Expected number of coin flips to get N Heads in a row:

# $$ 2 \times (2^N-1) $$

```python

```

## Find magic number N in range(1,1000000) that meets the criteria:
## N, 2xN, 3xN, 4xN, 5xN and 6xN have the same digits.

```
N  = abcde
2xN = bcaed
3xN = cdaeb
...

```

```python
from collections import Counter
def findNum(rng):
    """
    Find magic number N in range(1,1000000) that meets the criteria:
    N, 2xN, 3xN, 4xN, 5xN and 6xN have the same digits.
    """
    num_list = []
    for i in range(1,rng+1):
        if Counter(str(i)) == Counter(str(2*i)) \
        and Counter(str(2*i)) == Counter(str(3*i))\
        and Counter(str(3*i)) == Counter(str(4*i))\
        and Counter(str(4*i)) == Counter(str(5*i))\
        and Counter(str(5*i)) == Counter(str(6*i)):
            num_list.append(i)
    return num_list
```

```python
rng = 1000000
nums = findNum(rng)
nums
```

    [142857]

```python
# range(1:1,000,000)
[(num,num*2,num*3,num*4,num*5,num*6) for num in nums]
```

    [(142857, 285714, 428571, 571428, 714285, 857142)]

```python
# range(1:10,000,000)
# [(num,num*2,num*3,num*4,num*5,num*6) for num in nums]
```

    [(142857, 285714, 428571, 571428, 714285, 857142),
     (1428570, 2857140, 4285710, 5714280, 7142850, 8571420),
     (1429857, 2859714, 4289571, 5719428, 7149285, 8579142)]

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python
def bisearch(alist, item):
    first =0
    last =len(alist)-1
    found = False
    while first <= last and not found:
        midpoint =(first+last)//2
        if alist[midpoint] == item:
            found = True
            return midpoint+1
        else:
            if alist[midpoint] > item:
                last = midpoint-1
            else:
                first = midpoint+1
    return found
```

```python
L = [2,3,4,5,6,7,8,9,10,11,12,14,15,16]
num = 13
bisearch(L,num)
```

    False

```python
def searchList(alist, item):
    pos = 0
    found =False
    while pos<len(alist) and not found:
        if alist[pos] == item:
            found = True
        else:
            pos +=1
    return found
```

```python
L = [4,2,5,34,30,7,9,6,22]
num = 30
searchList(L,num)
```

    True

```python

```

```python
import turtle
import tkinter as TK
```

```python
t=turtle.Turtle()
```

```python
t.pencolor('red')
t.pensize(5)
```

```python
for i in range(5):
    t.forward(100)
    t.right(144)

t.hideturtle()
turtle.done()
```

```python
turtle.clearscreen()
```

```python
TK.mainloop()
```

```python
t1 = turtle.Turtle()
t1.pencolor('blue')
t1.pensize(5)
```

```python
def drawSpiral(t1, lineLen):
    if lineLen > 0:
        t.forward(lineLen)
        t.right(90)
        drawSpiral(t1, lineLen-5)

```

```python
drawSpiral(t1,100)
```

```python

```

```python
def tell_story(n):
    print("let me tell you a story:")
    while n>0:
        print("hahahahaha.....")
        n -=1
        tell_story()

```

```python

```

```python
from scipy import stats
priorR,priorG,priorB = (1,1),(1,1),(1,1)
```

```python
red_rv = stats.beta.rvs(priorR[0],priorR[1])
green_rv = stats.beta.rvs(priorG[0],priorG[1])
blue_rv = stats.beta.rvs(priorB[0],priorB[1])
```

```python
red_rv,green_rv,blue_rv
```

    (0.7794054386123787, 0.6482455141493157, 0.9204730278310114)

```python
words1 = ["hello","leetcode"]
words2 = ["word","world","row"]
order = "hlabcdefgijkmnopqrstuvwxyz"
```

```python
def isAlienSorted(words, order):
    dict1 = {}
    for idx, itm in enumerate(order):
        dict1[itm] = idx
    for i in range(len(words)-1):
        cur = words[i]
        nxt = words[i+1]
        for j in range(min(len(cur),len(nxt))):
            if cur[j] != nxt[j]:
                if dict1[cur[j]] > dict1[nxt[j]]:
                    return False
#                 break
            else:
                if len(cur) > len(nxt):
                    return False
            break
    return True

```

```python
isAlienSorted(words1, order)
```

    True

```python
isAlienSorted(words2, order)
```

    False

```python
import random
def simu_pi(N):
    inside = 0
    for i in range(N):
        x = random.uniform(0,1)
        y = random.uniform(0,1)
        if (x**2+y**2)<1:
            inside +=1
    pi = 4*inside/N
    return pi
```

```python
simu_pi(10000000)
```

    3.141764

```python
def isAlien(self,words, order):
    dict1 ={}
    for idx, itm in enumerate(words):
        dict[itm] = idx
    for i in range(len(words)-1):
        cur = words[i]
        nxt = words[i+1]
        for j in range(min(len(cur), len(nxt))):
            if cur[j] != nxt[j]:
                if dict[cur[j]] > dict1[nxt[j]]:
                    return False
                break
            else:
                if len(cur) > len(nxt)
                return False
    return True

```


# Characterization of Discrete Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Eigenfunctions

An [eigenfunction](https://en.wikipedia.org/wiki/Eigenfunction) of a discrete system is defined as the input signal $x[k]$ which produces the output signal $y[k] = \mathcal{H}\{ x[k] \} = \lambda \cdot x[k]$ with $\lambda \in \mathbb{C}$. The weight $\lambda$ associated with $x[k]$ is known as scalar eigenvalue of the system. Hence besides a weighting factor, an eigenfunction is not modified by passing through the system.

[Complex exponential signals](../discrete_signals/standard_signals.ipynb#Complex-Exponential-Signal) $z^k$ with $z \in \mathbb{C}$ are eigenfunctions of discrete linear time-invariant (LTI) systems. Let's assume a generic LTI system with input signal $x[k] = z^k$ and output signal $y[k] = \mathcal{H}\{ x[k] \}$. Due to the time-invariance of the system, the response to a shifted input signal $x(k-\kappa) = z^{k - \kappa}$ reads

\begin{equation}
y[k- \kappa] = \mathcal{H}\{ x[k - \kappa] \} = \mathcal{H}\{ z^{- \kappa} \cdot z^k  \}
\end{equation}

Due to the linearity of the system this can be reformulated as

\begin{equation}
y[k- \kappa] =  z^{- \kappa} \cdot \mathcal{H}\{ z^k  \} = z^{- \kappa} \cdot y[k]
\end{equation}

If the complex exponential signal $z^k$ is an eigenfunction of the LTI system, the output
signal is a weighted exponential signal $y[k] = \lambda \cdot z^k$. Introducing $y[k]$ into the left- and right-hand side of above equation yields

\begin{equation}
\lambda z^k z^{- \kappa} = z^{- \kappa} \lambda z^k
\end{equation}

which obviously is fulfilled. This proves that the exponential signal $z^k$ is an eigenfunction of LTI systems.

**Example**

The output signal of the previously introduced [second-order recursive LTI system](difference_equation.ipynb#Second-Order-System) with the difference equation

\begin{equation}
y[k] - y[k-1] + \frac{1}{2} y[k-2] = x[k]
\end{equation}

is computed for a complex exponential signal $x[k] = z^k$ at the input. The output signal should be a weighted complex exponential due to above reasoning.

```python
%matplotlib inline
import numpy as np
from scipy import signal
import matplotlib.pyplot as plt

a = [1.0, -1.0, 1/2]
b = [1.0]
z = np.exp(0.02 + .5j)

k = np.arange(30)
x = z**k
y = signal.lfilter(b, a, x)
```

The real and imaginary part of the input and output signal is plotted.

```python
plt.figure(figsize=(10,8))

plt.subplot(221)
plt.stem(k, np.real(x))
plt.xlabel('$k$')
plt.ylabel(r'$\Re \{ x[k] \}$')

plt.subplot(222)
plt.stem(k, np.imag(x))
plt.xlabel('$k$')
plt.ylabel(r'$\Im \{ x[k] \}$')
plt.tight_layout()

plt.subplot(223)
plt.stem(k, np.real(y))
plt.xlabel('$k$')
plt.ylabel(r'$\Re \{ y[k] \}$')

plt.subplot(224)
plt.stem(k, np.imag(y))
plt.xlabel('$k$')
plt.ylabel(r'$\Im \{ y[k] \}$')
plt.tight_layout()
```

**Exercise**

* From the in- and output signal only, can we conclude that the system is LTI?

## Transfer Function

The complex eigenvalue $\lambda$ characterizes the properties of the transfer of a complex exponential signal $z^k$ with [complex frequency $z$](../discrete_signals/standard_signals.ipynb#Complex-Exponential-Signal) through a discrete LTI system. It is commonly termed as [*transfer function*](https://en.wikipedia.org/wiki/Transfer_function) and denoted by $H(z)=\lambda(z)$. Using this definition, the output signal $y[k]$ of an LTI system with complex exponential signal at the input reads

\begin{equation}
y[k] = \mathcal{H} \{ z^k \} = H(z) \cdot z^k
\end{equation}

Note that the concept of the transfer function is directly linked to the linearity and time-invariance of a system. Only in this case, complex exponential signals are eigenfunctions of the system and $H(z)$ describes the properties of an LTI system with respect to these.

Above equation can be rewritten in terms of the magnitude $| H(z) |$ and phase $\varphi(z)$ of the complex transfer function $H(z)$

\begin{equation}
y[k] = | H(z) | \cdot z^k = | H(z) | \cdot e^{\Sigma k + j \Omega k + j \varphi(z)}
\end{equation}

where $z = e^{\Sigma + j \Omega}$ has been substituted to derive the last equality. The magnitude $| H(z) |$ provides the frequency dependent attenuation of the eigenfunction $z^k$ by the system, while $\varphi(z)$ provides the phase-shift introduced by the system.

## Link between Transfer Function and Impulse Response

In order to establish a link between the transfer function $H(z)$ and the impulse response $h[k]$, the output signal $y[k] = \mathcal{H} \{ x[k] \}$ of an LTI system with input signal $x[k]$ is computed. It is given by convolving the input signal with the impulse response

\begin{equation}
y[k] = x[k] * h[k] = \sum_{\kappa = -\infty}^{\infty} x[k-\kappa] \cdot h[\kappa]
\end{equation}

For a complex exponential signal as input $x[k] = z^k$ the output of the LTI system is given as $y[k] = \mathcal{H} \{ z^k \} = H(z) \cdot z^k$. Introducing both signals into the left- and right-hand side of the convolution yields

\begin{equation}
H(z) \cdot z^k = \sum_{\kappa = -\infty}^{\infty} z^k \, z^{- \kappa} \cdot h[\kappa]
\end{equation}

which after canceling out $z^k$ results in

\begin{equation}
H(z) = \sum_{\kappa = -\infty}^{\infty} h[\kappa] \cdot z^{- \kappa}
\end{equation}

The transfer function $H(z)$ can be computed from the impulse response by summing over the impulse response $h[k]$  multiplied with the complex exponential function $z^k$. This constitutes a transformation, which is later introduced in more detail as [$z$-transform](https://en.wikipedia.org/wiki/Z-transform).

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.