# Skeletonization using Fuzzy Logic

## Introduction
This demo shows how our skeletonization scheme works using fuzzy logic for pruning. For more information please read our papers. If you use this code or part of it for your research please respectfully cite these publications:

[1] Kamani, M.M., Farhat, F., Wistar, S. and Wang, J.Z., 2017. Skeleton Matching with Applications in Severe Weather Detection, Applied Soft Computing. [Link1](http://www.sciencedirect.com/science/article/pii/S1568494617303009) [Link2](https://pdfs.semanticscholar.org/a90c/ffa078ed566a1cb046ec68fd7195f01c7a76.pdf)

[2] Kamani, M.M., Farhat, F., Wistar, S. and Wang, J.Z., 2016, December. Shape matching using skeleton context for automated bow echo detection. In Big Data (Big Data), 2016 IEEE International Conference on (pp. 901-908). IEEE. [Link1](http://ieeexplore.ieee.org/abstract/document/7840685/) [Link2](https://pdfs.semanticscholar.org/ae62/d5802980859ead8288f6ff436754253882d6.pdf)

```python
import cv2
import numpy as np
from matplotlib import pyplot as plt
from fuzzyTransform import fuzzyTransform
from skeleton2Graph import *
import scipy.ndimage.morphology as morph
from skimage.morphology import thin
import matplotlib.patches
```

```python
BW = mat2gray(cv2.imread('BW.png', cv2.IMREAD_GRAYSCALE).astype('float64'))
plt.imshow(BW,cmap='gray')
plt.title('Initial Binary Image')
plt.show()
```

### Skeletonization
Skeleton of a shape is a low-level representation that can be used for matching and recognition purposes in various fields of study including image retrieval and shape matching or human pose estimation and recovery. Skeleton can provide a good abstraction of a shape, which contains topological structure and its features. Because it is the simplest representation of a shape, there has been an extensive effort among researchers to develop generic algorithms for skeletonization of shapes. However, since there is no “true” skeleton defined for an object, the literature in skeletonization lack of a robust evaluation. The vast majority of the algorithms are based on Blum’s “Grassfire” analogy and formulation for skeletonization. The most important key factor in skeletonization algorithms is to preserve the topology of the shape. One of the most widely used algorithms is based on measuring the net outward flux by using Euclidean Distance Transform (EDT) of the binary image followed by a topology preserving thinning algorithm. We calculate the net outward flux per unit area and detect the location of the pixels where conservation of energy principle is violated. EDT maps a binary image into a gray level image with value of each pixel represents its euclidean distance to the border of image. Given Euclidean distance of an image ($D_E$), we should first compute the gradient vector field ($\nabla D_E$), and then the divergence of this vector field [19]. Mathematically, the divergence of the gradient vector field ($\nabla .
(\nabla D_E)$) is defined as the limit of the net outward flow of the field across the boundary of the area around the given point, while the area is shrinking to zero:

\begin{equation*}
\nabla \cdot (\nabla D_E) = \displaystyle{\lim_{S \to 0}} \iint_{C}\frac{\nabla D_E\cdot \vec{n}}{S} dC\;
\end{equation*}

where $C$ is the boundary, $S$ is the area, and $\vec{n}$ is the normal vector of the boundary. Hence, we can calculate net outward flux at each point $P = (x, y)$ as follow:

\begin{equation}
Flux(P) = \sum_{i=1}^{8}\nabla D_E(Q_i) \cdot \vec{n}\;,
\end{equation}

where $Q_i$'s are neighbor points to point $P$. According to the direction of the normal vector, we can determine that positive or negative flux values are representing drain or source of energy, where energy-draining points are internal skeletal points, and energy-generating points are external skeletal points. In the following cell we calculate the EDT of the binary image and its gradient.

```python
BW = 1 - BW
M,N = BW.shape

'''Calculating Euclidean Distance of the Binary Image'''
D,IDX = morph.distance_transform_edt(BW,return_distances=True, return_indices=True)
D = mat2gray(D)
X,Y = np.meshgrid(range(N),range(M))
delD_x = -(IDX[1,:,:] - X)
delD_y = -(IDX[0,:,:] - Y)
# normalize the derivatives
delD_norm = np.sqrt(pow(delD_x,2) + pow(delD_y,2))
with np.errstate(divide='ignore',invalid='ignore'):
    delD_xn   = delD_x / delD_norm
    delD_yn   = delD_y / delD_norm

mir_delD_xn  = mirrorBW(delD_xn)
mir_delD_yn  = mirrorBW(delD_yn)

f, (ax1, ax2,ax3) = plt.subplots(1,3,figsize=(10,30))
ax1.imshow(D)
ax1.set_title('Euclidean Distance Transform')
ax2.imshow(delD_x)
ax2.set_title('X direction of the gradient of EDT')
ax3.imshow(delD_y)
ax3.set_title('Y direction of the gradient of EDT')
plt.tight_layout()
plt.show()
```

### Flux Map
Now using the formula introduced above, we can caculate the flux map of the image.

```python
#Calculate flux map
fluxMap = flux(mir_delD_xn,mir_delD_yn)
plt.imshow(np.nan_to_num(fluxMap))
plt.title('Flux Map')
plt.show()

```

By setting a threshold on flux values, initial binary skeleton map can be computed. Because we are dealing with highly boundary-variant shapes, the skeleton map would contain a large number of unwanted branches that make the subsequent matching steps complicated. We need to develop an automated method to remove all such branches while keeping the main skeleton intact. We introduce a new method to prune the skeleton using fuzzy logic, which will be developed in the next cell.

```python
#Calculate flux map
fluxMap = flux(mir_delD_xn,mir_delD_yn)
# Calculate flux threshold
fluxBWThreshold = (np.nanmax(fluxMap) - np.nanmean(fluxMap)) * 0.15 + np.nanmean(fluxMap)
with np.errstate(divide='ignore',invalid='ignore'):
    fluxThin = thin(fluxMap>fluxBWThreshold)

fluxLabeled,b = ndimage.label(fluxThin, np.array([[1,1,1], [1,1,1], [1,1,1]]))
labels ,pixelSize = np.unique(fluxLabeled,return_counts=True)
# Excluding the background
pixelSize = pixelSize[labels != 0]
labels = labels[labels != 0]
# Calculating the size threshold and filter out small objects
th = min(np.mean(pixelSize) + 3 * np.std(pixelSize), np.max(pixelSize))
selectedObjects = labels[np.where(pixelSize >= th)]

fluxTemp = np.zeros(fluxMap.shape)
fluxTemp[fluxLabeled == 1] = 1
plt.imshow(fluxTemp,cmap='gray')
plt.title('Initial Skeleton with branches')
plt.tight_layout()
plt.show()
```

### Fuzzy logic skeleton pruning
Our pruning algorithm needs to have the complete graph information of the skeleton including its vertices and edges' pixels coordinates. Therefore, the skeleton map is converted to a graph before the pruning step. Converting a skeleton map to a graph consists of two steps, namely extracting vertices and then points of each edge. We will go through each step as follow:

1. **Extracting Vertices**: For finding the vertices of the skeleton graph we need to scan the whole pixels of the image and based on the structure of other pixels around each pixel, we could decide weather it is a vertex or not. In this regard, we extract a $3\times3$ matrix around each pixel, which consists of its 8-connected neighbors. Then, in this matrix we build a local graph having pixels with value of 1 as its vertices. The connectivity of this graph is 4-connected neighbors, which means that 2 vertices are connected if and only if they are in one of the 4 main directions of each other (left, right, up, down). After that, in this graph we could calculate the Euler characteristic using its number of vertices and edges. If the Euler characteristic of the graph is greater than 2, it shows that this point is on the crossing of three or more edges, hence, it is a vertex with a branch type. If it is 1, it means that this point is an end point vertex of an edge. When it equals 2, the point is a simple edge point and not a vertex. Vertices extracted in this way for the skeleton is are depicted in output of the cell below (with a magnified part for a better representation). Red points in the image are branch points, and green ones are end points.
2. **Extracting Edge Points**: After finding vertices in the graph, we should form the edge list of the graph. We start with a random end point and traverse its neighbors to reach to a branch point or other endpoints. If the neighbor pixel is not a branch point nor an endpoint, it would be added to the current edge's pixel list. When we reach a branch point, we should add another edge to the edge list, having that branch point as its first point, and start searching edge points for the next edge in the edge list. If we reach an end point, we just start searching edge points for the next edge in the edge list. This approach will be continued until there is no edge left unprocessed. The result of finding edge points is also depicted in the output of the cell below.

```python
skeletonNew = np.zeros(fluxMap.shape)
adjacencyMatrix, edgeList, edgeProperties,edgeProperties2, verticesProperties, verticesProperties2, endPoints, branchPoints = skeleton2Graph(fluxTemp,fluxTemp*fluxMap)
vertices = np.concatenate((endPoints, branchPoints))
_,_,_, skeletonGraphPointsImg = findBranchPoints(fluxTemp,return_image=True)
f, (ax1, ax2,ax3) = plt.subplots(1,3,figsize=(10,30))
ax1.imshow(skeletonGraphPointsImg)
ax1.set_title('Vertices of the Skeleton\'s graph')
ax2.imshow(skeletonGraphPointsImg[50:100,125:170])
ax2.set_title('Vertices, close look')
ax3.imshow(graphDrawing(fluxTemp,edgeList,0.08))
ax3.set_title('Edges of the Skeleton\'s graph')

plt.tight_layout()
plt.show()
```

Having high sensitivity to border variations, almost all algorithms for skeletonization need to be followed by a pruning stage in order to remove thin branches caused by boundary deformations. These branches may significantly change the skeleton graph, and hence they should be treated carefully for the matter of topology preserving in skeletonization algorithms. Most of these algorithms use Boolean logic in their decision to remove or keep the branches. The output of these algorithms is a crisp value attributed to each edge distinguishing branch edges from main skeleton edges. However, if we ask a person to do pruning on a skeleton graph, he or she would extract the main skeleton with an uncertainty to some extent. On the other hand, fuzzy logic introduces many-valued logic in close proximity to human decision making system. Hence, we propose an approach based on fuzzy inference system to prune skeleton graph and extract the main skeleton.

In our method, we use the outward flux values of the pixels as an input to the fuzzy inference system. Heuristically from raw images of flux values, the higher the value of the outward flux in each pixel, the more probable that the pixel is in the main skeleton. Therefore, based on this observation we can extract a feature for every edge connected to a vertex in the skeleton graph. In order to record vertices properties in the flux images, we form an array, called $\Gamma$, for each vertex with the length of the number of the edges linked to it. The value attributed to each edge $e_j$ connected to the vertex $V_i$ could be computed as follow:
\begin{equation}
\Gamma\{V_i,e_j\} =\frac{1}{M_j-1} \sum_{P_j = 2}^{M_j}(W_G(P_j)\cdot Flux(P_j))\;,
\end{equation}
in which, $j=1,...,N_i$ indicating the $j^{th}$ edge connected to $V_i$. $N_i$ is the number of edges linked to $V_i$, $M_j$ is the number of pixels in the $j^{th}$ edge, $P_j$ is the index of pixels on the $j^{th}$ edge, starting from the vertex $V_i$, and $W_G(P_j)$ is the Gaussian weight for each pixel of edge $e_j$ computed as follow:
\begin{equation}
W_G(P_j) = \exp{\left(-\frac{{||P_j - V_i||^2}}{2 \sigma^2}\right)}\;.
\end{equation}

Our proposed fuzzy inference system (FIS) consists of two components:
1. FIS-1: Fuzzy inference system to compute degree of belief of each pixel to main skeleton edges.
2. FIS-2: Fuzzy inference system to compute degree of belief of each pixel to branch edges.

The FIS-1 output, indicates that to what extent we believe an edge belongs to main skeleton. Afterwards, we use this value as an input to FIS-2 to compute the extent to which that we believe an edge belongs to branch edges. These values are the same for the pixels of the edge and varies among different edges. In following subsections we introduce these two fuzzy inference systems, their inputs, rules, and outputs. And then we go through the details of our algorithm for pruning skeleton.

The input and output of the fuzzy inference systems are depicted here. For more information about these fuzzy inference systems, please refer to the papers listed in the first cell.

#### Main Skeleton Fuzzy Inference System

#### Branch Fuzzy Inference System

```python

skeletonNew, MSDB,BDB = fuzzyTransform(fluxTemp, vertices, edgeList, edgeProperties, verticesProperties, verticesProperties2, adjacencyMatrix, returnDB=True)
f, (ax1, ax2,ax3) = plt.subplots(1,3,figsize=(10,30))
ax1.imshow(MSDB,cmap ='gist_gray')
ax1.set_title('Main Skeleton Degree of Belief Map')
ax2.imshow(BDB,cmap ='gist_gray')
ax2.set_title('Branch Degree of Belief Map')
ax3.imshow(skeletonNew, cmap='gray')
ax3.set_title('Pruned Skeleton')
plt.tight_layout()
plt.show()
```


# Speed of symbolic computation of stability polynomials

Symbolic computation of stability polynomials and internal stability polynomials in SymPy can get very slow for methods with many stages.  The stability polynomial can be computed in a variety of ways, and different ways seem to work better for different classes of RK methdods.  Let's investigate.

```python
%pylab inline
import sympy
from nodepy import rk
import time
import matplotlib.pyplot as plt
import numpy as np
```

Next we have 3 functions for computing the stability polynomial in different ways.  The first uses the matrix-inverse formula, the second uses the ratio of determinants, and the third uses a power series for the matrix inverse.  They are implemented for both the Shu-Osher and Butcher forms.

Note that the ratio of determinants in the Shu-Osher form is still not implemented correcly.

Also note that for implicit methods the ratio of determinants is the preferred method, since it gives formulas for the numerator and denominator.  Here we focus on explicit methods, for which the denominator is 1.

```python
def lower_triangular_solve(rk,SO=False): # Using lower_triangular_solve, which is always faster than LUsolve
    start = time.perf_counter()
    z = sympy.var('z')
    I = sympy.eye(len(rk))
    if SO:
        v = 1 - rk.alpha.sum(1)
        vstar = sympy.Matrix(v[:-1])
        v_mp1 = sympy.Rational(v[-1])
        alphastar=sympy.Matrix(rk.alpha[:-1,:])
        betastar=sympy.Matrix(rk.beta[:-1,:])
        alpha_mp1 = sympy.Matrix(rk.alpha[-1,:])
        beta_mp1 = sympy.Matrix(rk.beta[-1,:])
        p1=(alpha_mp1 + z*beta_mp1).transpose()*(I-alphastar-z*betastar).lower_triangular_solve(vstar)
        p1=p1[0].expand()+v_mp1
    else:
        Asym=sympy.Matrix(rk.A)
        bsym=sympy.Matrix(rk.b)
        e = sympy.ones(len(rk),1)
        p1=z*bsym.transpose()*(I-z*Asym).lower_triangular_solve(e)
        p1=p1[0].expand()+1
    p1=p1.as_poly(z).all_coeffs()
    p1=p1[::-1]
    q1=[sympy.Rational(1)]
    p=np.poly1d(p1[::-1])    # Numerator
    q=np.poly1d(q1[::-1])    # Denominator
    t = time.perf_counter()-start
    return p,q,t

def determinants(rk,SO=False): #Using charpoly
    start = time.perf_counter()
    z=sympy.var('z')
    if SO:
        I = sympy.eye(len(rk))
        v = 1 - rk.alpha.sum(1)
        vstar = sympy.Matrix(v[:-1]).T
        v_mp1 = sympy.Rational(v[-1])
        alphastar=sympy.Matrix(rk.alpha[:-1,:])
        betastar=sympy.Matrix(rk.beta[:-1,:])
        alpha_mp1 = sympy.Matrix(rk.alpha[-1,:])
        beta_mp1 = sympy.Matrix(rk.beta[-1,:])
        xsym = I - alphastar - z*betastar + vstar/v_mp1 * (alpha_mp1+z*beta_mp1)
        p1=v_mp1*xsym.charpoly(z).coeffs()
    else:
        Asym=sympy.Matrix(rk.A)
        bsym=sympy.Matrix(np.tile(rk.b,(len(rk),1)))
        xsym=Asym-bsym
        p1=xsym.charpoly(z).coeffs()

    q1=[sympy.Rational(1)]
    p=np.poly1d(p1[::-1])    # Numerator
    q=np.poly1d(q1[::-1])    # Denominator
    t = time.perf_counter()-start
    return p,q,t

def series(rk,SO=False): # Using power series
    start = time.perf_counter()
    s = rk.num_seq_dep_stages()
    z=sympy.var('z')
    I = sympy.eye(len(rk))
    if SO:
        alphastarsym = sympy.Matrix(rk.alpha[0:-1,:])
        betastarsym  = sympy.Matrix(rk.beta[0:-1,:])

        matsym = alphastarsym + betastarsym*z
        vecsym = sympy.Matrix(rk.alpha[-1,:]+z*rk.beta[-1,:])

    else:
        Asym=sympy.Matrix(rk.A)
        bsym=sympy.Matrix(rk.b)
        matsym = z*sympy.Matrix(rk.A)
        vecsym = z*sympy.Matrix(rk.b)

    # Compute (I-zA)^(-1) = I + zA + (zA)^2 + ... + (zA)^(s-1)
    matpow = I
    matsum = I
    for i in range(1,s):
        matpow = matpow*matsym
        matsum = matsum + matpow
    p1 = vecsym.transpose()*matsum
    if SO:
        v = 1 - rk.alpha.sum(1)
        vstar = sympy.Matrix(v[:-1])
        v_mp1 = sympy.Rational(v[-1])
        p1 = p1*vstar
        p1=p1[0].expand()+v_mp1
    else:
        e = sympy.ones(len(rk),1)
        p1 = p1*e
        p1=p1[0].expand()+1

    p1=p1.as_poly(z).all_coeffs()
    p1=p1[::-1]
    q1=[sympy.Rational(1)]
    p=np.poly1d(p1[::-1])    # Numerator
    q=np.poly1d(q1[::-1])    # Denominator
    t = time.perf_counter()-start
    return p,q,t
```

The following function just runs each of the above approaches on a given family of methods and plots the run-times for each approach:

```python
def time_stability_polynomial_calculation(method,pmin=2,pmax=5,approaches=[lower_triangular_solve,determinants,series],SO=False):
    norders = pmax-pmin+1
    orders = range(pmin,pmax+1)

    times = []
    for j in range(len(approaches)):
        times.append(np.zeros((norders,1)))

    for i, order in enumerate(orders):
        myrk = method(order)
        for j, approach in enumerate(approaches):
            p,q,t = approach(myrk,SO=SO)
            times[j][i] = t

    for j in range(len(approaches)):
        plt.plot(orders,times[j],linewidth=3)

    leg_text = [approach.__name__ for approach in approaches]
    plt.legend(leg_text,loc='best')
    plt.ylabel('time (s)')
    plt.xlabel('order')
    plt.title('Computation of stability polynomial for '+method.__name__)
```

```python
time_stability_polynomial_calculation(rk.extrap,pmax=7)
```

```python
time_stability_polynomial_calculation(rk.extrap,pmax=7,SO=True,approaches=[lower_triangular_solve,series])
```

Clearly the `lower_triangular_solve` method is the fastest for extrapolation methods.  The other two methods will quickly reach unreasonable run-times if we increase $p$ further.  Using the Shu-Osher form is noticeably faster for all methods.

```python
time_stability_polynomial_calculation(rk.DC,pmax=5)
```

```python
time_stability_polynomial_calculation(rk.DC,pmax=5, )
```

For deferred correction methods, we see roughly the opposite behavior: `lower_triangular_solve` is extremely slow compared to either of the other two.  Series has a slight edge over charpoly.  Again, using the Shu-Osher form is significantly faster for some methods.

```python
time_stability_polynomial_calculation(rk.RKC1,pmax=12)
```

```python
time_stability_polynomial_calculation(rk.RKC1,pmax=12,SO=True,approaches=[lower_triangular_solve,series])
```

For RKC methods, things are complicated.  The `determinants` approach is fastest if using the Butcher form, but the other two approaches are reasonably fast if using the Shu-Osher form (I really need to implement the Shu-Osher ration of determinants!) (I should add that the x-axis label here is wrong; it should really be "stages")

```python
time_stability_polynomial_calculation(rk.SSPRK2,pmax=12)
```

```python
time_stability_polynomial_calculation(rk.SSPRK2,pmax=12,SO=True,approaches=[lower_triangular_solve,series])
```

SSPRK2 methods behave mostly like RKC methods, which is unsurprising since they have a very similar structure.  But they show much better results using `lower_triangular_solve` on the SO form.

So, we can implement all three options.  Which should be the default?  Let's try an "ordinary" Runge-Kutta method, not belonging to any of these families (but with a moderately large number of stages):

```python
approaches=[lower_triangular_solve,determinants,series]
SO=False
times = []
myrk = rk.loadRKM('PD8')
for j, approach in enumerate(approaches):
    p,q,t = approach(myrk,SO=SO)
    times.append(t)
list(zip([approach.__name__ for approach in approaches], times))
```

`lower_triangular_solve` is the clear loser.  So we'll go with `determinants` as the default if Butcher coefficients are used, `lower_triangular_solve` as the default if Shu-Osher arrays are used, but allow the user to pick any of the three.

For the implementation of `internal_stability_polynomials`, it seems not worthwhile to work out and implement a ratio of determinants formula, so we'll just provide the other two and default to Shu-Osher with `lower_triangular_solve`.


*Text provided under a Creative Commons Attribution license, CC-BY, code provided under an MIT license. (c) Aron J. Ahmadia, Lorena A. Barba, 2014. Thanks: Gilbert Forsyth and Olivier Mesnard, and NSF for support via CAREER award #1149784.*

# The Parabolic Particle Path

Adapted from Tutorial for HW #4 of GWU MAE 2117 by Chunlei Liang

Let us consider a particle sliding on a parabola which has a profile of:

\begin{align}
  y = x^2
\end{align}

The coordinates of the particle are shown in [Figure 1](#Figure-1:-Coordinates).

#### Figure 1: Coordinates

```
SVG("./Figure_1.svg")
```

The reference frames of the particle can be defined by an inertial reference frame $I = (O, \vec{e}_x, \vec{e}_y)$ and a local reference frame $B = (P, \vec{e}_t, \vec{e}_n)$ as shown in [Figure 2](#Figure-2:-Reference-Frame).

#### Figure 2: Reference Frame

```
SVG("./Figure_2.svg")
```

We can then draw a free-body diagram for this particle as depicted in [Figure 3](#Figure-3:-Free-body-diagram).

Figure 3: Free-body diagram

```
SVG("./Figure_3.svg")
```

The total force includes the gravitational force $-m_pg\vec{e}_y$ and the resultant force $N\vec{e}_n$.

The kinematics of the particle can be written as the following.

The position vector of the particle in the inertial reference frame is:

\begin{align}
    \vec{r} = x\vec{e}_x + y\vec{e}_y = x\vec{e}_x + x^2\vec{e}_y
\end{align}

The velocity vector of the particle is therefore:

\begin{align}
    \vec{v} = \dot{x}\vec{e}_x + \dot{y}\vec{e}_y = \dot{x}\vec{e}_x + 2x\dot{x}\vec{e}_y
\end{align}

The acceleration vector of the particle can be derived as:

\begin{align}
    \vec{a} = \ddot{x}\vec{e}_x + 2\left(\dot{x}^2 + x\ddot{x}\right)\vec{e}_y
\end{align}

Thus, according to Newton's Second Law, we have:

#### Equation 1

\begin{align}
    -m_pg\vec{e}_y + N\vec{e}_n = m_p\left\{{\ddot{x}\vec{e}_x + \left(\dot{x}^2 + x\ddot{x}\right)\vec{e}_y}\right\}
\end{align}

The tangent direction unit vector shown in Figure [2](#Figure-2:-Reference-Frame) can be related to the velocity vector through $\vec{e}_t = \frac{\vec{v}}{||\vec{v}||}$

One can compute the unit normal vector:

\begin{align}
    \vec{e_n} = \vec{e_z} \times \vec{e}_t = \frac{-2x\vec{e}_x + \vec{e}_y}{\sqrt{1+4x^2}}
\end{align}

Equation [(1)](#Equation-1) can now be split into two scalar equations:

#### Equation 2

\begin{align}
\frac{-2xN}{\sqrt{1+4x^2}} = m_p\ddot{x}
\end{align}

#### Equation 3

\begin{align}
    \frac{N}{\sqrt{1+4x^2}} -m_pg = m_p2(\dot{x}^2 + x\ddot{x})
\end{align}

Combine Equations [(2)](#Equation-2) and [(3)](#Equation-3) and eliminate N to obtain the equation of motion for the particle:

#### Equation 4

\begin{align}
\ddot{x} = \frac{-2x(g+2\dot{x}^2)}{1+4x^2}
\end{align}

Equation [(4)](#Equation-4) can be split into two first-order ODEs

#### Equation 5

\begin{align}
\dot{x} &= u
\end{align}

#### Equation 6

\begin{align}
\dot{u} &= \frac{-2x(g+2u^2)}{1+4x^2}
\end{align}

If we introduce the vector $\vec{q} = \left[\begin{array}{c}x\\u\end{array}\right] = \left[\begin{array}{c}q_0\\q_1\end{array}\right]$ we can write equations [(5)](#Equation-5) and [(6)](#Equation-1) in terms of $\vec{q}$.

#### Code Fragment 1

```
# As usual, we begin by importing the NumPy package
import numpy as np

# Then, we define the gravitational constant
g = 9.8

# Now, we define the derivative of our vector Q
def Q_Dot(Q):
    """Calculate the derivative of Q"""
    x, u = Q[0], Q[1]
    x_dot = u
    u_dot = -2*x*(g+2*u**2)/(1+4*x**2)
    return np.asarray([x_dot, u_dot])
```

Go ahead and execute the above cell fragment.  None of the three actions (importing a package, defining a variable, defining a function) produce any output, so IPython lets us know that the cell has executed by adding a number between the `In [X]` marker to the left of the cell.

*Recall that to execute a code fragment, we need to activate the cell by clicking on it, then either pressing the "Play" button, or holding down the Shift key and pressing Enter.*

Let's check our derivative function at a few initial conditions.  We'll do that with a simple test function that simply prints input and output values.  Once we've defined the test function, we can easily use it for a range of inputs without having to type the same print statements again and again.

#### Code Fragment 2

```
def check_Q_Dot(x_0, u_0):
    """Test function for Q_Dot"""
    print "initial velocity                 : %g" % x_0
    print "initial acceleration             : %g" % u_0
    x_dot, u_dot = Q_Dot([x_0, u_0])
    print "computed velocity derivative     : %g" % x_dot
    print "computed acceleration derivative : %g" % u_dot
```

First, if the parabola is at rest ($x=0, u=0$), we don't expect there to be any changes in velocity or acceleration.

```
check_Q_Dot(0.0, 0.0)
```

    initial velocity                 : 0
    initial acceleration             : 0
    computed velocity derivative     : 0
    computed acceleration derivative : -0

Good!  Note that the acceleration derivative is zero, but has a negative sign caused by the slight rounding in our computations.  Let's slightly improve our test.  The computed velocity derivative should always be equal to the current acceleration, so let's make it an error when it doesn't!

#### Code Fragment 3

```
def check_Q_Dot(x_0, u_0):
    """Test function for Q_Dot"""
    print "initial velocity                 : %g" % x_0
    print "initial acceleration             : %g" % u_0
    x_dot, u_dot = Q_Dot([x_0, u_0])
    print "computed velocity derivative     : %g" % x_dot
    print "computed acceleration derivative : %g" % u_dot
    assert(x_dot == u_0)

check_Q_Dot(0.0, 2.0)
```

    initial velocity                 : 0
    initial acceleration             : 2
    computed velocity derivative     : 2
    computed acceleration derivative : -0

#### Numerical Checkpoint 1

Use equations [(5)](#Equation-5) and [(6)](#Equation-1) to calculate $\dot{x}$ and $\dot{u}$, given $x_0=1$ and $u_0=1$.  Check your work using the code cell below.

```
#check_Q_Dot( , )
```

## Euler's Method

Now that we have our transformed our second-order scalar ODE into a system of first-order ODEs, we can solve this system using a numerical integrator.

The simplest integrator is Euler's method.  Recall that if we know that:

\begin{align}
\dot{y}(t) &= f(t,y(t)) \\
y(t_0) &= y_0
\end{align}

then we can solve for $y$ at a set of discrete time steps of size $h$, such that $t_n = t_0 + n*h$.  Euler's method relies on the recurrence relation:

\begin{align}
y_{n+1} = y_n + hf(t_n,y_n)
\end{align}

#### Code Fragment 4

```
def euler_step(f, y_n, n, h):
    """Take a single step using Euler's method.

    y_n - The current value of the vector y at t=n
    f   - function with with calling form f(y,t), computes the derivative of y at t
    """

    y_dot = f(y_n, n)
    y_n_plus_1 = y_n + h*y_dot
    return y_n_plus_1
```

As always, we verify with a test example.  Let's assume we are observing simple particle motion, with initial position $y(0)=0$ and constant velocity $\dot{y}(t) = 10$.  If we take a time step $h=0.5$, we expect $y(0.5) = 5$.

#### Code Fragment 5

```
def constant_velocity_10(y_n, n):
    """Derivative of particle's position that is at a constant velocity of 10"""
    return 10
euler_step(constant_velocity_10, 0, 0, 0.5)
```

    5.0

We could have also used a [`lambda` statement](http://docs.python.org/2/tutorial/controlflow.html#lambda-expressions) here to define the `constant_velocity_10` function, but function definitions in Python are so easy that we just created another one!

## Numerically Integrating

Now that we have a way to take a single time step, we'd like to be able to take several of them in order to calculate $y$ at some final time, $t_{final}$.  To do that, we'll need to use a for loop, once for each time step we'd like to take.  More sophisticated numerical integration techniques take time steps of varying length, but for this exercise we'll take $n=\frac{t_{final}-t_0}{h}$ Euler steps.

#### Code Fragment 6

```
def simple_integrator(f, y_0, t, h):
    """Compute trajectory of values of y using Euler's method

    f       - Computes the derivative of y at t, with calling form f(y,t)
    y_0     - Initial value of vector y
    t       - t_0, t_final (distance should be a multiple of h)
    h       - Requested distance between time steps
    """

    t_0, t_final = t
    # n - number of steps to take
    n = int((t_final-t_0)/h)

    # d - dimension of the problem, equal to number of values in vector y
    d = len(y_0)

    # y_traj - trajectory of y at each step i, represented as [i,:]
    # The trajectory stores y at every time, including the initial time t_0.
    # y_traj will store the output of $n$ steps + 1 for the initial value.

    y_traj = np.zeros((n+1,d))

    # t  - complete set of values of t at each step i
    t = np.linspace(t_0, t_final, n+1)

    y_traj[0,:] = y_0

    for i in range(n):
        y_traj[i+1,:] = euler_step(f, y_traj[i,:], t[i], h)
    return t, y_traj
```

Whew, that might seem like a lot of code, but the majority of the code fragment is comments, and as long as you're comfortable with for loops in Python and using `np.zeros` to create empty arrays and `np.linspace` to create sets of equally spaced points, this function should make sense.

Let's test it out, using a visual inspection this time.  Again, the following two lines of code tell IPython that we want to view our output figures in the Notebook and that we'll be using `plt` as shorthand for `matplotlib.pyplot`.

#### Code Fragment 7

```
%matplotlib inline
import matplotlib.pyplot as plt
```

One final trick.  Our general integrator calls functions that might do something with the current time (that is, they are functions of $y(t)$ *and* $t$).  Since our `Q_Dot` only relies on $y$, we need to create an adapter function around `Q_Dot` that acts like it will use $t$, but really doesn't.  Here's how to do that:

#### Code Fragment 8

```
def Q_Dot_t(y, t):
    """Adapter function to allow us to use general integrator with Q_Dot"""
    return Q_Dot(y)
```

The sign of well-designed code is that all of the complexity is hidden from the rest of the program.  Here's our call to `simple_integrator` for a test problem with initial velocity of 0.5 and initial acceleration of 0.5 from time 0 to 3, with time steps of size 0.001.

#### Code Fragment 9

```
t, Q_Traj = simple_integrator(Q_Dot_t, [0.5, 0.5], [0, 3], 0.001)

plt.plot(t, Q_Traj[:,0]);
```

## Instructor Notes / Code

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
/*
    div.input{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    div.output{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
*/
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    h2 {
        font-family: 'Fenix', serif;
    }
    h3{
		font-family: 'Fenix', serif;
        margin-top:12px;
        margin-bottom: 3px;
       }
	h4{
		font-family: 'Fenix', serif;
       }
    h5 {
        font-family: 'Alegreya Sans', sans-serif;
    }
    div.text_cell_render{
        font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 135%;
        font-size: 120%;
        width:600px;
        margin-left:auto;
        margin-right:auto;
    }
/*
    div.input_area{
        width:600px;
	margin-left:0
    }
*/
    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/
    .text_cell_render h1 {
        font-weight: 200;
        font-size: 50pt;
		line-height: 100%;
        color:#CD2305;
        margin-bottom: 0.5em;
        margin-top: 0.5em;
        display: block;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #CD2305;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```
from IPython.core.display import SVG
```

If needed, you can modify any of the Figures used in this IPython Notebook by modifying the Asymptote code below.  This code is not distributed to students.

```
%install_ext http://raw.github.com/jrjohansson/ipython-asymptote/master/asymptote.py
```

    Installed asymptote.py. To use it, type:
      %load_ext asymptote

```
%load_ext asymptote
```

```
%%asymptote Figure_1

defaultpen(1.5);
import graph;
size(200,200);

real f(real x) {return x^2;}
pair F(real x) {return (x,f(x));}

xaxis();
yaxis();

draw(graph(f,-2.5,2.5,operator ..));

pair zx=(2,0);
pair zy=(0,4);
pair P=(2,4);

dot("$P$",P);
draw(zx--P,red+dashed);
draw(zy--P,red+dashed);
label("$x$",(0.5*zx.x,4),N);
label("$y$",(2,0.5*zy.y),E);
```

```
%%asymptote Figure_2

defaultpen(1.5);
import graph;
size(200,200);

real f(real x) {return x^2;}
pair F(real x) {return (x,f(x));}

xaxis("$\vec{e}_x$");
yaxis("$\vec{e}_y$");

draw(graph(f,-2.5,2.5,operator ..));

pair P=(2,4);
pair et=P+0.5*(1,4);
pair en=P+0.5*(-4,1);

dot("$P$",P);
draw(P--et,blue,EndArrow(size=10));
label("$\vec{e}_t$", et, SE, blue);
draw(P--en,blue,EndArrow(size=10));
label("$\vec{e}_n$", en, 3*E, blue);
```

```
%%asymptote Figure_3

defaultpen(1.5);
import graph;
size(200,200);

real f(real x) {return x^2;}
pair F(real x) {return (x,f(x));}

draw(graph(f,-2.5,2.5,operator ..));

pair P=(2,4);
pair eg=P+0.5*sqrt(17)*(0,-1);
pair en=P+0.5*(-4,1);

dot("$P$",P);
draw(P--eg,blue,EndArrow(size=10));
label("$-m_pg\vec{e}_y$", eg, SE, blue);
draw(P--en,blue,EndArrow(size=10));
label("$N\vec{e}_n$", en, 3*E, blue);
```


# Bayesian Model Example - The Deregulation Act

Basic idea: have taxi- and non-taxi-related crime (i.e. what Jacob is doing) diverged after the deregulation act?

Method:
 - Calculate the difference in proportions between the two datasets.
 - Model the change in proportion following the text messaging example from Probabilistic Programming book. I.e.:
   - Model the difference using a Poisson distribution (defined by $\lambda$ parameter)
   - Assume that two distributions exist: one before the Act, and one after.
   - Define $\tau$ as the point at which they switch (i.e. when the Act came into force).
   - Create two $\lambda$ parameters - one before $\tau$ and one after.
   - See what the posterior distributions of $\lambda_1$, $\lambda_2$, and $\tau$ are. Do they correspond with the introduction of the Act?

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats as stats
from IPython.core.pylabtools import figsize

import pymc3 as pm
import theano.tensor as tt
```

    /Users/nick/anaconda3/envs/py36/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

## Data

Prepare the data

```python
data = pd.read_csv("./data/taxi_crime.csv")

date = np.array(data.loc[:,"Month"])
taxi_crime = np.array(data.loc[:,"Taxi Crime"])
all_crime = np.array(data.loc[:,"All Crime"])
```

Plot them

```python
figsize(12.5, 4)
plt.plot(np.arange(len(date)), taxi_crime*100, color="#348ABD", label="Taxi crime (*100)")
plt.plot(np.arange(len(date)), all_crime, color="#00AA00", label="All crime")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.title("Crime volumes")
plt.legend()
```

As _all crime_ has much larger volumes than _taxi crime_, index to the first point and then calculate the difference. This will give them comparable sizes.

```python
taxi_i = np.array( [ x /  taxi_crime[0]  for x in taxi_crime ] )
all_i  = np.array( [ x /  all_crime[0]  for x in all_crime ] )
diff = all_i - taxi_i
```

```python
figsize(12.5, 4)
plt.plot(np.arange(len(date)), taxi_i, color="#00AA00", label="Taxi indexed")
plt.plot(np.arange(len(date)), all_i, color="#348ABD", label="All indexed")
plt.plot(np.arange(len(date)), diff, color="#A60628", label="Difference")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.legend()
```

It doesn't work very well using these noisy data. So try calculating the difference between the two proportions as a 12-month rolling average

```python
avg_diff = []
avg_taxi = []
avg_all = []
for i in range(12, len(taxi_i)):
    taxi_mean =  np.mean(taxi_i[i-12:len(taxi_i)])
    all_mean =  np.mean(all_i[i-12:len(taxi_i)])
    avg_taxi.append(taxi_mean)
    avg_all.append(all_mean)
    avg_diff.append(all_mean - taxi_mean)

avg_diff = np.array(avg_diff)
avg_taxi = np.array(avg_taxi)
avg_all  = np.array(avg_all)
```

```python
figsize(12.5, 4)
plt.plot(np.arange(len(avg_diff)), avg_taxi, color="#00AA00", label="Taxi indexed 12-month average")
plt.plot(np.arange(len(avg_diff)), avg_all, color="#348ABD", label="All indexed 12-month average")
plt.plot(np.arange(len(avg_diff)), avg_diff, color="#A60628", label="Difference 12-month average")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.title("12-month rolling average differences")
plt.legend()
```

## Model Conception

We want to model the difference between the two indexed crime volumes. The difference is continuous and could be positive or negative. Therefore lets assume that the difference can be modelled using a normal distribution:

$$X \sim N(\mu, 1/\tau)$$

The values of $\mu$ and $\tau$ determine the shape of the distribution. The $\tau$ will give us information about the spread of the difference. $\mu$ is especially useful: a _larger_ value for $\mu$ will higher probability of _larger_ variable values being chosen. Therefore if a model suggests that $\mu$ increases after some point (i.e. the introduction of the Deregulation Act) then it would appear that the differnce has increased.

```python

nor = stats.norm
x = np.linspace(-8, 7, 150)
mu = (-2, 0, 3)
tau = (.7, 1, 2.8)
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip(mu, tau, colors)

for _mu, _tau, _color in parameters:
    plt.plot(x, nor.pdf(x, _mu, scale=1./_tau),
             label="$\mu = %d,\;\\tau = %.1f$" % (_mu, _tau), color=_color)
    plt.fill_between(x, nor.pdf(x, _mu, scale=1./_tau), color=_color,
                     alpha=.33)

plt.legend(loc="upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title("Probability distribution of three different Normal random \
variables");
```

We want to test whether at some point (e.g. at the introduction of the Deregulation Act) the difference in crime volumes changed. Call this point $z$. After $z$, the difference suddenly becomes larger; $\mu$ increases. Therefore there are two $\mu$ parameters, one before $z$ and one after (and each will have its own $\tau$ parameter):

$$
X \sim N
\begin{cases}
    (\mu_1, 1/\tau_1 ) & \text{if } t \lt z \cr
    (\mu_2, 1/\tau_2 ) & \text{if } t \ge z
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\mu_1 = \mu_2$, then the $\mu$s posterior distributions should look about equal.

We are interested in inferring the unknown $\mu$s and $\tau$s. To use Bayesian inference, we need to assign prior probabilities to their different possible values. $\mu$ is continuous and might be negative (this would happen if taxi crime were more abundant than all other crime), so use a normal distrubtion again. $\tau$ is also continuous, but cannot be negative, so use an exponential distribution.  Each of those distributions has their own parameters, so we now have six _hyper-parameters_ (two for each of the $\mu$s and one for each $\tau$). Name them $\alpha$, $\beta$ and $\gamma$:

$$
\begin{align}
& \mu_1 \sim \text{N} (\alpha_1, 1/\beta_1) \text{ , } \tau_1 \sim \text{Exp} (\gamma_1) \\\
& \mu_2 \sim \text{N} (\alpha_2, 1/\beta_2) \text{ , } \tau_2 \sim \text{Exp} (\gamma_2)
\end{align}
$$

The $\alpha$, $\beta$ and $\gamma$ parameters influence other parameters, so our initial guesses do not influence the model too strongly, so we have some flexibility in our choice.

We also need a distribution for the point at which the difference changes ($z$). Assign a *uniform prior belief* to every possible time point. This will reduce any personal bias.

\begin{align}
& z \sim \text{DiscreteUniform(1,N) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{N}
\end{align}

where $N$ is the number of time points in the data.

## Model Definition

Now define the model using PYMC. (_Note that we use the same values for the hyperparameters, although above I talked about them as separate variables for each of the two Normal distributions_).

```python

with pm.Model() as model:
    # The hyperparameters
    alpha = 1.0
    beta = 1.0
    gamma = 1.0/avg_diff.mean()

    # The two mean and sd variables parameters.
    # (We want to see if their posterior distributions change)
    mu_1 = pm.Normal("mu_1", mu = alpha, tau = 1.0/beta)
    mu_2 = pm.Normal("mu_2", mu = alpha, tau = 1.0/beta)

    tau_1 = pm.Exponential("tau_1", gamma)
    tau_2 = pm.Exponential("tau_2", gamma)
    #taus = pm.Exponential("taus", gamma, shape=2)

    # Z is the point at which the texting behaviour changed.
    # Don't know anything about this so assign a uniform prior belief
    z = pm.DiscreteUniform("z", lower=0, upper=len(avg_diff) - 1)
    #z = pm.Normal("z", mu = 20, sd=5)

    # Creates new functions using switch() to assign the correct mu or tau variables
    # These can be thought of as the random variables mu and tau definied initially.
    idx = np.arange(len(avg_diff)) # Index
    mu_ =  pm.math.switch(z > idx, mu_1, mu_2)
    tau_ = pm.math.switch(z > idx, tau_1, tau_2)

    # Combine the observation data (we assume that it has been generated by
    # a normal distribution with our mu and tau parameters).
    obs = pm.Normal("obs", mu=mu_, tau=tau_, observed=avg_diff)
```

The model has been defined, now we can perform MCMC to sample from the posterior distribution

```python
N = 10000 # Number of samples
with model:
    step = pm.Metropolis()
    #start = pm.find_MAP() # Help it to start from a good place
    #trace = pm.sample(N, tune=int(N*2), step=step, start=start, njobs=4)
    trace = pm.sample(N, tune=int(N/2), step=step, start=start, njobs=1)
```

     67%|██████▋   | 10044/15000 [00:10<00:05, 946.91it/s]

The model has finished. Now example our samples

```python
mu_1_samples = trace['mu_1']
mu_2_samples = trace['mu_2']
tau_1_samples = trace['tau_1']
tau_2_samples = trace['tau_2']
z_samples = trace['z']
```

```python
figsize(12.5, 10)
BINS = 50
#histogram of the samples:

# MUs

ax = plt.subplot(321) # rows, columns, index
#ax.set_autoscaley_on(False)
plt.hist(mu_1_samples, histtype='stepfilled', bins=BINS, alpha=0.85,
         label="posterior of $\mu_1$", color="#A60628", normed=True)
plt.vlines(np.mean(mu_1_samples), 0, 3, linestyle="--", label="mean $mu_1$")
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables""")
plt.xlim([-2,4])
#plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(322) # rows, columns, index
#ax.set_autoscaley_on(False)
plt.hist(mu_2_samples, histtype='stepfilled', bins=BINS, alpha=0.85,
         label="posterior of $\mu_2$", color="#A60628", normed=True)
plt.vlines(np.mean(mu_2_samples), 0, 3, linestyle="--", label="mean $mu_2$")
plt.legend(loc="upper left")
plt.xlim([-2,4])
#plt.xlabel("$\lambda_1$ value")

# TAUs

ax = plt.subplot(323)
#ax.set_autoscaley_on(False)
plt.hist(tau_1_samples, histtype='stepfilled', bins=BINS, alpha=0.85,
         label="posterior of $\tau_1$", color="#7A68A6", normed=True)
plt.legend(loc="upper left")
plt.xlim([0, 7])
plt.xlabel("$\tau_1$ value")

ax = plt.subplot(324)
#ax.set_autoscaley_on(False)
plt.xlim([0, 7])
plt.hist(tau_2_samples, histtype='stepfilled', bins=BINS, alpha=0.85,
         label="posterior of $\tau_2$", color="#7A68A6", normed=True)
plt.legend(loc="upper left")
plt.xlim([0, 7])
plt.xlabel("$\tau_2$ value")

plt.subplot(325)
plt.hist(z_samples, bins=len(diff), alpha=1,
         label=r"posterior of $z$",
         color="#467821", rwidth=2.)
plt.xticks(np.arange(len(diff)))

plt.legend(loc="upper left")
#plt.ylim([0, .75])
plt.xlim([0,len(avg_diff)])
plt.xlabel(r"$z$ (in months)")
plt.ylabel("probability");
```

Some more diagnostic visualisations

```python
pm.plots.traceplot(trace=trace, varnames=["mu_1", "mu_2", "tau_1", "tau_2", "z"])
#pm.plots.traceplot(trace=trace, varnames=["mu_1"])
```

```python
pm.plots.plot_posterior(trace=trace, varnames=["mu_1", "mu_2", "tau_1", "tau_2", "z"])
#pm.plots.plot_posterior(trace=trace["centers"][:,1])
#pm.plots.autocorrplot(trace=trace, varnames=["centers"]);
```

```python

```

```python

```

## Next steps:

Think about How to detect structural change in a timeseries.

E.g.: [https://stats.stackexchange.com/questions/16953/how-to-detect-structural-change-in-a-timeseries]

    Can involve Jose too.

```python

```


#  PROCESAMIENTO DIGITAL DE SEÑALES DE AUDIO
##  Análisis por Predicción Lineal

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

from scipy import signal
from scipy.io import wavfile

from scipy.linalg import toeplitz
from scipy.linalg import solve

import IPython.display as ipd
```

**NOTA:** *Las siguientes dos celdas solo son necesarias para descargar el archivo de ejemplo. Ignórelas si va a trabajar con sus propios archivos de audio.*

```python
!pip install wget
```

```python
import wget
```

### Cómo correr el notebook
Se puede bajar y correr el notebook de forma local en una computadora.

O también se puede correr en Google Colab usando el siguiente enlace.

    Run in Google Colab

### Descripción

Este ejercicio sirve de introducción al **análisis por predicción lineal**. Se implementa análisis por el método de **autocorrelación**. Se calculan los coeficientes del modelo, el error y la ganancia.

El análisis LPC extrae los parámetros del modelo de producción de voz a partir del análisis de un fragmento de tiempo corto de la señal de voz. Se plantea un modelo todo polos de la forma,

$$ H(z) = \frac{S(z)}{U(z)} = \frac{G}{1-\sum_{k=1}^{p}a_kz^{-k}}$$
$$ s[n] = \sum_{k=1}^{p}a_k s[n-k]+Gu[n]. $$

Se define un predictor de orden $p$ a partir de las muestras pasadas de la salida,

$$\tilde{s}[n] = \sum_{k=1}^{p}\alpha_k s[n-k].$$

La solución consiste en obtener los coeficientes que minimizan el error cuadrático de predicción para un fragmento de tiempo corto de la señal de voz $s_n[m]$ elegido en torno a la muestra $n$, i.e. $s_n[m]=s[m+n]$.

$$ \begin{align}\label{eq:mean_square_error}
		 E_n &= \sum_m e_n^2[m]\nonumber\\
		     &= \sum_m\left(s_n[m]-\tilde{s}_n[m]\right)^2\nonumber\\
		     &= \sum_m\left(s_n[m]-\sum_{k=1}^{p}\alpha_k s_n[m-k]\right)^2
		\end{align}
$$
El análisis es de tiempo corto, por lo que el rango de la sumatoria es finito. Hay dos enfoques para resolver el problema que difieren en la elección de los límites de la sumatoria, el método de la **autocorrelación** y el método de la **covarianza**.

A continuación se carga la señal de audio que será procesada.

```python
# download impulse responses as audio files
wget.download('https://github.com/mrocamora/audio-dsp/blob/main/audio/voice2.wav?raw=true')
```

```python
# read the audio file
filename = 'voice2.wav'

# read audio samples using scipy.io.wavfile
fs, x = wavfile.read(filename)

# normalize maximum (absolute) amplitude
x = x / np.max(abs(x)) * 0.9
```

```python
# time corresponding to the audio signal
time_x = np.arange(0, x.size)/fs

# plot the audio signal waveform
plt.figure(figsize=(12,6))
ax1 = plt.subplot(2, 1, 1)
plt.plot(time_x, x)
plt.xlabel('Time (s)')
plt.ylabel('Amplitude')
plt.tight_layout()
```

```python
ipd.Audio(x, rate=fs)
```

### Parte 1

En esta primera parte se extraer un frame de la señal de audio y se aplica una ventana suavizante.

Luego de ejecutar el código responda las siguientes preguntas.

1. ¿En el método de la autocorrelación que se asume sobre la señal fuera del intervalo de análisis?
2. ¿Qué efecto tiene este supuesto en la estimación LPC en los bordes de la trama de señal?
3. ¿Cuál es el objetivo de aplicar una ventana suavizante?

```python
# number of samples in window
N = 460

# initial sample of the audio frame
ind_ini = 94567

# signal frame
s = x[ind_ini:ind_ini+N]

# smoothing window
window = signal.windows.get_window('hann', N)

# windowed signal frame
s_win = s * window

# number of DFT points
Ndft = 1024

# spectrum of the signal frame
X = np.fft.fft(s_win, Ndft)
# frequency values
f = np.fft.fftfreq(Ndft) * fs
```

```python
# plot the original signal frame
plt.figure(figsize=(12,6))
ax1 = plt.subplot(3, 1, 1)
plt.plot(s, 'k')
plt.ylabel('Amplitude')
plt.title('Signal frame')

# plot the smoothing window
ax2 = plt.subplot(3, 1, 2)
plt.plot(window, 'r')
plt.xlabel('Time (samples)')
plt.ylabel('Amplitude')
plt.title('Smoothing window')

# plot the windowed frame
ax3 = plt.subplot(3, 1, 3)
plt.plot(s_win, 'k')
plt.xlabel('Time (samples)')
plt.ylabel('Amplitude')
plt.title('Windowed signal frame')
plt.tight_layout()

# magnitude spectrum
magX = np.abs(X)
ind_fmx = int(Ndft/2)

# plot the frequency response
plt.figure(figsize=(12,6))
ax1 = plt.subplot()
ax1.set_title('Signal frame spectrum')
ax1.plot(f[:ind_fmx], 20 * np.log10(magX[:ind_fmx]), 'k')
ax1.set_ylabel('Magnitude (dB)')
ax1.set_xlabel('Frequency (Hz)')
plt.tight_layout()

```

### Parte 2

El siguiente código implementa el **análisis LPC** usando el método de la **autocorrelación**.

Tenga en cuenta que el sistema de ecuaciones expresado en forma matricial es el siguiente.

$$
		\left( \begin{array}{ccccc}
		R_n[0] & R_n[1] & R_n[2] & \cdots & R_n[p-1]\\
		R_n[1] & R_n[0] & R_n[1] & \cdots & R_n[p-2]\\
		R_n[2] & R_n[1] & R_n[0] & \cdots & R_n[p-3]\\
		\vdots & \vdots & \vdots & \ddots & \vdots\\
		R_n[p-1] & R_n[p-2] & R_n[p-3] & \cdots & R_n[0]\\
		\end{array} \right) \left( \begin{array}{c}
		\alpha_1\\
		\alpha_2\\
		\alpha_3\\
		\vdots\\
		\alpha_p\\
		\end{array} \right) = \left( \begin{array}{c}
		R_n[1]\\
		R_n[2]\\
		R_n[3]\\
		\vdots\\
		R_n[p]\\
		\end{array} \right)
$$

Y que el error cuadrático medio mínimo es:

$$E_n = R_n[0]-\sum_{k=1}^p\alpha_kR_n[k].$$

Complete el código que se proporciona a continuación. Para ello, siga los siguientes pasos.

1. Calcular los valores de la autocorrelación que intervienen en el sistema de ecuaciones.
2. Plantear y resolver el sistema de ecuaciones, teniendo en cuenta que hay que invertir una matriz Toeplitz.
3. Calcular el valor mínimo del error cuadrático medio.

```python
def lpc_analysis(s, p=20):
    """ compute the LPC analysis using the autocorrelation method

    Parameters
    ----------
    x : numpy array
        windowed signal frame as a numpy 1D array.
    p : int
        model order.

    Returns
    -------
    ak : numpy array
         model coefficients.
    e : float
        minimum mean squared error.
    e_norm : float
             normalized minimum mean squared error.
    """
    # frame length
    N = s.shape[0]

    # compute autocorrelation values
    r = np.zeros((p+1, 1))
    for k in range(p+1):
        # r[k] = np.dot(s[??:??].T, s[??:??])

    # solve to compute model coefficients
    # ak = solve(toeplitz(r[??:??]), r[??:??]).squeeze()

    # compute mean squared error
    # e = ??

    # compute normalized mean squared error
    e_norm = e / r[0]

    return ak, e, e_norm
```

### Parte 3

Una vez completada la implementación de la función `lpc_analysis` siga los siguientes pasos para calcular la respuesta en frecuencia y la respuesta al impulso del sistema.

Tenga presente la forma de la función de transferencia del sistema

$$H(z) = \frac{S(z)}{U(z)} = \frac{G}{1-\sum_{k=1}^{p}\alpha_kz^{-k}}.$$

1. Ejecute la función `lpc_analysis` para la trama de audio.
2. Calcule los polinomios numerador y denominador del filtro resultante.
3. Calcule la ganancia G a partir del error cuadrático medio.
4. Calcule la prespuesta al impulso filtrando una delta con el filtro resultante.
5. Cambie el orden del modelo y analice el resultado obtenido.

```python
# model order
p = 26

# apply lpc analysis to signal frame
ak, e, e_norm = lpc_analysis(s_win, p)

# filter obtained from the lpc analysis
# S = ??
U = np.concatenate([[1], -ak])

# compute gain
# G = ??

# compute the frequency response of the digital filter
w, H = signal.freqz(G*S, U, worN=Ndft, whole=True)
fw = w / (2 * np.pi) * fs

# impulse response of the LPC filter
delta = np.zeros(N)
delta[0] = 1
h = signal.lfilter(G*S, U, delta)
```

```python
# magnitude spectrum
magH = np.abs(H)
ind_fmx = int(Ndft/2)

# plot the frequency response
plt.figure(figsize=(12,6))
ax1 = plt.subplot()
ax1.set_title('Signal frame spectrum and frequency response of LPC filter')
ax1.plot(f[:ind_fmx], 20 * np.log10(magX[:ind_fmx]), 'k')
ax1.plot(f[:ind_fmx], 20 * np.log10(magH[:ind_fmx]), 'r')
ax1.set_ylabel('Magnitude (dB)')
ax1.set_xlabel('Frequency (Hz)')
plt.tight_layout()

# plot the impulse response
plt.figure(figsize=(12,6))
ax1 = plt.subplot(2, 1, 1)
ax1.plot(h, 'k')
ax1.set_title('Impulse response of LPC filter')
ax1.set_ylabel('Amplitude')
ax1.set_xlabel('Time (samples)')
plt.tight_layout()

```

### Parte 4

A continuación se calcula la excitación $e[n] = G u[n]$ filtrando la señal de voz $s[n]$ con el filtro inverso.

Tenga presente la relación entre la función de transferencia del sistema

$$H(z) = \frac{S(z)}{U(z)} = \frac{G}{1-\sum_{k=1}^{p}\alpha_kz^{-k}}$$

y el filtro inverso

$$A(z) = \frac{E(z)}{S(z)} = 1-\sum_{k=1}^{p}\alpha_kz^{-k}.$$

Complete el código que se proporciona a continuación. Para ello, siga los siguientes pasos.

1. Calcule los polinomios numerador y denominador del filtro inverso.
2. Utilice adecuadamente la ganancia calculada anteriormente.
3. Aplique el filtro a la señal de voz para obtener la excitación.

```python
# inverse filter
# A = ??
# B = ??

# compute the excitation from the inverse filter
p = signal.lfilter(B, A, s_win)

# compute the spectrum of the excitation
P = np.fft.fft(p, Ndft)

```

```python
# magnitude spectrum
magP = np.abs(P)
ind_fmx = int(Ndft/2)

plt.figure(figsize=(12,6))
ax1 = plt.subplot(2, 1, 1)
plt.plot(s_win, 'k')
plt.xlabel('Time (samples)')
plt.ylabel('Amplitude')
plt.title('Windowed signal frame')
plt.tight_layout()
ax2 = plt.subplot(2, 1, 2)
plt.plot(p, 'k')
ax2.set_title('Inverse LPC filter applied to the signal frame')
ax2.set_ylabel('Amplitude')
ax2.set_xlabel('Time (samples)')
plt.tight_layout()

# plot the frequency response
plt.figure(figsize=(12,6))
ax1 = plt.subplot()
ax1.set_title('Spectrum of the signal frame and the estimated excitation')
ax1.plot(f[:ind_fmx], 20 * np.log10(magX[:ind_fmx]), 'k')
ax1.plot(f[:ind_fmx], 20 * np.log10(magP[:ind_fmx]), 'r')
ax1.set_ylabel('Magnitude (dB)')
ax1.set_xlabel('Frequency (Hz)')
plt.tight_layout()

```

```python

```


# Extension of DV Formula

$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}$

## $f$-divergence

Consider the more general problem of estimating the $f$-divergence in {eq}`f-D`:

$$
D_f(P_{\R{Z}}\|P_{\R{Z}'}) = E\left[f\left(\frac{dP_{\R{Z}}(\R{Z}')}{dP_{\R{Z}'}(\R{Z}')}\right)\right].
$$

**Exercise**

How to estimate $f$-divergence using the DV formula?

YOUR ANSWER HERE

Instead of using the DV bound, it is desirable to train a network to optimize a tight bound on the $f$-divergence because:

- *Estimating KL divergence well does not imply the underlying neural network approximates the density ratio well*:
  - While KL divergence is just a non-negative real number,
  - the density ratio is in a high dimensional function space.

- DV formula does not directly maximizes a bound on $f$-divergence, i.e.
  it does not directly minimize the error in estimating $f$-divergence.

- $f$-divergence may have bounds that are easier/more stable for training a neural network.

**How to extend the DV formula to $f$-divergence?**

The idea is to think of the $f$-divergence as a convex *function(al)* evaluated at the density ratio:

---

**Proposition**
:label: D->F

$f$-divergence {eq}`f-D` is

$$
\begin{align}
D_f(P_{\R{Z}}\|P_{\R{Z}'}) = F\left[ \frac{P_{\R{Z}}}{P_{\R{Z}'}}\right]
\end{align}
$$ (D->F)

where

$$
\begin{align}
F[r] := E [ \underbrace{(f \circ r)(\R{Z}')}_{f(r(\R{Z}'))}]
\end{align}
$$ (F)

for any function $r:\mc{Z} \to \mathbb{R}$.

---

This is more like a re-definition than a proposition as the proof is immediate:
{eq}`f-D` is obtained from {eq}`D->F` by substituting $r=\frac{dP_{\R{Z}}}{dP_{\R{Z}'}}$.

As mentioned before, the KL divergence $D(P_{\R{Z}}\|P_{\R{Z}'})$ is a special case of $f$-divergence:

$$
D(P_{\R{Z}}\|P_{\R{Z}'}) = F\left[r\right]
$$

where

$$
\begin{align*}
F[r] &:= E\left[ r(\R{Z}')\log r(\R{Z}')\right].
\end{align*}
$$ (KL:F)

**Exercise** When is $F[r]=0$ equal to $0$?

YOUR ANSWER HERE

**Exercise**

Show using the properties of $f$ that $F$ is strictly convex.

**Solution**

For $\lambda\in [0,1]$ and functions $r_1, r_2\in \Set{r:\mc{Z}\to \mathbb{R}}$,

$$
\begin{align*}
F[\lambda r_1 + (1-\lambda) r_2]
&= E[\underbrace{ f(\lambda r_1(\R{Z}') + (1-\lambda) r_2(\R{Z}'))}_{\stackrel{\text{(a)}}{\geq} \lambda f(r_1(\R{Z}'))+(1-\lambda) f(r_2(\R{Z}'))}]\\
&\stackrel{\text{(b)}}{\geq} \lambda E[f(r_1(\R{Z}'))] + (1-\lambda) E[f(r_2(\R{Z}'))]
\end{align*}
$$

where (a) is by the convexity of $f$, and (b) is by the linearity of expectation. $F$ is strictly convex because (b) is satisfied with equality iff (a) is almost surely.

For a clearer understanding, consider a different choice of $F$ for the KL divergence:

$$
D(P_{\R{Z}}\|P_{\R{Z}'}) = F'\left[r\right]
$$

where

$$
\begin{align*}
F'[r] &:= E\left[ \log r(\R{Z})\right].
\end{align*}
$$ (rev-KL:F)

Note that $F'$ in {eq}`rev-KL:F` defined above is concave in $r$. In other words, {eq}`D2` in {prf:ref}`DV2`

$$
\begin{align*}
D(P_{\R{Z}}\|P_{\R{Z}'}) & =  \sup_{\substack{r:\mc{Z}\to \mathbb{R}_+\\ E[r(\R{Z}')]=1}} E \left[ \log r(\R{Z}) \right]
\end{align*}
$$

is maximizing a concave function and therefore has a unique solution, namely, $r=\frac{dP_{\R{Z}}}{dP_{\R{Z}'}}$. Here comes the tricky question:

**Exercise**

Is KL divergence concave or convex in the density ratio $\frac{dP_{\R{Z}}}{dP_{\R{Z}'}}$? Note that $F$ defined in {eq}`KL:F` is convex in $r$.

YOUR ANSWER HERE

## Convex conjugation

Given $P_{\R{Z}'}\in \mc{P}(\mc{Z})$, consider
- a function space $\mc{R}$,

$$
\begin{align}
\mc{R} &\supseteq \Set{r:\mathcal{Z}\to \mathbb{R}_+\mid E\left[r(\R{Z}')\right] = 1},
\end{align}
$$ (R)

- a dual space $\mc{T}$, and

$$
\begin{align}
\mc{T} &\subseteq \Set{t:\mc{Z} \to \mathbb{R}}
\end{align}
$$ (T)

- the corresponding inner product $\langle\cdot,\cdot \rangle$:

$$
\begin{align}
\langle t,r \rangle &= \int_{z\in \mc{Z}} t(z) r(z) dP_{\R{Z}'}(z) = E\left[ t(\R{Z}') r(\R{Z}') \right].
\end{align}
$$ (inner-prod)

The following is a generalization of DV formula for estimating $f$-divergence {cite}`nguyen2010estimating`{cite}`ruderman2012tighter`:

---

**Proposition**
:label: convex-conjugate

$$
\begin{align}
D_{f}(P_{\R{Z}} \| P_{\R{Z}'}) = \sup _{t\in \mc{T}} E[g(\R{Z})] - F^*[t],
\end{align}
$$ (convex-conjugate2)

where

$$
\begin{align}
F^*[t] = \sup_{r\in \mc{R}} E[t(\R{Z}') r(\R{Z}')] - F[r].
\end{align}
$$ (convex-conjugate1)

---

---

**Proof**

Note that the supremums in {eq}`convex-conjugate1` and {eq}`convex-conjugate2` are [Fenchel-Legendre transforms][FL]. Denoting the transform as $[\cdot]^*$,

$$\underbrace{[[F]^*]^*}_{=F}\left[\frac{dP_{\R{Z}}}{dP_{\R{Z}'}}\right]$$

gives {eq}`convex-conjugate2` by expanding the outer/later transform. The equality is by the property that Fenchel-Legendre transform is its own inverse for strictly convex functional $F$. This completes the proof by {eq}`D->F`.

[FL]: https://en.wikipedia.org/wiki/Convex_conjugate

---

The proof is illustrated in the following figure:

Let's breakdown the details:

**Step 1**

For the purpose of the illustration, visualize the convex functional $F$ simply as a curve in 2D.

The $f$-divergence is then the $y$-coordinate of a point on the curve indicated above, with $r$ being the density ratio $\frac{dP_{\R{Z}}}{dP_{\R{Z}'}}$.

**Step 2**

To obtain a lower bound on $F$, consider any tangent of the curve with an arbitrary slope $t\cdot dP_{\R{Z}'}$

The lower bound is given by the $y$-coordinate of a point on the tangent with $r$ being the density ratio.

**Exercise**

Why is the $y$-coordinate of the tangent a lower bound on the $f$-divergence?

**Solution**

By the convexity of $F$, the tangent must be below $F$.

**Step 3**

To calculate the lower bound, denote the $y$-intercept as $-F^*[t]$:

Thinking of a function as nothing but a vector, the displacement from the $y$-intercept to the lower bound is given by the inner product of the slope and the density ratio.

**Step 4**

To make the bound tight, maximize the bound over the choice of the slope or $t$:

This gives the bound in {eq}`convex-conjugate2`. It remains to show {eq}`convex-conjugate1`.

**Step 5**

To compute the $y$-intercept or $F^*[t]$, let $r^*$ be the value of $r$ where the tangent touches the convex curve:

The displacement from the point at $r^*$ to the $y$-intercept can be computed as the inner product of the slope and $r^*$.

**Exercise**

Show that for the functional $F$ {eq}`KL:F` defined for KL divergence,

$$F^*[t]=\log E[e^{t(\R{Z}')}]$$

with $\mc{R}=\Set{r:\mc{Z}\to \mathbb{R}_+}$ and so {eq}`convex-conjugate2` gives the DV formula {eq}`DV` as a special case.

YOUR ANSWER HERE


# In Depth: Naive Bayes Classification

<!--BOOK_INFORMATION-->

*This notebook contains an excerpt from the [Python Data Science Handbook](http://shop.oreilly.com/product/0636920034919.do) by Jake VanderPlas; the content is available [on GitHub](https://github.com/jakevdp/PythonDataScienceHandbook).*

*The text is released under the [CC-BY-NC-ND license](https://creativecommons.org/licenses/by-nc-nd/3.0/us/legalcode), and code is released under the [MIT license](https://opensource.org/licenses/MIT). If you find this content useful, please consider supporting the work by [buying the book](http://shop.oreilly.com/product/0636920034919.do)!*

<!--NAVIGATION-->
< [Feature Engineering](05.04-Feature-Engineering.ipynb) | [Contents](Index.ipynb) | [In Depth: Linear Regression](05.06-Linear-Regression.ipynb) >

``Naive Bayes models`` are a group of **extremely fast and simple** classification algorithms that are often suitable for **very high-dimensional** datasets.

A quick-and-dirty baseline for a classification problem.
- so fast
- so few tunable parameters
- very useful

This section will focus on
- an intuitive explanation of how naive Bayes classifiers work
- followed by a couple examples of them in action on some datasets.

## Bayesian Classification

Naive Bayes classifiers are built on Bayesian classification methods.

Bayes's theorem is an equation describing the relationship of conditional probabilities of statistical quantities.

- We are intrested in finding the probability of a label given some observed features
    - which we can write as $P(L~|~{\rm features})$.

Bayes's theorem:

$$
P(L~|~{\rm features}) = \frac{P({\rm features}~|~L)P(L)}{P({\rm features})}
$$

$$
\mbox{posterior} =  \frac{\mbox{likelihood}\times \mbox{prior}}{\mbox{evidence}} \
$$

https://en.wikipedia.org/wiki/Naive_Bayes_classifier

In practice, there is interest only in the numerator of that fraction,
- because the denominator does not depend on $L$ and the values of the features $x_i$ are given
- so that the denominator is effectively constant.

The numerator is equivalent to the ``joint probability`` model

$$p(x_1, \dots, x_n|L_k) p(L_k)  = p(L_k, x_1, \dots, x_n)$$

$$
p(L_k \mid x_1, \dots, x_n)  \varpropto p(L_k, x_1, \dots, x_n)
$$

\begin{align}
p(L_k, x_1, \dots, x_n) & = p(x_1, \dots, x_n, L_k) \\
                        & = p(x_1 \mid x_2, \dots, x_n, L_k) p(x_2, \dots, x_n, L_k) \\
                        & = p(x_1 \mid x_2, \dots, x_n, L_k) p(x_2 \mid x_3, \dots, x_n, L_k) p(x_3, \dots, x_n, L_k) \\
                        & = \dots \\
                        & = p(x_1 \mid x_2, \dots, x_n, L_k) p(x_2 \mid x_3, \dots, x_n, L_k) \dots   p(x_{n-1} \mid x_n, L_k) p(x_n \mid L_k) p(L_k) \\
\end{align}

Now the "naive" conditional independenLe assumptions come into play: assume that each feature $x_i$ is conditionally statistical independence of every other feature $x_j$ for $j\neq i$, given the category $L_k$.  This means that

$$p(x_i \mid x_{i+1}, \dots ,x_{n}, L_k ) = p(x_i \mid L_k)$$

Thus, the joint model can be expressed as

\begin{align}
p(L_k \mid x_1, \dots, x_n) & \varpropto p(L_k, x_1, \dots, x_n) = \\
                            & = p(L_k) \ p(x_1 \mid L_k) \ p(x_2\mid L_k) \ p(x_3\mid L_k) \ \cdots \\
                            & = p(L_k) \prod_{i=1}^n p(x_i \mid L_k)\,.
\end{align}

The maximum a posteriori (MAP) decision rule

$$\hat{y} = \underset{k \in \{1, \dots, K\}}{\operatorname{argmax}} \ p(C_k) \displaystyle\prod_{i=1}^n p(x_i \mid C_k)$$

## Bayesian Classification

If we are trying to decide between two labels $L_1$ and $L_2$
- to compute the ratio of the posterior probabilities for each label:

$$
\frac{P(L_1~|~{\rm features})}{P(L_2~|~{\rm features})} = \frac{P({\rm features}~|~L_1)}{P({\rm features}~|~L_2)}\frac{P(L_1)}{P(L_2)}
$$

We can compute $P({\rm features}~|~L_i)$ for each label.
- Such a model is called a *generative model*
    - because it specifies the hypothetical random process that generates the data.

## Bayesian Classification

``Specifying this generative model`` for each label is the main piece of the training of such a Bayesian classifier.
- The general version of such a training step is a very difficult task,
- but we can make it simpler through the use of some simplifying assumptions about the form of this model.

This is where the "naive" in "naive Bayes" comes in:
- if we make very naive assumptions about the generative model for each label,
    - we can find a rough approximation of the generative model for each class, and then proceed with the Bayesian classification.

Different types of naive Bayes classifiers rest on different naive assumptions about the data, and we will examine a few of these in the following sections.

## Gaussian Naive Bayes

Perhaps the easiest naive Bayes classifier to understand is Gaussian naive Bayes.
- In this classifier, the assumption is that *data from each label is drawn from a simple Gaussian distribution*.

Imagine that you have the following data:

When dealing with continuous data, a typical assumption is that the continuous values associated with each class are distributed according to a Normal distribution (Gaussian) distribution.

The probability density of the normal distribution is

$$
f(x \mid \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2} } e^{ -\frac{(x-\mu)^2}{2\sigma^2} }
$$

For example, suppose the training data contains a continuous attribute, $x$.

We first segment the data by the class, and then compute the mean and variance of $x$ in each class.
- Let $\mu_k$ be the mean of the values in $x$ associated with class $C_k$
- Let $\sigma^2_k$ be the variance of the values in $x$ associated with class $C_k$.

Suppose we have collected some observation value $v$.
Then, the probability distribution of $v$ given a class $C_k$,

$p(x=v \mid C_k)$, can be computed by plugging $v$ into the equation for a Normal distribution parameterized by $\mu_k$ and $\sigma^2_k$. That is,

\begin{align}
p(x=v \mid C_k)=\frac{1}{\sqrt{2\pi\sigma^2_k}}\,e^{ -\frac{(v-\mu_k)^2}{2\sigma^2_k} }
\end{align}

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
```

```python
from sklearn.datasets import make_blobs
X, y = make_blobs(100, 2, centers=2, random_state=2, cluster_std=1.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu');
```

One extremely fast way to create a simple model is to assume that
- the data is described by a Gaussian distribution with no covariance between dimensions.

This model can be fit by
- simply finding the mean and standard deviation of the points within each label

The result of this naive Gaussian assumption is shown in the following figure:

[figure source in Appendix](06.00-Figure-Code.ipynb#Gaussian-Naive-Bayes)

The ellipses here represent the Gaussian generative model for each label
- with larger probability toward the center of the ellipses.

With this generative model in place for each class, we have a simple recipe to compute the likelihood $P({\rm features}~|~L_1)$ for any data point
- and thus we can quickly compute the posterior ratio and determine which label is the most probable for a given point.

This procedure is implemented in Scikit-Learn's ``sklearn.naive_bayes.GaussianNB`` estimator:

```python
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()
model.fit(X, y);
```

Now let's generate some new data and predict the label:

```python
rng = np.random.RandomState(0)
Xnew = [-6, -14] + [14, 18] * rng.rand(2000, 2)
ynew = model.predict(Xnew)
```

Now we can plot this new data to get an idea of where the decision boundary is:

```python
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='RdBu')
lim = plt.axis()
plt.scatter(Xnew[:, 0], Xnew[:, 1], c=ynew, s=20, cmap='RdBu', alpha=0.1)
plt.axis(lim);
```

We see a slightly curved boundary in the classifications—in general, the boundary in Gaussian naive Bayes is quadratic.

A nice piece of this Bayesian formalism is that it naturally allows for probabilistic classification, which we can compute using the ``predict_proba`` method:

```python
yprob = model.predict_proba(Xnew)
yprob[-8:].round(2)
# The columns give the posterior probabilities of the first and second label, respectively.
```

    array([[ 0.89,  0.11],
           [ 1.  ,  0.  ],
           [ 1.  ,  0.  ],
           [ 1.  ,  0.  ],
           [ 1.  ,  0.  ],
           [ 1.  ,  0.  ],
           [ 0.  ,  1.  ],
           [ 0.15,  0.85]])

If you are looking for estimates of uncertainty in your classification, Bayesian approaches like this can be a useful approach.

The final classification will only be as good as the model assumptions that lead to it
- This is why Gaussian naive Bayes often does not produce very good results.

## Multinomial Naive Bayes

- The Gaussian assumption is only one simple assumption that could be used to specify the generative distribution for each label.
- Multinomial naive Bayes assumes that the features are generated from a simple multinomial distribution.

$$
p(\mathbf{x} \mid L_k) = \frac{(\sum_i x_i)!}{\prod_i x_i !} \prod_i {p_{ki}}^{x_i}
$$

The multinomial distribution describes the probability of observing counts among a number of categories,
- multinomial naive Bayes is most appropriate for features that represent counts or count rates.
- The idea is precisely the same as before, but we model the data distribuiton with a best-fit multinomial distribution.

### Example: Classifying Text

One place where multinomial naive Bayes is often used is in ``text classification``
- the features are related to word counts or frequencies within the documents to be classified.
    - the extraction of such features from text in [Feature Engineering](05.04-Feature-Engineering.ipynb)

Using the sparse word count features from the 20 Newsgroups corpus to classify these documents.
- Let's download the data and take a look at the target names:

```python
from sklearn.datasets import fetch_20newsgroups

data = fetch_20newsgroups()
data.target_names
```

    Downloading dataset from http://people.csail.mit.edu/jrennie/20Newsgroups/20news-bydate.tar.gz (14 MB)

    ['alt.atheism',
     'comp.graphics',
     'comp.os.ms-windows.misc',
     'comp.sys.ibm.pc.hardware',
     'comp.sys.mac.hardware',
     'comp.windows.x',
     'misc.forsale',
     'rec.autos',
     'rec.motorcycles',
     'rec.sport.baseball',
     'rec.sport.hockey',
     'sci.crypt',
     'sci.electronics',
     'sci.med',
     'sci.space',
     'soc.religion.christian',
     'talk.politics.guns',
     'talk.politics.mideast',
     'talk.politics.misc',
     'talk.religion.misc']

For simplicity here, we will select just a few of these categories, and download the training and testing set:

```python
categories = ['talk.religion.misc', 'soc.religion.christian',
              'sci.space', 'comp.graphics']
train = fetch_20newsgroups(subset='train', categories=categories)
test = fetch_20newsgroups(subset='test', categories=categories)
```

Here is a representative entry from the data:

```python
print(train.data[5])
```

    From: dmcgee@uluhe.soest.hawaii.edu (Don McGee)
    Subject: Federal Hearing
    Originator: dmcgee@uluhe
    Organization: School of Ocean and Earth Science and Technology
    Distribution: usa
    Lines: 10

    Fact or rumor....?  Madalyn Murray O'Hare an atheist who eliminated the
    use of the bible reading and prayer in public schools 15 years ago is now
    going to appear before the FCC with a petition to stop the reading of the
    Gospel on the airways of America.  And she is also campaigning to remove
    Christmas programs, songs, etc from the public schools.  If it is true
    then mail to Federal Communications Commission 1919 H Street Washington DC
    20054 expressing your opposition to her request.  Reference Petition number

    2493.

To convert the content of each string into a vector of numbers.
- Use TF-IDF vectorizer (discussed in [Feature Engineering](05.04-Feature-Engineering.ipynb))
- Create a pipeline that attaches it to a multinomial naive Bayes classifier:

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.pipeline import make_pipeline

model = make_pipeline(TfidfVectorizer(), MultinomialNB())
```

With this pipeline, we can apply the model to the training data, and predict labels for the test data:

```python
model.fit(train.data, train.target)
labels = model.predict(test.data)
```

    /Users/datalab/Applications/anaconda/lib/python3.5/site-packages/sklearn/feature_extraction/text.py:1015: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      if hasattr(X, 'dtype') and np.issubdtype(X.dtype, np.float):

Evaluate the performance of the estimator.
- the confusion matrix between the true and predicted labels for the test data

```python
from sklearn.metrics import confusion_matrix
sns.set_context("notebook", font_scale=1.7)

mat = confusion_matrix(test.target, labels)
sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=False,
            xticklabels=train.target_names, yticklabels=train.target_names)
plt.xlabel('true label')
plt.ylabel('predicted label');
```

Evidently, even this very simple classifier can successfully separate space talk from computer talk,
- but it gets confused between talk about religion and talk about Christianity.
- This is perhaps an expected area of confusion!

### The very cool thing here
we now have the tools to determine the category for *any* string
- using the ``predict()`` method of this pipeline.

Here's a quick utility function that will return the prediction for a single string:

```python
def predict_category(s, train=train, model=model):
    pred = model.predict([s])
    return train.target_names[pred[0]]
```

Let's try it out:

```python
predict_category('sending a payload to the ISS')
```

    /Users/datalab/Applications/anaconda/lib/python3.5/site-packages/sklearn/feature_extraction/text.py:1015: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      if hasattr(X, 'dtype') and np.issubdtype(X.dtype, np.float):

    'sci.space'

```python
predict_category('discussing islam vs atheism')
```

    /Users/datalab/Applications/anaconda/lib/python3.5/site-packages/sklearn/feature_extraction/text.py:1015: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      if hasattr(X, 'dtype') and np.issubdtype(X.dtype, np.float):

    'soc.religion.christian'

```python
predict_category('determining the screen resolution')
```

    /Users/datalab/Applications/anaconda/lib/python3.5/site-packages/sklearn/feature_extraction/text.py:1015: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      if hasattr(X, 'dtype') and np.issubdtype(X.dtype, np.float):

    'comp.graphics'

Remember that this is nothing more sophisticated than a simple probability model for the (weighted) frequency of each word in the string;
- nevertheless, the result is striking.

> ##  Even a very naive algorithm, when used carefully and trained on a large set of high-dimensional data, can be surprisingly effective.

## When to Use Naive Bayes

Because naive Bayesian classifiers make such stringent assumptions about data, they will generally not perform as well as a more complicated model.

Advantages:

- They are extremely fast for both training and prediction
- They provide straightforward probabilistic prediction
- They are often very easily interpretable
- They have very few (if any) tunable parameters

## When to Use Naive Bayes

A good choice as an initial baseline classification.
- If it performs suitably, then congratulations: you have a very fast, very interpretable classifier for your problem.
- If it does not perform well, then you can begin exploring more sophisticated models, with some baseline knowledge of how well they should perform.

Naive Bayes classifiers tend to perform especially well in one of the following situations:

- When the naive assumptions actually match the data (very rare in practice)
- For very well-separated categories, when model complexity is less important
- For very high-dimensional data, when model complexity is less important

## When to Use Naive Bayes

The last two points seem distinct, but they actually are related:
- as the dimension of a dataset grows, it is much less likely for any two points to be found close together (after all, they must be close in *every single dimension* to be close overall).
- clusters in high dimensions tend to be more separated, on average, than clusters in low dimensions, assuming the new dimensions actually add information.

> ## simplistic classifiers like naive Bayes tend to work as well or better than more complicated classifiers as the dimensionality grows: once you have enough data, even a simple model can be very powerful.

<!--NAVIGATION-->
< [Feature Engineering](05.04-Feature-Engineering.ipynb) | [Contents](Index.ipynb) | [In Depth: Linear Regression](05.06-Linear-Regression.ipynb) >


```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

from chmp.ds import mpl_set, get_color_cycle
```

```python
# helper for gradient checking
def approximate_gradient(x, func, eps=1e-5):
    res = np.zeros(x.size)

    for i in range(x.size):
        d = np.zeros(x.size)
        d[i] = eps

        res[i] = (func(x + d) - func(x - d)) / (2 * eps)

    return res
```

# Linear models for regression

## Linear Basis Function Models

### The Bias Variance Decomposition

## Bayesian linear regression

Prior for $L_q$ regularization in 1 dimension (p. 156):

$$
    p(w|\alpha) =
        \frac{q}{2} \left( \frac{\alpha}{2} \right)^{1/q} \frac{1}{\Gamma(1/q)}
        \exp \left( -\frac{\alpha}{2} |w|^q \right)
$$

### Gaussian likelihood + known variance

$$
\begin{align}
    &\text{likelihood} &\;
    &p(t|\phi, w, \beta) = \mathcal{N}(t|w^T \phi, \beta^{-1})
\\
    &\text{prior} &\;
    &p(w|m_0, \alpha) = \mathcal{N}(w|m_0, \alpha^{-1})
\\
    &\text{posterior} &\;
    &p(w|\{t, \phi\}) = \mathcal{N}(w|m_N, S_N)
\\
    &\; &\;
    &m_N = S_N \left( \alpha m_0 + \beta \Phi^t T \right)
\\
    &\; &\;
    &S_N^{-1} = \alpha + \beta \Phi^T \Phi
\\
    &\text{predictive} &\;
    &p(t|\phi, \{y, \phi\}) = \mathcal{N}(t|m_N^T \phi, \sigma_N^2)
\\
    &\; &\;
    &\sigma_N^2 = \beta^{-1} + \phi^T S_N \phi
\end{align}
$$

Note, these equations are also valid if $\alpha$ is a matrix.

Note, notation is unified with Sec. 3.5 ($S_0^{-1} \rightarrow \alpha$).

```python
# TODO: check for inverse of alpha? does the definition match up?
def update_posterior(params, observed, beta):
    m, s = params
    x, y = observed

    s_new = np.linalg.inv(np.linalg.inv(s) + beta * x.T @ x)
    m_new = ((m @ np.linalg.inv(s) + beta * y.T @ x) @ s_new).reshape(-1)

    return m_new, s_new

def eval_predictive(x, params, beta):
    m, s = params
    return (
        x @ m,
        # TODO: optimize the diag part
        1.0 / beta + np.sum((x @ s_N) * x, axis=1)
    )

def empirical_bayes(observed, alpha, beta):
    x, y = observed
    N, M = x.shape
    lam = np.linalg.eigvals(x_eval.T @ x_eval)

    for _ in range(10):
        m_0, s_0 = np.zeros(M), np.diag([1 / alpha] * M)
        m_N, s_N = update_posterior((m_0, s_0), observed, beta)

        gamma = np.sum(lam / (lam + alpha))
        alpha = gamma / (m_N.T @ m_N)
        beta = np.sum((y - x @ m_N) ** 2.0) / (N - gamma)

    return (m_N, s_N), alpha, beta

def plot_gaussian2d(m, s, extent, bins=(10, 10)):
    # TODO: normalize s
    x = np.linspace(extent[0][0], extent[0][1], bins[0])
    y = np.linspace(extent[1][0], extent[1][1], bins[1])
    x, y = np.meshgrid(x, y)

    m_x, m_y = m
    (t_xx, t_xy), (t_yx, t_yy) = np.linalg.inv(s_N)

    norm = 1 / (2.0 * np.pi * np.linalg.det(s) ** 0.5)

    p = norm * np.exp(-0.5 * (
        t_xx * (x - m_x) * (x - m_x) +
        t_xy * (x - m_x) * (y - m_y) +
        t_yx * (y - m_y) * (x - m_x) +
        t_yy * (y - m_y) * (y - m_y)
    ))

    p = p.astype('complex128').real
    plt.imshow(
        p,
        extent=[extent[0][0], extent[0][1], extent[1][0], extent[1][1]],
        origin='lower',
        aspect='auto',
    )
```

```python
# Example (see fig. 3.7)
# beta = 25, alpha = 1 / 2.0

w_true = np.asarray([-0.3, 0.5]).reshape((2, 1))

np.random.seed(241)

n_samples = 20
x_obs = np.stack([np.ones(n_samples), np.random.uniform(low=-1, high=+1, size=n_samples)])
x_obs = x_obs.T
y_obs = np.random.normal(x_obs @ w_true, scale=0.2)

x_eval = np.stack([np.ones(100), np.linspace(-1, 1, 100)])
x_eval = x_eval.T
```

```python
beta = 25
alpha = 1 / 2.0
params_0 = np.zeros(2), np.diag([1 / alpha, 1 / alpha])
m_N, s_N = update_posterior(params_0, (x_obs, y_obs), beta)
p_m, p_s = eval_predictive(x_eval, (m_N, s_N), beta)

(m_emp, s_emp), alpha_emp, beta_emp = empirical_bayes((x_obs, y_obs), alpha, beta)
```

```python
c0, c1, c2 = get_color_cycle(3)

plt.figure(figsize=(16, 4))

plt.subplot(1, 3, 1)
plot_gaussian2d(m_N, s_N, extent=[[-0.75, +0.25], [-0.25, +0.75]], bins=(50, 50))
plt.plot([w_true[0]], [w_true[1]], 'wo')
mpl_set(xlabel='w_0', ylabel='w_1', colorbar=True, title='Posterior fixed hyperparameters')

plt.subplot(1, 3, 2)
plot_gaussian2d(m_emp, s_emp, extent=[[-0.75, +0.25], [-0.25, +0.75]], bins=(50, 50))
plt.plot([w_true[0]], [w_true[1]], 'wo')
mpl_set(xlabel='w_0', ylabel='w_1', colorbar=True, title='Posterior empirical Bayes')

plt.subplot(1, 3, 3)
plt.scatter(x_obs[:, 1], y_obs[:, 0], color=c0, label='data')
plt.plot(x_eval[:, 1], x_eval @ w_true, color=c1, label='truth')
plt.plot(x_eval[:, 1], p_m, label='predictive', color=c2)
plt.fill_between(x_eval[:, 1], p_m - p_s ** 0.5, p_m + p_s ** 0.5, color=c2, alpha=0.2)

mpl_set(xlabel='x', ylabel='y', legend=True)
pass
```

## Bayesian model comparison

Assume mutliple models $\mathcal{M}_i$ under investigation. The posterior for the model is given by

$$
\begin{align}
    \color{red}{p(\mathcal{M}_i|\mathcal{D})} &=
        \frac{P(\mathcal{M}_i) \color{blue}{p(\mathcal{D}|\mathcal{M}_i)}}{p(\mathcal{D})}
    &\; &\;
\\
    p(t|x, \mathcal{D}) &= \sum_i p(t|x, \mathcal{M}_i, \mathcal{D}) \color{red}{p(\mathcal{M}_i|\mathcal{D})}
    &\; &\;
\\
    \log \color{blue}{p(\mathcal{D}|\mathcal{M}_i)} &=
        \log \int \mathrm{d}w\; p(\mathcal{D}|w, \mathcal{M}_i) p(w|\mathcal{M}_i)
    &\; &\;
\\
    &\approx \log  \Delta w_\mathrm{posterior} \log  p(\mathcal{D}|w_\mathrm{MAP}, \mathcal{M}_i)
    p(w_\mathrm{MAP}|\mathcal{M}_i)
    &\; &\text{strongly peaked posterior}
\\
    &\approx \log  p(\mathcal{D}|w_\mathrm{MAP}, \mathcal{M}_i) +
        \log \frac{\Delta w_\mathrm{posterior} }{\Delta w_\mathrm{prior}}
    &\; &\text{flat prior}
\\
    &\approx \log p(\mathcal{D}|w_\mathrm{MAP}, \mathcal{M}_i) +
        M \log \frac{\Delta w_\mathrm{posterior} }{\Delta w_\mathrm{prior}}
    &\; &\text{for M parameters}
\end{align}
$$

For flat model prior $p(\mathcal{M}_i) = \mathrm{const}$ the relevant quantity is only the model evidence (or marginal likelihood) $\color{blue}{p(\mathcal{D}|\mathcal{M}_i)}$ is relevant.

Note that, in the approximation above, the model evidence is the sum between a likelihood term (how good the data is fitted) and a complexity term (how many parameters are included). This form favors a model that is complex enough to fit the data, but not too complex.

When approximating the predictive distribution by the most likely model, on selects on average the correct model, as can be seen as follows: Assume the correct model is $\mathcal{M}^\star$, then the average of the log model evidence is equal to

$$
    \mathbb{E}_\mathcal{D} \log p(\mathcal{D}|\mathcal{M}_i) =
    \int \mathrm{d} \mathcal{D}\; p(\mathcal{D}|\mathcal{M}^\star)
        \log \frac{p(\mathcal{D}|\mathcal{M}^\star)}{p(\mathcal{D}|\mathcal{M}_i)} + \mathrm{const}
$$

I.e., it is equal to the KL divergence to the correct model, which is minimized for the correct model choice.

## The Evidence Approximation

Marginalize over some parameters and choose the maximum likelihood value for the other paramters. Also called Empircal Bayes. For example: marginalize over the weights, but determie the scale parameters by maximizing the marginal likelihood.

$$
    p(\alpha, \beta|\mathcal{D}) \propto p(\mathcal{D}| \alpha, \beta) p(\alpha, \beta)
$$

Assume flat prior and search for $\alpha, \beta$ that maximize the marginal data likelihood

$$
    p(\mathcal{D}| \alpha, \beta) =
        \int\mathrm{d}w\; p(\mathcal{D}| \mathcal{w}, \beta)p(\mathcal{w}|\alpha)
$$

Note, that both $\alpha$ and $\beta$ are scalar here.

The maximum is given by

$$
\begin{align}
    \gamma &= \sum_i \frac{\lambda_i}{\lambda_i + \alpha} \\
    \alpha &= \frac{\gamma}{m_N^T m_N} \\
    \beta^{-1} &= \frac{1}{N - \gamma} \sum_n \left( t_n - m_N^T \phi_n \right)^2
\end{align}
$$

In principle it would also be possible to first margininalize out the scale parameters and the approximate in the weight variables. However, empirically this leads to poorer results.

## Limitations of Fixed Basis Functions

```python

```


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""
# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
# https://colab.research.google.com/github/kassbohm/wb-snippets/blob/master/ipynb/HTM_03/Selbst/3.3_cc.ipynb

from mpmath import radians, degrees, pi

prec = 0.001
pprint("\nNum. results rounded to:")
pprint(prec)

# SI Units:
l = 1*meter/10
a = 1*meter/10
b = 1*meter/100
w1 = 2*pi /s

dp2, dp3 = var("delta_2, delta_3")

pprint("\nphi1 / deg:")
p1_deg = 30
# p1_deg = 60
p1 = p1_deg *pi/180
pprint(p1_deg)

c1, s1 = cos(p1), sin(p1)

p2, p3 = var("phi2, phi3",positive=True)

c2, s2 = cos(p2), sin(p2)
c3, s3 = cos(p3), sin(p3)

rB = l/2 * Matrix([c1, s1, 0])
rD = Matrix([a, -b, 0])

dBC = l * Matrix([c2, s2, 0])
dDC = l * Matrix([c3, s3, 0])

dev = rB + dBC - (rD + dDC)
xdev = dev[0] /meter
ydev = dev[1] /meter

J11, J12 = diff(xdev, p2), diff(xdev, p3)
J21, J22 = diff(ydev, p2), diff(ydev, p3)

# Jacobian:
J = Matrix([[J11, J12],[J21, J22]])

pprint("\n--- Numerical Solution using SymPy's nsolve ---")
# Initial Values:
# pprint("\nInitial (phi2, phi3) / deg:")
# pprint(Matrix([0, 90]))

pprint("\nInit:")
p2ideg, p3ideg = 0, 90
# p2ideg, p3ideg = 180, 270

pprint("(φ₂, φ₃) / deg:")
pprint(Matrix([p2ideg, p3ideg]))
p2i, p3i = radians(p2ideg) , radians(p3ideg)

sol = nsolve( [xdev, ydev], [p2, p3], [p2i, p3i], dict=True )
sol = sol[0]
p2_exact, p3_exact = sol[p2], sol[p3]

sol =  Matrix([sol[p2], sol[p3]])

pprint("\nSolution:")
pprint("(φ₂, φ₃) / deg:")
tmp = degrees(sol)
pprint(iso_round(tmp,prec))

pprint("\n--- Step-by-Step Newton-Solution ---")
pprint("\nJacobian:")
pprint(J)

# n = 1 Iteration:
pprint("\n1: Initial (phi2, phi3):")
p = Matrix([p2i, p3i])

pprint("\nin rad (radians):")
pprint(iso_round(p, prec))
pprint("\nin deg (degrees):")
pprint(iso_round(degrees(p), prec))

F = Matrix([xdev, ydev])

sub_list = [
    (p2, p2i),
    (p3, p3i),
    ]

pprint("\n2: Jacobian for (phi2, phi3):")
Jn = J.subs(sub_list)
# pprint(Jn)
pprint(iso_round(Jn, prec))

pprint("\n3: Deviation Phi for (phi2, phi3):")
Fn = F.subs(sub_list)
pprint(iso_round(Fn, 0.01))

# Newton-Step:
pprint("\n4: New (phi2, phi3):")
dp = Matrix([dp2, dp3])
eq = Eq(Jn*dp, -Fn)
sol = solve(eq,[dp2, dp3], dict=True)
sol = sol[0]
dp = Matrix([sol[dp2], sol[dp3]])
p += dp

p2_step, p3_step = p[0], p[1]

pprint("\nin rad (radians):")
pprint(iso_round(p, prec))

pprint("\nin deg (degrees):")
pprint(iso_round(degrees(p), prec))

pprint("\n--- Results: ---")
exact = True
exact = False
if exact==True:
    pprint("\nNumerical Solution using SymPy's nsolve:")
    p2, p3 =  p2_exact, p3_exact
else:
    pprint("\nAfter one Iteration Step of Newton's Method:")
    p2, p3 =  p2_step, p3_step

c2, s2 = cos(p2), sin(p2)
c3, s3 = cos(p3), sin(p3)

dBC = l * Matrix([c2, s2,0])
rC = rB + dBC

pprint("\nrC / mm:")
tmp = rC
tmp /= mm
tmp = iso_round(tmp, prec)
pprint(tmp)

pprint("\nvB / (mm/s):")
O1 = Matrix([0,0,w1])
vB = O1.cross(rB)
tmp = vB
tmp /= mm/s
tmp = iso_round(tmp, prec)
pprint(tmp)

# (φ₂, φ₃) / deg:
# ⎡38.853⎤
# ⎢      ⎥
# ⎣77.774⎦
#
# --- Step-by-Step Newton-Solution ---
#
# Jacobian:
# ⎡-sin(φ₂)    sin(φ₃) ⎤
# ⎢─────────   ─────── ⎥
# ⎢    10         10   ⎥
# ⎢                    ⎥
# ⎢ cos(φ₂)   -cos(φ₃) ⎥
# ⎢ ───────   ─────────⎥
# ⎣    10         10   ⎦
#
# 1: Initial (phi2, phi3):
#
# in rad (radians):
# ⎡ 0.0 ⎤
# ⎢     ⎥
# ⎣1.571⎦
#
# in deg (degrees):
# ⎡0.0 ⎤
# ⎢    ⎥
# ⎣90.0⎦
#
# 2: Jacobian for (phi2, phi3):
# ⎡0.0  0.1⎤
# ⎢        ⎥
# ⎣0.1  0.0⎦
#
# 3: Deviation Phi for (phi2, phi3):
# ⎡0.04 ⎤
# ⎢     ⎥
# ⎣-0.06⎦
#
# 4: New (phi2, phi3):
#
# in rad (radians):
# ⎡0.65 ⎤
# ⎢     ⎥
# ⎣1.138⎦
#
# in deg (degrees):
# ⎡37.242⎤
# ⎢      ⎥
# ⎣65.19 ⎦
#
# --- Results: ---

# Numerical Solution using SymPy's nsolve:
#
# rC / mm:
# ⎡121.178⎤
# ⎢       ⎥
# ⎢87.732 ⎥
# ⎢       ⎥
# ⎣  0.0  ⎦
#
# vB / (mm/s):
# ⎡-157.08⎤
# ⎢       ⎥
# ⎢272.07 ⎥
# ⎢       ⎥
# ⎣  0.0  ⎦

# After one Iteration Step of Newton's Method:
#
# rC / mm:
# ⎡122.91⎤
# ⎢      ⎥
# ⎢85.519⎥
# ⎢      ⎥
# ⎣ 0.0  ⎦
#
# vB / (mm/s):
# ⎡-157.08⎤
# ⎢       ⎥
# ⎢272.07 ⎥
# ⎢       ⎥
# ⎣  0.0  ⎦

```


```python
from simba.core import linear_coupling_operator_from_k_matrix, hamiltonian_from_r_matrix, make_complex_ladder_state
from sympy import Matrix, init_printing
from IPython.display import display
init_printing(use_latex='mathjax')
```

```python
r_matrix = Matrix([[-2, 1], [1, 2]])
state = make_complex_ladder_state(1)
(state.H, r_matrix, state)
```

$\displaystyle \left( \left[\begin{matrix}\overline{a_{1}} & a_{1}\end{matrix}\right], \  \left[\begin{matrix}-2 & 1\\1 & 2\end{matrix}\right], \  \left[\begin{matrix}a_{1}\\\overline{a_{1}}\end{matrix}\right]\right)$

```python
hamiltonian_from_r_matrix(r_matrix)
```

$\displaystyle \left(a_{1} - 2 \overline{a_{1}}\right) a_{1} + \left(2 a_{1} + \overline{a_{1}}\right) \overline{a_{1}}$

```python
r_matrix = Matrix([[-2, 0, 3, 1], [0, 3, 0, 2], [0, 3, 3, 0], [0, 0, 5, 1]])
state = make_complex_ladder_state(2)
(state.H, r_matrix, state)
```

$\displaystyle \left( \left[\begin{matrix}\overline{a_{1}} & a_{1} & \overline{a_{2}} & a_{2}\end{matrix}\right], \  \left[\begin{matrix}-2 & 0 & 3 & 1\\0 & 3 & 0 & 2\\0 & 3 & 3 & 0\\0 & 0 & 5 & 1\end{matrix}\right], \  \left[\begin{matrix}a_{1}\\\overline{a_{1}}\\a_{2}\\\overline{a_{2}}\end{matrix}\right]\right)$

```python
hamiltonian_from_r_matrix(r_matrix)
```

$\displaystyle \left(a_{1} - 2 \overline{a_{1}}\right) a_{1} + \left(2 a_{1} + \overline{a_{1}}\right) \overline{a_{1}}$

```python
from simba.core import transfer_function_to_state_space, SLH, concat
from sympy import symbols

s, a, b = symbols('s a b')
tf = (s - 2) / (s + 2)
system = transfer_function_to_state_space(tf).extended_to_quantum().to_physically_realisable()
g_a, g_b = system.to_slh(a), system.to_slh(b)
g_ab = concat(g_a, g_b)
g_ab
```

$$\displaystyle \left(I_{4\times4}, \left[\begin{matrix}0 & 2 & 0 & 0\\2 & 0 & 0 & 0\\0 & 0 & 0 & 2\\0 & 0 & 2 & 0\end{matrix}\right] \left[\begin{matrix}a_{1}\\\overline{a_{1}}\\b_{1}\\\overline{b_{1}}\end{matrix}\right], \frac{1}{2} \left[\begin{matrix}\overline{a_{1}} & a_{1} & \overline{b_{1}} & b_{1}\end{matrix}\right] \left[\begin{matrix}0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right] \left[\begin{matrix}a_{1}\\\overline{a_{1}}\\b_{1}\\\overline{b_{1}}\end{matrix}\right]\right)$$

```python
g_a
```

$$\displaystyle \left(I_{2\times2}, \left[\begin{matrix}0 & 2\\2 & 0\end{matrix}\right] \left[\begin{matrix}a_{1}\\\overline{a_{1}}\end{matrix}\right], \frac{1}{2} \left[\begin{matrix}\overline{a_{1}} & a_{1}\end{matrix}\right] \left[\begin{matrix}0 & 0\\0 & 0\end{matrix}\right] \left[\begin{matrix}a_{1}\\\overline{a_{1}}\end{matrix}\right]\right)$$


```python
%matplotlib inline
%load_ext autoreload
%autoreload 2

import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
import ipywidgets as widgets
from ipywidgets import interact, interactive, fixed, interact_manual

from mueller_matrices import M_Retarder, M_Diattenuator, M_rotate, M_LinearPolarizer, stokes2ellipse
from plotting import plotEllipse
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

## Some examples

### linear and circular states

```python
ax=plt.subplots(3,2)[1];
ellipt,orient=stokes2ellipse([1,1,0,0]) # Llinear 0
plotEllipse(ellipt,orient,axes=ax[0,0])

ellipt,orient=stokes2ellipse([1,-1,0,0]) # Linear 90
plotEllipse(ellipt,orient,axes=ax[0,1])

ellipt,orient=stokes2ellipse([1,0,1,0]) # Linear 45
plotEllipse(ellipt,orient,axes=ax[1,0])

ellipt,orient=stokes2ellipse([1,0,-1,0]) # Linear 135
plotEllipse(ellipt,orient,axes=ax[1,1])

ellipt,orient=stokes2ellipse([1,0,0,1]) # Linear RCP
plotEllipse(ellipt,orient,axes=ax[2,0])

ellipt,orient=stokes2ellipse([1,0,0,-1]) # Linear LCP
plotEllipse(ellipt,orient,axes=ax[2,1])

```

### Universal compensator

```python
M_uni=  M_Retarder(theta=0,delta=np.pi) * M_Retarder(theta=np.pi/4,delta=np.pi/2)

inputpol = sp.Matrix([1, 1, 0, 0])
S_outuni = M_uni_np * inputpol
ellipt,orient=stokes2ellipse(np.array(S_outuni).astype(np.float64)) # Llinear 0
plotEllipse(ellipt,orient)
```

## Test interactive plotting

```python
interact(plotEllipse,ellipt=(-1,1,0.1),\
         orient=(0,np.pi,np.pi/180),size=fixed(5),axes=fixed(None));

```

    interactive(children=(FloatSlider(value=0.0, description='ellipt', max=1.0, min=-1.0), FloatSlider(value=1.570…

```python

```