```python
%matplotlib notebook

import numpy as np

import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()

import pyflow
from pyflow.potential import MurmanCole
from pyflow.potential import CircularAirfoil
```

# Full Potential Equation

If the flow is irrotational, $\nabla \times \mathbf{u} = 0$, then the governing equations of the fluid dynamics can be determined by a potential function. Starting with the steady Euler equations,

$$
\begin{align}
& \nabla \cdot (\rho \mathbf{u}) = 0 \\
& \rho \mathbf{u} \cdot \nabla \mathbf{u} = -\nabla p
\end{align}
$$

Using the above definition,

$$
\begin{align}
& \mathbf{u} \cdot \nabla \rho + \rho \nabla \cdot \mathbf{u} = 0 \\
& \rho \left( \mathbf{u} \cdot \nabla \right) \mathbf{u} = -a^2 \nabla \rho
\end{align}
$$

Multiplying the momentum equation with the velocity vector,

$$
\mathbf{u} \cdot \left( \rho \left( \mathbf{u} \cdot \nabla \right) \mathbf{u}\right) = a^2 \mathbf{u} \cdot \nabla \rho
$$

and then using the mass conservation equation,

$$
\rho \left( \mathbf{u} \cdot \left( \mathbf{u} \cdot \nabla\right) \mathbf{u}  \right) = a^2 \rho \nabla \cdot \mathbf{u}
$$

Dividing by the density, and the rearranging

$$
\nabla \cdot \mathbf{u} - \frac{ \mathbf{u} \cdot \left( \mathbf{u} \cdot \nabla\right) }{a^2} \mathbf{u} = 0
$$

Using the definition of the potential, $\mathbf{u} = \nabla \phi$, we get the following

$$
\nabla^2 \phi - \frac{ \nabla \phi \cdot \left( \nabla \phi \cdot \nabla\right) }{a^2} \nabla \phi = 0
$$

If $a$ is the speed of sound, then we can define a local Mach number vector as

$$
\mathbf{M} = \frac{\mathbf{u}}{a} = \frac{1}{a} \nabla \phi
$$

Using this definition,

$$
\nabla^2 \phi - \mathbf{M} \cdot \left( \mathbf{M} \cdot \nabla\right) \nabla \phi = 0
$$

$$
\nabla \times \mathbf{u} = \nabla \times \left( \nabla \phi \right) = 0
$$

where in Cartesian coordinates, the velocity vector is given by the gradient of the potential, $\phi$,

$$
\mathbf{u} = \nabla \phi = \left[ \frac{\partial \phi}{\partial x} \, , \frac{\partial \phi}{\partial y} \, , \frac{\partial \phi}{\partial z} \right]
$$

## Linearization of the potential equation

In two-dimensions, the full potential equation is written as

$$
(a^2 - \phi_x^2)\phi_{xx} - 2 \phi_x \phi_y \phi_{xy} +
(a^2 - \phi_y^2)\phi_{yy} = 0
$$

where

$$
\phi_x = \frac{\partial \phi}{\partial x} \,\, \textrm{and} \,\,
\phi_{xx} = \frac{\partial^2 \phi}{\partial x^2}
$$

Let us assume the freestream flow is uniform and steady and the flow conditions are defined by $M_{\infty}$, $a_{\infty}$, $\rho_{\infty}$, etc. Now consider what happens to the flow when the gas moves over a slender object. If the disturbance or pertubation of the flow is small, then we can simplify our governing equations. We assume that velocity components are perturbed by $\hat{u}$ and $\hat{v}$, such that

$$
\begin{align}
&u = u_{\infty} + \hat{u} \\
&v = \hat{v}
\end{align}
$$

where

$$
\frac{\hat{u}}{u_{\infty}} << 1
$$

Using the definition of the potential,

$$
\hat{u} = \frac{\partial \hat{\phi}}{\partial x} = \hat{\phi}_x
\,\, \textrm{and} \,\, \hat{v} = \hat{\phi}_y
$$

which results in

$$
\begin{align}
&u = u_{\infty} + \hat{\phi}_x \\
&v = \hat{v}
\end{align}
$$

or in terms of the potential $(\phi_{\infty} = u_{\infty}x)$,

$$
\begin{align}
&\phi_x = u_{\infty} + \hat{\phi}_x \\
&\phi_y = \hat{\phi}_y
\end{align}
$$

The second derivatives of the potential are

$$
\begin{align}
&\phi_{xx} = \hat{\phi}_{xx} \\
&\phi_{yy} = \hat{\phi}_{yy} \\
&\phi_{xy} = \hat{\phi}_{xy}
\end{align}
$$

Using these expressions, we get

$$
\Big( a^2 - (u_{\infty} + \hat{\phi}_x)^2 \Big) \hat{\phi}_{xx}  - 2 (u_{\infty} + \hat{\phi}_x) \hat{\phi}_y \hat{\phi}_{xy} +
(a^2 - \hat{\phi}_y^2)\hat{\phi}_{yy} = 0
$$

Since we require that $\hat{u}$ to be much less than $u_{\infty}$, terms like

$$
\hat{\phi}_x^2, \,\,\, \hat{\phi}_y^2, \,\,\, \hat{\phi}_x\hat{\phi}_y
$$

are assumed to be negilible. Removing those terms from the above equation results in

$$
\Big( a^2 - (u_{\infty} + \hat{\phi}_x)^2 \Big) \hat{\phi}_{xx} +
a^2 \hat{\phi}_{yy} = 0
$$

$$
\Big( a^2 - u_{\infty}^2  - \hat{\phi}_x^2 - 2 u_{\infty}\hat{\phi}_x \Big)\hat{\phi}_{xx} +
a^2 \hat{\phi}_{yy} = 0
$$

and assuming $\hat{\phi}_x^2  << 1$,

$$
\Big( a^2 - u_{\infty}^2 - 2 u_{\infty} \hat{\phi}_x \Big) \hat{\phi}_{xx} + a^2 \hat{\phi}_{yy} = 0
$$

With the assumption of isentropic flow, the energy equation can be written as

$$
\frac{u^2 + v^2}{2} + \frac{a^2}{\gamma - 1} = \frac{u^2_{\infty}}{2} + \frac{a^2_{\infty}}{\gamma - 1}
$$

which we can write as

$$
\frac{(u_{\infty} + \hat{u})^2 + \hat{v}^2}{2} + \frac{a^2}{\gamma - 1} = \frac{u^2_{\infty}}{2} + \frac{a^2_{\infty}}{\gamma - 1}
$$

and can be expanded as

$$
 \frac{a^2}{\gamma - 1} = \frac{u^2_{\infty}}{2} - \frac{u^2_{\infty} + \hat{u}^2 + 2u_{\infty}\hat{u} + \hat{v}^2}{2} + \frac{a^2_{\infty}}{\gamma - 1}
$$

simplifying

$$
a^2 = a^2_{\infty} - \left({\gamma - 1} \right)\frac{\hat{u}^2 + 2u_{\infty}\hat{u} + \hat{v}^2}{2}
$$

Since $\hat{u}^2$ and $\hat{v}^2$ are much smaller than one, we can assume they are negilible,

$$
a^2 = a^2_{\infty} - (\gamma -1) u_{\infty} \hat{\phi}_x
$$

Using this expression for $a^2$, let us plug it into the linearized potential equation,

$$
\Big(a^2_{\infty} - (\gamma -1) u_{\infty} \hat{\phi}_x - u_{\infty}^2  - 2 u_{\infty} \hat{\phi}_x \Big) \hat{\phi}_{xx} +
\Big(a^2_{\infty} - (\gamma -1) u_{\infty} \hat{\phi}_x\Big) \hat{\phi}_{yy} = 0
$$

Assume

$$
\hat{\phi}_x \hat{\phi}_{yy} << 1
$$

which results in

$$
\Big(a^2_{\infty} - (\gamma -1) u_{\infty} \hat{\phi}_x - u_{\infty}^2  - 2 u_{\infty} \hat{\phi}_x \Big) \hat{\phi}_{xx} +
a^2_{\infty}  \hat{\phi}_{yy} = 0
$$

Dividing by $a_{\infty}^2$ and simplifying results in the transonic small distrubance equation is

$$
\left[ 1 - M^2_{\infty} - \left(\gamma + 1\right) M^2_{\infty} \frac{\hat{\phi}_x}{u_{\infty}} \right] \hat{\phi}_{xx} + \hat{\phi}_{yy} = 0
$$

$$
\rho \approx \rho_{\infty}\left(1 - M^2_{\infty}\left(\frac{u}{u_{\infty}} - 1\right)\right)
$$

$$
p \approx p_{\infty}\left(1 - \gamma M^2_{\infty}\left(\frac{u}{u_{\infty}} - 1\right)\right)
$$

$$
c_p = \frac{p - p_{\infty}}{\frac{1}{2}\rho_{\infty} u^2_{\infty}} \approx -2 \frac{\hat{u}}{u_{\infty}}
$$

As a result of the linearization, the transonic small distrubance equation describes the full around slender bodies, for larger distrubances, e.g., blunt-nosed airfoils, the full potential equations must be used. Both equations, however, assume the flow to be isentropic. The assumption of isentropic is that the flow is reversible, requiring only gradual changes in the thermodynamic quantities. As such the formation of a shock wave violates this assumption, since the flow is now irreversible.

**Only weak shocks.** Strong shocks increase the entropy within the flow significantly, which would violate the assumption that the gas is isentropic. The isentropic equations are approximately valid to Mach numbers less than 1.3.

# Model Elliptic Equation

The second-order partial differential equation

$$
A \phi_{xx} + \phi_{yy} = 0
$$

where $A > 0$ is classified as an elliptic PDE. Elliptic equations have no characteristic curves. For both hyperbolic and parabolic equations, information ("perturbations") travels along characteristic curves. As a result, for elliptic equations, there is no mathematical mechanism present to support the concept of information propagation. In physics, elliptic equations often describe equilibrium states or potentials, e.g., gravity, electrical, etc.

We also note that if $A < 0$, then the equation is hyperbolic. We should expect this, since in the transonic small perturbation equation, we have

$$
A =1 - M^2_{\infty} - \left(\gamma + 1\right) M^2_{\infty} \frac{\hat{\phi}_x}{u_{\infty}}
$$

which is less than one when the Mach number is greater than 1.

## Boundary conditions

Since the PDE is elliptic, we must specify a boundary condition for the far-field boundary of the domain. We will apply two different types of boundary condition. A Dirchilet boundary condition,

$$
\phi = \theta_{bdry}
$$

where the value of $\phi$ along the boundary is specified directly and held constant throughout the simulation. The other type of boundary condition is a von Neumann boundary condition,

$$
\frac{\partial \phi}{\partial n}\bigg\rvert_{bdry} = 0
$$

where the gradient of $\phi$ normal to the boundary is specified. Notice that for the potential equation, specifying the gradient is conceptually equivalent to specifying the velocity. At the surface, the velocity vector should be tangential to the body, or stated another way perpendicular to the normal vector of the body. Let the function $f(x,y)$ represent the curve of the surface, then we can construct a boundary condition from the requirement that

$$
\mathbf{u} \cdot \nabla f = 0
$$

which for $u=u_{\infty} + \hat{u}$ and $v=\hat{v}$ is

$$
\Big(u_{\infty} + \hat{u}\Big) \frac{\partial f}{\partial x} + \hat{v}\frac{\partial f}{\partial y} = 0
$$

Since $u_{\infty} >> \hat{u}$, we can state

$$
\frac{\hat{v}}{u_{\infty}} \approx - \frac{\partial f / \partial x}{\partial f / \partial y} = -\frac{\textrm{d}y}{\textrm{d}x}
$$

Since the body must be "thin" in order for the perturbations to be small, then we can approximate the velocity perturbation using a Taylor series expansion about $y=0$,

$$
\hat{v}(x,y) = \hat{v}(x,0) + \left(\frac{\partial \hat{v}}{\partial y}\right)\bigg\rvert_{y=0}y + \dots
$$

Assuming the gradient of the $\hat{v}$ is small at the body,

$$
\frac{\partial \phi}{\partial y} = \hat{v}(x,y) = \hat{v}(x,0) = u_{\infty} \left( \frac{\textrm{d}y}{\textrm{d}x}\right)_{\textrm{body}}
$$

## Simulation Domain

The chord length is defined as $c$. We define the width and height of the domain as $L_x$ and $L_y$, respectively. Let us use, $L_x = 50c$ and $L_y = 50c$, where the airfoil is centered in domain along the $y=0$ boundary. Note that because we assume the airfoil is thin (small distrubance theory), then we do not need to use a body-fitted mesh.

Even though we do not need to use a body-fitted mesh, we can visualize the circular arc airfoil using the following code.

```python
mesh = CircularAirfoil(6, stretch_mesh=True, Nx=51, Ny=51)

xx, yy = mesh.plot_circular_arc()

fig = plt.figure(figsize=(9,6))
ax = fig.add_subplot(111)

line = ax.plot(xx, yy, c='k', ls='-', label="Symmetric Airfoil")

ax.legend(fontsize=12)
ax.set_xlabel(r'$x$', fontsize=12)
ax.set_ylabel(r'$y$', fontsize=12)

ax.set_ylim((0,0.1))
```

    <IPython.core.display.Javascript object>

    (0.0, 0.1)

### Two-dimensional Mesh

We will use a $N_x \times N_y$ mesh where $N_x = N_y = 51$. In the $x$-direction, the mesh is uniformally spaced across the chord of the airfoil using 21 points and then exponentially stretched to the left and right far-field boundaries $50c$ away. In the $y$-direction, the mesh is exponentially stretched to the top far-field boundary, also $50c$ away.

The mesh spacing in the x-direction across the airfoil is

$$
\Delta x = \frac{c}{N_{\textrm{chord}} - 1}
$$

where $N_{\textrm{chord}} = 21$. We will define the leading-edge of the airfoil as the origin of our simulation domain, thus we can state,

$$
\Delta x = \frac{c}{N_{\textrm{chord}} - 1} \qquad \textrm{for} \, 0 \leq x \leq c
$$

For $x < 0$ and $x > c$, we will exponentially stretch the mesh using the remaining points. Let us evenally distribute the remaining points to the left and right and define

$$
N_{\textrm{flow}} = \frac{1}{2} \big(N_x - N_{\textrm{chord}} \big)
$$

For $N_x = 51$ and $N_{\textrm{chord}}$, $N_{\textrm{flow}} = 15$. We will use the following function to define the mesh size

$$
x_i - x_0 = \big( L_{\textrm{flow}} \big) \frac{e^{\kappa_x \left(\frac{i-1}{N_{\textrm{flow}}\,-\,1}\right)} - 1}{e^{\kappa_x} - 1}
$$

However, the parameter $\kappa_x$ is unknown, since all we are given is $L_{\textrm{flow}}$, $N_{\textrm{flow}}$, and $x_0$. To find a value of $\kappa_x$, we will specify a value for the minimum spacing, $\Delta x_{\textrm{min}}$, and then solve for $\kappa_x$.

For the $y$-direction, we use the same exponential stretching function,

$$
y_i - y_0 = \big( L_y \big) \frac{e^{\kappa_y \left(\frac{i-1}{N_y\,-\,1}\right)} - 1}{e^{\kappa_y} - 1}
$$

where once again the parameter $\kappa_y$ is unknown, since all we are given is $L_y$, $N_y$, and $y_0$. To find a value of $\kappa_y$, we will specify a value for the minimum spacing at the airfoil, $\Delta y_{\textrm{min}}$ and then solve for $\kappa_y$. We will use

$$
\Delta y_{\textrm{min}} = 0.01 \, t
$$

where $t$ is the thickness of the circular-arc airfoil.

### Solving for the stretching factor, $\kappa$

Take the $x$-direction as an example. The same procedure described below applies the $y$-direction as well. The minimum

$$
\Delta x_{\textrm{min}} = x_1 - x_0 = \big( L_{\textrm{flow}} \big) \frac{e^{\kappa \left(\frac{1}{N_{\textrm{flow}}\,-\,1}\right)} - 1}{e^{\kappa} - 1}
$$

To use Newton's method, we can define the following function,

$$
f(\kappa) = \big( L_{\textrm{flow}} \big) \frac{e^{\kappa \left(\frac{1}{N_{\textrm{flow}}\,-\,1}\right)} - 1}{e^{\kappa} - 1} - \Delta x_{\textrm{min}}
$$

By inspection of the function above, if $f(\kappa)$ is zero, then the value of $\kappa$, corresponds to the desired value of $\Delta x_{\textrm{min}}$. Starting with some intial guess for $\kappa^n$, we can compute what our next value of $\kappa^{n+1}$ should be according to the following expansion,

$$
0 = \frac{\partial f}{\partial \kappa} \big(\kappa^{n+1} - \kappa^n\big) + f(\kappa^n) = f^{\prime}(\kappa^{n}) \big[\kappa^{n+1} - \kappa^n\big] + f(\kappa^n)
$$

or

$$
\kappa^{n+1} = \kappa^n - \frac{f(\kappa^n)}{f^{\prime}( \kappa)}
$$

# Murman-Cole Method

Use the Murman-Cole method to find a solution to the transonic small distrubance equation for flow past a thin airfoil. Consider two different freestream conditions

- Case 1: Subsonic flow with $M_{\infty} = 0.5$
- Case 2: Transonic flow with $M_{\infty} = 0.908$

The thin airfoil in the above figure is symmetric and defined as a circular arc with a thickness of $t$, where

$$
t = \frac{\textrm{Thickness Percentage}}{100}
$$

Using geometry, the radius of curvature is defined as

$$
R = \frac{t c}{4} + \frac{1}{4} \frac{c^2}{t * c}
$$

Using $x = R \cos(\theta)$ and $y = R \sin(\theta)$, we can define the geometry of the airfoil. However, because the airfoil is thin and we are using the transonic, small distrubance equation, we do not need to use a body-fitted mesh. The only geometrical definition we need occurs at the airfoil surface boundary, where the flow tangency boundary condition requires,

$$
\frac{\partial \hat{\phi}}{\partial y} = V_{\infty} \left[ \frac{\textrm{d} y(x)}{\textrm{d} x}\right]_{\textrm{body}}
$$

From geometry, we know that for $0 \leq x < c/2$

$$
\frac{\textrm{d} y(x)}{\textrm{d} x}\bigg\rvert_{\textrm{body}} = \frac{t}{c}
$$

and $c/2 \leq x < c$

$$
\frac{\textrm{d} y(x)}{\textrm{d} x}\bigg\rvert_{\textrm{body}} = -\frac{t}{c}
$$

Assume that the pressure and density is normalized such that $V_{\infty}=1$ and $\rho_{\infty} = 1$ with $\gamma = 1.4$. Using $V_{\infty} = M_{\infty}a_{\infty}$,

$$
p_{\infty} = \frac{1}{\gamma} \rho_{\infty} a^2_{\infty} = \frac{1}{\gamma M_{\infty}^2}
$$

Since $\hat{\phi}$ is the distrubance of the potential, we can set it to any constant initially. Let us use, $\hat{\phi} = 1$. Remember we are interested in the derivatives of $\hat{\phi}$ with respect to $x$ and $y$. Take the coefficient of pressure, it is computed as

$$
c_p = \frac{p - p_{\infty}}{q_{\infty}}
$$

where the dynamic pressure is given by

$$
q_{\infty} = \frac{1}{2} \rho_{\infty} V^2_{\infty}
$$

and the pressure as

$$
p = p_{\infty}\left[1 + \frac{\gamma - 1}{2} M_{\infty}^2 \left(\frac{u^2 + v^2}{V_{\infty}^2} -1 \right)\right]^{\frac{\gamma}{\gamma - 1}}
$$

The solutions to the transonic, small distrubance equation give the value of the distrubance potential, $\hat{\phi}$, which is related to the velocity by

$$
\begin{align}
u &= V_{\infty} + \hat{\phi}_x = V_{\infty} + \frac{\partial \hat{\phi}}{\partial x}\\
v &= \hat{\phi}_y = \frac{\partial \hat{\phi}}{\partial y}
\end{align}
$$

## Subsonic Flow

```python
mesh = CircularAirfoil(6, stretch_mesh=True, Nx=51, Ny=51)
```

```python
flow = MurmanCole(mesh, M_inf=0.5)
```

```python
flow.solve(print_residuals=100,max_residual=1.0e-5,max_iterations=2000)
```

    100 0.03374215694566374
    200 0.009988600329961628
    300 0.0036403257472557016
    400 0.0014564515695383307
    500 0.0006150066583359148
    600 0.0002690569126571063
    700 0.00012070361844962463
    800 5.517625322681852e-05
    900 2.5592515157768503e-05
    1000 1.2009677448077127e-05

```python
fig = plt.figure(figsize=(9,6))
ax = fig.add_subplot(111)

ax.plot(mesh.x, -flow.cp[:,2], marker='o')

ax.set_xlim((-0.1,1.1))
ax.set_xlabel(r'$x$', fontsize=12)
ax.set_ylabel(r'$-c_p$', fontsize=12)
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, '$-c_p$')

```python
fig = plt.figure(figsize=(9,6))
ax = fig.add_subplot(111)

CS = plt.contour(flow.X, flow.Y, flow.M)#, levels=[0.8,  0.9, 1.0, 1.1, 1.2 ], colors=['y', 'b', 'r', 'g','k'])

cbar = fig.colorbar(CS)
cbar.ax.set_ylabel('Mach Number')

ax.minorticks_on()

ax.set_xlim((-0.25,1.25))
ax.set_ylim((0,1))

ax.set_xlabel(r'$x$', fontsize=12)
ax.set_ylabel(r'$y$', fontsize=12)
```

    <IPython.core.display.Javascript object>

    Text(0, 0.5, '$y$')

```python

fig = plt.figure(figsize=(9,6))
ax = fig.add_subplot(111)

CS = plt.pcolor(flow.X, flow.Y, flow.phi)
cbar = fig.colorbar(CS)
cbar.ax.set_ylabel('Potential')

ax.minorticks_on()

ax.set_xlim((-0.25,1.25))
ax.set_ylim((0,0.1))

ax.set_xlabel(r'$x$', fontsize=12)
ax.set_ylabel(r'$y$', fontsize=12)
```

    <IPython.core.display.Javascript object>

    /var/folders/b_/pfp74yh957398t17tw7cby8h0000gr/T/ipykernel_91103/2767174129.py:4: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
      CS = plt.pcolor(flow.X, flow.Y, flow.phi)
    /var/folders/b_/pfp74yh957398t17tw7cby8h0000gr/T/ipykernel_91103/2767174129.py:5: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
      cbar = fig.colorbar(CS)

    Text(0, 0.5, '$y$')

```python

```


This notebook has my attempts at applying Bayesian parameter estimation onto the problem of source-level estimation.

```python
import arviz as az
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(100)
import pymc3 as pm
import pandas as pda
from scipy.special import jv as bessel_firstkind
from sympy import besselj, symbols, hankel2, legendre, sin, cos, summation, I, oo, diff, pi
from sympy import Abs, lambdify
```

```python
theta_values = np.linspace(0,np.pi/2,100)
v_sound = 330 # m/s
freq = 50*10**3 # Hz
k_value = 2*np.pi/(v_sound/freq)
a_value = 5*10**-3

# choose another higher frequency
freq2 = 80*10**3
k_value2 =2*np.pi/(v_sound/freq2)

```

```python
# from the online repo: https://github.com/thejasvibr/bat_beamshapes/

def absdivide_or_zero(a,b):
    try:
        out = abs(a/b)
    except:
        out = 0
    return out

def piston_in_infinite_baffle(theta, k=k_value, a=a_value):
    '''
    Generate predictions for the classic piston vibrating in an infinite baffle.

    Parameters
    ----------
    theta: 0<=float.<=2*pi
           The angle in azimuth/elevation at which the relative sound pressure is calculated.
           The relative soundpressure is calculated in comparison to the sound pressure level
           on-axis (0degrees/radians)
    k : float>0,optional
        The wavenumber of the sound (2*pi/wavelength)
    a : float>0, optional
        The diameter/radius(?) of the piston in m.
    Returns
    --------
    rp_theta : float >=0.
               The relative sound pressure level at the given theta
    '''
    ka_sintheta = k*a*np.sin(theta)
    numerator = 2*bessel_firstkind(1, ka_sintheta)#2*bessel_firstkind(1, ka_sintheta)
    try:
        rp_theta = np.apply_along_axis(absdivide_or_zero,0,numerator,ka_sintheta)
    except:
        rp_theta = absdivide_or_zero(numerator,ka_sintheta)
    return rp_theta

#PIBF = lambdify(piston_in_infinite_baffle)

db = lambda X: 20*np.log10(X)
```

```python

```

```python

rel_levels = np.zeros((theta_values.size,3))
rel_levels[:,0] = theta_values
rel_levels[:,1] = piston_in_infinite_baffle(theta_values)
rel_levels[:,2] = db(rel_levels[:,1])

```

    <ipython-input-40-2b7b0e36b7d7>:5: RuntimeWarning: invalid value encountered in true_divide
      out = abs(a/b)

```python
%matplotlib inline
```

```python
fig = plt.figure()
ax = plt.subplot(111, projection='polar')
ax.set_theta_zero_location("N")  # theta=0 at the top
plt.plot(rel_levels[:,0],rel_levels[:,2])
```

This is a coarse idea of the how beam-shape affects the calculated source-level of bat calls. Now, assuming this the beam-shape in general, let us sample a set of random-points, and generate a set of source-level measurements with beam-shape effects included.

```python
on_axis_SL = 100 # dB SPL re 20muPa at 1m -   the mean source-level
SL_sd = 1 # a +/- 1dB call-to-call standard deviation

random_thetas = np.random.choice(np.arange(rel_levels.shape[0]),25)
# generaate
measured_SL = np.random.normal(on_axis_SL,SL_sd,random_thetas.size) + rel_levels[random_thetas,2]
```

```python
plt.figure()
plt.hist(measured_SL);
```

```python
az.summary(measured_SL,kind='stats')
```

      mean
      sd
      hdi_3%
      hdi_97%

      x
      87.99
      8.701
      75.0
      99.796

```python
plt.figure()
az.plot_kde(measured_SL)
```

#### Beam-shape effects can have pretty strong results.
As we can see above, the beam-shape effects could mean that the measured source-level will be generally under-estimated. The mean is ~82 dB SPL, while the range is between 60-100 dB SPL!

#### How do we estimate the underlying mean source-level given all the prior information at hand?
$Measured\:level = On-axis\:source\:level - beam\:shape\:effect$

Here, the two parts are described by:

* $On-axis\:source\:level = N(\mu,\sigma)$
* $beam\:shape \:effect = f(\theta)$

```python
def fn_measured_level(on_axis,theta):
    return on_axis-db(piston_in_infinite_baffle(theta))
```

```python
m00 = pm.Model()
with m00:
    theta = pm.Uniform('theta',lower=0,upper=np.pi/2,shape=measured_SL.size)
    on_axis = pm.Normal('on_axis',mu=100,sigma=1,shape=measured_SL.size)
    sim = pm.Simulator('sim',fn_measured_level,params=(on_axis,theta),
                                   observed=measured_SL, epsilon=10)
    # -- delete the parallel=True as it always caused issues initially
    trace_lv = pm.sample_smc(draws=500,kernel="ABC")
    idata_lv = az.from_pymc3(trace_lv)

```

    Initializing SMC sampler...
    Sampling 2 chains in 2 jobs
    Stage:   0 Beta: 0.052
    Stage:   1 Beta: 0.112
    Stage:   2 Beta: 0.176
    Stage:   3 Beta: 0.246
    Stage:   4 Beta: 0.321
    Stage:   5 Beta: 0.405
    Stage:   6 Beta: 0.496
    Stage:   7 Beta: 0.621
    Stage:   8 Beta: 0.783
    Stage:   9 Beta: 0.987
    Stage:  10 Beta: 1.000
    Stage:   0 Beta: 0.056
    Stage:   1 Beta: 0.120
    Stage:   2 Beta: 0.187
    Stage:   3 Beta: 0.255
    Stage:   4 Beta: 0.327
    Stage:   5 Beta: 0.420
    Stage:   6 Beta: 0.519
    Stage:   7 Beta: 0.640
    Stage:   8 Beta: 0.787
    Stage:   9 Beta: 0.984
    Stage:  10 Beta: 1.000

```python
az.plot_trace(idata_lv, kind="rank_vlines");
```

```python
out_data = idata_lv.to_dataframe()
```

```python
r = az.summary(trace_lv,kind='stats',hdi_prob=0.89)
```

    /home/autumn/anaconda3/envs/statsre/lib/python3.8/site-packages/arviz/data/io_pymc3.py:88: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
      warnings.warn(

```python
theta_summary = r.loc['theta[0]':,:]
theta_summary['hdi_width'] = theta_summary['hdi_94.5%']-theta_summary['hdi_5.5%']
theta_summary['hdiwidth_deg'] = np.degrees(theta_summary['hdi_width'])
theta_summary['mean_deg'] = np.degrees(theta_summary['mean'])
theta_summary['actual_deg'] = np.degrees(theta_values[random_thetas])

theta_summary
```

    <ipython-input-58-6461f91ae797>:2: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      theta_summary['hdi_width'] = theta_summary['hdi_94.5%']-theta_summary['hdi_5.5%']
    <ipython-input-58-6461f91ae797>:3: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      theta_summary['hdiwidth_deg'] = np.degrees(theta_summary['hdi_width'])
    <ipython-input-58-6461f91ae797>:4: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      theta_summary['mean_deg'] = np.degrees(theta_summary['mean'])
    <ipython-input-58-6461f91ae797>:5: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      theta_summary['actual_deg'] = np.degrees(theta_values[random_thetas])

      mean
      sd
      hdi_5.5%
      hdi_94.5%
      hdi_width
      hdiwidth_deg
      mean_deg
      actual_deg

      theta[0]
      0.235
      0.185
      0.010
      0.473
      0.463
      26.527946
      13.464508
      40.000000

      theta[1]
      0.253
      0.194
      0.007
      0.501
      0.494
      28.304115
      14.495832
      36.363636

      theta[2]
      0.216
      0.141
      0.029
      0.425
      0.396
      22.689129
      12.375888
      60.909091

      theta[3]
      0.213
      0.157
      0.009
      0.429
      0.420
      24.064227
      12.204001
      65.454545

      theta[4]
      0.273
      0.191
      0.014
      0.545
      0.531
      30.424059
      15.641748
      32.727273

      theta[5]
      0.444
      0.341
      0.008
      0.903
      0.895
      51.279723
      25.439326
      3.636364

      theta[6]
      0.297
      0.221
      0.015
      0.561
      0.546
      31.283496
      17.016847
      37.272727

      theta[7]
      0.366
      0.269
      0.014
      0.705
      0.691
      39.591384
      20.970255
      11.818182

      theta[8]
      0.303
      0.217
      0.019
      0.569
      0.550
      31.512679
      17.360621
      22.727273

      theta[9]
      0.294
      0.195
      0.029
      0.557
      0.528
      30.252172
      16.844959
      37.272727

      theta[10]
      0.378
      0.276
      0.034
      0.703
      0.669
      38.330876
      21.657805
      2.727273

      theta[11]
      0.435
      0.325
      0.014
      0.804
      0.790
      45.263666
      24.923664
      0.909091

      theta[12]
      0.215
      0.142
      0.032
      0.438
      0.406
      23.262086
      12.318593
      66.363636

      theta[13]
      0.224
      0.150
      0.011
      0.421
      0.410
      23.491270
      12.834255
      61.818182

      theta[14]
      0.203
      0.159
      0.006
      0.410
      0.404
      23.147495
      11.631043
      68.181818

      theta[15]
      0.273
      0.207
      0.021
      0.514
      0.493
      28.246819
      15.641748
      39.090909

      theta[16]
      0.210
      0.136
      0.015
      0.380
      0.365
      20.912960
      12.032114
      50.000000

      theta[17]
      0.312
      0.224
      0.020
      0.586
      0.566
      32.429411
      17.876283
      30.909091

      theta[18]
      0.301
      0.227
      0.011
      0.599
      0.588
      33.689918
      17.246030
      28.181818

      theta[19]
      0.280
      0.189
      0.022
      0.530
      0.508
      29.106256
      16.042818
      38.181818

      theta[20]
      0.238
      0.158
      0.031
      0.459
      0.428
      24.522594
      13.636396
      86.363636

      theta[21]
      0.235
      0.157
      0.024
      0.452
      0.428
      24.522594
      13.464508
      73.636364

      theta[22]
      0.235
      0.159
      0.022
      0.455
      0.433
      24.809073
      13.464508
      43.636364

      theta[23]
      0.349
      0.260
      0.006
      0.662
      0.656
      37.586031
      19.996227
      18.181818

      theta[24]
      0.500
      0.398
      0.006
      1.194
      1.188
      68.067386
      28.647890
      13.636364

What about the on-axis source-levels?

```python
onaxis_summary = r.loc['on_axis[0]':'on_axis[24]',:]
onaxis_summary['hdi_width'] = onaxis_summary['hdi_94.5%']-onaxis_summary['hdi_5.5%']
onaxis_summary
```

    <ipython-input-57-bf186442427d>:2: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      onaxis_summary['hdi_width'] = onaxis_summary['hdi_94.5%']-onaxis_summary['hdi_5.5%']

      mean
      sd
      hdi_5.5%
      hdi_94.5%
      hdi_width

      on_axis[0]
      99.782
      0.819
      98.475
      101.066
      2.591

      on_axis[1]
      99.908
      0.903
      98.478
      101.368
      2.890

      on_axis[2]
      99.752
      0.863
      98.328
      101.048
      2.720

      on_axis[3]
      99.621
      0.839
      98.174
      100.858
      2.684

      on_axis[4]
      99.998
      0.806
      98.719
      101.279
      2.560

      on_axis[5]
      99.875
      0.855
      98.525
      101.241
      2.716

      on_axis[6]
      99.885
      0.836
      98.638
      101.235
      2.597

      on_axis[7]
      100.046
      0.882
      98.601
      101.315
      2.714

      on_axis[8]
      99.897
      0.856
      98.565
      101.277
      2.712

      on_axis[9]
      99.846
      0.876
      98.416
      101.124
      2.708

      on_axis[10]
      100.044
      0.824
      98.756
      101.386
      2.630

      on_axis[11]
      99.868
      0.851
      98.478
      101.112
      2.634

      on_axis[12]
      100.114
      0.814
      98.935
      101.516
      2.581

      on_axis[13]
      99.792
      0.859
      98.484
      101.204
      2.720

      on_axis[14]
      99.887
      0.859
      98.528
      101.258
      2.730

      on_axis[15]
      100.012
      0.863
      98.609
      101.232
      2.623

      on_axis[16]
      99.592
      0.844
      98.107
      100.814
      2.707

      on_axis[17]
      99.738
      0.812
      98.464
      101.011
      2.547

      on_axis[18]
      99.730
      0.872
      98.309
      101.106
      2.797

      on_axis[19]
      99.837
      0.807
      98.704
      101.180
      2.476

      on_axis[20]
      99.878
      0.869
      98.636
      101.337
      2.701

      on_axis[21]
      99.855
      0.888
      98.413
      101.199
      2.786

      on_axis[22]
      99.894
      0.804
      98.689
      101.257
      2.568

      on_axis[23]
      100.035
      0.826
      98.682
      101.307
      2.625

      on_axis[24]
      99.869
      0.841
      98.468
      101.081
      2.613

### Does the upper 90%ile issue help with source-level estimation?
We've seen that when we have recorded calls from +/- 90$^{\circ}$ of the on-axis direction, the mean SL has been estimated correctly, even though there is still an okay interval of about 3 dB. We can see however, that because any observed value can be reproduced by a very large combination of on-axis source-level and angle. The $\theta$ estimate intervals are *pretty* wide (~$\pm 40^{\circ}$)

Even if $\theta$ can't be estimated all that well, does choosing the loudest-calls help in decreasing the highest density interval of the on-axis source-level?

```python
louder_sls = measured_SL[measured_SL>np.quantile(measured_SL,0.9)]
```

```python
louder_sls
```

    array([100.16627914,  99.97834302,  98.75851708])

Here, in this case we are kind of lucky as the 90%ile has the true on-axis source level range in it. But we may not always be so lucky, especially if we have a smaller sample size.

```python
# simulate choosing the loudest calls - only the upper 10%ile of all calls are chosen
# and the on-axis source-level estimated

m00 = pm.Model()
with m00:
    theta = pm.Uniform('theta',lower=0,upper=np.pi/2,shape=louder_sls.size)
    on_axis = pm.Normal('on_axis',mu=100,sigma=1,shape=louder_sls.size)
    sim = pm.Simulator('sim',fn_measured_level,params=(on_axis,theta),
                                   observed=louder_sls, epsilon=10)
    # -- delete the parallel=True as it always caused issues initially
    trace_lv = pm.sample_smc(draws=500,kernel="ABC")
    idata_lv = az.from_pymc3(trace_lv)

```

    Initializing SMC sampler...
    Sampling 2 chains in 2 jobs
    Stage:   0 Beta: 0.439
    Stage:   1 Beta: 1.000
    Stage:   0 Beta: 0.436
    Stage:   1 Beta: 1.000

```python
np.degrees(0.5)
```

    28.64788975654116

```python
loudest_summary = az.summary(idata_lv,kind='stats')
loudest_summary['hdi_width'] = np.abs(loudest_summary['hdi_3%']-loudest_summary['hdi_97%'])
loudest_summary
```

      mean
      sd
      hdi_3%
      hdi_97%
      hdi_width

      on_axis[0]
      99.977
      0.957
      98.227
      101.716
      3.489

      on_axis[1]
      99.964
      1.017
      98.037
      101.827
      3.790

      on_axis[2]
      99.930
      1.025
      97.866
      101.660
      3.794

      theta[0]
      0.460
      0.385
      0.002
      1.359
      1.357

      theta[1]
      0.447
      0.367
      0.002
      1.302
      1.300

      theta[2]
      0.433
      0.370
      0.001
      1.313
      1.312

```python
np.degrees(1.2)
```

    68.75493541569878

### Choosing the loudest calls does not lead to much better estimates of source-level

```python
# simulate the case where all the loudest calls are actually right censored
# this leaves us with <= 75%ile of all calls.
fainter_sls = measured_SL[measured_SL<np.quantile(measured_SL,0.75)]

m00b = pm.Model()
with m00b:
    theta = pm.Uniform('theta',lower=0,upper=np.pi/2,shape=fainter_sls.size)
    on_axis = pm.Normal('on_axis',mu=100,sigma=1,shape=fainter_sls.size)
    sim = pm.Simulator('sim',fn_measured_level,params=(on_axis,theta),
                                   observed=fainter_sls, epsilon=10)
    # -- delete the parallel=True as it always caused issues initially
    trace_lvb = pm.sample_smc(draws=500,kernel="ABC")
    idata_lvb = az.from_pymc3(trace_lvb)
    mapb = pm.find_MAP()

```

    Initializing SMC sampler...
    Sampling 2 chains in 2 jobs
    Stage:   0 Beta: 0.051
    Stage:   1 Beta: 0.113
    Stage:   2 Beta: 0.180
    Stage:   3 Beta: 0.251
    Stage:   4 Beta: 0.334
    Stage:   5 Beta: 0.438
    Stage:   6 Beta: 0.559
    Stage:   7 Beta: 0.720
    Stage:   8 Beta: 0.962
    Stage:   9 Beta: 1.000
    Stage:   0 Beta: 0.054
    Stage:   1 Beta: 0.116
    Stage:   2 Beta: 0.187
    Stage:   3 Beta: 0.255
    Stage:   4 Beta: 0.335
    Stage:   5 Beta: 0.430
    Stage:   6 Beta: 0.543
    Stage:   7 Beta: 0.715
    Stage:   8 Beta: 0.959
    Stage:   9 Beta: 1.000

  <progress value='1' class='' max='1' style='width:300px; height:20px; vertical-align: middle;'></progress>
  100.00% [1/1 00:00<00:00 logp = -24.669, ||grad|| = 0]

```python
fainter_summary = az.summary(idata_lvb,kind='stats')
fainter_summary['hdi_width'] = np.abs(fainter_summary['hdi_3%']-fainter_summary['hdi_97%'])
fainter_summary
```

      mean
      sd
      hdi_3%
      hdi_97%
      hdi_width

      on_axis[0]
      99.806
      0.933
      98.207
      101.594
      3.387

      on_axis[1]
      99.793
      0.981
      98.086
      101.713
      3.627

      on_axis[2]
      99.782
      0.861
      98.194
      101.419
      3.225

      on_axis[3]
      99.301
      0.886
      97.596
      100.874
      3.278

      on_axis[4]
      99.760
      0.887
      98.231
      101.485
      3.254

      on_axis[5]
      99.741
      0.910
      98.040
      101.399
      3.359

      on_axis[6]
      99.859
      0.919
      98.109
      101.480
      3.371

      on_axis[7]
      99.804
      0.920
      98.061
      101.481
      3.420

      on_axis[8]
      99.848
      0.922
      98.010
      101.389
      3.379

      on_axis[9]
      99.887
      0.960
      98.032
      101.629
      3.597

      on_axis[10]
      99.712
      0.867
      98.084
      101.228
      3.144

      on_axis[11]
      99.753
      0.923
      98.177
      101.551
      3.374

      on_axis[12]
      99.765
      0.878
      97.937
      101.218
      3.281

      on_axis[13]
      99.714
      0.892
      97.840
      101.196
      3.356

      on_axis[14]
      99.895
      0.915
      98.260
      101.641
      3.381

      on_axis[15]
      99.732
      0.938
      97.926
      101.414
      3.488

      on_axis[16]
      99.788
      0.931
      98.037
      101.453
      3.416

      on_axis[17]
      99.964
      0.913
      98.175
      101.561
      3.386

      theta[0]
      0.225
      0.164
      0.010
      0.513
      0.503

      theta[1]
      0.236
      0.185
      0.008
      0.561
      0.553

      theta[2]
      0.224
      0.163
      0.008
      0.512
      0.504

      theta[3]
      0.146
      0.101
      0.012
      0.341
      0.329

      theta[4]
      0.238
      0.159
      0.007
      0.528
      0.521

      theta[5]
      0.232
      0.150
      0.009
      0.500
      0.491

      theta[6]
      0.221
      0.167
      0.009
      0.512
      0.503

      theta[7]
      0.231
      0.167
      0.010
      0.535
      0.525

      theta[8]
      0.229
      0.164
      0.003
      0.521
      0.518

      theta[9]
      0.296
      0.220
      0.004
      0.651
      0.647

      theta[10]
      0.226
      0.161
      0.012
      0.532
      0.520

      theta[11]
      0.230
      0.148
      0.019
      0.505
      0.486

      theta[12]
      0.203
      0.158
      0.007
      0.498
      0.491

      theta[13]
      0.227
      0.184
      0.010
      0.511
      0.501

      theta[14]
      0.225
      0.156
      0.004
      0.494
      0.490

      theta[15]
      0.225
      0.151
      0.012
      0.499
      0.487

      theta[16]
      0.240
      0.159
      0.023
      0.535
      0.512

      theta[17]
      0.255
      0.213
      0.011
      0.568
      0.557

```python
mapb
```

    {'theta_interval__': array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
            0.]),
     'on_axis': array([100., 100., 100., 100., 100., 100., 100., 100., 100., 100., 100.,
            100., 100., 100., 100., 100., 100., 100.]),
     'theta': array([0.78539816, 0.78539816, 0.78539816, 0.78539816, 0.78539816,
            0.78539816, 0.78539816, 0.78539816, 0.78539816, 0.78539816,
            0.78539816, 0.78539816, 0.78539816, 0.78539816, 0.78539816,
            0.78539816, 0.78539816, 0.78539816])}

```python
np.degrees(0.785)
```

    44.97718691776962

### Using fainter-calls results in the same interval width as using the loudest calls. However, the $\theta$ estimation interval is much more reduced.

## Using beam-shapes at different frequencies to estimate on-axis source-levels
The challenge in estimating source-level using the beam-shape for one frequency is that the same measured source-level can be the result of multiple on-axis levels and angle of recording. What if we try to estimate on-axis levels by using two or more beam-shapes? Two or more beam-shapes come about in a call when there is either a harmonic or when the call itself is frequency-modulated (FM).

```python
multi_rel_levels = np.zeros((theta_values.size,3))
multi_rel_levels[:,0] = theta_values
multi_rel_levels[:,1] = db(piston_in_infinite_baffle(theta_values)) # beam shape for 50kHz
multi_rel_levels[:,2] = db(piston_in_infinite_baffle(theta_values,k=k_value2))# beam shape for 80 kHz

```

```python
plt.figure()
ax0 = plt.subplot(111,projection='polar')
ax0.set_theta_zero_location("N")  # theta=0 at the top
plt.plot(multi_rel_levels[:,0],multi_rel_levels[:,1],label='50kHz')
plt.plot(multi_rel_levels[:,0],multi_rel_levels[:,2],label='80kHz')
plt.legend()
```

The 80kHz beam-shape is more directional, and shows more variation with off-axis angle. Let's see if including this other aspect of information helps us estimate the source-level and off-axis angle better.

*We are assuming that the sound pressure levels are the same at both frequencies. This may not necessarily be true in case of harmonics, but is likely to be true in case of an FM sweep. Here we are simulating something that matches a typical FM micro-bat call.*

```python
# choose a bunch of random angles of recording and recreate beam-shapes
on_axis_SL = 100 # dB SPL re 20muPa at 1m -   the mean source-level
SL_sd = 1 # a +/- 1dB call-to-call standard deviation

num_points = random_thetas.size
random_rows = np.random.choice(np.arange(multi_rel_levels.shape[0]),num_points)
random_angles = multi_rel_levels[random_rows,0]
off_axis_reduction = multi_rel_levels[random_rows,1:]
on_axis_levels = np.random.normal(on_axis_SL, SL_sd, num_points)
multifreq_measures = off_axis_reduction+np.column_stack((on_axis_levels,on_axis_levels))
```

```python
plt.figure()
plt.plot(multifreq_measures[:,0],label='50khz')
plt.plot(multifreq_measures[:,1],label='80khz')
plt.plot(multifreq_measures[:,0]-multifreq_measures[:,1],label='Difference')
plt.legend();plt.ylabel('Apparent source level (dB SPL re 20$\mu$Pa)');plt.xlabel('Random off-axis angles')

```

We see that there can be very huge differences in the apparent source-level as measured at various angles. For instance, the difference can be upto 40 dB! One lesson I can pick-up from this plot is that perhaps it's better to measure source-levels from lower frequencies than higher frequencies using peak-amplitude etc.

So, there can be strong differences in the apparent source-levels across different frequencies. This might actually help us estimate the $\theta$ and on-axis source-level better?

```python
def fn_measured_level_2freq(on_axis,theta):
    '''Calculates the expected apparent source-level
    for 50khz and 80kHz given the default piston size.
    '''
    f50khz = on_axis-db(piston_in_infinite_baffle(theta))
    f80khz = on_axis-db(piston_in_infinite_baffle(theta,k=k_value2))

    return np.column_stack((f50khz, f80khz))
```

```python
m01 = pm.Model()
with m01:
    theta = pm.Uniform('theta',lower=0,upper=np.pi/2,shape=num_points)
    on_axis = pm.Normal('on_axis',mu=100,sigma=1,shape=num_points)
    sim = pm.Simulator('sim',fn_measured_level_2freq,params=(on_axis,theta),
                                   observed=multifreq_measures, epsilon=10)
    # -- delete the parallel=True as it always caused issues initially
    m01_trace = pm.sample_smc(draws=500,kernel="ABC")
    #idata_lv = az.from_pymc3(trace_lv)
```

    Initializing SMC sampler...
    Sampling 2 chains in 2 jobs
    Stage:   0 Beta: 0.019
    Stage:   1 Beta: 0.039
    Stage:   2 Beta: 0.058
    Stage:   3 Beta: 0.078
    Stage:   4 Beta: 0.098
    Stage:   5 Beta: 0.123
    Stage:   6 Beta: 0.149
    Stage:   7 Beta: 0.179
    Stage:   8 Beta: 0.214
    Stage:   9 Beta: 0.263
    Stage:  10 Beta: 0.319
    Stage:  11 Beta: 0.391
    Stage:  12 Beta: 0.485
    Stage:  13 Beta: 0.614
    Stage:  14 Beta: 0.784
    Stage:  15 Beta: 1.000
    Stage:   0 Beta: 0.018
    Stage:   1 Beta: 0.037
    Stage:   2 Beta: 0.059
    Stage:   3 Beta: 0.079
    Stage:   4 Beta: 0.101
    Stage:   5 Beta: 0.124
    Stage:   6 Beta: 0.149
    Stage:   7 Beta: 0.179
    Stage:   8 Beta: 0.213
    Stage:   9 Beta: 0.256
    Stage:  10 Beta: 0.304
    Stage:  11 Beta: 0.369
    Stage:  12 Beta: 0.451
    Stage:  13 Beta: 0.560
    Stage:  14 Beta: 0.704
    Stage:  15 Beta: 0.881
    Stage:  16 Beta: 1.000

```python
mtrace_summary = az.summary(m01_trace,kind='stats',hdi_prob=0.89)

```

    /home/autumn/anaconda3/envs/statsre/lib/python3.8/site-packages/arviz/data/io_pymc3.py:88: FutureWarning: Using `from_pymc3` without the model will be deprecated in a future release. Not using the model will return less accurate and less useful results. Make sure you use the model argument or call from_pymc3 within a model context.
      warnings.warn(

### Adding another beam-shape decreases $\theta$ estimation intervals

```python
theta_summary = mtrace_summary.loc['theta[0]':,:]
theta_summary['hdi_width'] = theta_summary['hdi_94.5%']-theta_summary['hdi_5.5%']
theta_summary['hdiwidth_deg'] = np.degrees(theta_summary['hdi_width'])
theta_summary
```

    <ipython-input-31-245cfa5cc629>:2: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      theta_summary['hdi_width'] = theta_summary['hdi_94.5%']-theta_summary['hdi_5.5%']
    <ipython-input-31-245cfa5cc629>:3: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      theta_summary['hdiwidth_deg'] = np.degrees(theta_summary['hdi_width'])

      mean
      sd
      hdi_5.5%
      hdi_94.5%
      hdi_width
      hdiwidth_deg

      theta[0]
      0.125
      0.071
      0.019
      0.223
      0.204
      11.688339

      theta[1]
      0.111
      0.075
      0.009
      0.207
      0.198
      11.344564

      theta[2]
      0.118
      0.064
      0.025
      0.204
      0.179
      10.255945

      theta[3]
      0.131
      0.067
      0.031
      0.223
      0.192
      11.000790

      theta[4]
      0.171
      0.124
      0.021
      0.342
      0.321
      18.391945

      theta[5]
      0.216
      0.151
      0.018
      0.402
      0.384
      22.001579

      theta[6]
      0.226
      0.161
      0.026
      0.422
      0.396
      22.689129

      theta[7]
      0.134
      0.074
      0.020
      0.236
      0.216
      12.375888

      theta[8]
      0.137
      0.090
      0.012
      0.260
      0.248
      14.209353

      theta[9]
      0.130
      0.087
      0.016
      0.251
      0.235
      13.464508

      theta[10]
      0.123
      0.067
      0.020
      0.211
      0.191
      10.943494

      theta[11]
      0.138
      0.073
      0.025
      0.241
      0.216
      12.375888

      theta[12]
      0.118
      0.067
      0.025
      0.208
      0.183
      10.485128

      theta[13]
      0.147
      0.079
      0.026
      0.254
      0.228
      13.063438

      theta[14]
      0.186
      0.119
      0.028
      0.338
      0.310
      17.761692

      theta[15]
      0.122
      0.072
      0.023
      0.226
      0.203
      11.631043

      theta[16]
      0.144
      0.088
      0.020
      0.258
      0.238
      13.636396

      theta[17]
      0.124
      0.068
      0.026
      0.220
      0.194
      11.115381

      theta[18]
      0.136
      0.073
      0.035
      0.243
      0.208
      11.917522

      theta[19]
      0.101
      0.059
      0.021
      0.180
      0.159
      9.110029

      theta[20]
      0.118
      0.069
      0.023
      0.215
      0.192
      11.000790

      theta[21]
      0.121
      0.075
      0.019
      0.227
      0.208
      11.917522

      theta[22]
      0.103
      0.068
      0.010
      0.195
      0.185
      10.599719

      theta[23]
      0.127
      0.075
      0.011
      0.228
      0.217
      12.433184

      theta[24]
      0.115
      0.074
      0.014
      0.217
      0.203
      11.631043

We see that the 89% highest density interval for all $\theta$ values is now reduced to at most 20$^{\circ}$ from a max of 40$^{\circ}$ before with only one beam-shape included. This is quite the drop in uncertainty.

```python
onaxis_summary = mtrace_summary.loc['on_axis[0]':'on_axis[24]',:]
onaxis_summary['hdi_width'] = onaxis_summary['hdi_94.5%']-onaxis_summary['hdi_5.5%']
onaxis_summary
```

    <ipython-input-32-0addd4271c23>:2: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame.
    Try using .loc[row_indexer,col_indexer] = value instead

    See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
      onaxis_summary['hdi_width'] = onaxis_summary['hdi_94.5%']-onaxis_summary['hdi_5.5%']

      mean
      sd
      hdi_5.5%
      hdi_94.5%
      hdi_width

      on_axis[0]
      99.634
      0.796
      98.397
      100.846
      2.449

      on_axis[1]
      99.555
      0.855
      98.203
      100.848
      2.645

      on_axis[2]
      99.584
      0.881
      98.033
      100.773
      2.740

      on_axis[3]
      99.613
      0.803
      98.290
      100.817
      2.527

      on_axis[4]
      99.763
      0.919
      98.423
      101.290
      2.867

      on_axis[5]
      99.914
      0.873
      98.468
      101.299
      2.831

      on_axis[6]
      99.796
      0.859
      98.421
      101.078
      2.657

      on_axis[7]
      99.820
      0.777
      98.561
      101.055
      2.494

      on_axis[8]
      99.769
      0.842
      98.504
      101.221
      2.717

      on_axis[9]
      99.521
      0.822
      98.263
      100.815
      2.552

      on_axis[10]
      99.746
      0.853
      98.444
      101.135
      2.691

      on_axis[11]
      99.922
      0.803
      98.650
      101.170
      2.520

      on_axis[12]
      99.455
      0.831
      98.067
      100.740
      2.673

      on_axis[13]
      99.497
      0.810
      98.238
      100.807
      2.569

      on_axis[14]
      100.029
      0.796
      98.567
      101.159
      2.592

      on_axis[15]
      99.562
      0.874
      98.182
      100.990
      2.808

      on_axis[16]
      99.805
      0.761
      98.635
      101.051
      2.416

      on_axis[17]
      99.563
      0.866
      98.182
      100.871
      2.689

      on_axis[18]
      99.878
      0.796
      98.725
      101.176
      2.451

      on_axis[19]
      99.321
      0.849
      98.041
      100.722
      2.681

      on_axis[20]
      99.691
      0.819
      98.410
      100.990
      2.580

      on_axis[21]
      99.589
      0.844
      98.252
      100.862
      2.610

      on_axis[22]
      99.143
      0.903
      97.785
      100.678
      2.893

      on_axis[23]
      99.776
      0.808
      98.426
      101.017
      2.591

      on_axis[24]
      99.746
      0.845
      98.352
      101.009
      2.657

There is not much drop in the on-axis level uncertainty. The same ~3 dB interval remains as in the single beam-shape example.  Remember, we have maintained everything the same (sample size, on-axis level$\pm$sd), and the only new thing is the additional beam-shape information.

## Summary
Measuring on-axis source-levels for bats flying in the field is no easy task. Even if the bat is flying towards the microphones, the call may not be directed at the mics straight away. In this notebook I have shown how explicitly modelling beam-shape effects increases the accuracy of on-axis source level estimation, when we include all calls - and not only the 'loudest' calls. Till now, researchers have used an upper percentile threshold to estimate the on-axis source levels. Using the approach outlined in this notebook, we can thus actually estimate the on-axis levels even with relatively faint calls.

```python
%load_ext watermark
%watermark -n -u -v -iv `b -w
```

    Last updated: Sun Mar 14 2021

    Python implementation: CPython
    Python version       : 3.8.5
    IPython version      : 7.19.0

    numpy     : 1.19.5
    arviz     : 0.11.0
    pymc3     : 3.11.0
    matplotlib: 3.3.4
    pandas    : 1.2.1

    Watermark: 2.1.0

```python
import os

```

```python
os.getcwd()
```

    '/home/autumn/Documents/trying_out/me-statistical-rethinking/paramsims'

```python

```


# Projeto I - Computação Centífica II
### Solução de EDOs
> Autor: Gil Miranda

> Contato: gil.neto@ufrj.br

> Repo: [@mirandagil](https://github.com/mirandagil)

```python
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.colors as plt_color
```

### Encontrando a solução da EDO - refazer

$$\frac{\mathrm{d}u}{\mathrm{d}t} = Au + b$$
Como é uma equação separável, podemos resolver da seguinte forma:
$$
\begin{align}
\frac{\mathrm{d}u}{Au + b} &= \mathrm{d}t\\
\int \frac{\mathrm{d}u}{Au + b} &= \int \mathrm{d}t\\
ln(Au+b)&= A(t+c)\\
Au &= C_0 e^{At} - b\\
u &= \frac{C_0 e^{At}}{A} - \frac{b}{A}\\
\text{Mas como A é uma matriz, a solução terá a forma}\\
u(t) &= A^{-1} C_0 e^{At} - A^{-1} b\\
\text{Para descobrirmos $C_0$}\\
u(0) &= A^{-1} C_0 e^{0} - A^{-1} b\\
C_0 &= Au_0 + b
\end{align}
$$

$$
    u(t) = e^{At}u_0+ A^{-1} (e^{At}-I)g
$$

### Verificando sanidade das matrizes

```python
def verifica_defeituosa(A): ## input: A -> Matriz
    def cauchy_schwarz(u,v): ## input: u - vetor nx1, v -> vetor nx1
        norm_u = np.linalg.norm(u)
        norm_v = np.linalg.norm(v)
        scalar_product = np.dot(u,v)
        if abs(scalar_product) == abs(norm_u * norm_v):
            return 1
        else:
            return 0

    M_eigen_vects = np.linalg.eig(A)[1] ## Retorna a matriz de autovetores em forma de coluna
    eigen_vects = [M_eigen_vects[:,i] for i in range(0,M_eigen_vects.shape[1])] ## Retorna cada autovetor como um vetor linha
    for i in range(0, len(eigen_vects)):
        for j in range(1,len(eigen_vects)):
            if i != j:
                if cauchy_schwarz(eigen_vects[i],eigen_vects[j]):
                    return 1
                    break
    return 0
```

Caso matriz diagonalizavel
$$e^{At} = Se^{\Lambda t} S^{-1} $$

```python
def e_to_diag(A, t):
    eigen = np.linalg.eig(A)
    M_eigen_vects = eigen[1] ## Retorna a matriz de autovetores em forma de coluna
    eigen_values= eigen[0]*t
    e_to = np.e**eigen_values
    diag_A = np.diag(e_to)
    inv_M = np.linalg.inv(M_eigen_vects)
    return  M_eigen_vects @ (diag_A) @ inv_M
```

Caso matriz defeituosa
$$
e^{At} = I + \frac{At}{1!} + \frac{A^2t^2}{2!} +\frac{A^3t^3}{3} +\frac{A^4t^4}{4!} + \dots
$$

```python
def e_to_defect(A, t, termo = 10):
    I = np.identity(np.shape(A)[0])
    #e_to = I + A*t + A@A * t**2/2 + (A@A@A) * t**3 /6 + (A@A@A@A) * t**4 /24 + (A@A@A@A@A) *t**5/120 +(A@A@A@A@A@A) *t**6/720
    e_to = I
    for i in range(1,termo):
        e_to += np.linalg.matrix_power(A*t, i)/np.math.factorial(i)
    return e_to
```

### Solver para ED

```python
def ed_solver(A, b, c, ts):
    is_defect = verifica_defeituosa(A)
    I = np.identity(np.shape(A)[0])
    sol = [c]
    b_zero = not np.all(b)
    if b_zero:
        for t in ts:
            if is_defect:
                e_to_A = e_to_defect(A, t)
            else:
                e_to_A = e_to_diag(A,t)
            u_n = e_to_A@c
            sol.append(u_n)
    else:
        try:
            A_inv = np.linalg.inv(A)
            for t in ts:
                if is_defect:
                    e_to_A = e_to_defect(A, t)
                else:
                    e_to_A = e_to_diag(A,t)
                u_n = e_to_A@c + A_inv@(e_to_A-I)@b
                sol.append(u_n)
        except:
            print("A matriz A não é inversível, e portanto o problema não tem solução pois b é não nulo.")
    return np.array(sol)
```

### Função para exibir solução de uma edo dada a partir de uma Matriz

```python
def exibe_solucao_1(A, b, c, tf = 10):
    ts = np.arange(0, tf, 0.1)
    sol = ed_solver(A, b, c, ts)
    sol = sol[1:]
    sol_T = sol.T
    i = 0
    n = len(sol_T)
#     colors = plt.cm.rainbow(np.linspace(0,5,n))
    plt.figure(figsize = (10,5))
    for s in sol_T:
        i += 1
        plt.plot(ts, s, label = '$x_'+ str(i) + '$')
    plt.legend(bbox_to_anchor = (1,1))
    plt.title('Solução da Equação Diferencial Matricial', fontweight = 'bold')
    plt.xlabel('tempo (t)')
    plt.grid(alpha=0.3)
    plt.show()
```

```python
exibe_solucao_1(A,[0,0],np.array([3,0]))
```

```python
ts = np.arange(0,30,0.05)
A = np.array([[0,1],[-3,0]])
b = [0,0]
c = [-3,0]
exibe_solucao_1(A, b, c)

```

```python
ts = np.arange(0,30,0.05)
A = np.array([[0,1],[-1,1]])
b = [0,-1]
c = [0,1]
exibe_solucao_1(A, b, c)

```

```python
B = np.array([[-1,0,0],[1,-1,0],[0,1,-1]])
```

### Gerar matriz A, dada uma equação diferencial

A função `gen_mAb(eq)` gera a matriz $A$ e o vetor $b$ dada uma lista com coeficientes da equação diferencial, para a forma
$$\frac{\mathrm{d}u}{\mathrm{d}t} = Au + b$$

```python
def gen_mAb(eq): ## Input -> eq: lista de coeficientes da ed.
    col_len = len(eq) - 2 ## Armazena o tamanho da matrix A nxn, que é o tamanho do array retirados dois coeficientes
    A = np.zeros(shape=(col_len, col_len)) ## Gera a matriz A com zeros
    I = np.eye(col_len) ## Gera uma matriz identidade
    I[0] = I[0]*0 ## Transforma a primeira linha de I em zeros
    I_desl = np.roll(I, -1, 0) ## Desloca a matriz identidade para a direita
    b = np.zeros(col_len) ## Gera o vetor b com zeros
    b[-1] = eq[-1] ## Insere como último elemento de b a constante c
    for (i,j) in zip(np.flip(eq[1:-1]),range(0,col_len)): ## Insere os coeficientes na última linha da matriz A
        A[-1][j] = -i/eq[0]
    A = A + I_desl ## Soma A com a identidade deslocada
    return A, b ## Output -> A: Matriz do sistema de edos de primeira ordem. b: Vetor constante

```

$2y'' -y' + 5y = 0$

```python
eq = [-1,-2,1,-1]
c = [1,1]
```

### exibe_solucao_2

```python
def exibe_solucao_2(eq,c, tf = 10):
    A, b = gen_mAb(eq) ## Chama o metódo para gerar a matriz A associada ao sistema linear de ordem 1
    c = np.flip(c) ## Inverte o vetor de condições iniciais
    exibe_solucao_1(A, b, c, tf = tf) ## Chama o metódo de exibir solução dada uma Matriz
```

```python
exibe_solucao_2(eq,[1,1])
```

```python

```


# An Introduction to Bayesian Inference

### Agenda

* What is Bayesian inference ?
* Background
* Comparison with Frequentist approach
* Challenges in practical application
* Tools available
* Example-1
* Example-2
* Summary

## What is Bayesian inference ?

### Inference involves :

* Some **unknown quantities** about which we are interested in learning or testing. We call these parameters.
* Some **data** which have been observed, and hopefully contain information about these parameters.
* One or more **models** that relate the data to the parameters, and is the instrument that is used to learn.

### Bayesian Inference :

> Practical methods for making inferences from data using probability models for quantities we observe and about which we wish to learn.
*-- Gelman et al. 2013*

A Bayesian model is described by parameters, uncertainty in those parameters is described using probability distributions.

All conclusions from Bayesian statistical procedures are stated in terms of **probability statements**

## Background

As the name suggests, Bayesian inference and Bayesian method was first thought by Thomas Bayes. He was analyzing a problem: **Can we use evidence about the natural world around us to make rational conclusion about the existence of God.**

And he came up with a principle (in ~1763):

We modify our initial guess with objective new information to get a new and improved belief, which in turn carries a commitment to update that belief each time a new piece of information arrives.

Bayes' system was: **Initial Belief + New Data -> Improved Belief**

Laplace rederived it (in 1774) and called it **"Probability of Causes"**. We used this technique to calculate the mass of Saturn, amongst other things.

He came up with this formula:

\begin{equation} \Pr(M~|~D,I) = \dfrac{\Pr(D~|~M,I)\times\Pr(M ~|~I)}{\Pr(D~|~I)} \label{eqn:lapbayes} \end{equation}

where $M$ is the mass of the Saturn ,
       $D$ is data from various measurements of planetary orbits,
 and $I$ is existing background information, e.g., Newton’s laws of celestial mechanics

His estimate is just **0.6%** off of the currently accepted value for  𝑀

This is basis of the formula that we studied in high school

\\[P(A|B) = \frac{P(B|A) * P(A)}{P(B)}\\]

* P(B|A) is a conditional probability: the probability of event B occurring given that A has occured.
* P(A|B) is also a conditional probability: the probability of event A occuring given that B has occured
* P(A), P(B) are the probabilities of A and B occuring

### Bayesian vs Frequentist Statistics: *What's the difference?*

### The Frequentist World View

- The **data** that have been observed are considered **random**, because they are realizations of random processes, and hence will vary each time one goes to observe the system.
- Model **parameters** are considered **fixed**. A parameter's true value is uknown and fixed, and so we *condition* on them.

In mathematical notation, this implies a (very) general model of the following form:

\\[f(y | \theta)\\]

Here, the model \\(f\\) accepts data values \\(y\\) as an argument, conditional on particular values of \\(\theta\\).

### The Bayesian World View

- Data are considered **fixed**. They used to be random, but once they were written into your lab notebook/spreadsheet/IPython notebook they do not change.
- Model parameters themselves may not be random, but Bayesians use probability distribtutions to describe their uncertainty in parameter values, and are therefore treated as **random**.

This implies the following form:

\\[p(\theta | y)\\]

This formulation used to be referred to as ***inverse probability***, because it infers from observations to parameters, or from effects to causes.

Bayesians do not seek new estimators for every estimation problem they encounter. There is only one estimator for Bayesian inference: **Bayes' Formula**.

## Bayes' Formula

Bayesian inference is based on Bayes Formula

The equation expresses how our belief about the value of \\(\theta\\), as expressed by the **prior distribution** \\(P(\theta)\\) is reallocated following the observation of the data \\(y\\).

The innocuous denominator \\(P(y)\\) usually cannot be computed directly, and is actually the expression in the numerator, integrated over all \\(\theta\\):

\\[Pr(\theta|y) = \frac{Pr(y|\theta)Pr(\theta)}{\int Pr(y|\theta)Pr(\theta) d\theta}\\]

The intractability of this integral is one of the factors that has contributed to the under-utilization of Bayesian methods by statisticians.

### Priors

Once considered a controversial aspect of Bayesian analysis, the prior distribution characterizes what is known about an unknown quantity before observing the data from the present study. Thus, it represents the information state of that parameter. It can be used to reflect the information obtained in previous studies, to constrain the parameter to plausible values, or to represent the population of possible parameter values, of which the current study's parameter value can be considered a sample.

### Likelihood functions

The likelihood represents the information in the observed data, and is used to update prior distributions to posterior distributions. This updating of belief is justified becuase of the **likelihood principle**, which states:

> Following observation of \\(y\\), the likelihood \\(L(\theta|y)\\) contains all experimental information from \\(y\\) about the unknown \\(\theta\\).

Bayesian analysis satisfies the likelihood principle because the posterior distribution's dependence on the data is **only through the likelihood**. In comparison, most frequentist inference procedures violate the likelihood principle, because inference will depend on the design of the trial or experiment.

## Challenges

Even though Bayesian approach seems a very natual way of dealing with problems, it has historically had two main challenges

* Priors and the subjectivity associated with the Priors

* Calculating the normalizing constant

### Problem with the priors

Historically priors have been a problem. The main criticism for Bayesian inference and Priors has been

* They are subjective and we come up with them before seeing the data. Negates: 'Lets the data speak for itself'

* Probability as 'The degree of Uncertainty' was not the common view. Instead the frequentist view of probabilty as the 'Ratio of Occurences' held sway.

### Problem of computation

The innocuous denominator \\(P(y)\\) usually cannot be computed directly, and is actually the expression in the numerator, integrated over all \\(\theta\\).

\\[{\int Pr(y|\theta)Pr(\theta) d\theta}\\]

If we have n parameters we need to integrated over n variables and it can be solved analytically and this prevented Bayesian Inference to be applied to any complex real work problem.

The solution for this was Sampling and fast computers

### Monte Carlo sampling techniques

What the method produces for us is an approximation of the posterior distribution, p(θ|D), in the form of a large number of θ values sampled from that distribution. This heap of representative θ values can be used to estimate the central tendency of the posterior, its highest density interval (HDI), etc. The posterior distribution is estimated by randomly generating a lot of values from it

The goal in Bayesian inference is to get an accurate representation of the posterior distribution. One way to do that is to sample a large number of representative points from the posterior. The question then becomes this: How can we sample a large number of representative values from a distribution?

#### Metropolis

The simplest sampling solution is the **Metropolis-Hastings** algorithm

One of the simplest and most flexible algorithms for generating reversible Markov chains is the Metropolis-Hastings algorithm. Since we cannot sample directly from the (unknown) posterior distribution, this algorithm employs an **auxilliary distribution** that is easy to sample from. These samples generate candidate state transitions, which are accepted or rejected probabilistically.

Consider a simple Metropolis-Hastings algorithm for a single parameter, $\theta$. We will use a well-known statistical distribution like Normal distribution to produce candidate variables $q_t(\theta^{\prime} | \theta)$. Each generated value, $\theta^{\prime}$, is a *possible* next value for
$\theta$ at step $t+1$.

Whether or not $\theta^{\prime}$ is accepted depends on the relative probability of the new value versus the current value, weighted by the probabilities of the two values under the proposal distribution:

$$a(\theta^{\prime},\theta) = \frac{q_t(\theta^{\prime} | \theta) \pi(\theta^{\prime})}{q_t(\theta | \theta^{\prime}) \pi(\theta)}$$

This is the **acceptance ratio**, and is used as a threshold value for a uniform random draw that determines acceptance:

$$
\theta^{(t+1)} =
\begin{cases}
\theta^{\prime}  & \text{with prob. } p = \min(a(\theta^{\prime},\theta^{(t)}),1)\cr
\theta^{(t)}  & \text{with prob } 1 - p
\end{cases}
$$

This transition kernel implies that movement is not guaranteed at every step. It only occurs if the suggested transition is likely based on the acceptance ratio.

A single iteration of the Metropolis-Hastings algorithm proceeds as follows:

1.  Sample $\theta^{\prime}$ from $q(\theta^{\prime} | \theta^{(t)})$.

2.  Generate a Uniform[0,1] random variate $u$.

3.  If $a(\theta^{\prime},\theta) > u$ then
    $\theta^{(t+1)} = \theta^{\prime}$, otherwise
    $\theta^{(t+1)} = \theta^{(t)}$.

#### NUTS (No UTurns Sampler)

For a future session.

## Libraries

There are a bunch of libraries available which can insolving problem using Bayesian inference.

* WinBugs
* **Stan** and pyStan   (NOTE: TimeSeries libary Prophet from Facebook is based on Stan)
* Edward
* **Pymc3**
* Pymc4 (To be released)

## Summary

Bayesian inference provides a powerful methods to solve problem with data, inference and prediction.

In our context, Bayesian tools can be effective for use cases where

* There is a large domain expert knowledge which can be used as prior
* There is a inflow for new labelled data which can be used to update the model to create new posterior
* There is a need to interpret and reason the model predictions
* Sufficient HW resources are avaiable. Bayesian tools are more comptutionally than classical ML algorithms but less so than Deep Neural Net

References:

* [Doing Bayesian Data Analysis](http://www.r-5.org/files/books/computers/algo-list/statistics/data-mining/John_K_Kruschke-Doing_Bayesian_Data_Analysis-EN.pdf)

* [Introduction to Hamiltonian Monte Carlo](https://arxiv.org/pdf/1701.02434.pdf)

* [History of Bayes Theorem](https://www.lesswrong.com/posts/RTt59BtFLqQbsSiqd/a-history-of-bayes-theorem)


# Chapter 16:Training neural networks

```python
import matplotlib.pyplot as plt
import numpy as np
```

## 0.1 Classifying data with neural networks

# 16.1 Classifying images of handwritten digits

## 16.1.1 Building the 64-dimensional image vectors

```python
from sklearn import datasets
digits = datasets.load_digits()
```

```python
digits.images[0]
```

    array([[ 0.,  0.,  5., 13.,  9.,  1.,  0.,  0.],
           [ 0.,  0., 13., 15., 10., 15.,  5.,  0.],
           [ 0.,  3., 15.,  2.,  0., 11.,  8.,  0.],
           [ 0.,  4., 12.,  0.,  0.,  8.,  8.,  0.],
           [ 0.,  5.,  8.,  0.,  0.,  9.,  8.,  0.],
           [ 0.,  4., 11.,  0.,  1., 12.,  7.,  0.],
           [ 0.,  2., 14.,  5., 10., 12.,  0.,  0.],
           [ 0.,  0.,  6., 13., 10.,  0.,  0.,  0.]])

```python
plt.imshow(digits.images[0], cmap=plt.cm.gray_r)
```

FYI, a way to overlay numbers on the pixels showing their brightness values

```python
plt.imshow(digits.images[0], cmap=plt.cm.gray_r)
for i in range(0,8):
    for j in range(0,8):
        plt.gca().text(i-0.15,j,int(digits.images[0][i][j]))
```

```python
np.matrix.flatten(digits.images[0])
```

    array([ 0.,  0.,  5., 13.,  9.,  1.,  0.,  0.,  0.,  0., 13., 15., 10.,
           15.,  5.,  0.,  0.,  3., 15.,  2.,  0., 11.,  8.,  0.,  0.,  4.,
           12.,  0.,  0.,  8.,  8.,  0.,  0.,  5.,  8.,  0.,  0.,  9.,  8.,
            0.,  0.,  4., 11.,  0.,  1., 12.,  7.,  0.,  0.,  2., 14.,  5.,
           10., 12.,  0.,  0.,  0.,  0.,  6., 13., 10.,  0.,  0.,  0.])

```python
np.matrix.flatten(digits.images[0]) / 15
```

    array([0.        , 0.        , 0.33333333, 0.86666667, 0.6       ,
           0.06666667, 0.        , 0.        , 0.        , 0.        ,
           0.86666667, 1.        , 0.66666667, 1.        , 0.33333333,
           0.        , 0.        , 0.2       , 1.        , 0.13333333,
           0.        , 0.73333333, 0.53333333, 0.        , 0.        ,
           0.26666667, 0.8       , 0.        , 0.        , 0.53333333,
           0.53333333, 0.        , 0.        , 0.33333333, 0.53333333,
           0.        , 0.        , 0.6       , 0.53333333, 0.        ,
           0.        , 0.26666667, 0.73333333, 0.        , 0.06666667,
           0.8       , 0.46666667, 0.        , 0.        , 0.13333333,
           0.93333333, 0.33333333, 0.66666667, 0.8       , 0.        ,
           0.        , 0.        , 0.        , 0.4       , 0.86666667,
           0.66666667, 0.        , 0.        , 0.        ])

## 16.1.2 Building a random digit classifier

```python
def random_classifier(input_vector):
    return np.random.rand(10)
```

```python
v = np.matrix.flatten(digits.images[0]) / 15.
result = random_classifier(v)
result
```

    array([0.09609494, 0.75660412, 0.78242862, 0.43881297, 0.33470396,
           0.92745083, 0.9695724 , 0.12471611, 0.01307655, 0.61261849])

**NOTE** because this is random, you will get a different digit result when you re-run the code.

```python
list(result).index(max(result))
```

    6

```python
digits.target[0]
```

    0

## 16.1.3 Measuring performance of the digit classifier

```python
def test_digit_classify(classifier,test_count=1000):
    correct = 0 #<1>
    for img, target in zip(digits.images[:test_count], digits.target[:test_count]): #<2>
        v = np.matrix.flatten(img) / 15. #<3>
        output = classifier(v) #<4>
        answer = list(output).index(max(output)) #<5>
        if answer == target:
            correct += 1 #<6>
    return (correct/test_count) #<7>
```

```python
test_digit_classify(random_classifier)
```

    0.089

## 16.1.4 Exercises

**Exercise:** Suppose a digit classifier function outputs the following NumPy array.  What digit does it think the image represents?

```python
np.array([5.00512567e-06, 3.94168539e-05, 5.57124430e-09, 9.31981207e-09,
       9.98060276e-01, 9.10328786e-07, 1.56262695e-03, 1.82976466e-04,
       1.48519455e-04, 2.54354113e-07])
```

    array([5.00512567e-06, 3.94168539e-05, 5.57124430e-09, 9.31981207e-09,
           9.98060276e-01, 9.10328786e-07, 1.56262695e-03, 1.82976466e-04,
           1.48519455e-04, 2.54354113e-07])

**Mini project:** Find the average of all the images of nines in the data set, in the same way we took averages of images in Chapter 6.  Plot the resulting image. What does it look like?

```python
def average_img(i):
    imgs = [img for img,target in zip(digits.images[1000:], digits.target[1000:]) if target==i]
    return sum(imgs) / len(imgs)
```

```python
plt.imshow(average_img(9), cmap=plt.cm.gray_r)
```

**Mini project:** Build a better classifier than a random one by finding the average image of each kind of digit in the test data set, and comparing a target image with all of the averages.  Specifically, return a vector of the dot products of the target image with each average digit image.

```python
avg_digits = [np.matrix.flatten(average_img(i)) for i in range(10)]
def compare_to_avg(v):
    return [np.dot(v,avg_digits[i]) for i in range(10)]
```

```python
test_digit_classify(compare_to_avg)
```

    0.853

# 16.2 Designing a neural network

## 16.2.1 Organizing neurons and connections

## 16.2.2 Data flow through a neural network

## 16.2.3 Calculating activations

## 16.2.4 Calculating activations in matrix notation

## 16.2.5 Exercises

# 16.3 Building a neural network in Python

## 16.3.1 Implementing an MLP class in Python

```python
class MLP():
    def __init__(self,layer_sizes): #<1>
        self.layer_sizes = layer_sizes
        self.weights = [
            np.random.rand(n,m) #<2>
            for m,n in zip(layer_sizes[:-1],layer_sizes[1:]) #<3>
        ]
        self.biases = [np.random.rand(n) for n in layer_sizes[1:]] #<4>
```

```python
nn = MLP([2,3])
```

**NOTE** these numbers are randomly initialized, so your results below will vary.

```python
nn.weights
```

    [array([[0.04098182, 0.61121605],
            [0.99402839, 0.43278994],
            [0.14826355, 0.39370373]])]

```python
nn.biases
```

    [array([0.62871743, 0.55151304, 0.91445791])]

## 16.3.2 Evaluating the MLP

```python
from math import exp
def sigmoid(x):
    return 1 / (1+exp(-x))
```

```python
class MLP():
    def __init__(self,layer_sizes): #<1>
        self.layer_sizes = layer_sizes
        self.weights = [
            np.random.rand(n,m) #<2>
            for m,n in zip(layer_sizes[:-1],layer_sizes[1:]) #<3>
        ]
        self.biases = [np.random.rand(n) for n in layer_sizes[1:]] #<4>
    def feedforward(self,v):
        activations = [] #<1>
        a = v
        activations.append(a) #<2>
        for w,b in zip(self.weights, self.biases): #<3>
            z = w @ a + b #<4>
            a = [sigmoid(x) for x in z] #<5>
            activations.append(a) #<6>
        return activations
    def evaluate(self,v):
        return np.array(self.feedforward(v)[-1])
```

## 16.3.3 Testing the classification performance of an MLP

```python
nn = MLP([64,16,10])
```

```python
v = np.matrix.flatten(digits.images[0]) / 15.
```

```python
nn.evaluate(v)
```

    array([0.99996588, 0.99829151, 0.99979785, 0.9998958 , 0.99991507,
           0.99982444, 0.9999176 , 0.99863889, 0.99977906, 0.99887847])

```python
test_digit_classify(nn.evaluate)
```

    0.099

## 16.3.4 Exercises

# 16.4 Training a neural network using gradient descent

## 16.4.1 Framing training as a minimization problem

## 16.4.2 Calculating gradients with backpropagation

## 16.4.3 Automatic training with scikit-learn

```python
x = np.array([np.matrix.flatten(img) for img in digits.images[:1000]]) / 15.0
y = digits.target[:1000]
```

```python
from sklearn.neural_network import MLPClassifier

mlp = MLPClassifier(hidden_layer_sizes=(16,), #<1>
                    activation='logistic', #<2>
                    max_iter=100, #<3>
                    verbose=10, #<4>
                    random_state=1, #<5>
                    learning_rate_init=.1) #<6>
```

```python
mlp.fit(x,y)
```

    Iteration 1, loss = 2.21958598
    Iteration 2, loss = 1.56912978
    Iteration 3, loss = 0.98970277
    Iteration 4, loss = 0.57473464
    Iteration 5, loss = 0.34048448
    Iteration 6, loss = 0.21495855
    Iteration 7, loss = 0.14366771
    Iteration 8, loss = 0.11077020
    Iteration 9, loss = 0.08764273
    Iteration 10, loss = 0.07193546
    Iteration 11, loss = 0.06020348
    Iteration 12, loss = 0.04961899
    Iteration 13, loss = 0.03979645
    Iteration 14, loss = 0.03334502
    Iteration 15, loss = 0.02996006
    Iteration 16, loss = 0.02603968
    Iteration 17, loss = 0.02355514
    Iteration 18, loss = 0.02137348
    Iteration 19, loss = 0.01967878
    Iteration 20, loss = 0.01751214
    Iteration 21, loss = 0.01617330
    Iteration 22, loss = 0.01460386
    Iteration 23, loss = 0.01408517
    Iteration 24, loss = 0.01270504
    Iteration 25, loss = 0.01191634
    Iteration 26, loss = 0.01114222
    Iteration 27, loss = 0.01045989
    Iteration 28, loss = 0.00983648
    Iteration 29, loss = 0.00920912
    Iteration 30, loss = 0.00890851
    Iteration 31, loss = 0.00843426
    Iteration 32, loss = 0.00796039
    Iteration 33, loss = 0.00749839
    Iteration 34, loss = 0.00726271
    Iteration 35, loss = 0.00673963
    Iteration 36, loss = 0.00655405
    Iteration 37, loss = 0.00626207
    Iteration 38, loss = 0.00600639
    Iteration 39, loss = 0.00581857
    Iteration 40, loss = 0.00557529
    Iteration 41, loss = 0.00533573
    Iteration 42, loss = 0.00519479
    Iteration 43, loss = 0.00505128
    Iteration 44, loss = 0.00490121
    Iteration 45, loss = 0.00469161
    Iteration 46, loss = 0.00459590
    Iteration 47, loss = 0.00464844
    Iteration 48, loss = 0.00445157
    Iteration 49, loss = 0.00425515
    Iteration 50, loss = 0.00424934
    Iteration 51, loss = 0.00397800
    Iteration 52, loss = 0.00399927
    Iteration 53, loss = 0.00383932
    Iteration 54, loss = 0.00372439
    Iteration 55, loss = 0.00361744
    Iteration 56, loss = 0.00356447
    Iteration 57, loss = 0.00345899
    Iteration 58, loss = 0.00336792
    Iteration 59, loss = 0.00330330
    Iteration 60, loss = 0.00321734
    Training loss did not improve more than tol=0.000100 for two consecutive epochs. Stopping.

    MLPClassifier(activation='logistic', alpha=0.0001, batch_size='auto',
           beta_1=0.9, beta_2=0.999, early_stopping=False, epsilon=1e-08,
           hidden_layer_sizes=(16,), learning_rate='constant',
           learning_rate_init=0.1, max_iter=100, momentum=0.9,
           nesterovs_momentum=True, power_t=0.5, random_state=1, shuffle=True,
           solver='adam', tol=0.0001, validation_fraction=0.1, verbose=10,
           warm_start=False)

```python
mlp._predict(x)[0]
```

    array([9.99766643e-01, 8.43331208e-11, 3.47867059e-06, 1.49956270e-07,
           1.88677660e-06, 3.44652605e-05, 6.23829017e-06, 1.09043503e-04,
           1.11195821e-07, 7.79837557e-05])

```python
def sklearn_trained_classify(v):
    return mlp._predict([v])[0]
```

```python
test_digit_classify(sklearn_trained_classify)
```

    1.0

## 16.4.4 Exercises

**Exercise:** Modify the `test_digit_classify` function to work on a custom range of examples in the test set.  How does it do on the next 500 examples after the 1,000 training examples?

```python
def test_digit_classify(classifier,start=0,test_count=1000):
    correct = 0
    end = start + test_count #<1>
    for img, target in zip(digits.images[start:end], digits.target[start:end]): #<2>
        v = np.matrix.flatten(img) / 15.
        output = classifier(v)
        answer = list(output).index(max(output))
        if answer == target:
            correct += 1
    return (correct/test_count)
```

```python
test_digit_classify(sklearn_trained_classify,start=1000,test_count=500)
```

    0.962

**Exercise:** Using the squared distance cost function, what is the cost of your randomly-generated MLP for the first 1,000 training examples?  What is the cost of the scikit-learn MLP?

```python
def y_vec(digit):
    return np.array([1 if i == digit else 0 for i in range(0,10)])
```

```python
def cost_one(classifier,x,i):
    return sum([(classifier(x)[j] - y_vec(i)[j])**2 for j in range(10)])
```

```python
def total_cost(classifier):
    return sum([cost_one(classifier,x[j],y[j]) for j in range(1000)])/1000.
```

```python
total_cost(nn.evaluate)
```

    8.990834701722013

```python
total_cost(sklearn_trained_classify)
```

    5.670512721637246e-05

**Mini-project:** Extract the MLPClassifier weights and biases using its properties called `coefs_` and `intercepts_`, respectively.  Plug these weights and biases into the MLP class we built from scratch and show that your resulting MLP performs well on digit classification.

```python
nn = MLP([64,16,10])
```

```python
nn.weights = [w.T for w in mlp.coefs_]
```

```python
nn.biases = mlp.intercepts_
```

```python
test_digit_classify(nn.evaluate,start=1000,test_count=500)
```

    0.962

# 16.5 Calculating gradients with backpropagation

## 16.5.1 Finding the cost in terms of the last layer weights

## 16.5.1 Finding the cost in terms of the last layer weights

## 16.5.2 Calculating the partial derivatives for the last layer weights using the chain rule

## 16.5.3 Exercises

**Mini-project:** Use SymPy or your own code from chapter 10 to automatically find the derivative of the sigmoid function:

$$\sigma(x) = \frac{1}{1+e^{-x}}$$

Show that the answer you get is equal to $\sigma(x)(1-\sigma(x))$.

```python
from sympy import *
X = symbols('x')
diff(1 / (1+exp(-X)),X)
```

    exp(-x)/(1 + exp(-x))**2

```python

```


*Sebastian Raschka*
last modified: 03/31/2014

I am really looking forward to your comments and suggestions to improve and extend this tutorial! Just send me a quick note
via Twitter: [@rasbt](https://twitter.com/rasbt)
or Email: [bluewoodtree@gmail.com](mailto:bluewoodtree@gmail.com)

### Problem Category
- Statistical Pattern Recognition
- Supervised Learning
- Parametric Learning
- Bayes Decision Theory
- Univariate data
- 2-class problem
- equal variances
- equal priors
- Gaussian model (2 parameters)
- No Risk function

# Sections

• Given information

• Deriving the decision boundary

• Plotting the class conditional densities, posterior probabilities, and decision boundary

• Classifying some random example data

• Calculating the empirical error rate

## Given information:

[back to top]

####model: continuous univariate normal (Gaussian) model for the class-conditional densities

$ p(x | \omega_j) \sim N(\mu|\sigma^2) $

$ p(x | \omega_j) \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu}{\sigma}\bigg)^2 \bigg] } $

####Prior probabilities:

$ P(\omega_1) = P(\omega_1) = 0.5 $

#### Variances of the sample distributions

$ \sigma_1^2 = \sigma_2^2 = 1 $

#### Means of the sample distributions

$ \mu_1 = 4, \quad \mu_2 = 10 $

## Deriving the decision boundary
[back to top]

### Bayes' Rule:

$ P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)} $

###Bayes' Decision Rule:

Decide $ \omega_1 $ if $ P(\omega_1|x) > P(\omega_2|x) $ else decide $ \omega_2 $.

\begin{equation}
\Rightarrow \frac{p(x|\omega_1) * P(\omega_1)}{p(x)} > \frac{p(x|\omega_2) * P(\omega_2)}{p(x)}
\end{equation}

We can drop $ p(x) $ since it is just a scale factor.

$ \Rightarrow P(x|\omega_1) * P(\omega_1) > p(x|\omega_2) * P(\omega_2) $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{P(\omega_2)}{P(\omega_1)} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{0.5}{0.5} $

$ \Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > 1 $

$ \Rightarrow \frac{1}{\sqrt{2\pi\sigma_1^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } > \frac{1}{\sqrt{2\pi\sigma_2^2}} \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } $

Since we have equal variances, we can drop the first term completely.

$ Rightarrow \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_1}{\sigma_1}\bigg)^2 \bigg] } >  \exp{ \bigg[-\frac{1}{2}\bigg( \frac{x-\mu_2}{\sigma_2}\bigg)^2 \bigg] } \quad\quad \bigg| \;ln, \quad \mu_1 = 4, \quad \mu_2 = 10, \quad \sigma=1 $

$ \Rightarrow -\frac{1}{2} (x-4)^2  > -\frac{1}{2} (x-10)^2  \quad \bigg| \; \times(-2) $

$ \Rightarrow (x-4)^2  < (x-10)^2 $

$ \Rightarrow x^2 - 8x + 16 < x^2 - 20x + 100 $

$ \Rightarrow 12x < 84 $

$ \Rightarrow x < 7 $

## Plotting the class conditional densities, posterior probabilities, and decision boundary

[back to top]

```python
%pylab inline

import numpy as np
from matplotlib import pyplot as plt

def pdf(x, mu, sigma):
    """
    Calculates the normal distribution's probability density
    function (PDF).

    """
    term1 = 1.0 / ( math.sqrt(2*np.pi) * sigma )
    term2 = np.exp( -0.5 * ( (x-mu)/sigma )**2 )
    return term1 * term2

# generating some sample data
x = np.arange(0, 100, 0.05)

# probability density functions
pdf1 = pdf(x, mu=4, sigma=1)
pdf2 = pdf(x, mu=10, sigma=1)

# Class conditional densities (likelihoods)
plt.plot(x, pdf1)
plt.plot(x, pdf2)
plt.title('Class conditional densities (likelihoods)')
plt.ylabel('p(x)')
plt.xlabel('random variable x')
plt.legend(['p(x|w_1) ~ N(4,1)', 'p(x|w_2) ~ N(10,1)'], loc='upper right')
plt.ylim([0,0.5])
plt.xlim([0,20])
plt.show()
```

```python
def posterior(likelihood, prior):
    """
    Calculates the posterior probability (after Bayes Rule) without
    the scale factor p(x) (=evidence).

    """
    return likelihood * prior

# probability density functions
posterior1 = posterior(pdf(x, mu=4, sigma=1), 0.5)
posterior2 = posterior(pdf(x, mu=10, sigma=1), 0.5)

# Class conditional densities (likelihoods)
plt.plot(x, posterior1)
plt.plot(x, posterior2)
plt.title('Posterior Probabilities w. Decision Boundary')
plt.ylabel('P(w)')
plt.xlabel('random variable x')
plt.legend(['P(w_1|x)', 'p(w_2|X)'], loc='upper right')
plt.ylim([0,0.5])
plt.xlim([0,20])
plt.axvline(7, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(4, 0.3), xytext=(4, 0.3))
plt.annotate('R2', xy=(10, 0.3), xytext=(10, 0.3))
plt.show()
```

## Classifying some random example data

[back to top]

#### Note on generating univariate random data from a Normal Distribution

We can generate random samples drawn from a Normal distribution via the `np.random.randn()` function. Its default is a standard Normal distribution with $ \mu = 0 $ and $ \sigma^2 = 1 $. In order to draw random data from $ N(\mu, \sigma^2) $, we use
`sigma * np.random.randn(...) + mu`

```python
# Parameters
mu_1 = 4
mu_2 = 10
sigma_1_sqr = 1
sigma_2_sqr = 1

# Generating 10 random samples drawn from a Normal Distribution for class 1 & 2
x1_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_1
x2_samples = sigma_1_sqr**0.5 * np.random.randn(10) + mu_2
y = [0 for i in range(10)]

# Plotting sample data with a decision boundary

plt.scatter(x1_samples, y, marker='o', color='green', s=40, alpha=0.5)
plt.scatter(x2_samples, y, marker='^', color='blue', s=40, alpha=0.5)
plt.title('Classifying random example data from 2 classes')
plt.ylabel('P(x)')
plt.xlabel('random variable x')
plt.legend(['w_1', 'w_2'], loc='upper right')
plt.ylim([-0.1,0.1])
plt.xlim([0,20])
plt.axvline(7, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(4, 0.05), xytext=(4, 0.05))
plt.annotate('R2', xy=(10, 0.05), xytext=(10, 0.05))
plt.show()
```

## Calculating the empirical error rate

[back to top]

```python
w1_as_w2, w2_as_w1 = 0, 0
for x1,x2 in zip(x1_samples, x2_samples):
    if x1 >= 7:
        w1_as_w2 += 1
    if x2 < 7:
        w2_as_w1 += 1

emp_err =  (w1_as_w2 + w2_as_w1) / float(len(x1_samples) + len(x2_samples))

print('Empirical Error: {}%'.format(emp_err * 100))

```

    Empirical Error: 0.0%

```python

```

```python

```

```python

```

```python
test complete; Gopal
```


Eigenvalue/Eigenvector Analysis
for Undamped Systems

MCHE 485: Mechanical Vibrations

Dr. Joshua Vaughan

joshua.vaughan@louisiana.edu

http://www.ucs.louisiana.edu/~jev9637/

     Figure 1: A Two-Mass-Spring System

This notebook demonstrates the eigenvalue/eigenvector problem using a two-mass-spring-damper system shown in Figure 1. We'll just look at one example set of parameters. The same techniques apply for other parameters and for larger matrices.

The equations of motion for the system are:

$ \quad m_1 \ddot{x}_1 + (k_1+k_2)x_1 - k_2 x_2 = 0 $

$ \quad m_2 \ddot{x}_2 -k_2 x_1 +(k_2 + k_3)x_2 = 0 $

We could also write these equations in matrix form:

$ \quad \begin{bmatrix}m_1 & 0 \\ 0 & m_2\end{bmatrix}\begin{bmatrix}\ddot{x}_1 \\ \ddot{x}_2\end{bmatrix} + \begin{bmatrix}k_1 + k_2 & -k_2 \\ -k_2 & k_2 + k_3\end{bmatrix}\begin{bmatrix}x_1 \\ x_2\end{bmatrix} = \begin{bmatrix}0 \\ 0\end{bmatrix}$

Define

$ \quad M = \begin{bmatrix}m_1 & 0 \\ 0 & m_2\end{bmatrix} $

and

$ \quad K = \begin{bmatrix}k_1 + k_2 & -k_2 \\ -k_2 & k_2 + k_3\end{bmatrix} $

Using $M$ and $K$, we want to solve:

$ \quad \left[K - \omega^2 M\right]\bar{X} = 0 $

for $\bar{X}$. This is an eigenvalue problem.

For information on how to obtain these equations, you can see the lectures at the [class website](http://www.ucs.louisiana.edu/~jev9637/MCHE485.html).

```python
import numpy as np
import sympy

sympy.init_printing()
```

```python
# We want our plots to be displayed inline, not in a separate window
%matplotlib inline

# Import the plotting functions
import matplotlib.pyplot as plt
```

```python
# Let's also improve the printing of NumPy arrays.
np.set_printoptions(precision=3, suppress=True)
```

Using [SymPy](http://sympy.org), we can solve for the eigenvalues and eigenvalues symbolically. We'll first set up and solve the problem just as we would by hand, just replacing the algebra of the solution with SymPy. Then, we'll use the State-space and Symmetric forms of the equations of motion to allow us to directly use the Sympy eigenvalue and eigenvector methods.

```python
# Define the symbols
m1, m2, k1, k2, k3 = sympy.symbols('m_1 m_2 k_1 k_2 k_3')

w, w1, w2 = sympy.symbols('omega omega_1 omega_2')

x1, x2 = sympy.symbols('x_1 x_2')

M = sympy.Matrix([[m1, 0],
                 [0,  m2]])

K = sympy.Matrix([[k1 + k2, -k2],
                 [-k2,      k2 + k3]])
```

```python
# create the matrix to solve
KM = K - w**2 * M

eigenvalues = sympy.solve(KM.det(), w**2)
```

In SymPy, the easiest way to output the answer is to call it in its own cell.

```python
eigenvalues[0]
```

```python
eigenvalues[1]
```

We can substiute numerical values, to check is our symbolic solutions match the ones found numerically in the [notebook using the SciPy linalg library](http://nbviewer.jupyter.org/github/DocVaughan/MCHE485---Mechanical-Vibrations/blob/Spring2016/Jupyter%20Notebooks/Eigenvalue-Eigenvector%20Analysis.ipynb). In that notebook, we used

$ \quad m_1 = m_2 = 1.0 $ kg

and

$ \quad k_1 = k_2 = k_3 = 4.0 $ $N/m^2$

to find eigenalues of 4.00 and 12.00. The eigenvectors were found to be:

$ \quad X_1 = [1 \ 1]^T $

and

$ \quad X_2 = [1 \ -1]^T $

```python
eigenvalues[0].subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)])
```

```python
eigenvalues[1].subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)])
```

So, the symbolic eigenvalues we found match those found numerically. Now, let's look at the eigenvectors.

```python
KM1 = K - eigenvalues[0] * M
KM2 = K - eigenvalues[1] * M

X = sympy.Matrix([[x1],
                   [x2]])
```

```python
eigenvect1 = sympy.solve(KM1 * X, (x1, x2))
eigenvect2 = sympy.solve(KM2 * X, (x1, x2))
```

Because our system of equations is linearly dependent, SymPy will return one of the values in terms of the other, similar to how we solve these problems by hand. So, we can look at the relationship returned to generate the eigenvector.

```python
eigenvect1
```

```python
eigenvect2
```

We can again substitute the numerical values to check the solution we obtain.

```python
eigenvect1[x1].subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)])
```

We see that $x_1 = x_2$ for this eigenvector, matching our numerical solutions.

```python
eigenvect2[x1].subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)])
```

We see that $x_1 = -x_2$ for this eigenvector, matching our numerical solutions.

# Using the State-Space Form of the Equations of Motion
Now, let's define a new matrix $A$ that is generated using the state-space form of the equations of motion. We'll again substitute the parameters we used in the numerical solution to check for a match.

```python
A = sympy.Matrix([[0, 1, 0, 0],
            [-(k1 + k2) / m1, 0, k2 / m1, 0],
            [0, 0, 0, 1],
            [k2 / m2, 0, -(k2 + k3) / m2, 0]])

A.subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)]).eigenvals()
```

Remember that in the state-space form, for lightly damped systems the eigenvalues approximately represent:

$ \quad \lambda = -\zeta \omega_i \pm i \omega_{di} $

where

$\zeta_i$ is the damping ratio, $\omega_{i}$ is the natural frequency, $\omega_{di}$ is the damped natural frequency for the $i^{th}$ mode. Be careful not to confuse the $_i$ subscripts with $i$ as an imaginary number.

Here, there is no damping, so $\zeta = 0$ and $\omega_{d} = \omega_i$ for all modes. So, we again see that the calculated eigenvalues match those found numerically.

Now, let's use the SymPy eigenvector method on the state space matrix, after substituting our numerical values.

```python
A.subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)]).eigenvects()
```

The SymPy ```eigenvects()``` function returns the eigenvalue, its multiplicity, and the eigenvector. For our choice of states in the state vector, we want to look at the 1st and 3rd value of each eigenvectors, as they correspond to the motion of $x_1$ and $x_2$. We can also multiply each by $i$ to make them real.

```python
A.subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)]).eigenvects()[1][2][0] * sympy.I
```

```python
A.subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)]).eigenvects()[3][2][0] * sympy.I
```

We see that these vectors have the same proportions as those calculated by the purely numerical method.

# Using the Symmetric Form
Using the matrices from the equations of motion, we can define two new matrices, $A$ and $B$, by:

$ \quad A = \left[\begin{array}{cc}\hphantom{-}0 & -K \\-K & -C\end{array}\right] \ \ \ \ \ \ \ B = \left[\begin{array}{cc}-K & 0 \\ \hphantom{-}0 & M\end{array}\right]$

In this case, $ C = 0 $, so the equations reduce to:

$ \quad A = \left[\begin{array}{cc}\hphantom{-}0 & -K \\-K & 0\end{array}\right] \ \ \ \ \ \ \ B = \left[\begin{array}{cc}-K & 0 \\ \hphantom{-}0 & M\end{array}\right]$

```python
A = sympy.Matrix([[0, 0, -(k1 + k2), k2],
                  [0, 0, k2, (-k2 + k3)],
                  [-(k1 + k2), k2, 0, 0],
                  [k2, (-k2 + k3), 0, 0]])

B = sympy.Matrix([[-(k1 + k2), k2, 0, 0],
                  [k2, (-k2 + k3), 0, 0],
                  [0, 0, m1, 0],
                  [0, 0, 0, m2]])

A.subs([(m1, 1.0), (m2, 1.0), (k1, 4.0), (k2, 4.0), (k3, 4.0)]).eigenvals()
```

#### Licenses
Code is licensed under a 3-clause BSD style license. See the licenses/LICENSE.md file.

Other content is provided under a [Creative Commons Attribution-NonCommercial 4.0 International License](http://creativecommons.org/licenses/by-nc/4.0/), CC-BY-NC 4.0.

```python
# This cell will just improve the styling of the notebook
from IPython.core.display import HTML
import urllib.request
response = urllib.request.urlopen("https://cl.ly/1B1y452Z1d35")
HTML(response.read().decode("utf-8"))
```

Jupyter Notebooks/Eigenvalue-Eigenvector Analysis - Symbolic.ipynb