###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth. Partly based on David Ketcheson's pendulum lesson, also under CC-BY.

# Phugoid Oscillation

Welcome back! This is the second IPython Notebook of the series _"The phugoid model of glider flight"_, the first learning module of the course [**"Practical Numerical Methods with Python."**](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about)

In the first notebook, _"Phugoid Motion"_, we described the physics of an aircraft's oscillatory trajectory, seen as an exchange of kinetic and potential energy. This analysis goes back to Frederick Lanchester, who published his book _"Aerodonetics"_ on aircraft stability in 1909. We concluded that first exposure to our problem of interest by plotting the flight paths predicted by Lanchester's analysis, known as _phugoids_.

Here, we will look at the situation when an aircraft is initially moving on the straight-line phugoid (obtained with the parameters $C=2/3$, $\cos\theta=1$, and $z=z_t$ in the previous analysis), and experiences a small upset, a wind gust that slightly perturbs its path. It will then enter into a gentle oscillation around the previous straight-line path: a _phugoid oscillation_.

If the aircraft experiences an upward acceleration of $-d^2z/dt^2$, and we assume that the perturbation is small, then $\cos\theta=1$ is a good approximation and Newton's second law in the vertical direction is:

$$\begin{equation}
L - W = - \frac{W}{g}\frac{d^2 z}{dt^2}
\end{equation}$$

In the previous notebook, we saw that the following relation holds for the ratio of lift to weight, in terms of the trim velocity $v_t$:

$$\begin{equation}
\frac{L}{W}=\frac{v^2}{v_t^2}
\end{equation}$$

This will be useful: we can divide Equation (1) by the weight and use Equation (2) to replace $L/W$. Another useful relation from the previous notebook expressed the conservation of energy (per unit mass) as $v^2 = 2 gz$. With this, Equation (1) is rearranged as:

$$\begin{equation}
\frac{d^2z}{dt^2} + \frac{gz}{z_t} = g
\end{equation}$$

Look at Equation (3) for a moment. Does it ring a bell? Do you recognize it?

If you remember from your physics courses the equation for _simple harmonic motion_, you should see the similarity!

Take the case of a simple spring. Hooke's law is $F=-kx$, where $F$ is a restoring force, $x$ the displacement from a position of equilibrium and $k$ the spring constant. This results in the following ordinary differential equation for the displacement:

$$\begin{equation}
 \frac{d^2 x}{dt^2}= -\frac{k}{m}x
\end{equation}$$

which has the solution $x(t) = A \cos(\omega t- \phi)$, representing simple harmonic motion with an angular frequency $\omega=\sqrt{k/m}=2\pi f$ and phase angle $\phi$.

Now look back at Equation (3): it has the same form and it represents simple harmonic motion with angular frequency $\omega=\sqrt{g/z_t}$.

Think about this for a moment ... we can immediately say what the period of the oscillation is: exactly $2 \pi \sqrt{z_t/g}$ — or, in terms of the trim velocity, $\pi \sqrt{2} v_t/g$.

_This is a remarkable result!_ Think about it: we know nothing about the aircraft, or the flight altitude, yet we can obtain the period of the phugoid oscillation simply as a function of the trim velocity. For example, if trim velocity is 200 knots, we get a phugoid period of about 47 seconds—over that time, you really would not notice anything if you were flying in that aircraft.

Next, we want to be able to compute the trajectory of the aircraft for a given initial perturbance. We will do this by numerically integrating the equation of motion.

## Prepare to integrate

We want to integrate the differential equation and plot the trajectory of the aircraft. Are you ready?

The equation for the phugoid oscillation is a second-order, ordinary differential equation (ODE). Let's represent the time derivative with a prime, and write it like this:

$$\begin{equation}
z(t)'' + \frac{g \,z(t)}{z_t}=g
\end{equation}$$

There's a convenient trick when we work with ODEs: we can turn this 2nd-order equation into a system of two 1st-order equations. Like this:

$$\begin{eqnarray}
z'(t) &=& v(t)\\
v'(t) &=& g\left(1-\frac{z(t)}{z_t}\right)
\end{eqnarray}$$

Are you following? Make sure you are following the derivations, even if it means writing the equations down in your own notes! (Yes, the old-fashioned paper way.)

Another way to look at a system of two 1st-order ODEs is by using vectors. You can make a vector with your two independent variables,

$$\begin{equation}
\vec{u}  = \begin{pmatrix} z \\ v \end{pmatrix}
\end{equation}$$

and write the differential system as a single vector equation:

\begin{equation}
\vec{u}'(t)  = \begin{pmatrix} v \\ g-g\frac{z(t)}{z_t} \end{pmatrix}
\end{equation}

If you call the right-hand-side $\vec{f}(\vec{u})$, then the equation is very short: $\vec{u}'(t) = \vec{f}(\vec{u})$—but let's drop those arrows to denote vectors from now on, as they are a bit cumbersome: just remember that $u$ and $f$ are vectors in the phugoid equation of motion.

Next, we'll prepare to solve this problem numerically.

## Initial value problems

Let's step back for a moment. Suppose we have a first-order ODE $u'=f(u)$. You know that if we were to integrate this, there would be an arbitrary constant of integration. To find its value, we do need to know one point on the curve $(t, u)$. When the derivative in the ODE is with respect to time, we call that point the _initial value_ and write something like this:

$$u(t=0)=u_0$$

In the case of a second-order ODE, we already saw how to write it as a system of first-order ODEs, and we would need an initial value for each equation: two conditions are needed to determine our constants of integration. The same applies for higher-order ODEs: if it is of order $n$, we can write it as $n$ first-order equations, and we need $n$ known values. If we have that data, we call the problem an _initial value problem_.

Remember the definition of a derivative? The derivative represents the slope of the tangent at a point of the curve $u=u(t)$, and the definition of the derivative $u'$ for a function is:

$$u'(t) = \lim_{\Delta t\rightarrow 0} \frac{u(t+\Delta t)-u(t)}{\Delta t}$$

If the step $\Delta t$ is already very small, we can _approximate_ the derivative by dropping the limit. We can write:

$$\begin{equation}
u(t+\Delta t) \approx u(t) + u'(t) \Delta t
\end{equation}$$

With this equation, and because we know $u'(t)=f(u)$, if we have an initial value, we can step by $\Delta t$ and find the value of $u(t+\Delta t)$, then we can take this value, and find $u(t+2\Delta t)$, and so on: we say that we _step in time_, numerically finding the solution $u(t)$ for a range of values: $t_1, t_2, t_3 \cdots$, each separated by $\Delta t$. The numerical solution of the ODE is simply the table of values $t_i, u_i$ that results from this process.

## Discretization

In order to execute the process described above and find the numerical solution of the ODE, we start by choosing the values $t_1,t_2,t_3 \cdots t_n$—we call these values our *grid* in time. The first point of the grid is given by our _initial value_, and the small difference between two consecutive times is called the _time step_, denoted by $\Delta t$.  The solution value at time $t_n$ is denoted by $u_n$.

Let's build a time grid for our problem. We first choose a final time $T$ and the time step $\Delta t$. In code, we'll use readily identifiable variable names: `T` and `dt`, respectively. With those values set, we can calculate the number of time steps that will be needed to reach the final time; we call that variable `N`.

Let's write some code. The first thing we do in Python is load our favorite libraries: NumPy for array operations, and the Pyplot module in Matplotlib, to later on be able to plot the numerical solution. The line `%matplotlib inline` tells IPython Notebook to show the plots inline.

```
import numpy
import matplotlib.pyplot as plt
%matplotlib inline
```

Now, initialize `T` and `dt`, calculate `N` and build a NumPy array with all the values of time that make up the grid.

```
T = 100.0
dt = 0.01
N = int(T/dt)
t = numpy.linspace(0.0, T, N)
```

We have our grid! Now it's time to apply the numerical time stepping represented by Equation (10).

##### Challenge!

* Write the code above using the NumPy function `arange()` instead of `linspace()`. If you need to, read the documentation for these functions.

##### Pro tip:

Enter a question mark followed by any function, e.g., `?numpy.linspace`, into a code cell and execute it, to get a help pane on the notebook.

## Euler's method

The approximate solution at time $t_n$ is $u_n$, and the numerical solution of the differential equation consists of computing a sequence of approximate solutions by the following formula, based on Equation (10):

$$u_{n+1} = u_n + \Delta t \,f(u_n).$$

This formula is called **Euler's method**.

For the equations of the phugoid oscillation, Euler's method gives the following algorithm that we need to implement in code:

\begin{align}
z_{n+1} & = z_n + \Delta t \, v_n \\
v_{n+1} & = v_n + \Delta t \left(g - \frac{g}{z_t} \, z_n \right).
\end{align}

### And solve!

To apply the numerical solution method, we need to set things up in code: define the parameter values needed in the model, initialize a NumPy array to hold the discrete solution values, and initialize another array for the angular values.

```
# initial conditions
z0 = 100.  #altitude
v  = 10.   #upward velocity resulting from gust
zt = 100.
g  = 9.81

u = numpy.array([z0, v])

# initialize an array to hold the changing angle values
z = numpy.zeros(N)
z[0] = z0
```

You should pay attention to a couple of things: (1) See how there is a dot after the numbers used to define our parameters? We just want to be explicit (as a good habit) that these variables are real numbers, called "floats." (2) We both _created_ and _initialized_ with zeros everywhere the solution vector `z`. Look up the documentation for the handy NumPy function `zeros()`, if you need to. (3) In the last line above, we assign the _initial value_ to the first element of the solution vector: `z[0]`.

Now we can step in time using Euler's method. Notice how we are time stepping the two independend variables at once in the time iterations.

```
# time-loop using Euler's method
for n in range(N-1):
    u = u + dt*numpy.array([u[1], g*(1-u[0]/zt)])
    z[n] = u[0]
```

Make sure you understand what this code is doing. This is a basic pattern in numerical methods: iterations in a time variable that apply a numerical scheme at each step.

## Plot the solution

If the code is correct, we have stored in the array `z` the position of the glider at each time. Let's use Matplotlib to examine the flight path of the arcraft.

You should explore the [Matplotlib tutorial](http://matplotlib.org/users/pyplot_tutorial.html) (if you need to) and familiarize yourself with the command-style functions that control the size, labels, line style, and so on. Creating good plots is a useful skill: it is about communicating your results effectively.

Here, we set the figure size, the limits of the vertical axis, the format of tick-marks, and axis labels. The final line actually produces the plot, with our chosen line style (continuous black line).

```
plt.figure(figsize=(10,4))   #set plot size
plt.ylim(40,160)             #y-axis plot limits
plt.tick_params(axis='both', labelsize=14) #increase font size for ticks
plt.xlabel('t', fontsize=14) #x label
plt.ylabel('z', fontsize=14) #y label
plt.plot(t,z, 'k-');
```

##### Explore and think

Try changing the value of `v` in the initial conditions.

* What happens when you have a larger gust?
* What about a smaller gust?
* What happens if there isn't a gust (`v = 0`)?

## Exact solution

The equation for phugoid oscillations is a 2nd-order, linear ODE and it has an exact solution of the following form:

$$
\begin{equation}
z(t) = A \sin \left(\sqrt{\frac{g}{z_t}} t \right) + B \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}
$$

where $A$ and $B$ are constants that we solve for using initial conditions.

Our numerical solution used the initial conditions:

$$\begin{eqnarray}
z(0) = z_0 \\
v(0) = v_0
\end{eqnarray}$$

Applying these to the exact solution and solving for $A$ and $B$, we get:

$$\begin{equation}
z(t) = v_0 \sqrt{\frac{z_t}{g}} \sin \left(\sqrt{\frac{g}{z_t}} t \right) + (z_0-z_t) \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}$$

We already defined all of these variables for our numerical solution, so we can immediately compute the exact solution.

##### Pro tip:

The expression is a  bit long —if you don't feel like scrolling left and right, you can insert a line break in the code using a backslash '\' and Python will treat the next line as a continuation of the first.

```
z_exact = v*(zt/g)**.5*numpy.sin((g/zt)**.5*t)+\
            (z0-zt)*numpy.cos((g/zt)**.5*t)+zt
```

## Plot the exact solution

Now we can plot our exact solution!  Even better, we can plot _both_ the numerical solution *and* the exact solution to see how well Euler's method approximated the phugoid oscillations.

To add another curve to a plot, simply type a second `plt.plot()` statement.  We also added a legend using the format

```Python
plt.legend(["line 1","line 2"])
```

```
plt.figure(figsize=(10,4))
plt.ylim(40,160)             #y-axis plot limits
plt.tick_params(axis='both', labelsize=14) #increase font size for ticks
plt.xlabel('t', fontsize=14) #x label
plt.ylabel('z', fontsize=14) #y label
plt.plot(t,z)
plt.plot(t, z_exact)
plt.legend(['Numerical Solution','Analytical Solution']);
```

That looks like pretty good agreement, but what's happening towards the end? We'll come back to this.

Euler's method, like all numerical methods, introduces some errors.  If the method is *convergent*, the approximation will get closer and closer to the exact solution as we reduce the size of the step, $\Delta t$. The error in the numerical method should tend to zero, in fact, when $\Delta t\rightarrow 0$—when this happens, we call the method _consistent_. We'll define these terms more carefully in the theory components of this course.

## Convergence

To compare the two solutions, we need to use a **norm** of the difference, like the $L_1$ norm, for example.

$$ E = \Delta t \sum_{n=0}^N \left|z(t_n) - z_n\right|$$

The $L_1$ norm is the sum of the individual differences between the exact and the numerical solutions, at each mesh point.

We check for convergence by calculating the numerical solution using progressively smaller values of `dt`. We already have most of the code that we need.  We just need to add an extra loop and an array of different $\Delta t$ values to iterate through.

You should read the documentation for the function [enumerate](https://docs.python.org/2/library/functions.html#enumerate) and make sure you understand how we're using it below.

##### Warning

The cell below can take a little while to finish (the last $\Delta t$ value alone requires 1 million iterations!).  If the cell is still running, the input label will say `In [*]`.  When it finishes, the `*` will be replaced by a number.

```
# time-increment array
dt_values = numpy.array([0.1, 0.05, 0.01, 0.005, 0.001, 0.0001])

# array that will contain solution of each grid
z_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):
    N = int(T/dt)    # number of time-steps
    ### discretize the time using numpy.linspace() ###
    t = numpy.linspace(0.0, T, N)

    # initial conditions
    u = numpy.array([z0, v])
    z = numpy.empty_like(t)
    z[0] = z0

    # time loop - Euler method
    for n in range(N-1):
        ### compute next solution using Euler method ###
        u[:] = u[:] + dt*numpy.array([u[1], g*(1-u[0]/zt)])
        z[n] = u[0]   # store the elevation at time-step n+1

    z_values[i] = z.copy()    # store the total elevation calculation grid i
```

### Calculate the error

We now have numerical solutions for each $\Delta t$ in the array `z_values`.  To calculate the error corresponding to each $\Delta t$, we can write a function!

```
def get_error(z, dt):
    """Returns the error relative to analytical solution using L-1 norm.

    Parameters
    ----------
    z : array of float
        numerical solution.
    z_exact : array of float
        analytical solution.
    dt : float
        time increment.

    Returns
    -------
    err : float
        L_{1} norm of the error with respect to the exact solution.
    """
    N = len(z)
    t = numpy.linspace(0.0, T, N)

    z_exact = v*(zt/g)**.5*numpy.sin((g/zt)**.5*t)+\
                (z0-zt)*numpy.cos((g/zt)**.5*t)+zt

    return dt * numpy.sum(numpy.abs(z-z_exact))
```

**Note**: in the last line of the function, we perform an 'array operation':

`z - z_exact`

We are *not* subtracting one value from another.  Instead, we are taking the difference between elements at each corresponding index in both arrays.  Here is a quick example:

```
a = numpy.array([1, 2, 3])
b = numpy.array([4, 4, 4])

b - a
```

    array([3, 2, 1])

Now, we iterate through each $\Delta t$ value and calculate the corresponding error.

```
error_values = numpy.empty_like(dt_values)

for i, dt in enumerate(dt_values):
    ### call the function get_error() ###
    error_values[i] = get_error(z, dt)
```

Remember, *if* the method is convergent then the error should get smaller as  $\Delta t$ gets smaller.  To visualize this, let's plot $\Delta t$ vs. error.  If you use `plt.plot` you won't get a very useful result.  Instead, use `plt.loglog` to create the same plot with a log-log scale.  This is what we do almost always to assess the errors of a numerical scheme graphically.

```
plt.figure(figsize=(10, 6))
plt.tick_params(axis='both', labelsize=14) #increase tick font size
plt.grid(True)                         #turn on grid lines
plt.xlabel('$\Delta t$', fontsize=16)  #x label
plt.ylabel('Error', fontsize=16)       #y label
plt.loglog(dt_values, error_values, 'ko-')  #log-log plot
plt.axis('equal')                      #make axes scale equally;
```

This is the kind of result we like to see!  As $\Delta t$ shrinks (towards the left), the error gets smaller and smaller, like it should.

##### Challenge!

We calculated the error for several different timestep sizes using two nested `for` loops.  That worked, but whenever possible, we like to re-use code (and not just copy and paste it!).

Create a function that implements Euler's method and re-write the error analysis cell to use your function.

---

######The cell below loads the style of this notebook.

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>


### Problem Statement

Consider three reactions for the production of ammonia

1. Steam-reforming of methane

CH4 + H2O → CO + 3 H2

2. Water-gas shift

CO + H2O → CO2 + H2

3. Haber-Bosch reaction

N2 + 3 H2 → 2 NH3

Determine if it is possible to construct a process for the production of ammonia with no wasted hydrogen and no release of carbon monoxide.

Let's Solve this.

First of all we will construct a matrix where reaction co-effcicent of each process and each molecule will be shown.

| Species         | R1 | R2 | R3 | Net                 |
| :-------------: | :-----------: | :-----------: | :-----------: | :-----------------: |
|                 | χ1 | χ2 | χ3 | $\sum_k\nu_k\chi_k$ |
| CH4  | -1            |  0            |  0            | ≤ 0             |
| H2O  | -1            | -1            |  0            | ≤ 0             |
| CO              |  1            | -1            |  0            | 0                   |
| H2   |  3            |  1            | -3            | 0                   |
| CO2  |  0            |  1            |  0            | ≥ 0             |
| N2   |  0            |  0            |  1            | ≤ 0             |
| NH3  |  0            |  0            |  2            | 1                   |

which includes three equality constraints which need to be solved for
χ1, χ2, and χ3.

```python
# importing sympy and numpy
import numpy as np
import sympy as sm
```

```python
# creating three variables for three reactions
sm.var('r1 r2 r3')
```

    (r1, r2, r3)

```python
# creating dictionary which stores the equation for each substance
v = {}

v['CH4'] = -r1
v['H20'] = -r1 -r2
v['CO'] = r1 -r2
v['H2'] = 3*r1 +r2 -3*r3
v['CO2'] = r2
v['N2'] = -r3
v['NH3'] = 2*r3
```

```python
# balance equations

balance_eqn = [
    sm.Eq(v['NH3'], 1),
    sm.Eq(v['CO'], 0),
    sm.Eq(v['H2'], 0)
]
```

```python
#solving balance equation using sympy
result = sm.solve(balance_eqn)

print(result)
```

    {r1: 3/8, r2: 3/8, r3: 1/2}

```python
# printing the final output of process for given conditions

for molecule in v.keys():
    print(molecule, v[molecule].subs(result))

```

    CH4 -3/8
    H20 -3/4
    CO 0
    H2 0
    CO2 3/8
    N2 -1/2
    NH3 1

Those are the freaction of substances that will be left at the end of the process (basis 1 unit of ammonia), given that all hydorgen and carbon monoxide are consumed.

Reference:
https://nbviewer.jupyter.org/github/jckantor/CBE20255/blob/master/notebooks/02.02-Generation-Consumption-Analysis-for-Ammonia-Production.ipynb

```python

```

Generation Consumption Analysis - Basics/generation-consumption-basics.ipynb


```python
from sympy import init_session
init_session(use_latex=True)
```

    IPython console for SymPy 1.7.1 (Python 3.7.10-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.7.1/

Ecuacion lineal

```python
a, b, c, d, x, y = symbols ('a b c d x y')
a,b,c,d,x,y
```

```python
x = symbols('x', real = True , positive = True)
x
```

```python
x.assumptions0
```

    {'commutative': True,
     'complex': True,
     'extended_negative': False,
     'extended_nonnegative': True,
     'extended_nonpositive': False,
     'extended_nonzero': True,
     'extended_positive': True,
     'extended_real': True,
     'finite': True,
     'hermitian': True,
     'imaginary': False,
     'infinite': False,
     'negative': False,
     'nonnegative': True,
     'nonpositive': False,
     'nonzero': True,
     'positive': True,
     'real': True,
     'zero': False}

Sympy define los números en Reales, Racionales y Enteros

```python
Rational(5,2)
```

sumando enteros racionales

```python
x = symbols('x',real= True, positive = True)
x
```

```python
x = 1/2 + 3/2
x
```

```python
x = Rational(1,2) + Rational (3,2)
x
```

```python
y,d = symbols=('y,d')
```

```python
d = 2 * y
d
```

```python
d = y **2
d
```


```python
"""
Examples on approximating functions by global basis functions,
using the approx2D.py module.
"""
from approx2D import *
import sympy as sym
import sys
x, y = sym.symbols('x y')

def sines(x, y, Nx, Ny):
    return [sym.sin(sym.pi*(i+1)*x)*sym.sin(sym.pi*(j+1)*y)
            for i in range(Nx+1) for j in range(Ny+1)]

def taylor(x, y, Nx, Ny):
    return [x**i*y**j for i in range(Nx+1) for j in range(Ny+1)]

# ----------------------------------------------------------------------

def run_linear():
    f = (1+x**2)*(1+2*y**2)
    psi = taylor(x, y, 1, 1)
    print(psi)
    Omega = [[0, 2], [0, 2]]
    u = least_squares(f, psi, Omega)
    comparison_plot(f, u, Omega, plotfile='approx2D_bilinear')
    print('\n\n**** Include second order terms:')
    psi = taylor(x, y, 2, 2)
    u = least_squares(f, psi, Omega)

if __name__ == '__main__':
    # Run from command line:
    # python ex_approx2D.py run_linear
    cmd = sys.argv[1]
    args = ''
    if len(sys.argv) > 2:
        args = ','.join(sys.argv[2:])
    eval('{0}({1})'.format(cmd, args))

```

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/SRC/17_EX_APPROX2D.ipynb


```python
# HIDDEN
# flake8: noqa t
import sys
import os
projroot = lambda p: p if os.path.isfile(os.path.join(p, "Pipfile")) or p == '/' else projroot(os.path.dirname(p))
sys.path.append(projroot(os.getcwd()))

import pandas as pd
import urllib
import math
import numpy as np
from typing import Tuple, Any
from IPython.display import display, Math, Latex, display_latex, HTML
```

 # Bayes formula

```python
# HIDDEN
display(Latex(r"""
\begin{equation}
P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}
\end{equation}"""))

```

\begin{equation}
P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}
\end{equation}

 Over the years I've encountered numerous probability puzzles.  These puzzles are usually against human intuitions with
 surprising answers.  The common thread behind these puzzles are that they can all be solved using Baye's theorem in a
 consistent way.  Below we will go through three examples in the

 # Example 1

  Probability of a disease in the population is 1%.  A diagnose is 90% accurate on a diseased person, with a 5%
  mis-diagnose rate on a normal person.  If a randomly selected person is positively diagnosed, what's the probability
  of the actual disease?

  - Let A: has disease
  - Let B: positively diagnosed
  - $P(B \mid A)$: probability of positively diagnosed given having disease = 0.9
  - $P(A)$: has disease = 0.01
  - $P(B)$: positively diagnosed = 0.01 * 0.9 + 0.99 * 0.05 =0.0585
  - $P(A \mid B)$: =  (0.9 * 0.01) / 0.0585 = 0.154 = 15%

 A position diagnose result in 15% of probability of having the actual disease.  This seems slow, but makes sense intuitively:
 because the disease is rare, a randomly chose person is more likely to be disease free and mis diagnosed.  A common sense
 thing to do is to get a second test.  Now that we can replace $P(A)$ with 0.15 instead of 0.01, reapply the Bayes theorem
 again will mean a second positive diagnose results in 74% of chance of having the disease.

 # Example 2

 Monty hall.  3 closed doors, 1 with prize behind it.  Contestant randomly selects 1 door.  The host, knowing what's
 behind the doors, pick one of the remaining two doors that is empty and reveal it to the contestant.  Host then ask the contestant
 if he/she wants to switch.  Should the contestant switch?

 Before we reframe the question in Bayes, lets fix the parameters: say contestant picks door1, and
 host reveals door3.  So the question becomes

 1. $P(Prize=door1 \mid Open=door3)$:  probability of prize behind door1 given host opens door3
 2. $P(Prize=door2 \mid Open=door3)$:  probability of prize behind door2 given host opens door3

 Which of above two is greater?  If 2 is greater, then definitely switch!

 Note $P(Prize=door3 \mid Open=door3)$ is not among consideration, by the question definition this is eliminated from
 the probability space.

 Next we solve for 1 and 2

 1. $P(Prize=door1 \mid Open=door3)$ = $\frac{P(Open=door3 \mid Prize=door1) P(Prize=door1)}{P(Open=door3)}$
     - $P(Open=door3 \mid Prize=door1)$, probability of host open door3 given prize is behind door1, = 1/2
     - $P(Prize=door1)$, probability of prize is behind door1, = 1/3
     - $P(Open=door3)$, probability of host open door3 in general, = 1/2
     - result = (1/2 * 1/3) / 1/2 = 1/3
     - This also matches intuition: contestant picked randomly 1 out of 3 and stayed with the pick, the probability
       of winning should remain 1/3.

 2. $P(Prize=door2 \mid Open=door3)$ = $\frac{P(Open=door3 \mid Prize=door2) P(Prize=door2)}{P(Open=door3)}$
     - $P(Open=door3 \mid Prize=door2)$, probability of host open door3 given prize is behind door2, = 1.  In other words, the host has no choice.
     - $P(Prize=door2)$, probability of prize is behind door2, = 1/3
     - $P(Open=door3)$, probability of host open door3 in general, = 1/2
     - result = (1 * 1/3) / 1/2 = 2/3

 2/3 > 1/3, switch is a good idea!

 Another way to think of this problem is that when the host removed an empty door, the host removed the uncertainty:
 2 doors out of three by definition has 2/3 chance of containing the prize, now that the empty door is removed, the entire
 2/3 probability is assigned to the only remaining closed door.

 Now suppose the question is modified as such:  host's toddler son wondered in and accidentally knocked over door3
 which is revealed to be empty,  does switching door enhances the contestant's probability of winning?

 1. $P(Prize=door1 \mid Open=door3)$ = $\frac{P(Open=door3 \mid Prize=door1) P(Prize=door1)}{P(Open=door3)}$
     - $P(Open=door3 \mid Prize=door1)$, probability of toddler knocks door3 given prize is behind door1, = 1/2
     - $P(Prize=door1)$, probability of prize is behind door1, = 1/3
     - $P(Open=door3)$, probability of toddler knocks over door3 in general, = 1/2
     - result = (1/2 * 1/3) / 1/2 = 1/3

 2. $P(Prize=door2 \mid Open=door3)$ = $\frac{P(Open=door3 \mid Prize=door2) P(Prize=door2)}{P(Open=door3)}$
     - $P(Open=door3 \mid Prize=door2)$, probability of toddler knocks door3 given prize is behind door2, = 1/2
     - $P(Prize=door2)$, probability of prize is behind door2, = 1/3
     - $P(Open=door3)$, probability of toddler knocks over door3 in general, = 1/2
     - result = (1/2 * 1/3) / 1/2 = 1/3

 1/3 = 1/3, switching makes no difference!

 From contestant's perspective, both scenarios appear the same.  However, the actual probability of winning the prize
 after switching the door varies a great deal depending on the **circumstances** behind the scenario: whether the host
 chose the empty door with or without the knowledge of the prize location.  Contemplate on this for a moment,
 then proceed to the next example.

 # Example 3

 ## Version 1
 We have a random sample of parents with exactly two children.  We then ask one parent if at least one of the child
 is a daughter.  If the parent says yes,  what's the probability that the other child is also a daughter?

 If the other child is also a daughter, then both children are daughters,  we denote this as P(DD).  We then denote that
 P(D) is at least one child is daughter.  So we are seeking $P(DD | D)$.

 $P(DD | D)$ = $\frac{P(D \mid DD) P(DD)}{P(D)}$

   - $P(D \mid DD)$, probability one is daughter given both are daughters: 1
   - $P(DD)$, probability both are daughters: 1/4
   - $P(D)$, probability at least one daughter = 1 - probability of both sons = 1 - 1/4 = 3/4
   - result = (1/2 * 1/3) / 1/2 = 1/3

 The probability of the other child is a daughter is 1/3.

 ## Version 2
 Same as version 1, except we ask if each parent if they have a daughter named Lucy.  If the parent says yes,  what's the
 probability that the other child is also a daughter?
r
 Now we are looking for $P(DD | Lucy)$.

 $P(DD | Lucy)$ = $\frac{P(Lucy \mid DD) P(DD)}{P(Lucy)}$

   - $P(DD)$, probability bother are daughters: 1/4
   - $P(Lucy)$, probability of a girl is named Lucy, lets assume it's L (the actual value doesn't really matter).
   - $P(Lucy \mid DD)$, probability of at least one girl is named Lucy given there are two daughters, this is 2 * $P(Lucy)$ = 2 * L.
   - result = (2 \* P(Lucy) * 1/4) / P(Lucy) = (2 * L * 1/4) / L  = 1/2

 So the probability of other child is a daughter is 1/2.

 This example also illustrates the essence of bayesian reasoning:  the posterior probability is affected by the input (priors), or
 our knowledge of the world!


# Revisão de conceitos estatísticos II

Vamos explorar alguns conceitos estatísticos aplicados à análise de sinais.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
from scipy.stats import norm
plt.rcParams.update({'font.size': 14})
import IPython.display as ipd # to play signals
import sounddevice as sd
```

# Distribuição de probabilidade de uma VA discreta

A distribuição de probabilidade de uma VA discreta é caracterizada pelas probabilidades de $x$(valores individuais) ser igual a $x_{i}$ (valor escolhido) para cada valor possível.

Consideremos o lançamento de uma dado de 6 façes não viciado. O universo amostral é:

$\Omega = {1, 2, 3, 4, 5, 6}$,
com 6 elementos.

A distribuição de probabilidade é dada por:

\begin{equation}
p(x=x_i) = \frac{1}{6}
\end{equation}

```python
plt.figure(figsize = (8,6))
plt.stem(np.arange(1,7), (1/6)*np.ones(6), '-b', markerfmt = 'ob', basefmt=" ", use_line_collection=  True)
plt.xticks(np.arange(1,7))
#plt.xticks(np.arange(1,7))
plt.grid(linestyle = '--', which='both')
plt.xlabel(r'Eventos - $x_i$')
plt.ylabel(r'$p(x)$')
plt.xlim((0, 7))
plt.ylim((0, 1/5));
```

# Distribuição cumulativa de probabilidade de uma VA discreta

A distribuição cumulativa especifica a probabilidade de $x \leq x_{i}$.

\begin{equation}
F(x) = P(x \leq x_i)
\end{equation}

Consideremos o lançamento do dado não viciado do exemplo anterior. Dentro do universo amostral ($\Omega = {1, 2, 3, 4, 5, 6}$, com 6 elementos), a probabilidade de obtermos um número $\leq 4$ ($\Omega = {1, 2, 3,}$) é $F(4) = P(x \leq 4) = 4/6 = 2/3$

```python
n = np.arange(-5, 10)
Fx = (1/6)*n
Fx[n<1] = 0
Fx[n>6] = 1

plt.figure(figsize = (8,6))
plt.stem(n, Fx, '-b', markerfmt = 'ob', basefmt=" ", use_line_collection=  True)
plt.xticks(n)
#plt.xticks(np.arange(1,7))
plt.grid(linestyle = '--', which='both')
plt.xlabel(r'Eventos - $x_i$')
plt.ylabel(r'$F(x) = p(x \leq x_i)$')
plt.xlim((n[0], n[-1]))
plt.ylim((0, 1.2));
```

# Distribuição cumulativa de probabilidade de uma VA contínua

A distribuição cumulativa especifica a probabilidade de $x \leq x_{i}$.

\begin{equation}
F(x) = P(x \leq x_i)
\end{equation}

As propriedades da distribuição cumulativa são:

- $F(-\infty)=0$
- $F(+\infty)=1$
- $F(x_{2}) \geq  F(x_{1})$, se $x_{2}\geq x_{1}$

Tomando a última propriedade, podemos definir a densidade de probabilidade por:

\begin{equation}
p(x) = \frac{\mathrm{d}F(x)}{\mathrm{d} x},
\end{equation}
que nos dá a probabilidade de encontrar o valor $x$ num intervalo infinitesimal.

```python
# parâmetros de uma distribuição normal
mu_x = 0.0
sigma_x = 1.0

# A distribuição cumulativa e a densidade de probabilidade
x = np.arange(-10, 10, 0.01)
Fx = norm.cdf(x, loc= mu_x, scale = sigma_x)
px = norm.pdf(x, loc= mu_x, scale = sigma_x)

# A relação entre F(x) e p(x)
x_1 = -1
x_2 = 1.5
x_section = np.arange(x_1, x_2, 0.01)

# plots
plt.figure()
plt.plot(x, Fx, color = 'b', linewidth = 2)
plt.fill_between(x_section, norm.cdf(x_section, loc= mu_x, scale = sigma_x), color = 'blue', alpha = 0.3)
plt.grid(linestyle = '--', which='both')
plt.title('Dist. Cumulativa')
plt.xlabel('x')
plt.ylabel(r'$F(x)$')
plt.xlim((x[0], x[-1]))
plt.ylim((0, 1.2))

plt.figure()
plt.plot(x, px, color = 'b', linewidth = 2)
plt.fill_between(x_section, norm.pdf(x_section, loc= mu_x, scale = sigma_x), color = 'blue', alpha = 0.3)
plt.grid(linestyle = '--', which='both')
plt.title('Dist. de densidade de probabilidade')
plt.xlabel('x')
plt.ylabel(r'$p(x)$')
plt.xlim((x[0], x[-1]))
plt.ylim((0, 1.2*np.amax(px)))
plt.tight_layout();
```

# Ruído aleatório com distribuição normal

Neste exemplo nós consideramos um sinal gerado a partir do processo de obter amostras de uma distribuição normal, dada por:

\begin{equation}
p(x) = \mathcal{N}(\mu_x, \sigma_x) = \frac{1}{\sqrt{2\pi}\sigma_x}\mathrm{e}^{-\frac{1}{2\sigma_x^2}(x-\mu_x)^2}
\end{equation}
em que $\mu_x$ é a média e $\sigma_{x}$ é o desvio padrão.

Imaginemos então, que a cada instante de tempo $t$, nós sorteamos um valor da distribuição normal.

```python
fs = 200
time = np.arange(0,2, 1/fs)

# sinal
mu_x = 0
sigma_x = 1.0
xt = np.random.normal(loc = mu_x, scale = sigma_x, size=len(time))

# A distribuição cumulativa e a densidade de probabilidade
x = np.arange(-4, 4, 0.01)
Fx = norm.cdf(x, loc= mu_x, scale = sigma_x)
px = norm.pdf(x, loc= mu_x, scale = sigma_x)

# Sombreando suas probabilidades
x_1 = -0.25
x_2 = 0.25
x_section = np.arange(x_1, x_2, 0.01)

# Figura
fig, axs = plt.subplots(1, 2, gridspec_kw={'width_ratios': [8, 4]}, figsize = (14, 4))
axs[0].plot(time, xt, '-b', linewidth = 1)
axs[0].fill_between(time, x_1, x_2,  color = 'lightcoral', alpha = 1)
axs[0].set_title(r'Sinal aleatório com dist. Normal ($\mu_x$ e $\sigma_x$ ctes.)')
axs[0].grid(linestyle = '--', which='both')
axs[0].set_xlabel('Tempo [s]')
axs[0].set_ylabel('Amplitude [-]')
axs[0].set_xlim((0, time[-1]))
axs[0].set_ylim((-4, 4))

axs[1].plot(x, px, color = 'b', linewidth = 2)
axs[1].fill_between(x_section, norm.pdf(x_section, loc= mu_x, scale = sigma_x),  color = 'lightcoral', alpha = 1)
axs[1].set_title(r'Dens. Prob. Normal')
axs[1].grid(linestyle = '--', which='both')
axs[1].set_ylabel(r'$p(x)$ [-]')
axs[1].set_xlabel(r'x [-]')
axs[1].set_xlim((x[0], x[-1]))
axs[1].set_ylim((-0.01, 1.2*np.amax(px)))
plt.tight_layout()

```

Aula 50 - Rev conceitos estatisticos II/conceitos estatisticos II.ipynb


# Introduction to Matrices in Python

## Some very basic notions

We can think of a *matrix* as a rectangular box full of numbers, like
$$ \begin{bmatrix} 1 & 2 & 3 & 4 \\ 5 & 6 & 7 & 8 \end{bmatrix}$$

These numbers are called the entries of the matrix, and can be *real* or *complex* (or many other, but we will focus on those, mostly, on matrices with real entries).

As you surely know, the entries of a matrix form *rows* (horizontally) and *columns* (vertically).

For example, the matrix above has two rows, and four columns. We usually say this is a 4 by 2 matrix.

The set of matrices of real numbers, having $m$ rows and $n$ columns will be denoted by $\R^{m \times n}$.

### Representing matrices in code

In Python, there are several different ways of representing matrices.

Here we will introduce two very useful packages, $\texttt{numpy}$ and $\texttt{sympy}$. They are usually imported into our code like this:

```python
import numpy as np
import sympy as sp
```

With $\texttt{numpy}$, we represent matrices with the type $\texttt{np.array}$. The method

```python
A = np.array([[1,2,3,4],[5,6,7,8]])
```

```python

```

notebooks/Introduction to matrices in Python.ipynb


# Likevektskonsentrasjoner i en syre-base likevekt
Her skal vi regne på en syre-baselikevekt, vi tar utgangspunkt i eksempel 16.8 (side 562) fra læreboken der vi blir bedt om å finne pH i 0.036 M HNO$_2$:

$$\text{HNO}_2 \rightleftharpoons \text{NO}_2^{-} + \text{H}^{+},$$

$$K_{a} = 4.5 \times 10^{-4}.$$

Vi skal løse denne oppgaven ved å bruke Python. For å kunne regne symbolsk skal vi bruke et bibliotek som heter [SymPy](https://www.sympy.org/).

```python
import sympy as sym  # Importer SymPy
```

```python
# Definer størrelsene vi kjenner
START_KONSENTRASJON = 0.036
KA = 4.5e-4
```

Over har vi listet opp hva vi kjenner. La oss også liste opp alle de ukjente som vi skal bestemme:
- $[\text{HNO}_2]$ ved likevekt.
- $[\text{NO}_2^{-}]$ ved likevekt.
- $[\text{H}^{+}]$ ved likevekt.

Vi har altså tre ukjente. La oss definere de som størrelser (spesifikt som [SymPy-symboler](https://docs.sympy.org/latest/tutorial/intro.html#a-more-interesting-example)) slik at vi kan regne med de (dette blir litt som når vi introduserer $x$ osv. for ukjente størrelser i ligninger vi skriver for hånd):

```python
# Vi definerer de ukjente størrelsene. For å spare litt skriving bruker vi
# - HA for syren HNO2
# - A for den korresponderende basen NO_2^-
# - H for H^+
c_HA, c_A, c_H = sym.symbols('c_HA c_A c_H')
```

Vi har nå definert konsentrasjonene. Disse er foreløpig ukjente. For å bestemme de, så trenger vi noen ligninger som relaterer de til hverandre. Mulige slike ligninger er:
- syre-basekonstanten
- elektronøytralitet
- massebalanser

```python
# La oss begynne med syre-basekonstanten. # Her sier vi at (c_A * c_H)/c_HA skal være lik ("sym.Eq") KA:
ligning1 = sym.Eq((c_A * c_H)/c_HA, KA)
```

Vi kan be SymPy skrive ut hva denne ligningen er, for å sjekke at den ser ut som vi hadde tenkt:

```python
ligning1
```

Den neste ligningen vi kan benytte oss av, er at det må være like mye negativ og positiv ladning. Her er det bare to ladede forbindelser, og de har motsatt fortegn. Det betyr at $[\text{A}]^- = [\text{H}]^+$. La oss skrive det som en ligning:

```python
# Elektronøytralitet:
ligning2 = sym.Eq(c_A, c_H)
```

```python
# Skriv ut denne ligningen også, for å dobbeltsjekke:
ligning2
```

I kjemiske reaksjoner er massen bevart (med mindre det skjer noen kjernereaksjoner som endrer typen av grunnstoffer). Det er litt omstendelig å jobbe med massebalanser, så vi skriver det her heller som en molbalanse.
Tankegangen er som følger: Vi har oppgitt startkonsentrasjonen av HNO$_2$, hvis vi vet volumet, så vet vi også hvor mange mol HNO$_2$ vi har i starten. Men da vet vi også hvor mange mol av grunnstoffene H, N og O vi har i systemet vårt. Disse moltallene endres ikke, men mengden av HNO$_2$ kan endres. Altså: generelt i kjemiske reaksjoner så endres ikke grunnstoffene, men hvordan de er koblet sammen endres.

Vi kan derfor her lage tre massebalanser:
- En for hydrogen,
- En for nitrogen,
- En for oksygen.

La oss bruke nitrogen:
- Antall mol av nitrogen totalt er lik antall mol av HNO$_2$ vi startet med.
- Ved likevekt er antall mol nitrogen totalt i systemet lik antall mol av HNO$_2$ pluss antall mol av NO$_2^-$
  (siden det er bare i disse to stoffene vi finner nitrogen, og det er ett nitrogen i hver av disse forbindelsene).

Med symboler (vi deler på et volumet for å gjøre om til konsentrasjon):

$$[\text{HNO}_3]_{\text{start}} = [\text{HNO}_3]_{\text{likevekt}} + [\text{NO}_2^-]_{\text{likevekt}} $$

La oss formulere det som en ligning:

```python
ligning3 = sym.Eq(START_KONSENTRASJON, c_HA + c_A)
```

```python
# Skriv ut ligning3 for dobbeltsjekk:
ligning3
```

Vi har nå tre ligninger og vi har tre ukjente. Dette kan vi (eller i dette tilfellet, SymPy) løse:

```python
løsninger = sym.solve([ligning1, ligning2, ligning3], [c_HA, c_A, c_H], dict=True)
```

Her får vi to løsninger:

```python
løsninger
```

En av løsningene SymPy fant gir negative konsentrasjoner. Dette er en ugyldig løsning og vi beholder bare den som har kun positive løsninger:

```python
# Vis gyldige løsninger:
gyldige = []
for løsning in løsninger:
    if all(i > 0 for i in løsning.values()):
        gyldige.append(løsning)
        print('Gyldig løsning:')
        print(f'- [HA] = {løsning.get(c_HA)}')
        print(f'- [A^-] = {løsning.get(c_A)}')
        print(f'- [H^+] = {løsning.get(c_H)}')
```

```python
# La oss finne pH:
ph = -sym.log(gyldige[0].get(c_H), 10)
```

```python
# La oss skrive ut verdien, for å få numerisk verdi, ber vi SymPy evaluere uttrykket:
print(f'pH = {ph.evalf()}')
```

Til sammenligning sier læreboka: $\text{pH} = 2.42$.


# Probabilistic Modelling using the Infinite Mixture Model

In many applications it is desirable to allow the model to adjust its complexity to the amount the data. Consider for example the task of assigning objects into clusters or groups. This task often involves the specification of the number of groups. However, often times it is not known beforehand how many groups exist. Moreover, in some applictions, e.g. modelling topics in text documents or grouping species, the number of examples per group is heavy tailed. This makes it impossible to predefine the number of groups and requiring the model to form new groups when data points from previously unseen groups are observed.

A natural approach for such applications is the use of non-parametric models. This tutorial will introduce how to use the Dirichlet process in a mixture of infinitely many Gaussians using Turing. For further information on Bayesian nonparametrics and the Dirichlet process we refer to the [introduction by Zoubin Ghahramani](http://mlg.eng.cam.ac.uk/pub/pdf/Gha12.pdf) and the book "Fundamentals of Nonparametric Bayesian Inference" by Subhashis Ghosal and Aad van der Vaart.

```julia
using Turing
```

## Mixture Model

Before introducing infinite mixture models in Turing, we will briefly review the construction of finite mixture models. Subsequently, we will define how to use the [Chinese restaurant process](https://en.wikipedia.org/wiki/Chinese_restaurant_process) construction of a Dirichlet process for non-parametric clustering.

#### Two-Component Model

First, consider the simple case of a mixture model with two Gaussian components with fixed covariance.
The generative process of such a model can be written as:

$$
\begin{align}
\pi_1 &\sim Beta(a, b) \\
\pi_2 &= 1-\pi_1 \\
\mu_1 &\sim Normal(\mu_0, \Sigma_0) \\
\mu_2 &\sim Normal(\mu_0, \Sigma_0) \\
z_i &\sim Categorical(\pi_1, \pi_2) \\
x_i &\sim Normal(\mu_{z_i}, \Sigma)
\end{align}
$$

where $\pi_1, \pi_2$ are the mixing weights of the mixture model, i.e. $\pi_1 + \pi_2 = 1$, and $z_i$ is a latent assignment of the observation $x_i$ to a component (Gaussian).

We can implement this model in Turing for 1D data as follows:

```julia
@model two_model(x) = begin

    # Hyper-parameters
    μ0 = 0.0
    σ0 = 1.0

    # Draw weights.
    π1 ~ Beta(1,1)
    π2 = 1-π1

    # Draw locations of the components.
    μ1 ~ Normal(μ0, σ0)
    μ2 ~ Normal(μ0, σ0)

    # Draw latent assignment.
    z ~ Categorical([π1, π2])

    # Draw observation from selected component.
    if z == 1
        x ~ Normal(μ1, 1.0)
    else
        x ~ Normal(μ2, 1.0)
    end
end
```

    DynamicPPL.ModelGen{var"###generator#282",(:x,),(),Tuple{}}(##generator#282, NamedTuple())

#### Finite Mixture Model

If we have more than two components, this model can elegantly be extend using a Dirichlet distribution as prior for the mixing weights $\pi_1, \dots, \pi_K$. Note that the Dirichlet distribution is the multivariate generalization of the beta distribution. The resulting model can be written as:

$$
\begin{align}
(\pi_1, \dots, \pi_K) &\sim Dirichlet(K, \alpha) \\
\mu_k &\sim Normal(\mu_0, \Sigma_0), \;\; \forall k \\
z &\sim Categorical(\pi_1, \dots, \pi_K) \\
x &\sim Normal(\mu_z, \Sigma)
\end{align}
$$

which resembles the model in the [Gaussian mixture model tutorial](https://turing.ml/dev/tutorials/1-gaussianmixturemodel/) with a slightly different notation.

## Infinite Mixture Model

The question now arises, is there a generalization of a Dirichlet distribution for which the dimensionality $K$ is infinite, i.e. $K = \infty$?

But first, to implement an infinite Gaussian mixture model in Turing, we first need to load the `Turing.RandomMeasures` module. `RandomMeasures` contains a variety of tools useful in nonparametrics.

```julia
using Turing.RandomMeasures
```

We now will utilize the fact that one can integrate out the mixing weights in a Gaussian mixture model allowing us to arrive at the Chinese restaurant process construction. See Carl E. Rasmussen: [The Infinite Gaussian Mixture Model](https://www.seas.harvard.edu/courses/cs281/papers/rasmussen-1999a.pdf), NIPS (2000) for details.

In fact, if the mixing weights are integrated out, the conditional prior for the latent variable $z$ is given by:

$$
p(z_i = k \mid z_{\not i}, \alpha) = \frac{n_k + \alpha/K}{N - 1 + \alpha}
$$

where $z_{\not i}$ are the latent assignments of all observations except observation $i$. Note that we use $n_k$ to denote the number of observations at component $k$ excluding observation $i$. The parameter $\alpha$ is the concentration parameter of the Dirichlet distribution used as prior over the mixing weights.

#### Chinese Restaurant Process

To obtain the Chinese restaurant process construction, we can now derive the conditional prior if $K \rightarrow \infty$.

For $n_k > 0$ we obtain:

$$
p(z_i = k \mid z_{\not i}, \alpha) = \frac{n_k}{N - 1 + \alpha}
$$

and for all infinitely many clusters that are empty (combined) we get:

$$
p(z_i = k \mid z_{\not i}, \alpha) = \frac{\alpha}{N - 1 + \alpha}
$$

Those equations show that the conditional prior for component assignments is proportional to the number of such observations, meaning that the Chinese restaurant process has a rich get richer property.

To get a better understanding of this property, we can plot the cluster choosen by for each new observation drawn from the conditional prior.

```julia
# Concentration parameter.
α = 10.0

# Random measure, e.g. Dirichlet process.
rpm = DirichletProcess(α)

# Cluster assignments for each observation.
z = Vector{Int}()

# Maximum number of observations we observe.
Nmax = 500

for i in 1:Nmax
    # Number of observations per cluster.
    K = isempty(z) ? 0 : maximum(z)
    nk = Vector{Int}(map(k -> sum(z .== k), 1:K))

    # Draw new assignment.
    push!(z, rand(ChineseRestaurantProcess(rpm, nk)))
end
```

```julia
using Plots

# Plot the cluster assignments over time
@gif for i in 1:Nmax
    scatter(collect(1:i), z[1:i], markersize = 2, xlabel = "observation (i)", ylabel = "cluster (k)", legend = false)
end;
```

    ┌ Info: Saved animation to
    │   fn = /home/cameron/code/TuringTutorials/tmp.gif
    └ @ Plots /home/cameron/.julia/packages/Plots/Xnzc7/src/animation.jl:104

Further, we can see that the number of clusters is logarithmic in the number of observations and data points. This is a side-effect of the "rich-get-richer" phenomenon, i.e. we expect large clusters and thus the number of clusters has to be smaller than the number of observations.

$$
E[K \mid N] \approx \alpha \cdot log \big(1 + \frac{N}{\alpha}\big)
$$

We can see from the equation that the concentration parameter $\alpha$ allows us to control the number of clusters formed *a priori*.

In Turing we can implement an infinite Gaussian mixture model using the Chinese restaurant process construction of a Dirichlet process as follows:

```julia
@model infiniteGMM(x) = begin

    # Hyper-parameters, i.e. concentration parameter and parameters of H.
    α = 1.0
    μ0 = 0.0
    σ0 = 1.0

    # Define random measure, e.g. Dirichlet process.
    rpm = DirichletProcess(α)

    # Define the base distribution, i.e. expected value of the Dirichlet process.
    H = Normal(μ0, σ0)

    # Latent assignment.
    z = tzeros(Int, length(x))

    # Locations of the infinitely many clusters.
    μ = tzeros(Float64, 0)

    for i in 1:length(x)

        # Number of clusters.
        K = maximum(z)
        nk = Vector{Int}(map(k -> sum(z .== k), 1:K))

        # Draw the latent assignment.
        z[i] ~ ChineseRestaurantProcess(rpm, nk)

        # Create a new cluster?
        if z[i] > K
            push!(μ, 0.0)

            # Draw location of new cluster.
            μ[z[i]] ~ H
        end

        # Draw observation.
        x[i] ~ Normal(μ[z[i]], 1.0)
    end
end
```

    DynamicPPL.ModelGen{var"###generator#800",(:x,),(),Tuple{}}(##generator#800, NamedTuple())

We can now use Turing to infer the assignments of some data points. First, we will create some random data that comes from three clusters, with means of 0, -5, and 10.

```julia
using Plots, Random

# Generate some test data.
Random.seed!(1)
data = vcat(randn(10), randn(10) .- 5, randn(10) .+ 10)
data .-= mean(data)
data /= std(data);
```

Next, we'll sample from our posterior using SMC.

```julia
# MCMC sampling
Random.seed!(2)
iterations = 1000
model_fun = infiniteGMM(data);
chain = sample(model_fun, SMC(), iterations);
```

    Sampling: 100%|█████████████████████████████████████████| Time: 0:00:00

Finally, we can plot the number of clusters in each sample.

```julia
# Extract the number of clusters for each sample of the Markov chain.
k = map(t -> length(unique(chain[:z].value[t,:,:])), 1:iterations);

# Visualize the number of clusters.
plot(k, xlabel = "Iteration", ylabel = "Number of clusters", label = "Chain 1")
```

If we visualize the histogram of the number of clusters sampled from our posterior, we observe that the model seems to prefer 3 clusters, which is the true number of clusters. Note that the number of clusters in a Dirichlet process mixture model is not limited a priori and will grow to infinity with probability one. However, if conditioned on data the posterior will concentrate on a finite number of clusters enforcing the resulting model to have a finite amount of clusters. It is, however, not given that the posterior of a Dirichlet process Gaussian mixture model converges to the true number of clusters, given that data comes from a finite mixture model. See Jeffrey Miller and Matthew Harrison: [A simple example of Dirichlet process mixture inconsitency for the number of components](https://arxiv.org/pdf/1301.2708.pdf) for details.

```julia
histogram(k, xlabel = "Number of clusters", legend = false)
```

One issue with the Chinese restaurant process construction is that the number of latent parameters we need to sample scales with the number of observations. It may be desirable to use alternative constructions in certain cases. Alternative methods of constructing a Dirichlet process can be employed via the following representations:

Size-Biased Sampling Process

$$
j_k \sim Beta(1, \alpha) * surplus
$$

Stick-Breaking Process
$$
v_k \sim Beta(1, \alpha)
$$

Chinese Restaurant Process
$$
p(z_n = k | z_{1:n-1}) \propto \begin{cases}
        \frac{m_k}{n-1+\alpha}, \text{ if } m_k > 0\\\
        \frac{\alpha}{n-1+\alpha}
    \end{cases}
$$

For more details see [this article](https://www.stats.ox.ac.uk/~teh/research/npbayes/Teh2010a.pdf).


# Introduction
In the falling parachutist problem we encountered an error - there was a difference between the numerical calculation and the analytical solution. There are numerous types of errors associated with an engineering mathematical model:
1. Modeling errors: These have to do with the fact that we can never describe reality exactely and/or our model neglects certain effects (e.g. sidewind in the parachutist example).
2. Numerical errors: These have to do directly with the numerical method used and can be generally categorized as
    * Round-off errors: due to the limited representation of numbers on computers
    * Truncation error: due to approximating exact mathematical operators

## Errors
The basic definition of an error is
\begin{equation}
\text{True Error = True Value - Approximation}
\end{equation}

For simplicity, denote the "True Error" as $E_\text{t}$.
There is a problem with this definition however - it does not take into account the "magnitude" of the true value. For example, an error of one centimeter is much more significant if we are measuring a rivet rather than a bridge. To fix this, we define the true relative error, $\epsilon_\text{t}$:
\begin{equation}
\text{True Relative Error} = \frac{\text{True Error}}{\text{True Value}}
\end{equation}
or
\begin{equation}
\epsilon_\text{t} = \frac{E_\text{t}}{\text{True Value}}
\end{equation}

### Example
Suppose that you have the task of measuring the lengths of a bridge and a rivet and come up with 9999 and 9 cm, respectively. If the true values are 10,000 and 10 cm, respectively, compute (a) the true error and (b) the true percent relative error for each case.

#### Solution
For the bridge, the True Error is:
\begin{equation}
E_\text{t} = 10,000 - 9999 = 1 \text{cm}
\end{equation}
while for the rivet
\begin{equation}
E_\text{t} = 10 - 9 = 1 \text{cm}
\end{equation}
Both measurements have the same true error. But we known intuitively that the 1 cm is a bigger deal for the rivet. We can show this by using the relative error.
For the bridge, we have
\begin{equation}
\epsilon_\text{t} = \frac{E_\text{t}}{\text{True Value}} \times 100\%= \frac{1}{10,000} \times 100\% = 0.01 \%
\end{equation}
while for the rivet,
\begin{equation}
\epsilon_\text{t} = \frac{1}{10}\times 100\% = 10 \%
\end{equation}

Note that we used the words "True Error" - this is because the definition is based on the "True Value" - but this doesn't make sense since if we know the true value then we don't need an approximation!

This definition is still important however because it lays the foundations of error analysis. In addition, when testing any numerical method, we first try it out on problems where the True Value is known (e.g. analytical solution). What we often do is figure out an approximation for the true value or find the best estimate of the true value.

Another common situation in numerical methods is iteration: repeating the same task over and over again. What we can do in that case is define an iterative error as the difference between current and previous iterations:
\begin{equation}
\epsilon_\text{a} = \frac{ \text{current approximation} - \text{previous approximation} }{ \text{current approximation} }\times 100\%
\end{equation}

The usefulness of this definition is that it allows us to get an idea of how good our approximation is. Specifically, one can use a well known formula to determine the number of significant digits in an error. If your computed error $|\epsilon_\text{a}|$ (in \%) is less than
\begin{equation}
\epsilon_\text{s} = (0.5\times 10^{2-n})\%
\end{equation}
then your results are accurate to at least $n$ significant digits.

For example, if you want your results to be significant up to at least four digits, ($n = 4$) then your results must satify $ |\epsilon_\text{a}| < 0.5\times10^{-2}\%$ or $ |\epsilon_\text{a}| < 0.005\%$.

This formula can be inverted to obtain the number of significant digits. Taking the logarigthm
\begin{equation}
\log \epsilon_\text{s} = \log 0.5 + 2 - n
\end{equation}
and
\begin{equation}
n = \log 0.5 + 2 - \log \epsilon_\text{s}
\end{equation}

## Example
The exponential function $e^x$ can be approximated with the following (Taylor) series:
\begin{equation}
e^x = 1 + x + \frac{x^2}{2} + \frac{x^3}{3!} + \frac{x^4}{4!} + \frac{x^5}{5!} + \ldots
\end{equation}
Compute the true and approximate relative errors when using this formula for 1, 2, 3, 4, 5, and 6 terms. Use this formula to compute $e^{0.5}$ up to four significant digits

### Solution
We will first need to define a function that computes the exponential up to a certain number of terms

```python
# need to import the factorial and exponential functions from the math module
import math
from math import factorial, exp
def my_exponential(x,n):
    '''x is the exponent and n is the number of terms'''
    result = 0.0 # initialize the result to zero
    for i in range(0,n): # loop over the number of terms: this goes from 0 to n-1 (n terms)
        result = result + x**i/factorial(i) # add each term at a time
    return result
```

```python
def sig_fig(error):
    return math.floor(math.log10(0.5) + 2 - math.log10(error))
```

```python
# Now run the function for different number of terms
oldval = 0.0
x = 0.5
trueval = exp(x)
for n in range(1,10):
    val = my_exponential(x,n)
    ϵt = abs( (val - trueval)/trueval ) * 100
    ϵa = abs( (val - oldval)/val ) * 100
    oldval = val
    print (n,'\t', "%.8f" % val, '\t', "%.8f" % ϵt, '\t', "%.8f" % ϵa, '\t', sig_fig(ϵa))
```

    1 	 1.00000000 	 39.34693403 	 100.00000000 	 -1
    2 	 1.50000000 	 9.02040104 	 33.33333333 	 0
    3 	 1.62500000 	 1.43876780 	 7.69230769 	 0
    4 	 1.64583333 	 0.17516226 	 1.26582278 	 1
    5 	 1.64843750 	 0.01721156 	 0.15797788 	 2
    6 	 1.64869792 	 0.00141649 	 0.01579529 	 3
    7 	 1.64871962 	 0.00010024 	 0.00131626 	 4
    8 	 1.64872117 	 0.00000622 	 0.00009402 	 5
    9 	 1.64872127 	 0.00000034 	 0.00000588 	 6

```python
from math import factorial, exp
oldval = 0.0
x = 2.0
nterms = 10
for n in range(0,nterms):
    newval = oldval + x**n/ factorial(n)
    ϵa = abs( (newval - oldval)/newval) * 100
    oldval = newval
    print(n, ϵa,'%')
```

    0 100.0 %
    1 66.66666666666666 %
    2 40.0 %
    3 21.052631578947363 %
    4 9.523809523809527 %
    5 3.669724770642201 %
    6 1.2084592145015065 %
    7 0.34408602150537515 %
    8 0.08594757198109124 %
    9 0.019095813242941077 %

```python
from math import factorial, exp
oldval = 0.0
x = 2.0
nterms = 100
tol = 0.01 # percent
ϵa = 10000
n = 0
while ϵa > tol:
    newval = oldval + x**n/ factorial(n)
    ϵa = abs( (newval - oldval)/newval) * 100
    oldval = newval
    n = n + 1
    print(n, ϵa,'%')
```

    1 100.0 %
    2 66.66666666666666 %
    3 40.0 %
    4 21.052631578947363 %
    5 9.523809523809527 %
    6 3.669724770642201 %
    7 1.2084592145015065 %
    8 0.34408602150537515 %
    9 0.08594757198109124 %
    10 0.019095813242941077 %
    11 0.0038190167941204627 %

```python
def mycos(x, nterms):
    result = 0.0
    for n in range(0,nterms):
        result += (-1)**n * x**(2*n) / factorial(2*n)
    return result
```

```python
print(mycos(3, 5), math.cos(3))
```

```python

```