# 10 ODE integrators: Verlet

```python
from importlib import reload

import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import integrators2_solution as integrators2
```

## Velocity Verlet

Use expansion *forward* and *backward* (!) in time (Hamiltons (i.e. Newton without friction) equations are time symmetric)

\begin{align}
r(t + \Delta t) &\approx r(t) + \Delta t\, v(t) + \frac{1}{2m} \Delta t^2 F(t)\\
r(t) &\approx r(t + \Delta t) - \Delta t\, v(t + \Delta t) + \frac{1}{2m} \Delta t^2 F(t+\Delta t)
\end{align}

Solve for $v$:
\begin{align}
v(t+\Delta t) &\approx v(t) + \frac{1}{2m} \Delta t \big(F(t) + F(t+\Delta t)\big)
\end{align}

Complete **Velocity Verlet** integrator consists of the first and third equation.

In practice, split into three steps (calculate the velocity at the half time step):
\begin{align}
v(t+\frac{\Delta t}{2}) &= v(t) + \frac{\Delta t}{2} \frac{F(t)}{m} \\
r(t + \Delta t) &= r(t) + \color{blue}{\Delta t\, v(t+\frac{\Delta t}{2})}= r(t) + \color{blue}{\Delta t\, v(t) + \frac{1}{2m} \Delta t^2 F(t)} \\
v(t+\Delta t) &= \color{blue}{v(t+\frac{\Delta t}{2})} + \frac{\Delta t}{2} \frac{F(t+\Delta t)}{m} = \color{blue}{v(t) + \frac{\Delta t}{2} \frac{F(t)}{m}} + \frac{\Delta t}{2} \frac{F(t+\Delta t)}{m}
\end{align}

**velocity Verlet integrator** equations
\begin{align}
v(t+\frac{\Delta t}{2}) &= v(t) + \frac{\Delta t}{2} \frac{F(t)}{m} \\
r(t + \Delta t) &= r(t) + \Delta t\, v(t+\frac{\Delta t}{2})\\
v(t+\Delta t) &= v(t+\frac{\Delta t}{2}) + \frac{\Delta t}{2} \frac{F(t+\Delta t)}{m}
\end{align}
or with steps
\begin{align}
v_{n+1/2} &= v_n + \frac{\Delta t}{2} \frac{F(r_n)}{m} \\
r_{n+1} &= r_n + \Delta t\, v_{n+1/2}\\
v_n &= v_{n+1/2} + \frac{\Delta t}{2} \frac{F(r_{n+1})}{m}
\end{align}

When writing production-level code, remember to re-use $F(t+\Delta t)$ als the "new" starting $F(t)$ in the next iteration (and don't recompute).

### Integration of planetary motion
Gravitational potential energy:
$$
U(r) = -\frac{GMm}{r}
$$
with $r$ the distance between the two masses $m$ and $M$.

#### Central forces
$$
U(\mathbf{r}) = f(r) = f(\sqrt{\mathbf{r}\cdot\mathbf{r}})\\
\mathbf{F} = -\nabla U(\mathbf{r}) = -\frac{\partial f(r)}{\partial r} \, \frac{\mathbf{r}}{r}
$$

#### Force of gravity
\begin{align}
\mathbf{F} &= -\frac{G m M}{r^2} \hat{\mathbf{r}}\\
\hat{\mathbf{r}} &= \frac{1}{\sqrt{x^2 + y^2}} \left(\begin{array}{c} x \\ y \end{array}\right)
\end{align}

#### Integrate simple planetary orbits
Set $M = 1$ (one solar mass) and $m = 3.003467×10^{-6}$ (one Earth mass in solar masses) and try initial conditions

\begin{alignat}{1}
x(0) &= 1,\quad  y(0) &= 0\\
v_x(0) &= 0,\quad  v_y(0) &= 6.179
\end{alignat}

Note that we use the following units:
* length in astronomical units (1 AU = 149,597,870,700 m )
* mass in solar masses (1 $M_☉ = 1.988435×10^{30}$ kg)
* time in years (1 year = 365.25 days, 1 day = 86400 seconds)

In these units, the gravitational constant is $G = 4\pi^2$ (in SI units $G = 6.674×10^{-11}\, \text{N}\cdot\text{m}^2\cdot\text{kg}^{-2}$).

```python
M_earth = 3.003467e-6
M_sun = 1.0
```

```python
G_grav = 4*np.pi**2

def F_gravity(r, m=M_earth, M=M_sun):
    rr = np.sum(r*r)
    rhat = r/np.sqrt(rr)
    return -G_grav*m*M/rr * rhat

def U_gravity(r, m=M_earth, M=M_sun):
    return -G_grav*m*M/np.sqrt(np.sum(r*r))
```

Let's now integrate the equations of motions under gravity with the **Velocity Verlet** algorithm:

```python
def planet_orbit(r0=np.array([1, 0]), v0=np.array([0, 6.179]),
                 mass=M_earth, dt=0.001, t_max=1):
    """2D planetary motion with velocity verlet"""
    dim = len(r0)
    assert len(v0) == dim

    nsteps = int(t_max/dt)

    r = np.zeros((nsteps, dim))
    v = np.zeros_like(r)

    r[0, :] = r0
    v[0, :] = v0

    # start force evaluation for first step
    Ft = F_gravity(r[0], m=mass)
    for i in range(nsteps-1):
        vhalf = v[i] + 0.5*dt * Ft/mass
        r[i+1, :] = r[i] + dt * vhalf
        Ftdt = F_gravity(r[i+1], m=mass)
        v[i+1] = vhalf + 0.5*dt * Ftdt/mass
        # new force becomes old force
        Ft = Ftdt

    return r, v
```

```python
r, v = planet_orbit(dt=0.1, t_max=10)
```

```python
rx, ry = r.T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

These are not closed orbits (as we would expect from a $1/r$ potential). But it gets much besser when stepsize is reduced to 0.01 (just rerun the code with `dt = 0.01` and replot):

```python
r, v = planet_orbit(dt=0.01, t_max=10)
```

```python
rx, ry = r.T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rx, ry)
```

## Velocity Verlet vs RK4: Energy conservation
Assess the stability of `rk4` and `Velocity Verlet` by checking energy conservation over longer simulation times.

The file `integrators2.py` contains almost all code that you will need.

### Implement gravity force in `integrators2.py`
Add `F_gravity` to the `integrators2.py` module. Use the new function `unitvector()`.

### Planetary orbits with `integrators2.py`

```python
r0 = np.array([1, 0])
v0 = np.array([0, 6.179])
```

```python
from importlib import reload
reload(integrators2)
```

    <module 'integrators2_solution' from '/Volumes/ASU/oliver/Documents/Teaching/ASU/CompPhys_PHY432/PHY432-resources/10_ODEs/integrators2_solution.py'>

Use the new function `integrators2.integrate_newton_2d()` to integrate 2d coordinates.

#### RK4

```python
trk4, yrk4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=integrators2.F_gravity,
                                       integrator=integrators2.rk4)
```

```python
rxrk4, ryrk4 = yrk4[:, 0, 0], yrk4[:, 0, 1]
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxrk4, ryrk4)
```

```python
integrators2.analyze_energies(trk4, yrk4, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(trk4),
        integrators2.energy_conservation(trk4, yrk4, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation RK4 for 3000 steps: 3.5366039779498594e-06

#### Euler

```python
te, ye = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                         h=0.01,
                            force=F_gravity,
                            integrator=integrators2.euler)
rex, rey = ye[:, 0].T
```

```python
ax = plt.subplot(1,1,1)
ax.plot(rx, ry, label="RK4")
ax.plot(rex, rey, label="Euler")
ax.legend(loc="best")
ax.set_aspect(1)
```

```python
integrators2.analyze_energies(te, ye, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Euler for {} steps: {}".format(
        len(te),
        integrators2.energy_conservation(te, ye, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Euler for 3000 steps: 0.6763156457080265

*Euler* is just awful... but we knew that already.

#### Velocity Verlet

```python
tv, yv = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=30, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
rxv, ryv = yv[:, 0].T
ax = plt.subplot(1,1,1)
ax.set_aspect(1)
ax.plot(rxv, ryv, label="velocity Verlet")
ax.plot(rxrk4, ryrk4, label="RK4")
ax.legend(loc="best")
```

```python
integrators2.analyze_energies(tv, yv, integrators2.U_gravity, m=M_earth)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv),
        integrators2.energy_conservation(tv, yv, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Velocity Verlet for 3000 steps: 6.384226992694312e-05

*Velocity Verlet* only has moderate accuracy, especially when compared to *RK4*.

However, let's look at energy conservation over longer times:

#### Longer time scale stability
Run RK4 and Velocity Verlet for longer.

```python
tv2, yv2 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.velocity_verlet)
```

```python
print("Energy conservation Velocity Verlet for {} steps: {}".format(
        len(tv2),
        integrators2.energy_conservation(tv2, yv2, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation Velocity Verlet for 100000 steps: 6.390030863482895e-05

```python
t4, y4 = integrators2.integrate_newton_2d(x0=r0, v0=v0, t_max=1000, mass=M_earth,
                                       h=0.01,
                                       force=F_gravity,
                                       integrator=integrators2.rk4)
```

```python
print("Energy conservation RK4 for {} steps: {}".format(
        len(t4),
        integrators2.energy_conservation(t4, y4, integrators2.U_gravity, m=M_earth)))
```

    Energy conservation RK4 for 100000 steps: 0.00011860288209883538

Velocity Verlet shows **good long-term stability** but relative low precision. On the other hand, RK4 has high precision but the accuracy decreases over time.

* Use a *Verlet* integrator when energy conservation is important and long term stability is required (e.g. molecular dynamics simulations). It is generally recommended to use an integrator that conserves some of the inherent symmetries and structures of the governing physical equations (e.g. for Hamilton's equations of motion, time reversal symmetry and the symplectic and area-preserving structure).
* Use *RK4* for high short-term accuracy (but may be difficult to know what "short term" should mean) or when solving general differential equations.

```python

```


# Schwarzschild Solution to the Einstein's equation

We will rederive the equations from *General Theory of Relativity*, chapter 18, by Dirac. They describe a spherically symmetric solution to the Einstein's equation in vacuum.

We will not try to solve the resulting equations.

This version of the notebook uses functions that explicitly calculate the Christoffel symbols, hence the work is coordinate-system-dependent.

Import the necessary modules.

```
from sympy.diffgeom import *
TP = TensorProduct
```

Define a 4D manifold with a patch and a coordinate chart on which we will work.

```
m = Manifold('Schwarzschild', 4)
p = Patch('origin', m)
cs = CoordSystem('spherical', p, ['t', 'r', 'theta', 'phi'])
```

```
m, p, cs
```

Prepare the variables containing the scalar fields and the 1-form fields.

```
t, r, theta, phi = cs.coord_functions()
t, r, theta, phi
```

```
dt, dr, dtheta, dphi = cs.base_oneforms()
dt, dr, dtheta, dphi
```

The most general spherically-symmetric metric has the following form.

```
metric = exp(2*f(r))*TP(dt, dt) - exp(2*g(r))*TP(dr, dr) - r**2*TP(dtheta, dtheta) - r**2*sin(theta)**2*TP(dphi, dphi)
metric
```

The matrix $M$ representing the two-form as a bilinear map $V,U\to V^tMU$ over the column vectors $V$ and $U$ in the canonical basis of the chosen coordinate system is:

```
twoform_to_matrix(metric)
```

Now we will calculate the components in the same basis of the Ricci tensor.

```
ricci = metric_to_Ricci_components(metric)
```

```
ricci = [[simplify(ricci[i][j])
            for j in range(4)] for i in range(4)]

```

The diagonal components give the equations we are interested in.

```
ricci[0][0]
```

```
ricci[1][1]
```

```
ricci[2][2]
```

```
ricci[3][3]
```

The off-diagonal components are zero.

```
all(ricci[i][j]==0 for i in range(4) for j in range(4) if i!=j)
```

    True

For completeness we can also check out the Christoffel symbol of 2nd kind. We will print only the non-zero components, and only one of the symmetric components (symmetric in the last two indices).

```
ch_2nd = metric_to_Christoffel_2nd(metric)
filt = [((i,j,k), simplify(ch_2nd[i][j][k]))
            for i in range(4) for j in range(4) for k in range(j,4)
            if ch_2nd[i][j][k]!=0]
```

```
filt[0:3]
```

```
filt[3:6]
```

```
filt[6:9]
```

We can also confirm that the Christoffel symbol is symmetric.

```
all([ch_2nd[k][i][j] == ch_2nd[k][j][i] for k in range(4) for i in range(4) for j in range(4)])
```

    True


# Naive Bayes classifier

The first steps are the same as previously, we start up by importing libraries.

```python
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import random
```

Defining number of points, covariance matrix and means in respective classes.

```python
n = {'A': 40, 'B': 30}
cov = np.array([[4, 0], [0, 4]])
means = {'A': np.array([-3, -1]), 'B': np.array([2, 2])}
```

Generating the points from multivariate normal distribution and inserting them into dataframe.

```python
data = pd.DataFrame(index=range(sum(n.values())), columns=['x', 'y', 'label'])
data.loc[:n['A']-1, ['x', 'y']] = np.random.multivariate_normal(means['A'], cov, n['A'])
data.loc[:n['A']-1, ['label']] = 'A'
data.loc[n['A']:, ['x', 'y']] = np.random.multivariate_normal(means['B'], cov, n['B'])
data.loc[n['A']:, ['label']] = 'B'
```

Function that plots the generated points.

```python
def plot_points(data):
    fig = plt.figure(figsize=(8, 6))
    sns.set_style('white')
    sns.set_palette('muted')
    sns.scatterplot(data=data, x='x', y='y', hue='label', legend=False, edgecolor='black').set(xlim=(-9, 9), ylim=(-9, 9))
```

```python
plot_points(data)
```

## Gaussian Naive Bayes model from *Scikit-learn*

We need to split our data into features and target variable. Then we create model with embedded `GaussianNB()` method and fit it according to the variables.

```python
from sklearn.naive_bayes import GaussianNB

X = data[['x', 'y']]
y = data['label']
clf = GaussianNB()
clf.fit(X, y);
```

With `predict_proba()` method we can compute posterior probability for any point. So to plot decision boundary for two classes we will create a rectangular meshgrid out of two arrays and compute the probability in every node of the grid.

```python
x_grid, y_grid = np.meshgrid(np.arange(-9, 9.1, 0.1), np.arange(-9, 9.1, 0.1))
prob = clf.predict_proba(np.c_[x_grid.ravel(), y_grid.ravel()])
prob = prob[:, 1].reshape(x_grid.shape)
```

Now we will define the function that plots the points with decision boundary and areas which belong to corresponding class.

```python
def plot_classified_points(data, xx, yy, prob):
    plot_points(data)
    plt.contour(xx, yy, prob, [0.5], zorder=0)
    plt.contourf(xx, yy, prob, [0, 0.5, 1], cmap='Pastel2', zorder=-1)
```

```python
plot_classified_points(data, x_grid, y_grid, prob)
```

## Own implementation of Naive Bayes classifier

Before we start implementing the algorithm we need to update the means and covariance matrices of the generated points.

```python
means = {label: np.array(np.mean(data.loc[data['label'] == label])) for label in means.keys()}
cov = {label: np.cov(data.loc[data['label'] == label][['x', 'y']].values.T.astype(float)) for label in means.keys()}
```

To implement our own classifier we need to make an assumption that all the features are conditionally independent to each other. With this assumption the posterior probability can be expressed by the following equation:

\begin{equation}
P(C|\boldsymbol{X}) = \frac{\pi_{C_{i}}P(\boldsymbol{X}|C_{i})}{P(\boldsymbol{X})} = \frac{\pi_{C_{i}}P(x|C_{i})P(y|C_{i})}{\pi_{C_{1}}P(x|C_{1})P(y|C_{1}) + \pi_{C_{2}}P(x|C_{2})P(y|C_{2})}
\end{equation}

where $\pi_{C_{i}}$ is the prior probability of the given class.

```python
from scipy.stats import multivariate_normal

def posterior_prob(prior_prob_1, prior_prob_2, mean_1, mean_2, cov_1, cov_2, x, y):
    likelihood_x_1 = multivariate_normal.pdf(x, mean_1[0], cov_1[0, 0])
    likelihood_y_1 = multivariate_normal.pdf(y, mean_1[1], cov_1[1, 1])
    likelihood_x_2 = multivariate_normal.pdf(x, mean_2[0], cov_2[0, 0])
    likelihood_y_2 = multivariate_normal.pdf(y, mean_2[1], cov_2[1, 1])

    nominator = prior_prob_2 * likelihood_x_2 * likelihood_y_2
    denominator = prior_prob_1 * likelihood_x_1 * likelihood_y_1 + prior_prob_2 * likelihood_x_2 * likelihood_y_2

    return nominator / denominator
```

We estimate class prior probabilities by dividing number of class observations by total number of observations.

```python
prior_prob_A = n['A'] / sum(n.values())
prior_prob_B = n['B'] / sum(n.values())

prob = posterior_prob(prior_prob_A, prior_prob_B, means['A'], means['B'], cov['A'], cov['B'], x_grid.ravel(), y_grid.ravel())
prob = prob.reshape(x_grid.shape)
```

Finally we can plot the decision boundary determined by our classifier.

```python
plot_classified_points(data, x_grid, y_grid, prob)
```

As we can see the results are identical both for embedded model and our own implementation.


## Introduction
-----
You (an electrical engineer) wish to determine the resistance of an electrical component by using Ohm's law. You remember from your high school circuit classes that $$V = RI$$ where $V$ is the voltage in volts, $R$ is resistance in ohms, and $I$ is electrical current in amperes. Using a multimeter, you collect the following data:

| Current (A) | Voltage (V) |
|-------------|-------------|
| 0.2         | 1.23        |
| 0.3         | 1.38        |
| 0.4         | 2.06        |
| 0.5         | 2.47        |
| 0.6         | 3.17        |

Your goal is to
1. Fit a line through the origin (i.e., determine the parameter $R$ for $y = Rx$) to this data by using the method of least squares. You may assume that all measurements are of equal importance.
2. Consider what the best estimate of the resistance is, in ohms, for this component.

## Getting Started
----

First we will import the neccesary Python modules and load the current and voltage measurements into numpy arrays:

```python
import numpy as np
from numpy.linalg import inv
import matplotlib.pyplot as plt

# Store the voltage and current data as column vectors.
I = np.mat([0.2, 0.3, 0.4, 0.5, 0.6]).T
V = np.mat([1.23, 1.38, 2.06, 2.47, 3.17]).T
```

Now we can plot the measurements - can you see the linear relationship between current and voltage?

```python
plt.scatter(np.asarray(I), np.asarray(V))

plt.xlabel('Current (A)')
plt.ylabel('Voltage (V)')
plt.grid(True)
plt.show()
```

```python

```

## Estimating the Slope Parameter
----
Let's try to estimate the slope parameter $R$ (i.e., the resistance) using the least squares formulation from Module 1, Lesson 1 - "The Squared Error Criterion and the Method of Least Squares":

\begin{align}
\hat{R} = \left(\mathbf{H}^T\mathbf{H}\right)^{-1}\mathbf{H}^T\mathbf{y}
\end{align}

If we know that we're looking for the slope parameter $R$, how do we define the matrix $\mathbf{H}$ and vector $\mathbf{y}$?

```python
# Define the H matrix, what does it contain?
# H = ...
H = np.mat([1,1,1,1,1]).T

# Now estimate the resistance parameter.
# R = ...
R_oi = np.mat(V/I)
R = (1/((H.T)*H))*(H.T)*R_oi
print('The slope parameter (i.e., resistance) for the best-fit line is:')
print(R)

```

    The slope parameter (i.e., resistance) for the best-fit line is:
    [[5.22466667]]

## Plotting the Results
----
Now let's plot our result. How do we relate our linear parameter fit to the resistance value in ohms?

```python
I_line = I
V_line = np.multiply(R,I_line)

plt.scatter(np.asarray(I), np.asarray(V))
plt.plot(I_line, V_line)
plt.xlabel('current (A)')
plt.ylabel('voltage (V)')
plt.grid(True)
plt.show()
```

If you have implemented the estimation steps correctly, the slope parameter $\hat{R}$ should be close to the actual resistance value of $R = 5~\Omega$. However, the estimated value will not match the true resistance value exactly, since we have only a limited number of noisy measurements.

```python

```


Simulación matemática 2018

Lázaro Alonso

Email:   `alonsosilva@iteso.mx, lazarus.alon@gmail.com`

### Por favor, den click al siguiente link. Será una forma fácil de que me hagan llegar sus dudas, además de que todos podremos estar al tanto de los problemas tanto de tarea como de clase.

https://join.slack.com/t/sm-grupo/shared_invite/enQtMzcxMzcxMjY1NzgyLWE5ZDlhYjg4OGJhMGE2ZmY2ZGUzZWIxYzQzOTUxZWU2ZGM5YjUyYWMyZGUzNzZjMDE5ZDIxYTA4YTI2ZWQ1NTU

### Máximos y mínimos

##### 1. Encuentre los valores máximo y mínimo locales de la función
$$g(x) = x + 2 \sin x$$

Elementos que debe de contener su respuesta:
   - Gráfica de la funcion $g(x)$
   - Gráfica de la primera y segunda derivada, ($g(x)'$, $g(x)''$)
   - Indicar en la gráfica los máximos y mínimos (_utilizar plt.scatter_)

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import sympy as sym
sym.init_printing(use_latex='mathjax')
sym.var("x")
sym.var("x", real = True)
g = x + 2*sym.sin(x)
dg = sym.diff(g,x,1)
d2g = sym.diff(g,x,2)
```

```python

x1c = sym.solve(dg, x)
x2c = sym.solve(d2g, x)
print("Valores de g´(x)=0 --> ")
print(x1c)
print("Valores de g´´(x)=0 --> ")
print(x2c)
```

    Valores de g´(x)=0 -->
    [2*pi/3, 4*pi/3]
    Valores de g´´(x)=0 -->
    [0, pi]

```python
g_num = sym.lambdify([x], g, 'numpy')
dg_num = sym.lambdify([x], dg, 'numpy')
d2g_num = sym.lambdify([x], d2g, 'numpy')
x_vec = np.linspace(0, 2*np.pi, 100)

plt.figure(figsize=(10,4))
plt.plot(x_vec, g_num(x_vec), color = "cyan", label = g)
plt.plot(x_vec, dg_num(x_vec), color = "green", label = dg)
plt.plot(x_vec, d2g_num(x_vec), color = "red", label = d2g)
plt.scatter(2*np.pi/3,(2*np.pi/3) + 2*np.sin(2*np.pi/3), color="cyan")
plt.scatter(4*np.pi/3,(4*np.pi/3) + 2*np.sin(4*np.pi/3), color="cyan")
plt.grid()
plt.legend()
plt.xlabel('$x$', fontsize = 18)
plt.ylabel('$y$', fontsize = 18)
plt.show()
```

##### 2. Discuta la curva $f(x) = x^4 - 4x^3$. Puntos de inflexión, máximos y mínimos. Su respuesta debe de incluir los mismos puntos que el caso anterior.

```python
sym.var("z")
sym.var("z", real = True)
f=z**4-4*z**3
df = sym.diff(f,z,1)
d2f = sym.diff(f,z,2)
z1c = sym.solve(df, z)
z2c = sym.solve(d2f, z)
print("Valores de f´(z)=0 --> ")
print(z1c)
print("Valores de f´´(z)=0 --> ")
print(z2c)
```

    Valores de f´(z)=0 -->
    [0, 3]
    Valores de f´´(z)=0 -->
    [0, 2]

```python
f_num = sym.lambdify([z], f, 'numpy')
df_num = sym.lambdify([z], df, 'numpy')
d2f_num = sym.lambdify([z], d2f, 'numpy')
x_vec = np.linspace(-2, 4, 100)

plt.figure(figsize=(10,4))
plt.plot(x_vec, f_num(x_vec), color = "cyan", label = f)
plt.plot(x_vec, df_num(x_vec), color = "green", label = df)
plt.plot(x_vec, d2f_num(x_vec), color = "red", label = d2f)
plt.scatter(0,0, color="black")
plt.scatter(3,3**4-4*3**3, color="orange")
plt.scatter(2,2**4-4*2**3, color="blue")
plt.grid()
plt.legend()
plt.xlabel('$x$', fontsize = 18)
plt.ylabel('$y$', fontsize = 18)
plt.show()
```

##### 3. Se va a fabricar una lata que ha de contaner 1L de aceite. Encuentre las dimensiones que debe de tener la lata de manera que minimicen el costo del metal para fabricarla.

- La solución debe de incluir los siguientes elementos
    - Ecuación del sistema
    - Números críticos
    - Dibujar la lata

```python
sym.var("w")
sym.var("w", real = True)

#1000cm3=1lt=h*pi*r**2
h = 1000/(np.pi*(w**2))
area = 2*np.pi*w**2 +h*2*np.pi*w
derarea = sym.diff(area,w,1)
print("Derivada de area: ")
print(derarea)
w1c = sym.solve(derarea, w)
print("Valores de area´(w)=0 --> ")
print(w1c)
```

    10.838764212436917
    Derivada de area:
    12.5663706143592*w - 2000.0/w**2
    Valores de area´(w)=0 -->
    [5.41926070139289]

```python

area_num = sym.lambdify([w], area, 'numpy')
derarea_num = sym.lambdify([w], derarea, 'numpy')
x_vec = np.linspace(2, 8, 100)
punto= (2*np.pi*5.4192**2) + (1000/(np.pi*(5.4192**2)))*2*np.pi*5.4192
plt.figure(figsize=(10,4))
plt.plot(x_vec, area_num(x_vec), color = "red", label = area)
plt.grid()
plt.scatter(5.4192,punto)
plt.legend()
plt.xlabel('$Radio$', fontsize = 18)
plt.ylabel('$Area Total$', fontsize = 18)
plt.show()
```

Para saber como minimizar el costo al hacer una lata, necesitamos encontrar una formula la cual nos diga cuanto metal usaremos, en este caso, necesitaremos el area de las tapas y el cilindro, por lo que podemos decir que:

$Area$ $total = 2π(r^2) + 2000/r$

Entonces, ahora que conocemos la funcion del area, necesitamos saber el punto donde el area es minima, manteniendo el volumen de 1 litro, con el teorema de Fermat, nos da que para optimizar y reducir los costos, el radio tendra que ser de 5.4192 cm y la altura de 10.8387 cm.

```python

```


# Pendulum
## Analytical small angle approximation
We start with the simple pendulum; assuming the small amplitude approximation, and the initial condition that at t=0 the pendulum is at $\theta_0=\pi/3$ and $\dot{\theta}=0$.  Then
\begin{align}
\theta&=\theta_0\cos{\omega t}\\
\omega&=\sqrt{g/l}\\
\end{align}

# A file we want to include-- pendulumParameters.py

```python
#File pendulumParameters.py

stringDiameter=0.031*0.0254 #meters
stringLength=0.627 #meters, from pivot to top of mass
stringDensity=0.000273/0.738 #kg/m
mMass=0.024245-1.260*stringDensity #kg just the mass (mass+string=24.245 g)
dMass=0.02539 #m diameter of mass, measured with caliper
stringDensity=0.00133/1.31 #kg/m, mass of entire length of string-1.33 g = mass of 1.31 m of string
stringMass=stringDensity*stringLength

#calculate length of pendulum
l=stringLength+dMass/2

#moments of Inertia
IMass=mMass*l*l  #moment of inertia of mass about pivot
IString=1.0/3*stringMass*stringLength**2 #1/3 mL**2 is Moment of intertia of rod about one end
I=IMass+IString

#center of mass
m=mMass+stringMass
lcm=(mMass*l+stringMass*stringLength/2)/m

print('Total Mass ',m)
print('lcm ',lcm,' l ',l)
print('I, IMass, IString:',I, IMass, IString)
```

    Total Mass  0.02441547495810836
    lcm  0.6311902802180227  l  0.639695
    I, IMass, IString: 0.009813975740162914 0.009730557367544595 8.34183726183206e-05

```python
import numpy as np
import matplotlib.pyplot as plt
import pendulumParameters as pend

theta_0=np.pi/3
#l=0.635 #length of pendulum, meters
l=pend.l
g=9.8 # acceleration of gravity on Earth's surface, meters/s**2
omega=np.sqrt(g/l)
T=2*np.pi/omega  #  period, seconds

nsteps0=1000
t = np.linspace(0.0, 25.0*T, nsteps0)
y0=theta_0*np.cos(omega*t)
figno=1
plt.figure(figno)
figno+=1

plt.plot(t, y0, 'b')
plt.title('Pendulum angle versus time, small angle model')
plt.xlabel('Time[s]')
plt.ylabel('Angle [radians]')
plt.show()

print('l=',l)
```

## Without the small angle approximation: simple finite difference solution

This is called "Euler integration" in NR, chapter 17.  It is good to read chapter 17 at this point.
The original equation for the pendulum is:
\begin{equation}
\ddot{\theta}=-\frac{g}{l}\sin{\theta}
\end{equation}

We write this second order equation as coupled first order equations:
\begin{equation}
\frac{d}{dt}\left(
\begin{array}{c}
\omega\\
\theta\\
\end{array}
\right)=
\left(
\begin{array}{c}
-\frac{g}{l}\sin{\theta}\\
\omega\\
\end{array}\right)
\end{equation}

Expanding the derivatives in terms of $\Delta$ we have
\begin{equation}
\left(
\begin{array}{c}
\frac{\omega(t+\Delta)-\omega(t)}{\Delta}\\
\frac{\theta(t+\Delta)-\theta(t)}{\Delta}\\
\end{array}
\right)=
\left(
\begin{array}{c}
-\frac{g}{l}\sin{\theta}\\
\omega\\
\end{array}\right)
\end{equation}

This ends up giving us a recursion relationship:
\begin{equation}
\left(
\begin{array}{c}
\omega(t+\Delta)\\
\theta(t+\Delta)\\
\end{array}
\right)=
\left(
\begin{array}{c}
-\Delta\frac{g}{l}\sin{\theta(t)}+\omega(t)\\
\Delta\omega(t)+\theta(t)\\
\end{array}\right)
\end{equation}

We write a little Python loop to calculate $\omega$ and $\theta$ using this equation:

```python
factor=100
nsteps=nsteps0*factor
t2=np.linspace(0,25*T,nsteps)
x=np.zeros(2*len(t2)).reshape(len(t2),2)
Delta=t2[1]
print('Number of steps =%d and size of steps=%f between 0 and %f s'%(len(t2),Delta,25*T))

# we will use the 0-row for theta, and the 1-row for omega
x[0,1]=0  #initial condition is pendulum at rest
x[0,0]=theta_0 #initial condition
for i in np.arange(0,len(t2)-1):
    x[i+1,1]=-Delta*g/l*np.sin(x[i,0])+x[i,1]
    x[i+1,0]=Delta*x[i,1]+x[i,0]
y=x[::factor,0]
r=y/y0
plt.figure(figno)
figno+=1
plt.plot(t,y,'k')
plt.plot(t, y0, 'b')
plt.title('Euler integrated(black) versus analytic small angle (blue) pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Angle[radian]')

plt.figure(figno)
figno+=1
plt.plot(t,r,'k')
plt.title('Ratio of Euler integration and small angle solutions to pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Ratio of amplitudes')
plt.show()

```

```python
factor=1000
nsteps=nsteps0*factor
t2=np.linspace(0,25*T,nsteps)
x=np.zeros(2*len(t2)).reshape(len(t2),2)
Delta=t2[1]
print('Number of steps =%d and size of steps=%f between 0 and %f s'%(len(t2),Delta,25*T))

# we will use the 0-row for theta, and the 1-row for omega
x[0,1]=0  #initial condition is pendulum at rest
x[0,0]=theta_0 #initial condition
for i in np.arange(0,len(t2)-1):
    x[i+1,1]=-Delta*g/l*np.sin(x[i,0])+x[i,1]
    x[i+1,0]=Delta*x[i,1]+x[i,0]
y=x[::factor,0]
r=y/y0
plt.figure(figno)
figno+=1
plt.plot(t,y,'k')
plt.plot(t, y0, 'b')
plt.title('Euler integrated(black) versus analytic small angle (blue) pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Angle[radian]')
plt.figure(figno)
figno+=1
plt.plot(t,r,'k')
plt.title('Ratio of Euler integration and small angle solutions to pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Ratio of amplitudes')
plt.show()

```

```python
factor=10000
nsteps=nsteps0*factor
t2=np.linspace(0,25*T,nsteps)
x=np.zeros(2*len(t2)).reshape(len(t2),2)
Delta=t2[1]
print('Number of steps =%d and size of steps=%f between 0 and %f s'%(len(t2),Delta,25*T))

# we will use the 0-row for theta, and the 1-row for omega
x[0,1]=0  #initial condition is pendulum at rest
x[0,0]=theta_0 #initial condition
for i in np.arange(0,len(t2)-1):
    x[i+1,1]=-Delta*g/l*np.sin(x[i,0])+x[i,1]
    x[i+1,0]=Delta*x[i,1]+x[i,0]
y=x[::factor,0]
r=y/y0
plt.figure(figno)
figno+=1
plt.plot(t,y,'k')
plt.plot(t, y0, 'b')
plt.title('Euler integrated(black) versus analytic small angle (blue) pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Angle[radian]')
plt.figure(figno)
figno+=1
plt.plot(t,r,'k')
plt.title('Ratio of Euler integration and small angle solutions to pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Ratio of amplitudes')
plt.show()

```

Our next step will be more sophisticated- using algorithms that have been worked out more carefully.  But first- some thoughts about an important principle in coding

# Explicit Verification of Code
After figuring out the model-- always!-- think of ways to test the numerical solution.  Testing and debugging code is almost always more than half the work.  Therefore- never give up a chance to do things differently/redundantly- check at the end to make sure that the algorithm is doing what you intended it to do.   Reading it over and over is natural but usually, for most people, not productive.  Looking at the output line by line-- and there are python debuggers that allow you to do that (or cut and paste the code one line at a time into the ipython window) work.  But I always try to explicitly think of tests that we can do inside the project to double check that the code is right.

For this code, what can we do?
1.  Do the second numerical derivative and compare it to the equation (ie.  substitute back in)
2.  Do time reversal, and see if we come back to the starting point.
3.  Add an explicit calculation of the total energy, and see that it is constant

Why do we do all three rather than just one?

We start with doing the second order numerical derivative:
\begin{equation}
\ddot{\theta}=\frac{g}{l}\sin{\theta}=\frac{1}{h}\left(\frac{\theta(t+h)-\theta(t)}{h}-\frac{\theta(t)-\theta(t-h)}{h}\right)=\frac{\theta(t+h)+\theta(t-h)-2\theta(t)}{h^2}
\end{equation}

```python
thetaOfT=x[1:-1,0]
thetaTPlusH=x[2:,0]
thetaTMinusH=x[:-2,0]
h=Delta
thetaDotDot=(1/h**2)*(thetaTPlusH+thetaTMinusH-2*thetaOfT)
plt.plot(t2[1:-1:factor],thetaDotDot[::factor],'k')
plt.plot(t2[1:-1:factor],-g/l*np.sin(thetaOfT[1:-1:factor]),'r')
plt.title("Numerical Derivative in black; analytic derivative in red")
plt.xlabel("Time[s]")
plt.ylabel(r'$\ddot\theta$[rad/s^2]')
plt.show()

diff=thetaDotDot[::factor]+g/l*np.sin(thetaOfT[1:-1:factor])
plt.plot(t2[1:-1:factor],diff,'b')
plt.title("Difference between desired and actual second derivative")
plt.xlabel("Time[s]")
plt.ylabel(r'Difference in $\ddot\theta$[rad/s^2]')
plt.show()
```

This would indicate that our precision is about 0.0004/10 or 4 e-5.  It needs to be a bit better than this because theta is also an output of the calculation- that is we are checking the derivative of $\theta$, which is $\omega$!

To time reverse we start with theta0, phi0 as the final value, and make the derivative be - the old derivative.

```python
x[0,1]=x[-1,1]  #initial condition is pendulum at rest
x[0,0]=x[-1,0] #initial condition
for i in np.arange(0,len(t2)-1):
    x[i+1,1]=Delta*g/l*np.sin(x[i,0])+x[i,1]
    x[i+1,0]=-Delta*x[i,1]+x[i,0]
y=x[::factor,0]
r=y/y0
plt.figure(figno)
figno+=1
plt.plot(t,y,'k')
plt.title('Euler integrated(black) versus analytic small angle (blue) pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Angle[radian]')
plt.figure(figno)
figno+=1
plt.plot(t,r,'k')
plt.title('Ratio of Euler integration and small angle solutions to pendulum')
plt.xlabel('Time[s]')
plt.ylabel('Ratio of amplitudes')
plt.show()

print('Final theta=%f omega=%f'%(x[-1,0],x[-1,1]))
print('Original theta0=%f omega0=%f'%(theta_0,0))
print('Fractional Difference=',(x[-1,0]-theta_0)/(50*np.pi))

```

Notice that this also says we have something like 2e-3.

The total energy is $E=1/2 mv^2+mgh=1/2ml^2\omega^2-mgl\cos\theta$.

```python

m=pend.m
Energy=(0.5*m*l**2)*x[::factor,1]**2-(m*g*l)*np.cos(x[::factor,0])
plt.figure(figno)
figno+=1
plt.plot(t,Energy,'r')
plt.show()
print('Energy is', Energy)
```

```python

```

But also note that we already have a physics result-- the large angle pendulum has a longer period than the "small angle" approximation by about 1 cycle in 23- or about 4-5% in this case.  That makes sense because the acceleration is smaller.  Of course, this will depend on the amplitude, and as the amplitude drops the approximation will get closer to reality.

```python

```

```python

```

Notebook Folder-20190925/.ipynb_checkpoints/02 Pendulum Calculations With Eulers Method-checkpoint.ipynb


# Taylor Series for Approximations

Taylor series are commonly used in physics to approximate functions making them easier to handle specially when solving equations. In this notebook we give a visual example on how it works and the biases that it introduces.

## Theoretical Formula

Consider a function $f$ that is $n$ times differentiable in a point $a$. Then by Taylor's theorem, for any point $x$ in the domain of f, we have the Taylor expansion about the point $a$ is defined as:
\begin{equation}
f(x) = f(a) + \sum_{k=1}^n \frac{f^{k}(a)}{k!}(x-a)^k + o\left((x-a)^n\right) \quad,
\end{equation}
where $f^{(k)}$ is the derivative of order $k$ of $f$. Usually, we consider $a=0$ which gives:
\begin{equation}
f(x) = f(0) + \sum_{k=1}^n \frac{f^{k}(0)}{k!}(x)^k + o\left((x)^n\right) \quad.
\end{equation}

For example, the exponential, $e$ is infinitely differentiable with $e^{(k)}=e$ and $e^0=1$. This gives us the following Taylor expansion:
\begin{equation}
e(x) = 1 + \sum_{k=1}^\infty \frac{x^k}{k!} \quad.
\end{equation}

## Visualising Taylor Expansion Approximation and its Bias

Let us see visually how the Taylor expansion approximatees a given function. We start by defining our function below, for example we will consider the exponential function, $e$ again up to order 3.

```python
#### FOLDED CELL
%matplotlib inline
import matplotlib.pyplot as plt
from IPython.display import Markdown as md
from sympy import Symbol, series, lambdify, latex
from sympy.functions import *
from ipywidgets import interactive_output
import ipywidgets as widgets
from sympy.parsing.sympy_parser import parse_expr
import numpy as np

x = Symbol('x')
```

```python
order = 3
func = exp(x)
```

```python
#### FOLDED CELL
taylor_exp = series(func,x,n=order+1)
approx = lambdify(x, sum(taylor_exp.args[:-1]), "numpy")
func_np = lambdify(x, func, "numpy")
latex_func = '$'+latex(func)+'$'
latex_taylor = '\\begin{equation} '+latex(taylor_exp)+' \end{equation}'
```

The Taylor expansion of  {{ latex_func }} is :
{{latex_taylor}}

Now let's plot the function and its expansion while considering a point, noted $p$, to study the biais that we introduce when we approximate the function by its expansion:

```python
#### FOLDED CELL
order = widgets.IntSlider(min=0,max=20,step=1,value=3,description='Order')
x_min = -4
x_max = 4
x1 = widgets.FloatSlider(min=x_min,max=x_max,value=3,step=0.2,description='Point Absciss')
func = widgets.Text('exp(x)',description='Function')
text_offset = np.array([-0.15,2.])

ui = widgets.HBox([x1, order, func])

def f(order=widgets.IntSlider(min=1,max=10,step=1,value=3)
      ,x1=1.5
      ,func='exp(x)'):
    func_sp = parse_expr(func)
    taylor_exp = series(func_sp,x,n=order+1)
    approx = lambdify(x, sum(taylor_exp.args[:-1]), "numpy")
    func_np = lambdify(x, func_sp, "numpy")
    n_points = 1000
    x_array = np.linspace(x_min,x_max,n_points)
    approx_array = np.array([approx(z) for z in x_array])
    func_array = np.array([func_np(z) for z in x_array])
    func_x1 = func_np(x1)
    approx_x1 = approx(x1)
    plt.figure(42,figsize=(10,10))
    plt.plot(x_array,approx_array,color='blue',label='Taylor Expansion')
    plt.plot(x_array,func_array,color='green',label=func)
    plt.plot(0,approx(0),color='black',marker='o')
    plt.annotate(r'(0,0)',[0,approx(0)],xytext=text_offset)
    plt.plot([x1,x1]
             ,[-np.max(np.abs([np.min(func_array),np.max(func_array)])),min(approx_x1, func_x1)]
             ,'--',color='black',marker='x')
    plt.plot([x1,x1],[approx_x1, func_x1],'r--',marker='x')
    plt.annotate(r'$p_{approx}$',[x1,approx(x1)],xytext=[x1,approx(x1)]-text_offset)
    plt.annotate(r'$p$',[x1,func_np(x1)],xytext=[x1,func_np(x1)]-text_offset)
    plt.xlim([x_min,x_max])
    plt.ylim(-np.max(np.abs([np.min(func_array),np.max(func_array)]))
             ,np.max(np.abs([np.min(func_array),np.max(func_array)])))
    plt.legend()
    plt.show()
    print('Approximation  bias : {}'.format(func_x1-approx_x1))
    return None
interactive_plot = widgets.interactive_output(f, {'order': order, 'x1': x1, 'func': func})
interactive_plot.layout.height = '650px'
display(interactive_plot, ui)
```

    Output(layout=Layout(height='650px'))

    HBox(children=(FloatSlider(value=3.0, description='Point Absciss', max=4.0, min=-4.0, step=0.2), IntSlider(val…

Notice that the further $p$ gets away from the point of the expansion (in that case $0$), the higher the approximation bias gets. Samely, the lower the order of approximation is, the higher the approximation bias gets.


Determine a polynomial $f(x)$ of degree 3 from the following information:

- $f(x)$ has a stationary point at $P(0|4)$
- $f(x)$ has an inflection point at $Q(2|2)$

Plot the graph of the resulting function for $-2.1 \le x \le 6.1$.

## Solution

The first step consists of some initialisations

```python
# Initialisations
from sympy import *
init_printing()

import matplotlib.pyplot as plt
%matplotlib inline
%config InlineBackend.figure_format='retina' # for OSX only

import numpy as np

from IPython.display import display, Math

from fun_expr import Function_from_Expression as FE
```

Then, the function $f(x)$ is defined as

$$
  f(x) = a\,x^3 + b\,x^2 + c\,x + d
$$

with unknown coefficients $a\cdots d$.

```python
# define f

x = Symbol('x')

a,b,c,d = symbols('a,b,c,d')

f = FE(x, a*x**3 + b*x**2 + c*x + d)
f_1 = f.diff(x)
f_2 = f.diff(x,2)

display(Math("f(x)="+latex(f(x))))
display(Math("f'(x)="+latex(f_1(x))))
display(Math("f''(x)="+latex(f_2(x))))
```

$$f(x)=a x^{3} + b x^{2} + c x + d$$

$$f'(x)=3 a x^{2} + 2 b x + c$$

$$f''(x)=6 a x + 2 b$$

The unknown coefficients are determined by the conditions

\begin{align*}
  f(x_s) &= y_s \\
  f(x_i) &= y_i \\
  f'(x_s) &= 0 \\
  f''(x_i) &= 0 \\
\end{align*}

Here, $(x_s|y_s)$ is the stationary point and $(x_i|y_i)$ the inflection point.

```python
# known information
x_s, y_s = 0,4
x_i, y_i = 2,2
p_s = (x_s,y_s) # stationary point
p_i = (x_i,y_i) # inflection point

# equations
eqns = [Eq(f(x_s),y_s),
        Eq(f_1(x_s),0),
        Eq(f(x_i),y_i),
        Eq(f_2(x_i),0)]

for eq in eqns:
    display(eq)
```

The resulting system of equations is solved

```python
# solve equations
sol = solve(eqns)
sol
```

... and the solution substituted into $f$, $f'$ and $f''$.

```python
# substitute solution
f = f.subs(sol)
f_1 = f_1.subs(sol)
f_2 = f_2.subs(sol)
display(Math('f(x)='+latex(f(x))))
display(Math("f'(x)="+latex(f_1(x))))
display(Math("f''(x)="+latex(f_2(x))))
```

$$f(x)=\frac{x^{3}}{8} - \frac{3 x^{2}}{4} + 4$$

$$f'(x)=\frac{3 x^{2}}{8} - \frac{3 x}{2}$$

$$f''(x)=\frac{3 x}{4} - \frac{3}{2}$$

The resulting function $f(x)$ is plotted over $-2.1 \le x \le 6.1$

```python
# define new plot
fig, ax = plt.subplots()

# x-values
lx = np.linspace(-2.1,6.1)

# plot
ax.plot(lx,f.lambdified(lx),label=r'$y={f}$'.format(f=latex(f(x))))
ax.scatter(*zip(*[p_s,p_i]))

# refine plot
ax.axhline(0,c='k')
ax.axvline(0,c='k')
ax.grid(True)
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend(loc='best')

# show plot
plt.show()
```


# 解方程

和计算算式不同,解方程是知道结果求变量,作为符号运算工具,sympy在解方程方面非常直观

```python
from sympy import init_printing
init_printing(use_unicode=True)
```

```python
from sympy import symbols
x, y, z = symbols('x y z')
```

## 相等

python中`=`是赋值,`==`是判断相等,而在SymPy中使用函数`Eq(exp1,exp2)`来表示两个算式相等

```python
from sympy import Eq
```

```python
Eq(x,y)
```

## 求解方程

求解方程使用`solveset(Eq(expr, result), var)`来实现.求解方程本质上是求方程的解集,因此其返回值是一个集合,集合计算会在后面介绍

```python
from sympy import solveset
```

```python
solveset(Eq(x**2, 1), x)
```

因为就一个参数所以x可以省略

```python
solveset(Eq(x**2, 1))
```

如果第一个参数不是等式,那么默认的`solveset`将会把它作为等于0处理
`solveset(expr, var)`

```python
solveset(x**2-1, x)
```

其实`solveset(equation, variable=None, domain=S.Complexes)->Set`才是solveset的完整接口,而domain则表示域(取值范围集合),SymPy支持的域包括:

+ `S.Naturals`表示自然数(或计数数),即从1开始的正整数($ℕ$)
+ `S.Naturals0`非负整数($ℕ0$)
+ `S.Integers`整数($Z$)
+ `S.Reals`实数($R$)
+ `S.Complexes`复数($C$)
+ `S.EmptySet`空域($\emptyset$)

```python
from sympy import S,sin,cos,exp
```

```python
solveset(x - x, x, domain=S.Reals)
```

```python
solveset(sin(x) - 1, x, domain=S.Reals)
```

如果无解,那么结果就是一个空集

```python
solveset(exp(x), x)     # No solution exists
```

```python
solveset(cos(x) - x, x)
```

## 求方程组

求方程组可以用`linsolve(exps...,(vars...))-> Set`函数

```python
from sympy import linsolve
```

```python
linsolve([x + y + z - 1,
          x + y + 2*z - 3 ], (x, y, z))
```

当然也可以使用矩阵的方式,矩阵将在下一部分学习

```python
from sympy import Matrix
```

```python
A=Matrix([[1,1,1],[1,1,2]])
```

```python
b = Matrix([[1],[3]])
```

```python
linsolve((A,b,),(x,y,z,))
```

## 求解微分方程

`dsolve()`是微分方程的求解函数

```python
from sympy import Function,dsolve
```

```python
f, g = symbols('f g', cls=Function)
```

```python
f(x)
```

```python
f(x).diff(x)
```

```python
diffeq = Eq(f(x).diff(x, x) - 2*f(x).diff(x) + f(x),
            sin(x))
```

```python
diffeq
```

```python
dsolve(diffeq, f(x))
```

```python
f(x).diff(x)*(1 - sin(f(x)))
```

```python
dsolve(f(x).diff(x)*(1 - sin(f(x))),
       f(x))
```


# Revisão de conceitos estatísticos I

Vamos explorar alguns conceitos estatísticos aplicados à análise de sinais.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
import IPython.display as ipd # to play signals
import sounddevice as sd
```

# Exemplo 1: Três pessoas lançam uma moeda 10 vezes

Assim, teremos um registro "temporal" do lançamento da moeda. Vamos dizer que

- Ao conjunto de registros temporais chamamos de ***Ensemble***
- A cada $n$ em $n_{\text{eventos}}$, temos um ***evento***. Cada ***evento*** (cara ou coroa) tem associado a si uma ***variável aleatória***, definida como
    - Cara = 0
    - Coroa = 1

- O universo amostral é: $\Omega = (0, 1)$
- Podemos calcular a probavilidade do evento 'cara' tomando $p(\text{cara}) = n(\text{cara})/n_{\text{eventos}}$. Esta é uma visão frequentista do fenômeno.

```python
n_eventos = np.arange(1,11)
p1 = np.random.randint(2, size=len(n_eventos))
p2 = np.random.randint(2, size=len(n_eventos))
p3 = np.random.randint(2, size=len(n_eventos))

print("Prob. de caras em p1 é de: {:.2f}".format(len(p1[p1==1])/len(p1)))
print("Prob. de caras em p2 é de: {:.2f}".format(len(p2[p2==1])/len(p2)))
print("Prob. de caras em p3 é de: {:.2f}".format(len(p3[p3==1])/len(p3)))

plt.figure(figsize = (12,6))
plt.subplot(3,1,1)
plt.bar(n_eventos, p1, color = 'lightblue')
plt.xticks(n_eventos)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Eventos')
plt.ylabel('V.A. [-]')
plt.xlim((0, len(n_eventos)+1))
plt.ylim((0, 1.2))

plt.subplot(3,1,2)
plt.bar(n_eventos, p2, color = 'lightcoral')
plt.xticks(n_eventos)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Eventos')
plt.ylabel('V.A. [-]')
plt.xlim((0, len(n_eventos)+1))
plt.ylim((0, 1.2))

plt.subplot(3,1,3)
plt.bar(n_eventos, p3, color = 'lightgreen')
plt.xticks(n_eventos)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Eventos')
plt.ylabel('V.A. [-]')
plt.xlim((0, len(n_eventos)+1))
plt.ylim((0, 1.2))
plt.tight_layout();
```

# Exemplo 2: Ruído aleatório com distribuição normal

Neste exemplo nós consideramos um sinal gerado a partir do processo de obter amostras de uma distribuição normal, dada por:

\begin{equation}
p(x) = \mathcal{N}(\mu_x, \sigma_x) = \frac{1}{\sqrt{2\pi}\sigma_x}\mathrm{e}^{-\frac{1}{2\sigma_x^2}(x-\mu_x)^2}
\end{equation}
em que $\mu_x$ é a média e $\sigma_{x}$ é o desvio padrão.

Imaginemos então, que a cada instante de tempo $t$, nós sorteamos um valor da distribuição normal.

```python
fs = 44100
time = np.arange(0,2, 1/fs)

# sinal
mu_x = 0
sigma_x = 1.0
xt = np.random.normal(loc = mu_x, scale = sigma_x, size=len(time))

# Figura
fig, axs = plt.subplots(1, 2, gridspec_kw={'width_ratios': [8, 2]}, figsize = (12, 4))
axs[0].plot(time, xt, '-b', linewidth = 1, color = 'lightcoral')
axs[0].grid(linestyle = '--', which='both')
axs[0].set_xlabel('Tempo [s]')
axs[0].set_ylabel('Amplitude [-]')
axs[0].set_xlim((0, 0.05))
axs[0].set_ylim((-4, 4))

axs[1].hist(xt, density = True, orientation='horizontal', bins=np.linspace(-4*sigma_x, 4*sigma_x, 20), color = 'lightcoral')
axs[1].grid(linestyle = '--', which='both')
axs[1].set_xlabel('Densidade [-]')
axs[1].set_ylabel(r'Valores de $x(t)$ [-]')
axs[1].set_ylim((-4, 4))

plt.tight_layout()

ipd.Audio(xt, rate=fs) # load a NumPy array
```

```python

```

Aula 49 - Rev conceitos estatisticos I/.ipynb_checkpoints/conceitos estatisticos I-checkpoint.ipynb


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.0 (Python 2.7.12-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

# Euler Equations

The Euler equations in primitive variable form, $q = (\rho, u, p)^\intercal$ appear as:

$$q_t + A(q) q_x = 0$$

with the matrix $A(q)$:

$$A(q) = \left ( \begin{array}{ccc} u  & \rho     & 0 \\
                                  0  &  u       & 1/\rho \\
                                  0  & \gamma p & u \end{array} \right )
$$

The sound speed is related to the adiabatic index, $\gamma$, as $c^2 = \gamma p /\rho$.

We can represent this matrix symbolically in SymPy and explore its eigensystem.

```python
from sympy.abc import rho
rho, u, c = symbols('rho u c')

A = Matrix([[u, rho, 0], [0, u, rho**-1], [0, c**2 * rho, u]])
A
```

The eigenvalues are the speeds at which information propagates with.  SymPy returns them as a
dictionary, giving the multiplicity for each eigenvalue.

```python
A.eigenvals()
```

The right eigenvectors are what SymPy gives natively.  For a given eigenvalue, $\lambda$, these
satisfy:

$$A r = \lambda r$$

## Right Eigenvectors

```python
R = A.eigenvects()   # this returns a tuple for each eigenvector with multiplicity -- unpack it
r = []
lam = []
for (ev, _, rtmp) in R:
    r.append(rtmp[0])
    lam.append(ev)

# we can normalize them anyway we want, so let's make the first entry 1
for n in range(len(r)):
    v = r[n]
    r[n] = v/v[0]
```

### 0-th right eigenvector

```python
r[0]
```

this corresponds to the eigenvalue

```python
lam[0]
```

### 1-st right eigenvector

```python
r[1]
```

this corresponds to the eigenvalue

```python
lam[1]
```

### 2-nd right eigenvector

```python
r[2]
```

this corresponds to the eigenvalue

```python
lam[2]
```

Here they are as a matrix, $R$, in order from smallest to largest eigenvalue

```python
R = zeros(3,3)
R[:,0] = r[1]
R[:,1] = r[0]
R[:,2] = r[2]
R
```

## Left Eigenvectors

The left eigenvectors satisfy:

$$l A = \lambda l$$

SymPy doesn't have a method to get left eigenvectors directly, so we take the transpose of this expression:

$$(l A)^\intercal = A^\intercal l^\intercal = \lambda l^\intercal$$

Therefore, the transpose of the left eigenvectors, $l^\intercal$, are the right eigenvectors of transpose of $A$

```python
B = A.transpose()
B
```

```python
L = B.eigenvects()
l = []
laml = []
for (ev, _, ltmp) in L:
    l.append(ltmp[0].transpose())
    laml.append(ev)

```

Traditionally, we normalize these such that $l^{(\mu)} \cdot r^{(\nu)} = \delta_{\mu\nu}$

```python
for n in range(len(l)):
    if lam[n] == laml[n]:
        ltmp = l[n]
        p = ltmp.dot(r[n])
        l[n] = ltmp/p
```

### 0-th left eigenvector

```python
l[0]
```

### 1-st left eigenvector

```python
l[1]
```

### 2-nd left eigenvector

```python
l[2]
```

```python

```

# Entropy formulation

here we write the system in terms of $q_s = (\rho, u, s)^\intercal$, where the system is

$${q_s}_t + A_s(q_s) {q_s}_x = 0$$

and

$$
A_s = \left (\begin{matrix}u & \rho & 0\\
      \frac{c^{2}}{\rho} & u & \frac{p_{s}}{\rho}\\
         0 & 0 & u\end{matrix}\right)
         $$

```python
ps = symbols('p_s')

As = Matrix([[u, rho, 0], [c**2/rho, u, ps/rho], [0, 0, u]])
As
```

```python
As.eigenvals()
```

```python
R = As.eigenvects()   # this returns a tuple for each eigenvector with multiplicity -- unpack it
r = []
lam = []
for (ev, _, rtmp) in R:
    r.append(rtmp[0])
    lam.append(ev)

# we can normalize them anyway we want, so let's make the first entry 1
for n in range(len(r)):
    v = r[n]
    r[n] = v/v[0]
```

```python
r[0], lam[0]
```

```python
r[1], lam[1]
```

```python
r[2], lam[2]
```

### left eigenvectors

```python
Bs = As.transpose()
L = B.eigenvects()
l = []
laml = []
for (ev, _, ltmp) in L:
    l.append(ltmp[0].transpose())
    laml.append(ev)

```

normalization

```python
for n in range(len(l)):
    if lam[n] == laml[n]:
        ltmp = l[n]
        p = ltmp.dot(r[n])
        l[n] = ltmp/p
```

```python
simplify(l[0])
```

```python
l[1]
```

```python
l[2]
```

# 2-d system

```python
rho, u, v, c = symbols('rho u v c')

A = Matrix([[u, rho, 0, 0], [0, u, 0, rho**-1], [0,0, u, 0], [0, c**2 * rho, 0, u]])
A
```

```python
A.eigenvals()
```

```python
R = A.eigenvects()   # this returns a tuple for each eigenvector with multiplicity -- unpack it
r = []
lam = []
for (ev, _, rtmp) in R:
    for rv in rtmp:
        r.append(rv)
        lam.append(ev)

# we can normalize them anyway we want, so let's make the first entry 1
for n in range(len(r)):
    v = r[n]
    if not v[0] == 0:
        r[n] = v/v[0]
```

```python
r[0], lam[0]
```

```python
r[1], lam[1]
```

```python
r[2], lam[2]
```

```python
r[3], lam[3]
```

```python

```


## Finding expressions for elastic moduli

I want to find more expressions to fill in the gaps in [my matrix](http://www.subsurfwiki.org/wiki/Template:Elastic_modulus):

```python
from IPython.display import Image
Image("data/moduli.png")
```

We can use [`sympy`](http://docs.sympy.org/dev/tutorial/basic_operations.html) to help!

```python
import sympy
sympy.__version__
```

    '1.0'

```python
sympy.init_printing(use_latex='mathjax')
```

We must define our symbols. We'll use `alpha` and `beta` for Vp and Vs.

```python
alpha, beta, gamma = sympy.symbols("alpha, beta, gamma")
lamda, mu, E, K, M, rho = sympy.symbols("lamda, mu, E, K, M, rho")
nu = sympy.symbols("nu")
```

## Expression for Vp (alpha) in terms of K, E

```python
from sympy import sqrt
alpha_expr = sqrt((mu * (E - 4*mu)) / (rho * (E - 3*mu)))
alpha_expr
```

$$\sqrt{\frac{\mu \left(E - 4 \mu\right)}{\rho \left(E - 3 \mu\right)}}$$

```python
print(sympy.latex(alpha_expr))
```

    \sqrt{\frac{\mu \left(E - 4 \mu\right)}{\rho \left(E - 3 \mu\right)}}

```python
mu_expr = (3 * K * E) / (9 * K - E)
```

```python
subs = alpha_expr.subs(mu, mu_expr)
subs
```

$$\sqrt{3} \sqrt{\frac{E K \left(- \frac{12 E K}{- E + 9 K} + E\right)}{\rho \left(- E + 9 K\right) \left(- \frac{9 E K}{- E + 9 K} + E\right)}}$$

```python
print(sympy.latex(subs))
```

    \sqrt{3} \sqrt{\frac{E K \left(- \frac{12 E K}{- E + 9 K} + E\right)}{\rho \left(- E + 9 K\right) \left(- \frac{9 E K}{- E + 9 K} + E\right)}}

```python
from sympy import simplify
simplify(subs)
```

$$\sqrt{3} \sqrt{\frac{K \left(E + 3 K\right)}{\rho \left(- E + 9 K\right)}}$$

```python
print(sympy.latex(simplify(subs)))
```

    \sqrt{3} \sqrt{\frac{K \left(E + 3 K\right)}{\rho \left(- E + 9 K\right)}}

## Expression for Vs (beta) in terms of K, E

```python
beta_expr = sqrt(mu/rho)
simplify(beta_expr.subs(mu, mu_expr))
```

$$\sqrt{3} \sqrt{- \frac{E K}{\rho \left(E - 9 K\right)}}$$

```python
simpl = simplify(beta_expr.subs(mu, mu_expr))
```

```python
print(sympy.latex(simpl))
```

    \sqrt{3} \sqrt{- \frac{E K}{\rho \left(E - 9 K\right)}}

## Expression for Vp/Vs (gamma) in terms of K, E

```python
gamma_expr = sqrt((K + (4*mu/3)) / mu)
simpl = simplify(gamma_expr.subs(mu, mu_expr))
simpl
```

$$\sqrt{\frac{1}{E} \left(E + 3 K\right)}$$

```python
print(sympy.latex(simpl))
```

    \sqrt{\frac{1}{E} \left(E + 3 K\right)}

## Expression for Vp/Vs (gamma) in terms of E, mu

Not totally sure why I have to use that hacky way to get the terms to cnacel properly.

```python
gamma_emu_expr = alpha_expr / beta_expr
simpl = sqrt(gamma_emu_expr**2)
simpl
```

$$\sqrt{\frac{E - 4 \mu}{E - 3 \mu}}$$

```python
print(sympy.latex(simpl))
```

    \sqrt{\frac{E - 4 \mu}{E - 3 \mu}}

## Expression for Vp/Vs (gamma) in terms of PR, mu

```python
e_expr = 2 * mu * (1 + nu)
```

```python
simplify(sqrt(gamma_emu_expr.subs(E, e_expr)**2))
```

$$\sqrt{2} \sqrt{\frac{\nu - 1}{2 \nu - 1}}$$

```python
print(sympy.latex(simplify(sqrt(gamma_emu_expr.subs(E, e_expr)**2))))
```

    \sqrt{2} \sqrt{\frac{\nu - 1}{2 \nu - 1}}

## Expression for Vp in terms of E, lamda

```python
vp_expr = sympy.Eq(alpha, sqrt(mu*(E - 4*mu) / (rho*(E - 3*mu))))
vp_expr
```

$$\alpha = \sqrt{\frac{\mu \left(E - 4 \mu\right)}{\rho \left(E - 3 \mu\right)}}$$

```python
alpha_expr = sqrt(mu*(E - 4*mu) / (rho*(E - 3*mu)))
```

```python
mu_expr = (rho * alpha**2 - lamda)/2
```

```python
new_expr = simplify(vp_expr.subs(mu, mu_expr))
```

```python
new_expr.subs(alpha, alpha_expr)
```

$$\sqrt{\frac{\mu \left(E - 4 \mu\right)}{\rho \left(E - 3 \mu\right)}} = \sqrt{\frac{\left(- \lambda + \frac{\mu \left(E - 4 \mu\right)}{E - 3 \mu}\right) \left(E + 2 \lambda - \frac{2 \mu \left(E - 4 \mu\right)}{E - 3 \mu}\right)}{\rho \left(2 E + 3 \lambda - \frac{3 \mu \left(E - 4 \mu\right)}{E - 3 \mu}\right)}}$$

OK, I give up. Trying Wolfram Alpha...

```python
vp_wolfram = simplify(sqrt(-lamda/rho+sqrt(9*lamda**2+E**2+2*lamda*E)/rho+E/rho)/sqrt(2))**2
sqrt(vp_wolfram)
```

$$\frac{\sqrt{2}}{2} \sqrt{\frac{1}{\rho} \left(E - \lambda + \sqrt{E^{2} + 2 E \lambda + 9 \lambda^{2}}\right)}$$

That's better!

```python
print(sympy.latex(sqrt(vp_wolfram)))
```

    \frac{\sqrt{2}}{2} \sqrt{\frac{1}{\rho} \left(E - \lambda + \sqrt{E^{2} + 2 E \lambda + 9 \lambda^{2}}\right)}

Another for Wolfram Alpha: Vs

    solve V = sqrt((a/(2*((y-2 r V^2)/(2 r V^2))*r)) - (a/r)) for V

```python
simplify(sqrt(sqrt(9*lamda**2+2*lamda*E+E**2)/rho-(3*lamda)/rho+E/rho)/2)**2
```

$$\frac{1}{4 \rho} \left(E - 3 \lambda + \sqrt{E^{2} + 2 E \lambda + 9 \lambda^{2}}\right)$$

Another for Wolfram Alpha: Gamma, or Vp/Vs

    solve G = sqrt(((4/3)*a - 2*((y*G^2 - y)/3))/(a - ((y*G^2 - y)/3))) for G

```python
simplify(sqrt(sqrt(9*lamda**2+2*lamda*E+E**2)/E+(3*lamda)/E+3)/sqrt(2))**2
```

$$\frac{1}{2 E} \left(3 E + 3 \lambda + \sqrt{E^{2} + 2 E \lambda + 9 \lambda^{2}}\right)$$

## Expression for Vs (beta) in terms of nu, K

```python
mu_expr = 3*K*(1 - 2*nu)/(2 + 2*nu)
simpl = simplify((sqrt(mu/rho)).subs(mu,mu_expr)**2)
simpl
```

$$- \frac{3 K \left(2 \nu - 1\right)}{2 \rho \left(\nu + 1\right)}$$

```python
print(sympy.latex(simpl))
```

    - \frac{3 K \left(2 \nu - 1\right)}{2 \rho \left(\nu + 1\right)}

## Expression for Vp (alpha) in terms of nu, K

```python
vp_expr = sqrt(lamda * (1 - nu) / (rho * nu))
l_expr = 3 * K * nu / (1 + nu)
vp_expr.subs(lamda, l_expr)
```

$$\sqrt{3} \sqrt{\frac{K \left(- \nu + 1\right)}{\rho \left(\nu + 1\right)}}$$

```python

```


# HW 03

1.  The lowest and highest bins are filled.  We expect clipping in the data.

2.  Increase sampling frequency till spectra does not change anymore.

3.  Here, we want the resolution $\Delta V$ to be 0.1% of the range (full scale: $FS = V_{max} - V_{min}$).

\begin{align}
\frac{\Delta V}{FS} & = \frac{1}{V_{max} - V_{min}}  \cdot  \frac{V_{max} - V_{min}}{2^{N}} \\
 & = \frac{1}{2^{N}} \lt 0.001
\end{align}

This implies $N = 10$ bit,  ($2^{10} = 1,024$).

4. The signal is:

\begin{align}
y(t) = 1.1 + 3.9 \sin(20,000 \pi t)
\end{align}
Its amplitude will be between 2.8 and 5 V, and the frequency is $f = 10,000$ Hz.  We need to make sure the DAQ system has a high enough sampling rate to prevent aliasing and large enough input rate to prevent clipping.  Then we will select the DAQ system with the best resolution (or the smallest quantization error).

(a): sampling rate is too low to satisfy Nyquiest criterion

(b): input range is too small, we would like to have a safety margin of 20-40 %.

(c) \& (d): sampling rate and input range are ok.

```python
Q_c = 20/2**(8+1)

Q_d = 20/2**(16+1)

print('Q (c) = ', Q_c, ' V,  Q (d) = ', Q_d, ' V')
```

    Q (c) =  0.0390625  V,  Q (d) =  0.000152587890625  V

Case (d) is the best design.

5.
\begin{align}
y(t) = 0.03 + 0.05 \cos (1,000 \pi t)
\end{align}

First, we select the gain:  Voltage range: $A_{PT} = 0.02 - 0.08$ V.  The smallest range of $\pm 0.01$ V is the best and has adequate safety margin to guaranty that there will be not clipping.

Then, we select the sampling rate: The signal is periodic and we will optimize the frequency resolution to make sure we do not have aliazing.  The signal frequency is $f = 500$ Hz, to satisfy Nyquiest criterion, we would a sample rate $f_s > 2\cdot f > 1,000$ Hz.  An appropriate sampling frequency would be in the range $f_s = [1,024 - 4,000]$ Hz.

6.  (a) The ideal input range on the DAQ would be: $\pm 60-80$ mV.  We will use an amplifier with an output voltage: $A_{amp}\pm 80$ mV for the analysis here.

The gain should be: $G = A_{amp} / A_{PT} = 80/8 = 10$

(b) Inverting amplifiers are less susceptible to electromagnetic noise, so we should build an amplifier stage based on inverting amplifiers.  However, inverting amplifiers are susceptible to impedance loading, so it would be safe to use a buffer on the input of the amplifier stage.

7.  (a) The desired bandwidth is $f = 100,000$ Hz.  The amplifier gain bandwidth product is $GBP = 1 MHz$.  Remembering the definition of $GBP$:

\begin{align*}
GBP & = G_{theoretical} \times f_c
\end{align*}
So for us the $f_c = f = 100,000$ kHz.  This forces a theoretical gain per amplifier stage that should not exceed $G_{theoretical} = GBP / f_c = 10^6/10^5 = 10$.

So for a total desired gain of $G=100$, we should use at least two stages of $G_{stage} = G_{theoretical} = 10$ in series.

(b)  You have two options to build your circuit:

1- Use two non-inverting amplifiers of gain $G=10$ in series.  Non-inverting amplifiers have infinite input impedance and not susceptible to impedance loading.

2- Use two inverting amplifiers of gain $G=-10$ in series.  Now you have have to worry about impedance loading, so you should put a buffer first, then the two inverting amplifiers.

8.  For a Butterworth high-pass filter of order $n$ the gain is:

\begin{align*}
G = \frac{1}{\sqrt{1+\left( \frac{f_{cutoff}}{f} \right)^{2n} }}
\end{align*}

Here, $f$ is the noise: $f = 60$ Hz, the cutoff frequency is $f_{cutoff} = 300$ Hz.

```python
import numpy

f = 60 # Hz
f_c = 300 # Hz

G_1 = 1/numpy.sqrt(1+(f_c/f)**(2*1))
G_2 = 1/numpy.sqrt(1+(f_c/f)**(2*2))
G_4 = 1/numpy.sqrt(1+(f_c/f)**(2*4))

print('the gains in percentage are:')

print("order 1: G = %2.4f " % (G_1*100), '%')
print("order 2: G = %2.4f " % (G_2*100), '%')
print("order 4: G = %2.4f " % (G_4*100), '%')

# convert to dB:
G_1dB = 20*numpy.log10(G_1)
G_2dB = 20*numpy.log10(G_2)
G_4dB = 20*numpy.log10(G_4)

print('the gains in dB are: ')

print("order 1: G = %2.4f " % (G_1dB), 'dB')
print("order 2: G = %2.4f " % (G_2dB), 'dB')
print("order 4: G = %2.4f " % (G_4dB), 'dB')

```

    the gains in percentage are:
    order 1: G = 19.6116  %
    order 2: G = 3.9968  %
    order 4: G = 0.1600  %
    the gains in dB are:
    order 1: G = -14.1497  dB
    order 2: G = -27.9657  dB
    order 4: G = -55.9176  dB

9.  The signal could be written analytically as:
\begin{align}
y(t) = 1.50 \sin(40 \pi t) + 0.20 \sin(20,000 \pi t)
\end{align}
So the carrier frequency is $f_{carrier} = 20$ Hz, and the noise frequency is $f_{noise} = 10,000$ Hz.  We acquire $N=2^{12}$ points.  The DAQ has $n=16$ bits.
We also have $f_s = 30,000$ Hz.

(a) The frequency resolution is: $\Delta f = f_s/N$.

The quantization error is: $Q = (V_{max}-V_{min})/2^{n+1}$

```python
f_s = 30000 # Hz
N = 2**12 # points
n = 16 # # bits
Q = 20/2**(n+1) # quantization error in V

print("\u0394 f = %4.4f" %(f_s/N), 'Hz')
print("Q = %1.4f" %(Q*1000), 'mV')
```

    Δ f = 7.3242 Hz
    Q = 0.1526 mV

(b)  This takes 3 steps.

_Step 1_ Select the type of filter and cutoff frequency, $f_c$: Here we want to remove high frequency noise, so we will use a low-pass filter.  As a rule of thumb, it should be at least $10>f_{carrier}$ and $10<f_{noise}$.  Here any values between: $f_c = [200-1,000]$ Hz is acceptable.  I would select the lowest to more efficiently filter the noise: $f_c = 200$ Hz.

_Step 2_  Select the gain of the amplifier to make the most of the DAQ system: Ideally, the amplitude of the carrier should be 60-80% of the input range of the DAQ.  So the gain should be in the range: $G = [6-8]/1.5 = 4-5.3$.  Let's select: $G = 5$.

_Step 3_  Filter the signal: We wish for the noise amplitude to be smaller than the quantization error $Q$ of our DAQ system.  Remember that the low pass filter has to be applied to the amplified signal.  So after amplification, the noise has amplitude: $A_{noise} = 5\times 0.20 = 1$ V.

```python
A_noise = 1 #V

f_c = 200 # Hz
f_noise = 10000 # Hz

# try LPF of order 1
n_LPF = 1
G = 1/numpy.sqrt(1+(f_noise/f_c)**(2*n_LPF))
print('order ',n_LPF,' : A_noise', (G*A_noise), 'V, Q/A_noise = ',Q/(G*A_noise))
# too big

# try LPF of order 2
n_LPF = 2
G = 1/numpy.sqrt(1+(f_noise/f_c)**(2*n_LPF))
print('order ',n_LPF,' : A_noise', (G*A_noise), 'V, Q/A_noise = ',Q/(G*A_noise))
# still too big

# try LPF of order 3
n_LPF = 3
G = 1/numpy.sqrt(1+(f_noise/f_c)**(2*n_LPF))
print('order ',n_LPF,' : A_noise', (G*A_noise), 'V, Q/A_noise = ',Q/(G*A_noise))

print('I need a filter of order 3')
```

    order  1  : A_noise 0.01999600119960014 V, Q/A_noise =  0.007630920257598869
    order  2  : A_noise 0.0003999999680000038 V, Q/A_noise =  0.38146975708007697
    order  3  : A_noise 7.999999999744e-06 V, Q/A_noise =  19.073486328735353
    I need a filter of order 3

```python

```


# Random Number Generation

This notebook is an introduction to a somewhat tangential and perhaps advanced topic: How to generate random numbers. Many of you have probably used libraries like `random` or `np.random` before to quickly generate a random number. But what are they doing underneath the hood? While you may never in practice write your own random number generators from scratch, if for anything, knowing something about how to do so gives you a deeper understanding of the data sciences, and is a useful skill to know for data science interviews, where variants on such questions are often asked.

Note: This tutorial assumes a certain knowledge of probability theory, particularly what probability distributions are. For a brief introduction to this subject, see my [Probability and Naive Bayes](https://nbviewer.jupyter.org/github/rkingery/ml_tutorials/blob/master/notebooks/naive_bayes.ipynb) tutorial.

Now, let's talk about how one might go about generating random numbers from scratch in python. It turns out this is a subtle topic, because computers are deterministic devices (more or less). Programs follow a well-defined sequence of instructions, which is a deterministic process, so how can one generate anything approximating random behavior from a deterministic process? It turns out you can, kind of. We call the "random" numbers generated in such a way *pseudo-random*, because they're not really random (they can't be), but they none the less behave as random in most practical respects.

Let's see how this can be done. The only libraries we allow ourselves to use here are `numpy` (but not `np.random`) , `pandas` (mainly for visualization), and `matplotlib`, except for some special util functions later on. Let's start by loading these libraries in as usual.

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

seed = 123
np.random.seed(seed)
```

```python
def summarize(sample, title=None):
    df = pd.Series(samples)
    print(df.describe())
    plt.hist(sample);
    plt.title(title)
    plt.show()
```

## Sampling From a Standard Uniform Distribution (rand)

I've mentioned that random number generators are in fact not random, but deterministic, yet behave as random for many practical purposes. How do we achieve this? The trick is to define a function that generates outputs in a sequence that *look like* they have no discernible relationship to each other, while at the same time the (shuffled) sequence looks like it comes from some distribution. It will work as follows, we specify an initial value, called the *seed*, denoted $x_0$. Suppose we'd like to generate $n$ pseudo-random numbers $x_1, x_2, \cdots, x_n$. For a suitable function $f$, we can do this via a recursion like this:
$$x_1 = f(x_0)$$
$$x_2 = f(x_1)$$
$$x_3 = f(x_2)$$
$$ \dots $$
$$x_n = f(x_{n-1}).$$

The first goal will be to define a function $f$ that can generate a sequence of numbers $x_1, x_2, \cdots, x_n$ that *look like* they were uniformly sampled from the continuous range $[0,1]$. That is, each sample is equally likely to take on any continuous value between 0 and 1, but nothing else. One simple, though non-intuitive type of function that can do this is the *linear congruence generator* (LCG). An LCG function has the form
$$f(x) = (ax + c) \mod m,$$
where $a,c,m$ are suitably large integer constants fixed ahead of time.

Let's see what a function like this outputs on a simple example, where $a=5, c=3, m=8$, and seed is $x_0=123$. Does it look random to you? Does the set look uniformly sampled from $[0,1]$?

```python
a = 5
c = 3
m = 8
f = lambda x: (a * x + c) % m
```

```python
x = 123
for i in range(10):
    x = f(x)
    print(x)
```

    2
    5
    4
    7
    6
    1
    0
    3
    2
    5

The set does look kind of random. Could you predict the sequence without knowing $a, c, m, x_0$? It's obvious each number isn't between 0 and 1, yet, but we can fix that pretty easily. Does the sequence look uniformly random? Hard to say. Let's sample some more numbers and plot their histogram.

Looks pretty uniform to me. Notice the values though. It turns out that this function only samples (pseudo) uniformly between 0 and 7, i.e. in the integer range $0, \cdots, m-1$. This suggests two observations: 1) we can scale samples to be between 0 and 1 by dividing them by $m$, and 2) the size of $m$ controls the granularity of how fine we can sample (higher $m$ means finer sampling in $[0,1]$).

Remark: It's actually very important that $a, c, m$ be chosen a certain way. You can't just pick any values or you'll get things like repeating sequences. See [here](https://en.wikipedia.org/wiki/Linear_congruential_generator) for details.

```python
samples = []
x = 123
for i in range(1000):
    x = f(x)
    samples.append(x)

# plotting a nicer histogram than plt.hist would give
u, counts = np.unique(samples, return_counts=True);
plt.bar(np.arange(len(u)), counts);
plt.xticks(np.arange(len(u)), u);
```

In practice, it's a good idea to choose $a, c, m$ to all be large integers. A standard choice (though by no means the only one) is to take $a=1664525, c=1013904223, m=2^{31}$. Let's sample again with these constants, and then rescale by $m$ so the samples are between 0 and 1. There's certainly some visible variation, but on the whole the histogram looks rectangular, which is what you'd expect from a standard uniform sample.

We can sanity check the generator some more by verifying the statistics look as expected. For a standard uniform, we'd expect samples to have

\begin{equation}
\begin{aligned}
\text{min}&=0, \\
\text{max}&=1, \\
\text{mean}&= \frac{1}{2} = 0.5, \\
\text{std dev}&=\sqrt{\frac{1}{12}} \approx 0.289.
\end{aligned}
\end{equation}

Notice also that by taking more samples that the samples start to appear more uniform. In the first example below, I generate 1000 samples. In that case the histogram looks kind of uniform, but a bit bumpy. In the second example, I generate 10,000 samples. In that case the histogram looks significantly less bumpy, and the statistic estimates are 1-2 orders of magnitude closer to the true values above for a standard uniform.

```python
a = 1664525
c = 1013904223
m = 2**31
f = lambda x: (a * x + c) % m

samples = []
x = 123
size = 1000
for i in range(size):
    x = f(x)
    u = x / m
    samples.append(u)

summarize(samples)
```

```python
samples = []
x = 123
size = 10000
for i in range(size):
    x = f(x)
    u = x / m
    samples.append(u)

summarize(samples)
```

We finally have everything we need to implement the most fundamental random number generator of all, the `rand` function. But first, a brief word on generating seeds.

There are two common situations when it comes to seeds. In one case, the usual case in data science situations, the seed should always be fixed and pre-specified before hand. This allows for seemingly random programs to be reproduced by someone else, which is useful for verifying that the program is running as intended. In data science, this (sort of) allows one to be able to train a new model that reproduces the behavior of a previously trained model. As a rule of thumb, if you're trying to do *reproducible research*, you should always fix and specify your seed in advance.

In the other case, fixing and publishing the seed is bad, as knowledge of the seed takes what *should* be a random process and makes it completely deterministic. If one knows the seed, the generator used, and how many samples have been sampled so far, one can in principle predict with certainty what the next sample will be. This is dangerous in certain fields like cryptography and security, where it's essential that one not easily be able to deconstruct a random function (as it might result in a very unfortunate hacking situation).

In the latter case, it's essential that the seed be as hard as possible for a third party to guess. One often done way to generate hard-to-guess seeds is by choosing them based on the system clock the program is running on. For example, suppose the system clock reads `29344.743828348`. One can generate a seed by taking the sequence after the decimal point as an integer value for the seed, so in this case we'd have `seed = 743828348`. System clock seeds are hard to guess because it's hard for a human to figure out the *exact* time the seed was generated at the millisecond level. A simple such a seed generator is shown below as `get_seed`.

```python
def get_seed():
    """Generates random seed using decimal part of system clock"""
    import time
    t = time.perf_counter()
    return int((t % 1) * 1e16)

for i in range(10):
    print(get_seed())
```

    9370733899995684
    9373539611697196
    9373864270746708
    9374186121858656
    9374494790099562
    9374783290550112
    9440171001479030
    9442198751494288
    9442589031532406
    9442903371527792

With this aside on seeds addressed, we return to the `rand` function, which is implemented below using the above linear congruence generator. The job of `rand` is to output a standard uniform number as described above. We will implement `rand` such that it takes constants `a, c, m`, a seed `seed`, and a size `size` specifying how many numbers to sample. To verify it works as intended we use `rand` to generate 1000 samples and show their summary statistics. To promote reproducibility, I fix all the seeds below in advance to the value defined at the beginning of the notebook.

```python
def rand(a=1664525, c=1013904223, m=2**32, size=1, seed=None):
    """Generates Uniform(0,1) samples using linear congruence generator"""
    f = lambda x: (a * x + c) % m
    x = get_seed() if seed is None else seed
    u = []
    for i in range(size):
        x = f(x)
        u.append(x / m)
    return np.array(u)

size = 1000
samples = rand(size=size, seed=seed)
summarize(samples, title=f'rand')
```

## Sampling From Other Probability Distributions

Armed with `rand`, it turns out we can generate numbers from pretty much any probability distribution we want just by doing transformations on standard uniform samples. There are a few common ways to generate new distributions from a uniform random sample: simple probability transformations, inverse transform sampling, and rejection sampling.

Simple probability transformations involve applying some kind of transformation to samples from the standard uniform to turn them into samples from a different desired distribution. Such transformations might involve translation, scaling, or simple operations like applying a common function (exp, log, power, trig, etc). We'll see a few examples of such techniques below.

[Inverse transform sampling](https://en.wikipedia.org/wiki/Inverse_transform_sampling) works by making use of the following simple theorem from probability: If a [random variable](https://en.wikipedia.org/wiki/Continuous_random_variable) $X$ has a [cdf](https://en.wikipedia.org/wiki/Cumulative_distribution_function) $F$, then the random variable $U=F(X)$ has a standard uniform distribution. We can make use of the inverse of this theorem. Namely, if $U$ is standard uniform, then $F^{-1}(U)$ has the same distribution as $X$. This means we can use a uniform sample to generate samples with the intended distribution by applying that distribution's inverse cdf to it, if we know it.

[Rejection Sampling](https://en.wikipedia.org/wiki/Rejection_sampling) is a more advanced technique that involves sampling from an easier "proposal" distribution, and then keep only samples that satisfy some acceptance criterion that relates the proposal to the original distribution. Such an approach falls under the general umbrella of [Markov Chain Monte Carlo](https://en.wikipedia.org/wiki/Markov_chain_Monte_Carlo) (MCMC) methods. Perhaps the most powerful algorithm for doing rejection sampling is the [Metropolis-Hastings Algorithm](https://en.wikipedia.org/wiki/Metropolis_algorithm), which allows one to sample from arbitrary distributions (even if you don't know its functional form). I won't touch on these techniques here, but may come back to it later.

### Uniform Distribution

Perhaps the simplest distribution we can create from `rand` is the (arbitrary) uniform distribution. A number sampled from `uniform` takes parameters $low$ and $high$, and should return any continuous number in the interval $[low, high]$ with equal probability. It's just a rescaling of the standard uniform distribution. We can easily transform a standard uniform sample $\text{Uniform}(0,1)$ to a $\text{Uniform}(low, high)$ sample by applying the following rescaling:

$$\text{Uniform}(low, high) = (high - low)\text{Uniform}(0,1)+low.$$

This is implemented below. The expected statistics for this distribution are given below. How well do these values match the ones in the example below?

\begin{equation}
\begin{aligned}
\text{min}&=low, \\
\text{max}&=high, \\
\text{mean}&= \frac{low + high}{2}, \\
\text{std dev}&=\sqrt{\frac{(high-low)^2}{12}}.
\end{aligned}
\end{equation}

```python
def uniform(low=0, high=1, size=1, seed=None):
    """Generates Uniform(low, high) samples"""
    return ((high - low) * rand(seed=seed, size=size) + low)

size = 1000
samples = uniform(size=size, seed=seed, low=0, high=10)
summarize(samples, title=f'uniform(low=0, high=10)')
```

### Discrete Uniform Distribution (randint)

The next distribution we'll discuss is the discrete uniform distribution, which is like the uniform distribution but only allowed allowed to take on integer values in the interval. Namely, given parameters $low$ and $high$, a $\text{DiscreteUniform}(low, high)$ sample can take on any integer value between $low$ and $high$ with equal probability.

The discrete uniform implementation is frequently called `randint`. It works simply by sampling from `uniform(low, high)` and rounding each number down to the nearest integer. Note this means in particular that `low` is inclusive while `high` is exclusive, so values take a range `low, ..., high-1`.

The expected statistics for $\text{DiscreteUniform}(low, high)$ with support $[low,high)$ are below. How well do these values match the ones in the example below?

\begin{equation}
\begin{aligned}
\text{min}&=low, \\
\text{max}&=high-1, \\
\text{mean}&= \frac{low + high - 1}{2}, \\
\text{std dev}&=\sqrt{\frac{(high-low)^2-1}{12}}.
\end{aligned}
\end{equation}

```python
def randint(low=0, high=2, size=1, seed=None):
    """Generates DiscreteUniform(low, high) samples, i.e. random ints [low, high)"""
    u = uniform(low=low, high=high, size=size, seed=seed)
    return np.floor(u).astype(int)

size = 1000
samples = randint(size=size, seed=seed, low=0, high=5)
summarize(samples, title=f'randint(low=1, high=5)')
```

### Sampling an Array (choice)

Before continuing with the other distributions let's make a quick detour and talk about randomly sampling from an arbitrary array. When doing so it's necessary to distinguish between randomly sampling *with replacement* (WR) and randomly sampling *without replacement* (WOR). When sampling WR, we imagine putting that value back into the array so we can sample it again later if we wish with an equal probability. Whereas when sampling WOR, we imagine holding the value out so it can't be sampled again later (which means among other things that sampling can't be uniform since the probabilities change as a result of removing elements from the array).

In the case of sampling from an array we usually mean sampling WR (though not always). The implementation of sampling from an array is called `choice` in python. Below is an implementation of `choice` to allow sampling WR from an arbitrary array. Notice all we have to do is randomly sample the index of the array with `randint`, and then just return the values corresponding to those indexes.

In the sample below, we sample from `choice` `size` times on the string `abracadabra`, and display the counts for each letter and the histogram. Do the frequencies shown appear to match the frequency of the letters as they appear in `abracadabra`? They should.

```python
def choice(array, size=1, seed=None):
    """Randomly samples from given array, with replacement"""
    array = np.array(array)
    r = randint(low=0, high=len(array), size=size, seed=seed)
    return array[r]

array = list('abracadabra')

size = 1000
samples = choice(array, size=size, seed=seed)
print(pd.Series(samples).value_counts())
plt.hist(samples);
plt.title(f'choice(\'{"".join(array)}\')');
```

### Bernoulli Distribution

Returning to distributions, the Bernoulli distribution is a basic distribution that only takes on binary values 0 and 1. Sampling from $\text{Bernoulli}(p)$ returns 1 with probability $p$ or 0 with probability $1-p$. As you probably know, this distribution is frequently used to model real-world binary processes like coin flips, and is the theoretical basis of (binary) logistic regression. Its density is given by

$$p(x) = p^x (1-p)^{1-x}.$$

We can again implement $\text{Bernoulli}(p)$ using `rand`. The idea here is to use $p$ as a threshold to decide whether to round a `rand` number down to 0 or up to 1. This is implemented below as `bernoulli`, which takes a parameter `p` and returns a $\text{Bernoulli}(p)$ sample.

The expected statistics for $\text{Bernoulli}(p)$ are below. How well do these values match the ones in the example below?

\begin{equation}
\begin{aligned}
\text{min}&=0, \\
\text{max}&=1, \\
\text{mean}&= p, \\
\text{std dev}&=\sqrt{p(1-p)}.
\end{aligned}
\end{equation}

```python
def bernoulli(p=0.5, size=1, seed=None):
    """Generates Bernoulli(p) samples"""
    u = rand(size=size, seed=seed)
    return (u <= p).astype(int)

size = 1000
samples = bernoulli(size=size, seed=seed, p=0.2)
summarize(samples, title=f'bernoulli(p=0.2)')
```

### Binomial Distribution

As many of you know, the [binomial distribution](https://en.wikipedia.org/wiki/Binomial_distribution#Normal_approximation) is a model for the number of successful outcomes from $n$ binary (Bernoulli) trials. A $\text{Binomial}(n,p)$ random variable is just a sum of $n$ Bernoulli random variables, and can thus take on integer values between $0$ and $n$. Its density is given by

$$p(x) = {n \choose x} p^x (1-p)^{n-x}.$$

Given the binomial is just a sum of bernoullis, implementation is pretty easy. To create a function `binomial(n, p)`, we basically just call `bernoulli(p)` `n` times and sum the results together to get a single binomial sample.

One subtlety in the implementation is worth addressing though. It's *very* important to make sure each bernoulli sample be taken with a different seed. If you use the same seed all of the bernoulli samples will end up being the same, which means each sample of `binomial` will be the same number! To get around this, and still keep things reproducible, the easiest thing to do is to increment the seed in each iteration of the loop.

The expected statistics for $\text{Binomial}(n,p)$ are below. How well do these values match the ones in the example below?

\begin{equation}
\begin{aligned}
\text{min}&=0, \\
\text{max}&=n, \\
\text{mean}&= np, \\
\text{std dev}&=\sqrt{np(1-p)}.
\end{aligned}
\end{equation}

Aside: Notice the histogram for this binomial sample looks almost like that of a Gaussian distribution. This is not a coincidence. Turns out $\text{Binomial}(n,p) \approx \mathcal{N}(np, np(1-p))$ under certain regularity conditions. See [here](https://en.wikipedia.org/wiki/De_Moivre%E2%80%93Laplace_theorem) for details.

```python
def binomial(n, p=0.5, size=1, seed=None):
    """Generates Binomial(n, p) samples"""
    x = []
    for i in range(size):
        b = bernoulli(p=p, size=n, seed=seed)
        x.append(b.sum())
        seed += 1
    return np.array(x)

size = 1000
samples = binomial(size=size, seed=seed, n=10, p=0.5)
summarize(samples, title=f'binomial(n=10, p=0.5)')
```

### Poisson Distribution

The [Poisson distribution](https://en.wikipedia.org/wiki/Poisson_distribution) is used to model counting processes. $\text{Poisson}(\lambda)$ takes a parameter value $\lambda > 0$, and can take on any positive integer value, and its density tends to have a "hump" character, where it peaks around the parameter value $\lambda$. In math, its density is given by

$$p(x) = \frac{\lambda^x e^{-\lambda}}{x!}.$$

Implementing a sampler for $\text{Poisson}(\lambda)$ is a bit different. Here I choose to implement `poisson` using a [sampling algorithm](https://en.wikipedia.org/wiki/Poisson_distribution#Generating_Poisson-distributed_random_variables) created by Donald Knuth. The idea here is to make use of a fact about Poisson distributions that states the time between new events is exponentially distributed. (More explanation to be added later.)

Note: This algorithm only works well when `lambda_` is not too large (a rule of thumb is under `lambda_=30`). This is due to the fact that the inner loop depends on `np.exp(-lambda_)` taking on a non-zero value, which is impractical when `lambda_` is large due to numerical underflow. For larger values of `lambda_` the algorithm should be tweaked. See [here](https://www.johndcook.com/blog/2010/06/14/generating-poisson-random-values/) for an example of how to do this.

The expected statistics for $\text{Poisson}(\lambda)$ are below. How well do these values match the ones in the example below?

\begin{equation}
\begin{aligned}
\text{min}&=0, \\
\text{max}&=\infty, \\
\text{mean}&= \lambda, \\
\text{std dev}&=\sqrt{\lambda}.
\end{aligned}
\end{equation}

```python
def poisson(lambda_=1, size=1, seed=None):
    """Generates Poisson(lambda) samples using Knuth algorithm"""
    array = []
    L = np.exp(-lambda_)
    for j in range(size):
        x = 0
        p = 1
        while p > L:
            x += 1
            p = p * rand(seed=seed)
        array.append(x - 1)
        seed += 1
    return np.array(array)

size = 1000
samples = poisson(size=size, seed=seed, lambda_=10)
summarize(samples, title=f'poisson(lambda=10)')
```

### Exponential Distribution

The exponential distribution is often used to model arrival times of processes following a Poisson distribution. Given that it's used to model times, an exponential random variable can take on any positive real value. It's parametrized by a rate parameter $\lambda > 0$ that determines the (inverse) expected arrival time. Its density is given by

$$p(x) = \lambda e^{-\lambda x}.$$

An $\text{Exponential}(\lambda)$ random sample can easily be generated using inverse transform sampling. The CDF of $\text{Exponential}(\lambda)$ is
$$F(x) = 1 - e^{-\lambda x},$$
which means its inverse CDF is
$$F^{-1}(u) = -\frac{1}{\lambda} \log(1-u).$$
Thus, we merely have to generate a `rand()` and apply the inverse CDF to it to get an exponential random sample. This is implemented as `exponential` below, and takes in the rate parameter as `lambda_`. I merely sample a uniform sample via `rand()` and apply the above inverse CDF transformation to get an exponential sample.

The expected statistics for $\text{Exponential}(\lambda)$ are below. How well do these values match the ones in the example below?

\begin{equation}
\begin{aligned}
\text{min}&=0, \\
\text{max}&=\infty, \\
\text{mean}&= \frac{1}{\lambda}, \\
\text{std dev}&=\sqrt{\frac{1}{\lambda^2}}.
\end{aligned}
\end{equation}

```python
def exponential(lambda_=1, size=1, seed=None):
    """Generates Exp(lambda) samples using Inverse Transform Sampling"""
    u = rand(seed=seed, size=size)
    x = - (1 / lambda_) * np.log(1 - u)
    return x

size = 1000
samples = exponential(size=size, seed=seed, lambda_=1)
summarize(samples, title=f'exponential(lambda=1)')
```

### Gaussian / Normal Distribution

The Gaussian distribution (aka the Normal distribution) is without a doubt the most important distribution out there. It's used to model all kinds of things: measurement errors, the parameter values in a neural net, the velocities of a gas in a container, approximating other distributions via the central limit theorem, and much more. This distribution takes on the characteristic "bell curve", where a mean value is centered at the top of the bell, and values away from the mean become less and less likely to get sampled. The Gaussian, represented as $\mathcal{N}(\mu, \sigma^2)$, takes two parameters $\mu$ and $\sigma$, each of which can take on any real value. Its density is given by

$$ p(x) = \frac{1}{\sqrt{2 \pi \sigma^2}} e^{-\frac{(x-\mu)^2}{\sigma^2}}.$$

Generating samples from a Gaussian is most frequently done using the [Box Muller Transform](https://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform). The idea is to apply a transformation to 2 uniform random variables $U_1, U_2$:

$$ Z_0 = \sqrt{-2 \log U_1} \cos (2 \pi U_2)$$
$$ Z_1 = \sqrt{-2 \log U_1} \sin (2 \pi U_2).$$

It turns out that both $Z_0$ and $Z_1$ are (independent) $\mathcal{N}(0, 1)$ random variables. It's a simple transformation to do, but may take a bit to wrap your head around as to why it works. We can then recover any $\mathcal{N}(\mu, \sigma^2)$ variable by applying the transformation

$$ X = \sigma Z + \mu.$$

I have this coded up in the `normal` function below, which takes as input the 2 parameters `mu` and `sigma`. The two uniform samples `u1, u2` are generated by `rand()`. I only return the Gaussian variable generated by `z0`, but code up `z1` anyway so you can see it.

The expected statistics for $\mathcal{N}(\mu, \sigma^2)$ are below. How well do the sample statistics match?

\begin{equation}
\begin{aligned}
\text{min}&=-\infty, \\
\text{max}&=\infty, \\
\text{mean}&= \mu, \\
\text{std dev}&=\sigma.
\end{aligned}
\end{equation}

```python
def normal(mu=0, sigma=1, size=1, seed=None):
    """Generates N(mu, sigma) samples using Box-Muller Transform"""
    u1 = rand(seed=seed, size=size)
    u2 = rand(seed=seed+1, size=size)
    z0 = np.sqrt(-2 * np.log(u1)) * np.cos(2 * np.pi * u2)
    z1 = np.sqrt(-2 * np.log(u1)) * np.sin(2 * np.pi * u2)
    return z0 * sigma + mu

size = 1000
samples = normal(size=size, seed=seed, mu=0, sigma=1)
summarize(samples, title=f'normal(mu=0, sigma=1)')
```

## References

[1] https://towardsdatascience.com/how-to-generate-random-variables-from-scratch-no-library-used-4b71eb3c8dc7


sergazy.nurbavliyev@gmail.com © 2021

## Minimum of two random variables

Question: Assume we have $X \sim Uniform(0, 1)$ and $Y \sim Uniform(0,1)$ and independent of each other. What is the expected value of the minimum of $X$ and $Y$?

### Intuition

Before going to show the exact theoretical result, we can think heuristically,we have two uniform random variables  that are independent of each other, we would expect two of these random variables to divide the interval (0,1) equally into 3 equal subintervals. Then the minimum would occur at 1/3 and the other one (which is maximum in this case) occurs at 2/3. We can even generalize our intuition for $n$ independent uniform random variables. In that case minimum would occur at $1/(n+1)$ and maximum would occur at $n/(n+1)$. Now let us verify intution theoretically.

### Theoritical result

Let $Z$  be minimum of X and Y. We write this as $Z=\min(X,Y)$.
\begin{equation}
\mathbb{P}(Z\leq z)= \mathbb{P}(\min(X,Y)\leq z) =1-\mathbb{P}(\min(X,Y)\geq z)=1-\mathbb{P}(X> z, Y>z)
\end{equation}
Since our distribution is between 0 and 1 the following is true for uniform distribution
\begin{equation}
\mathbb{P}(X\leq z)= z \text{ and } \mathbb{P}(X> z)=1- z
\end{equation}
Also same goes for $Y$. Now since they are independent we have
\begin{equation}
\mathbb{P}(X> z, Y>z)=\mathbb{P}(X> z)\mathbb{P}(Y>z)=(1-z)^{2}
\end{equation}
Then equation (1) becomes
\begin{equation}
\mathbb{P}(Z\leq z)=1-\mathbb{P}(X> z, Y>z)=1-(1-z)^{2}
\end{equation}
We just calculated cumulative distribution function of $z$. Usually denoted as
\begin{equation}
F_{Z}(z)= \mathbb{P}(Z\leq z)
\end{equation}
If we take derivative of this $ F_{Z}(z)$, we will get density function of z. In this case it would be
\begin{equation}
F_{Z}'(z)=f_Z(z)=2(1-z)
\end{equation}
As last part we would take integral of $zf_Z(z)$ between 0 and 1 to find an expected value of minimum of two uniform random variables.
\begin{equation}
\mathbb{E}[Z]=\int_{0}^1 zf_Z(z)dz=\int_{0}^1 2z(1-z)dz=2\left(\frac{1}{2}-\frac{1}{3}\right)=\frac{1}{3}
\end{equation}

```python
# If you are allowed to use any programming language then you can simulate.
import numpy as np
x = np.random.uniform(0,1,100000)
y = np.random.uniform(0,1,100000)
z =np.minimum(x,y)
u =np.maximum(x,y)
```

```python
np.mean(z), np.mean(u) # as you can see z is very close to 1/3
```

    (0.33389155844548174, 0.6670118602826249)

### To see the histogram of Z and U

```python
import matplotlib.pyplot as plt
count, bins, ignored = plt.hist(z, 100, density=True)
plt.plot(bins, np.ones_like(bins), linewidth=2, color='r')
plt.show()
```

```python
import matplotlib.pyplot as plt
count, bins, ignored = plt.hist(u, 100, density=True)
plt.plot(bins, np.ones_like(bins), linewidth=2, color='r')
plt.show()
```

```python

```

```python

```

Minimim of two random variables March 3 2021.ipynb


```python
from scipy import *
import sympy as sym
from approx1D import *
import matplotlib.pyplot as plt

Ns = [2, 4, 8, 16]
Taylor     = [0.0983, 0.00263,  7.83e-07, 3.57e-10]
Sinusoidal = [0.0027, 0.00061,  0.00012,  2.17e-05]
Bernstein  = [0.0021, 4.45e-05, 8.73e-09, 4.49e-15]
Lagrange   = [0.0021, 4.45e-05, 8.73e-09, 2.45e-12]

x = sym.Symbol('x')
psi = [1, x]

u, c = regression_with_noise(log2(Sinusoidal), psi, log2(Ns))
print(("estimated model for sine: %3.2e*N**(%3.2e)" % \
      (2**(c[0]), c[1])))

# check the numbers estimated by the model by manual inspection
for N in Ns:
  print((2**c[0] * N **c[1]))

X = log2(Ns)
U = sym.lambdify([x], u)
UU = U(X)

plt.plot(X, log2(Sinusoidal))
plt.plot(X, UU)
plt.legend(["data", "model"])
plt.show()

u, c = regression_with_noise(log(Bernstein), psi, Ns)
print(("estimated model for Bernstein: %3.2e*exp(%3.2e*N)" % (exp(c[0]), c[1])))

# check the numbers estimated by the model by manual inspection
for N in Ns:
  print((exp(c[0]) * exp(N * c[1])))

X = Ns
U = sym.lambdify([x], u)
UU = U(array(X))

plt.plot(X, log(Bernstein))
plt.plot(X, UU)
plt.legend(["data", "model"])
plt.show()

CPU_Taylor    = [0.0123, 0.0325, 0.108,  0.441]
CPU_sine      = [0.0113, 0.0383, 0.229,  1.107]
CPU_Bernstein = [0.0384, 0.1100, 0.3368, 1.187]
CPU_Lagrange  = [0.0807, 0.3820, 2.5233, 26.52]

plt.plot(log2(Ns), log2(CPU_Taylor))
plt.plot(log2(Ns), log2(CPU_sine))
plt.plot(log2(Ns), log2(CPU_Bernstein))
plt.plot(log2(Ns), log2(CPU_Lagrange))
plt.legend(["Taylor", "sine", "Bernstein", "Lagrange"])
plt.show()

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/SRC/11_CONVERGENCE_RATE_LOCAL.ipynb


```python
import numpy as np
import scipy as sp
from matplotlib import pyplot as plt
from diff_exp import *
```

# Linear Systems of Differential Equations

Let $y = (y_1(t),..., y_n(t))$ and $M = M(t)$ matrix with dimensions $n \times n$. The linear system of differential equations:

\begin{equation}
\frac{dy}{dt} = M y
\end{equation}

with initial point values $y(t = t_0) = y_0$ can be solved nummericaly using integrators (e.g. Runge-Kutta). But it has analytical solution using matrix exponent:

\begin{equation}
y(t) = e^{\int_{t_0}^t M(t) dt} y_0
\end{equation}

The simple script *diff_exp.py* implements both ways for obtaining solutions, and here is the comparison of the so solutions.

First two examples cover the case where $M$ is constant matrix.

```python
M = np.array([[0,1], [1,-2]])
y0 = np.array([[1],[2]])
ti = 2
tf = 5
dt = 0.01
```

```python
y, t = sol_lin_system_ode_c(y0, ti, tf, dt, M)
```

```python
plt.plot(t, y[0])
plt.plot(t, y[1])
plt.show()
```

```python
y0 = np.array([1, 2])
x, t = lin_system_ivp_c(y0, ti, tf, dt, M)
```

```python
plt.plot(t, x[0])
plt.plot(t, x[1])
plt.show()
```

```python
M = np.array([[0, 1, 0, -1], [1,-2, 5, 0], [0, 1, 2, 3], [-1, -2, 0, -5]])
y0 = np.array([[1],[2],[5],[0]])
ti = 2
tf = 5
dt = 0.01
```

```python
y, t = sol_lin_system_ode_c(y0, ti, tf, dt, M)
```

```python
plt.plot(t, y[0])
plt.plot(t, y[1])
plt.plot(t, y[2])
plt.plot(t, y[3])
plt.show()
```

```python
y0 = np.array([1, 2, 5, 0])
x, t = lin_system_ivp_c(y0, ti, tf, dt, M)
```

```python
plt.plot(t, x[0])
plt.plot(t, x[1])
plt.plot(t, x[2])
plt.plot(t, x[3])
plt.show()
```

Let's compare two solutions:

```python
plt.plot(t, x[0]-y[0])
plt.plot(t, x[1]-y[1])
plt.plot(t, x[2]-y[2])
plt.plot(t, x[3]-y[3])
plt.show()
```

And the relative errors: (*Warning* division with $0$ possible)

```python
plt.plot(t, np.abs(x[0]-y[0])/np.abs(y[0]))
plt.plot(t, np.abs(x[1]-y[1])/np.abs(y[1]))
plt.plot(t, np.abs(x[2]-y[2])/np.abs(y[2]))
plt.plot(t, np.abs(x[3]-y[3])/np.abs(y[3]))
plt.show()
```

Next example covers the case where $M$ is function of $t$.

```python
M = lambda t: np.array([[-t, 1], [np.exp(-t), -t**2]])
y0 = np.array([[1], [3]])
ti = 0
tf = 3
dt = 0.001
```

```python
y, t = sol_lin_system_ode_l(y0, ti, tf, dt, M)
```

```python
plt.plot(t, y[0])
plt.plot(t, y[1])
plt.show()
```

```python
y0 = np.array([1, 3])
x, t = lin_system_ivp_l(y0, ti, tf, dt, M)
```

```python
plt.plot(t, x[0])
plt.plot(t, x[1])
plt.show()
```

To conclude, the Runge-Kutta method is way more efficient than matrix exponent method.

```python

```

Linear systems of differential equations/differential_equations.ipynb