# Module 1: Setting up the problem

## Introduction

Geophysical surveys consist of a similar basic framework. An energy source is delivered into the earth, which can be natural (for example, the Earth's magnetic field) or human-made (current in the ground, acoustic wave energy, etc.), and this stimulates a response according to the variation in physical properties in the subsurface. At the surface, receivers pick up the signal and record this as data.

The goal of inversion is to find a model of the physical property distribution in the earth that produced the data. This is a difficult process because (1) information about a physical property for each datum is encoded in a complex way, and (2) we have a finite amount of data and cannot represent the physical property distribution everywhere.

Inversion is a multistep process, often represented as a workflow.

The goal of this module is to cover the first section of the workflow, which discretizes the data and places the values of the functions onto a mesh. This will be done using the following steps:

(1) Start with an expression that relates a kernel function with the continuous distribution of a physical property.

(2) Discretize this expression, and introduce a simple example problem to illustrate the mathematics in detail.

(3) Define a mesh that organizes our information.

(4) Build up the matrix equation $d = Gm$.

(5) Generalize the form of the problem from the example.

(6) Implement the example problem in Python as a forward problem.

But first, here are some fundamental definitions:

The general mathematical description of the inverse problem can be written as follows:

\begin{equation} F_i[m]= d_i +n_i \quad \text{for} \quad i=1,..,N \; \text{where}\end{equation}

$F[m]$ is a forward modeling operator. $F[m]$ incorporates the survey design and the physics of the problem. In the linear case we represent this as the matrix $G$.

$\bf{m}$ is a generic symbol for a physical property distribution, i.e. the model.

$\bf{d}$ represents the observed data, (sometimes also represented as $\bf{d^{obs}}$).

$\bf{n}$ is a term that represents additive noise.

This is, of course, the most general formulation of the problem. In this module we will consider the simplest case, which is (a) one dimensional (this can be likened to a survey that varies as a function of depth only) and (b) linear, in which our forward modeling operator becomes a matrix **G** in the matrix equation:

\begin{equation}d = Gm\end{equation}

## Step 1: Physical property distribution and the kernel function

Each datum ($d_i$) collected is a volumetric response, that is to say, every datum measures the response of the whole volume (within the range of the system); it records the superposition of effects caused by all the material in the ground, and is therefore naturally represented as an integral. A "kernel function" or "sensitivity function", $g(x,y,z)$, shows how a datum is affected by all the subsurface. It describes the physics of the problem. The model, $m(x,y,z)$, represents the distribution of a physical property in the volume. Since each datum measures the response of the kernel function with the physical property distribution in the volume, for a continuous medium we can express this relationship as the inner product of the kernel function and the model. In the one dimensional case the expression for the *ith* datum is written as:

\begin{equation}d_i = \int_a^b g_i(x) m(x) dx \end{equation}

where again:
\begin{equation}
    d :=\text{measured data} \\
    g :=\text{kernel function} \\
    m :=\text{physical property}\\
    a,b := \text{the domain of consideration}\\
\end{equation}

## Step 2: Discretize the expression

While the integral expression describes the inner product in a continuum, it is not possible to maintain this representation on a computer, so the above equation must be discretized. Let us consider the case where we have $N$ data, and as we build up the matrix equation, let $i$ be the rows and $j$ be the columns of our matrix:

\begin{equation}d_i = \int_a^b g_i(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_i (x_j) m_j \Delta x\end{equation}

**A "Toy Problem"**

Consider a simple case where we have a one dimensional, linear problem that will generate two data points, $d_1$ and $d_2$, from five physical property values
 ($m_1, m_2, m_3, m_4, m_5$), and these two data points are generated using two kernel functions $g_1$ and $g_2$. Further, let us assume that our domain of interest lies on the interval [0,1]. The equation above is then expressed as the following two equations, one for each datum:

\begin{equation}d_1 = \int_0^1 g_1(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_1 (x_j) m_j \Delta x\end{equation}

\begin{equation}d_2 = \int_0^1 g_2(x) m(x) dx \; \Rightarrow \; \sum_{j=1}^N g_2 (x_j) m_j \Delta x\end{equation}

Given that our problem is small by design, it is instructive to visualize the summation notation as matrix-vector products. Doing so yields the following expressions:

(a) Data. We can collect our two data points into a column vector. Our data in vector notation is given by:

\begin{equation}
d = \left[
\begin{array}{c}
d_1\\
d_2
\end{array}
\right]
\end{equation}

(b) The x-spacing. The x-spacings, $\Delta x$, are represented by a diagonal matrix. Here we are in a one dimensional case, but in general our information will reside in a volume, so let this matrix be represented as $V$. In the most general situation, the x-spacings need not be equal, and there can be significant variation in the distances within a grid, depending on the amount of resolution one desires at a particular location. For the moment, let's ignore this complexity and assume equal spacing in our grid. Then let $V=diag(\Delta x)$, and given the dimensions of our problem, $V$ appears as follows:

\begin{equation}
V=\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\end{equation}

(c) The model. The
model, $m$, is a column vector with five rows:
\begin{equation}
m=
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

(d) The kernel functions. Recall that a kernel function shows how a datum is affected by all the subsurface. The full expression for the kernel function will be developed further in the next sections, but for the moment, let us put each kernel function on a row of a matrix, such that $g_1$ will be on row 1, and $g_2$ on row 2, and define this matrix as $\widetilde{G}$. This will yield the matrix:
\begin{equation}
\widetilde{G}=
\begin{bmatrix}
    g_{11} &g_{12} &g_{13} &g_{14} &g_{15}\\
    g_{21} &g_{22} &g_{23} &g_{24} &g_{25}
\end{bmatrix}
\end{equation}

Using the above, the assembled matrix vector forms for these equations becomes:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\end{array}
\right]
=
\begin{bmatrix}
    g_{11} &g_{12} &g_{13} &g_{14} &g_{15}\\
    g_{21} &g_{22} &g_{23} &g_{24} &g_{25}
\end{bmatrix}
\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

Or to put it succinctly, $d = \widetilde{G} \, diag(\Delta x)\, m$.

## Step 3: Set up the mesh

Now that the data has been discretized, let's look at where we put our information. In the three dimensional cases, our information is stored in a cube, where we can put values in the cell centers (the red circle below), on the cube faces (red triangles), the edges (blue triangles), and on the nodes (the apices). This representation is useful because it allows us to represent physical quantities that are conserved across boundaries from on cell to the other. For example, fluxes can be put across the faces of the cube.

Taken from http://docs.simpeg.xyz/en/latest/api_Mesh.html

In a one dimensional case, however, we are only dealing with two elements, cell centers and cell nodes. First, subdivide the one dimensional domain into cell centers and nodes, with spacings of $\Delta x$:

In this instance the model values reside in the cell centers, while the values for the kernel functions reside on the nodes. Schematically, we can think of our 1D mesh as the following picture, where $g$ represents a kernel function and $m$ represents the model parameters.

Note that the kernel function in our example (represented above) has six values and the model values are five; moreover, the x-coordinates where $g(x) $ and $m(x)$ are evaluated are not coincident, and this leads to a complication when we want to perform the inner product of **m** and **g**. What is needed is a way to evaluate the kernel functions at the cell centers. To do this, we employ the trapezoidal rule for approximating integration. Simply put, to obtain the values of the kernel function on the cell centers, take the average of the kernel function values on the adjacent nodes. Let us then define two kernel function matrices, one with values evaluated on the cell centers, $G_c$ (represented by the black circles below), and another with the values evaluated on the nodes, $G_n$ (in white).

The relationship between $G_c $ and $G_n$ is an "averaging matrix", $A_v$, such that $G_c = A_v G_n$. Putting each kernel function in the columns for both $G_c$ and $G_n$, and using again the dimensions of our toy problem, this relation appears as follows:

\begin{equation}
\begin{bmatrix}
    g_{c1}(x_1)    & g_{c2}(x_1)  \\
    g_{c1}(x_2)    & g_{c2}(x_2)  \\
    g_{c1}(x_3)    & g_{c2}(x_3)  \\
    g_{c1}(x_4)    & g_{c2}(x_4)  \\
    g_{c1}(x_5)    & g_{c2}(x_5)  \\
\end{bmatrix}
=
\frac{1}{2}
\begin{bmatrix}
    1 & 1 & 0 & 0 & 0 & 0\\
    0 & 1 & 1 & 0 & 0 & 0\\
    0 & 0 & 1 & 1 & 0 & 0\\
    0 & 0 & 0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0 & 1 & 1\\
\end{bmatrix}
\begin{bmatrix}
    g_{n1}(x_1)    & g_{n2}(x_1)  \\
    g_{n1}(x_2)    & g_{n2}(x_2)  \\
    g_{n1}(x_3)    & g_{n2}(x_3)  \\
    g_{n1}(x_4)    & g_{n2}(x_4)  \\
    g_{n1}(x_5)    & g_{n2}(x_5)  \\
    g_{n1}(x_6)    & g_{n2}(x_6)  \\
\end{bmatrix}
\end{equation}

Meanwhile, the relationship between $G_c$ and $\widetilde{G}$ in step 2 is such that $G_c = \widetilde{G}^T$.

## Step 4: Build up the matrix equation $d=Gm$

We have all the required building blocks to assemble the matrix equation. At the end of step 2 we arrived at $d = \widetilde{G} \, diag(\Delta x)\, m$, and from the previous step we have $G_c = \widetilde{G}^T = A_v G_n$. Put more orderly, $\widetilde{G} = (A_v G_n)^T$, so substituting  this into our expression gives:

\begin{equation}
d = (A_v G_n)^T \, diag(\Delta x)\, m
\end{equation}

If we group matrices together and let $G = (A_v G_n)^T diag(\Delta x)$ then we arrive at $d=Gm$, our desired form. Referring again to our example, this would appear as follows:

\begin{equation}
\left[
\begin{array}{c}
d_1\\
d_2\\
\end{array}
\right]
=
\frac{1}{2}
\left(
\begin{bmatrix}
    1 & 1 & 0 & 0 & 0 & 0\\
    0 & 1 & 1 & 0 & 0 & 0\\
    0 & 0 & 1 & 1 & 0 & 0\\
    0 & 0 & 0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0 & 1 & 1\\
\end{bmatrix}
\begin{bmatrix}
    g_{n1}(x_1)    & g_{n2}(x_1)  \\
    g_{n1}(x_2)    & g_{n2}(x_2)  \\
    g_{n1}(x_3)    & g_{n2}(x_3)  \\
    g_{n1}(x_4)    & g_{n2}(x_4)  \\
    g_{n1}(x_5)    & g_{n2}(x_5)  \\
    g_{n1}(x_6)    & g_{n2}(x_6)  \\
\end{bmatrix}
\right)^T
\begin{bmatrix}
    \Delta x &0  &0 &0  &0 \\
    0 &\Delta x &0  &0 &0  \\
    0  &0  &\Delta x &0 &0 \\
    0  &0  &0 &\Delta x   &0 \\
    0  &0  &0  &0 &\Delta x\\
\end{bmatrix}
\left[
\begin{array}{c}
m_1\\
m_2\\
m_3\\
m_4\\
m_5
\end{array}
\right]
\end{equation}

## Step 5: Generalize the form of the problem

In the above case we had two data points and five model values. We can generalize this to larger data sets easily. In the case where we have $N$ measured data and $M$ model values, we obtain a matrix equation of the following dimensions:

\begin{equation}
(N \times 1) = [(M \times (M+1)) \; ((M+1) \times N)]^T (M  \times M) (M \times 1)\\
d \qquad = \qquad \qquad  \qquad \quad   [A_v G_n]^T  \qquad \qquad diag(\Delta x) \quad m
\end{equation}

## Step 6: Implement the example problem in Python.

Next, we will build up our matrices one by one in Python. First we will do the small problem exactly as above, and then we will use a larger, but still one-dimensional, problem. To begin, import the SimPEG and numpy packages:

```python
# Import Packages
import numpy as np
import matplotlib.pyplot as plt
```

Let start by formulating the forward problem, that is, let us assume that we have our model values $m$ and seek to generate synthetic data $d$. Recall that we are building up the matrix equation: $d = (A_v G_n)^T \, diag(\Delta x)\, m$, so let's start with the right hand side and go over each matrix, one by one.

(1) The $A_v$ matrix. We can build the "averaging matrix" as follows:

```python
n_cells = 5  # Recall is the number of values we have in m
n_nodes = n_cells + 1  # Define n as the N+1 dimension of the matrix

# Create a matrix of zeros of the correct dimensions
# and fill in with elements usin the loop below (note the 1/2 is included in here).

Av = np.zeros((n_cells, n_nodes))

for i in range(n_cells):
    Av[i, i] = 0.5
    Av[i, i+1] = 0.5

print(Av)
```

    [[0.5 0.5 0.  0.  0.  0. ]
     [0.  0.5 0.5 0.  0.  0. ]
     [0.  0.  0.5 0.5 0.  0. ]
     [0.  0.  0.  0.5 0.5 0. ]
     [0.  0.  0.  0.  0.5 0.5]]

(2) The kernel matrix on the nodes, $G_n$. To build this matrix, we must first define a sensitivity function that describes some physical phenomenon. For the sake of our example, let us define our kernel function for the toy problem using the following equations. From a physical perspective, one can think of these equations as representing the decay of electromagnetic waves with depth. Note also that $j$ in this case is simply index number (i.e. $j=1,2,3,...$) and is *not* the imaginary number. The $j$ for the kernel functions affects the frequency and decay for each kernel function. Our kernel functions are as follows:

\begin{equation}
g_j(x) = e^{-jpx} \cos(2 \pi j q x)
\end{equation}

```python
def kernel_function(x, j, p, q):
    """
    Kernel functions for an example linear inverse problem
    """
    return np.exp(-p*j*x) * np.cos(2*np.pi*q*j*x)
```

```python
xn = np.linspace(0, 1, n_nodes)  # define the nodes of our x-array
p = 0.01
q = 0.1
j_values = np.array([1, 2])
Gn = np.zeros((len(xn), len(j_values)))

for i, j in enumerate(j_values):
    Gn[:, i] = kernel_function(xn, j, p, q)

print(Gn)
```

    [[1.         1.        ]
     [0.99013245 0.96471657]
     [0.96471657 0.86932419]
     [0.92421453 0.72027328]
     [0.86932419 0.52732179]
     [0.80096714 0.30289805]]

(3) The volume matrix $V$. This simply consists of making an $M \times M$ array of x-spacings, $\Delta x.$

```python
# Make the delta x array

delta_x = np.diff(xn)  # set x-spacings
V = np.diag(delta_x)  # create diagonal matrix
print(V)
```

    [[0.2 0.  0.  0.  0. ]
     [0.  0.2 0.  0.  0. ]
     [0.  0.  0.2 0.  0. ]
     [0.  0.  0.  0.2 0. ]
     [0.  0.  0.  0.  0.2]]

(4) Input the model values, $m$. Given that we are making a forward problem, we assume these values as given, so we input fictitious values:

```python
m = np.array([0.02, 0.05, 0.09, 0.07, 0.04])
print(m)
```

    [0.02 0.05 0.09 0.07 0.04]

(5) Generate our two data values. The remaining step is simply to perform the matrix-vector multiplication:

```python
Gc = (Av.dot(Gn)).T  # cell center matrix Gc. equivalent to np.transpose(Av.dot(Gn))
G = Gc.dot(V)  # make G
d = G.dot(m)

print(d)
```

    [0.04999083 0.03946006]

Equivalently, we can use the `@` operator in Python to perform matrix multiplications.

```python
Gc = (Av @ Gn).T  # cell center matrix Gc. equivalent to np.transpose(Av.dot(Gn))
G = Gc @ V  # make G
d = G @ m

print(d)
```

    [0.04999083 0.03946006]

and the same thing in one line

```python
d = ((Av @ Gn).T @ V) @ m

print(d)
```

    [0.04999083 0.03946006]

and one more! thinking about computational efficiency - put the brackets so that we are always doing matrix-vector products rather than matrix-matrix products

```python
d =  Gn.T @ (Av.T @ (V @ m))

print(d)
```

    [0.04999083 0.03946006]

## A Bigger Toy Problem

We have successfully created two synthetic data points. Hopefully this has provided some intuition about how this is put together. But two data points are not interesting to view. Let's rebuild the problem using a larger system and using more model values and several kernel functions. Let's clear everything and begin again:

```python
clear
```

This time, let's use a similar setup as the one provided in the the *Inversion for Applied Geophysics* tutorial, found here: http://www.eos.ubc.ca/ubcgif/iag/tutorials/tutorial-v9.pdf. For this exercise, let's assume that we have 1000 model parameter values and 20 kernel functions, using the same form for the kernel functions as above. Also, let us create a model that is the combination of a Gaussian and a boxcar on a 1D domain between zero and one, similar to the model in the tutorial. Again, we will be performing the forward problem, which means that we will create synthetic data. The general setup of the domain and a plot of the model the model function could go something like this:

```python
# Begin by creating a ficticious set of model data

n_cells = 1000              # Set number of model parameters
n_nodes = n_cells + 1
xn = np.linspace(0, 1, n_nodes)  # Define 1D domain on nodes
xc = 0.5*(xn[1:] + xn[:-1])   # Define 1D domain on cell centers
```

```python
# Define Gaussian function:
def gauss(x, amplitude, mean, std):
    """Define a gaussian function"""
    return amplitude * np.exp(-((x-mean)/std)**2 / 2)

# Choose parameters for Gaussian, evaluate, and store in an array, f.
std = 6e-2
mean = 0.7
amplitude_gaussian = 1
gaussian = gauss(xc, amplitude_gaussian, mean, std)

fig, ax = plt.subplots(1, 1)
ax.plot(xc, gaussian)
ax.set_title("Gaussian")
```

```python
# Define a boxcar function:
x_boxcar = np.r_[0.2, 0.35]
amplitude_boxcar = 1
boxcar = np.zeros(n_cells)  # initialize an array of all zeros
boxcar_inds = (xc >= x_boxcar.min()) & (xc <= x_boxcar.max())  # find the indices of the boxcar
boxcar[boxcar_inds] = amplitude_boxcar

# construct the model
m = gaussian + boxcar

# Plot
fig, ax = plt.subplots(1, 1)
ax.plot(xc, m)
ax.set_xlabel('x')
ax.set_ylabel('m(x)')
ax.set_title('Model, $m(x)$')

```

Next, let's build the kernel functions. In this case, let's define the kernel functions as before, with p, q, and the number of output data, $N$, as follows. We can then create a set of kernel functions on the nodes.

```python
# Make the set of kernel functions

def kernel_functions(x, j, p, q):
    return np.exp(-p*j*x) * np.cos(2*np.pi*q*j*x)

# Set values for p, q
p = 0.01
q = 0.15
n_data = 20  # specify number of output data

j_min = 0
j_max = n_data
j_values = np.linspace(j_min, j_max, n_data)

# create the G matrix
Gn = np.zeros((n_nodes, n_data))

for i, j in enumerate(j_values):
    Gn[:, i] = kernel_functions(xn, j, p, q)

# Plot
fig, ax = plt.subplots(1, 1)
ax.plot(xn, Gn)
ax.set_xlabel('x')
ax.set_ylabel('g(x)')
ax.set_title('Kernel functions, $g(x)$')
```

The averaging matrix can be made much the same as before, although it will be larger in this case. I will print a subsample of the matrix and show it as a plot below:

```python
# Make Averaging Matrix
Av = np.zeros((n_cells, n_nodes)) # Create a matrix of zeros of the correct dimensions

# and fill in with elements usin the loop below (note the 1/2 is included in here).
for i in range(n_cells):
    Av[i, i] = 0.5
    Av[i, i+1] = 0.5

print(Av.shape)

# Plot
fig, ax = plt.subplots(1, 1)
img = ax.imshow(Av[:5, :6], cmap='Greys')
plt.colorbar(img, ax=ax)
ax.set_title("subset of the averaging matrix")
```

The next step is to make our volume matrix (note its shape, $M \times M$):

```python
# make the Volume, "delta x" array
delta_x = np.diff(xn)  # set x-spacings
V = np.diag(delta_x)  # create diagonal matrix

print(V.shape)
```

    (1000, 1000)

The last step is to create and plot our synthetic data:

```python
%%time
d = Gn.T @ (Av.T @ (V @ m))
```

    CPU times: user 7.96 ms, sys: 9.92 ms, total: 17.9 ms
    Wall time: 5.86 ms

```python
# Plot
fig, ax = plt.subplots(1, 1)
ax.plot(d, '-o')
ax.set_title('Synthetic Data $d$')
```

## Harder, Bettter, Stronger, Faster

Now we have created our first 1D set of synthetic data using numpy exclusively. In real world situations, however, storing matrices explicitly like this is inefficient if not infeasible. For example, for a large data set, the volume matrix consists mainly of zeros, save the diagonal elements (this is a similar case for the averaging matrix). It is not practical to hold on to superfluous information when the operation can be specified without storing so many values. In Python there is a package available capable of performing these operations much more effectively: the Simulation and Parameter Estimation in Geophysics, or "SimPEG" for short (see http://www.simpeg.xyz). As a last exercise, let's see how the same process can be done using SimPEG.

```python
import discretize
```

```python
mesh = discretize.TensorMesh([n_cells])  # create a tensor mesh

# create the G matrix
Gn = np.zeros((n_nodes, n_data))
for i, j in enumerate(j_values):
    Gn[:, i] = kernel_functions(xn, j, p, q)

Av = mesh.aveF2CC # averages from faces to cell centers (faces = nodes in 1D)
V = discretize.utils.sdiag(mesh.vol) # build volume array
```

```python
%%time
d = Gn.T @ (Av.T @ (V @ m))
```

    CPU times: user 415 µs, sys: 88 µs, total: 503 µs
    Wall time: 401 µs

```python
fig, ax = plt.subplots(1, 1)

ax.plot(d, "-o")
ax.set_title("Synthetic Data $d$")
```

Thus far we have created a set of synthetic data. Essentially what we have done is make a forward problem. In the next module, we are going to look at a method for performing and inversion, which will involve casting our inverse problem in terms of an optimization.


sergazy.nurbavliyev@gmail.com © 2021

## fair coins and double head coin

Question: A jar has 1000 coins, of which 999 are fair and 1 is double headed. Pick a coin at random, and toss it 10 times. Given that you see 10 heads, what is the probability that the next toss of that coin is also a heads?

## Answer

Let us understand what the question is. We need to find probability of the toss that shows head given it showed 10 heads already. We will write this with mathematical term  as $\mathbb{P}(H\mid 10 H)$, here $H$ means head and also $T$ would mean tail. By using law of total probability we can write this
\begin{equation}
\mathbb{P}(H | 10H) = \mathbb{P}(H | U) * \mathbb{P}(U | 10H) + \mathbb{P}(H | F) * \mathbb{P}(F | 10H)
\end{equation}
Here $U$ represents unfair coin which is double headed and $F$ shows fair coin. Now let us try to calculate each of these probabilities in equation (1).

$\mathbb{P}(H | U)=1$ (because it is unfair coin)

 $\mathbb{P}(H | F)=\frac{1}{2}$ (because it is fair coin)

Now, for other two probability terms we will use definition of conditional probabilities.

\begin{equation}
\mathbb{P}(U | 10H) =\dfrac{ \mathbb{P}(U,10H)}{\mathbb{P}(10H)}=\dfrac{ \mathbb{P}(10H,U)}{\mathbb{P}(10H)}=  \dfrac{ \mathbb{P}(10H | U) * \mathbb{P}(U)}{\mathbb{P}(10H)}
\end{equation}

\begin{equation}
\mathbb{P}(F | 10H) =\dfrac{ \mathbb{P}(F,10H)}{\mathbb{P}(10H)}=\dfrac{ \mathbb{P}(10H,F)}{\mathbb{P}(10H)}=  \dfrac{ \mathbb{P}(10H | F) * \mathbb{P}(F)}{\mathbb{P}(10H)}
\end{equation}
Now to calculate the equation (1), we need to calculate the equation (2) and (3). Let us first calculate the numerator of equation (2), then numerator of equation (3).

$\mathbb{P}(10H | U)=1$

$\mathbb{P}(U)=\frac{1}{1000}$

$\mathbb{P}(10H | F)=\frac{1}{2^{10}}$

$\mathbb{P}(F)=\frac{999}{1000}$

Now the last thing we need to calculate is the denominator of both equation (2) and (3). For that term we again use the law of total probability.
\begin{equation*}
\mathbb{P}(10H)  = \mathbb{P}(10H | U) * \mathbb{P}(U) + \mathbb{P}(10H | F) * \mathbb{P}(F)=1*\frac{1}{1000}+\frac{999}{1000}*\frac{1}{2^{10}}
\end{equation*}
So equation (2) becomes

\begin{equation*}
\mathbb{P}(U | 10H)=\dfrac{1*\frac{1}{1000}}{1*\frac{1}{1000}+\frac{999}{1000}*\frac{1}{2^{10}}}
\end{equation*}
and equation (3) becomes

\begin{equation*}
\mathbb{P}(F | 10H)=\dfrac{\frac{999}{1000}*\frac{1}{2}}{1*\frac{1}{1000}+\frac{999}{1000}*\frac{1}{2^{10}}}
\end{equation*}
Combining all, equation (1) becomes
\begin{equation}
\mathbb{P}(H | 10H) = \mathbb{P}(H | U) * \mathbb{P}(U | 10H) + \mathbb{P}(H | F) * \mathbb{P}(F | 10H)=1*\dfrac{1*\frac{1}{1000}}{1*\frac{1}{1000}+\frac{999}{1000}*\frac{1}{2^{10}}}+\frac{1}{2}\dfrac{\frac{999}{1000}*\frac{1}{2}}{1*\frac{1}{1000}+\frac{999}{1000}*\frac{1}{2^{10}}}
\end{equation}

```python
equ_2 = (1/1000)/(1/1000 + (1/2)**10 * 999/1000)
equ_2
```

    0.5061789421651013

```python
equ_3 = ((1/2)**10 * 999/1000)/(1/1000 + (1/2)**10 * 999/1000)
equ_3
```

    0.49382105783489866

```python
#our result that is equation 1
equ_1=1*equ_2+1/2*equ_3
equ_1
```

    0.7530894710825506

## Python code for simulation

```python
coins = ["F", "U"]
outcomes = ["H", "T"]
numerator = 0
denominator = 0
```

```python
import random
def simulator(n):
    random.seed(42)
    numerator=0
    denominator=0
    for i in range(n):
        #pick a coin randomly
        rand_int = random.randint(1,1000)
        # assume the double sided coin is the first one
        if rand_int == 1:
            coin= coins[1]
        else:
            coin = coins[0]

        heads = True   # this is always the case for "U" unfair coin
        #print(coin)
        if coin == "F":
            k=0
            while k<10:
                #print(k)
                if random.choice(outcomes) == "T":
                    heads = False
                    break
                k+=1
        if heads==True:
            denominator += 1
            #for the last toss
            if coin == "U" or (coin == "F" and random.choice(outcomes) == "H"):
                numerator += 1
    prob = numerator / denominator
    return prob,numerator,denominator
```

```python
simulator(10000)
```

    (0.6, 12, 20)

```python
simulator(100000)
```

    (0.7, 140, 200)

```python
simulator(1000000)
```

    (0.7442094662638469, 1478, 1986)

```python
trial_list= [simulator(i)[0] for i in range(10000,5000001,10000)]
```

```python
#trial_list
```

```python
from matplotlib import pyplot as plt
plt.plot(trial_list)
plt.axhline(y=0.7530894710825506, color='r', linestyle='-')
plt.show()
```

For further discussions:
https://www.quora.com/A-jar-has-1000-coins-of-which-999-are-fair-and-1-is-double-headed-Pick-a-coin-at-random-and-toss-it-10-times-Given-that-you-see-10-heads-what-is-the-probability-that-the-next-toss-of-that-coin-is-also-a-heads

```python

```

fair coins and double head coin April 6 2021.ipynb


# Random walk and diffusion

We are going to look at a 1D process: a particle of mass $m$ moves in solution (e.g., a small bead or virus in water or oil).

We will use numpy and matplotlib.

```python
import numpy as np
import matplotlib.pyplot as plt
plt.matplotlib.style.use("ggplot")
```

## Newton's equation of motion

Start with Newton's equations of motion for intertial motion:

$$
m\frac{dx^2}{dt^2} = f
$$

The force is derived from a conservative potential $U(x)$

$$
f = -\frac{dU}{dx}.
$$

## Langevin equation

But that's only the particle. Where is the water?

Model the solvent _implicitly_ by its influence on the bead
1. **friction** $-\gamma m v$ ($\gamma > 0$ is the *friction coefficient*)
2. random kicks $f_\text{random}$ from collisions

leading to the **Langevin equation**:

$$
m\frac{dx^2}{dt^2} = f - \gamma m \frac{dx}{dt} + f_\text{random}
$$

## Overdamped Langevin equation

If the friction is high then there is no inertial motion:

$$
0 = f - \gamma m \frac{dx}{dt} + f_\text{random}
$$

so the **overdamped Langevin equation** is

$$
\frac{dx}{dt} = \frac{1}{m\gamma}(f + f_\text{random})
$$

## Discretized dynamics

* Simulate over discrete time steps $\Delta t$
* $v = \Delta x/\Delta t$

$$
\Delta x = \frac{\Delta t}{m\gamma}(f + f_\text{random})
$$

$$
\Delta x = \Delta x_\text{deterministic} + \Delta x_\text{random}
$$

### Trajectories

- We are interested in *trajectories*, i.e. $x(t)$.
- Start at $x(t=0) = x_0$ (initial condition).
- Compute $\Delta x$ for subsequent times and add up steps:

  $$
  x(t) = x_0 + \Delta x(\Delta t) + \Delta x(2\Delta t) + \dots + \Delta x(t-\Delta t)
  $$
- Rewrite:

  \begin{align}
  x_j &:= x(t=j\Delta t)\\
  x_{j+1} &= x_j + \Delta x_{j+1}
  \end{align}

- Deterministic force $f = f(x)$, i.e., $f_j = f(x_j)$ so
  $$
  \Delta x_\text{deterministic} = \frac{f(x_j)}{m\gamma} \Delta t
  $$
  (taking $f$ to be approximately constant over $\Delta t$)
- Random force: zero mean Gaussian distribution

### Random force/random displacement

* Zero mean: $\langle \Delta x_\text{random} \rangle = 0$
* Gaussian distributed: $\Delta x_\text{random} \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{\Delta x_\text{random}^2}{2\sigma^2}\right)$
* Variance *must* be $\sigma^2 = \frac{2 k_B T \Delta t}{m \gamma}$ (to satisfy a fluctuation-dissipation theorem)

$k_B T$ is the typical energy in thermal fluctuations where $k_B = 1.380649 × 10^{−23}\,\text{J/K}$ is the [Boltzmann constant](https://en.wikipedia.org/wiki/Boltzmann_constant) and $T$ the temperature in Kelvin.

#### Normal distribution
What does the Gaussian or **normal distribution** $\mathcal{N}$ look like?

$$
\mathcal{N}(x; \mu, \sigma) = \frac{1}{\sqrt{ 2 \pi \sigma^2 }}e^{ - \frac{ (x - \mu)^2 } {2 \sigma^2} }.
$$

Python function for the normal probability density for mean $\mu$ and standard deviation $\sigma$:

```python
def p_normal(x, mu=0, sigma=1):
    return np.exp(-0.5*((x - mu)/sigma)**2) / (np.sqrt(2*np.pi) * sigma)
```

Plot normal distribution for $\mu=0$, $\sigma=0.5$ over a range of $x$ values:

```python
X = np.linspace(-2, 2, 100)
P = p_normal(X, mu=0, sigma=0.5)
plt.plot(X, P, color="blue")
plt.xlabel("x"); plt.ylabel("p(x)");
```

## Simple diffusion

No external force:

$$
f = 0
$$

so only random movement

$$
\Delta x = \Delta x_\text{random}
$$

## Generating random steps

$$
\Delta x_\text{random} \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{\Delta x_\text{random}^2}{2\sigma^2}\right)
$$

with $\sigma^2 = \frac{2 k_B T \Delta t}{m \gamma}$.

I.e. the probability distribution $p(\Delta x_\text{random})$ is the normal distribution $\mathcal{N}(\Delta x_\text{random}; 0, \sigma)$

### Generating random numbers with numpy

How to generate Gaussian random displacements, i.e. **normally distributed** random numbers?

Use [numpy's random number generation](https://numpy.org/doc/stable/reference/random/index.html) facility:

Always start like this (create a random number generator instance `rng`) at the beginning of your code:

```python
import numpy as np

rng = np.random.default_rng()
```

You then use `rng` throughout your code.

#### Samples from the normal distribution

$$
p(x) = \mathcal{N}(x; \mu, \sigma) = \frac{1}{\sqrt{ 2 \pi \sigma^2 }}e^{ - \frac{ (x - \mu)^2 } {2 \sigma^2} }
$$

Get 1000 samples for $\mu = 0$ and $\sigma = 0.5$ using the [normal()](https://numpy.org/doc/stable/reference/random/generated/numpy.random.Generator.normal.html#numpy.random.Generator.normal) method:

```python
mu = 0
sigma = 0.5

x_samples = rng.normal(loc=mu, scale=sigma, size=1000)
```

<small>(`default_rng()` is the new and recommended way to generate random numbers in numpy. Previously, one used functions in [numpy.random](https://numpy.org/doc/stable/reference/random/) such as [numpy.random.normal()](https://numpy.org/doc/stable/reference/random/generated/numpy.random.RandomState.normal.html#numpy.random.RandomState.normal) or [numpy.random.randn()](https://numpy.org/doc/stable/reference/random/generated/numpy.random.RandomState.randn.html#numpy.random.RandomState.randn). New code, however, should use the above approach that starts with a [numpy.random.Generator](https://numpy.org/doc/stable/reference/random/generator.html#simple-random-data).)</small>

### Plotting random numbers

What do the samples and the distribution look like?

#### Samples

Plot the samples $x$:

```python
plt.plot(x_samples)
plt.ylabel("x");
```

#### Probability distribution from samples

How do we get an estimate for the **distribution** $p(x)$ from our samples?

##### Histogramming
**Histogram** the data with the [numpy.histogram()](https://numpy.org/doc/stable/reference/generated/numpy.histogram.html#numpy-histogram) function (and then turn the counts into a probability density):

```python
np.histogram(x_samples, bins=10)
```

Work with the `counts` (the histogram itself, related to $p(x)$) and the `edges` of the bins (related to the samples, our $x$ values)

```python
counts, edges = np.histogram(x_samples, bins=10)
```

```python
len(counts)
```

```python
len(edges)
```

```python
edges
```

##### Step-plot of histogram counts

Plot the lower edge of each bin:

```python
plt.step(edges[:-1], counts, color="red", where="post")
plt.xlabel("x")
plt.ylabel("counts")
plt.plot(edges[:-1], counts, color="red", marker="o", linestyle="", alpha=0.3)
```

##### Midpoints of bins

Get the *midpoints* $x_k$ of bins $0 \le k < N$ from the edges $e_k, 0 \le k < N+1$:

$$
x_k = \frac{e_k + e_{k+1}}{2}
$$

Python implementation with a loop:

```python
x_values = [0.5*(edges[k] + edges[k+1]) for k in range(len(edges) - 1)]
np.array(x_values)
```

Using fast **numpy array operations** for $x_k = (e_k + e_{k+1})/2$

```
    +---+---+---+---+------+---+----+
e = | 0 | 1 | 2 | 3 | .... | 9 | 10 |
    +---+---+---+---+------+---+----+
```

"Thinking in arrays":

Calculate `e[k] + e[k+1]` as
```
           +---+---+---+---+------+---+
"e[k]" =   | 0 | 1 | 2 | 3 | .... | 9 |  = e[:-1]
           +---+---+---+---+------+---+
             +   +   +   +          +
           +---+---+---+---+------+---+
"e[k+1]" = | 1 | 2 | 3 | 4 | .... |10 |  = e[1:]
           +---+---+---+---+------+---+
```

```python
x_values = 0.5 * (edges[:-1] + edges[1:])
x_values
```

Plot estimate histogram at the midpoints of the bins:

```python
plt.step(x_values, counts, color="red", where="mid")
plt.xlabel("x")
plt.ylabel("counts")
plt.plot(x_values, counts, color="red", marker="o", linestyle="", alpha=0.3)
```

##### Probability distribution from samples

A probability distribution $p(x)$ over a continuous variable $x$ should be a *normalized density* , i.e.,

$$
\int_{-\infty}^{+\infty} \!\! p(x)\, dx = 1
$$

or for discrete bins with a value $p_k$ in bin $k$:

$$
\sum_{k=0}^{N-1} p_k \Delta x_k = 1
$$

where $\Delta x_k = (e_{k+1} - e_k)$ is the bin width of bin $k$.

Given counts $h_k$, one can compute the normalized probability density as

$$
p_k = \frac{h_k}{\sum_k h_k}\frac{1}{\Delta x_k}
$$

```python
Delta_x = edges[1:] - edges[:-1]   # = np.diff(edges)
p_k = counts / (counts.sum() * Delta_x)
```

Check normalization:

```python
np.sum(p_k * Delta_x)
```

Plot estimate for the distribution $p(x)$ of our samples:

```python
plt.step(x_values, p_k, where="mid")
plt.xlabel("x")
plt.ylabel("p(x)");
```

##### `numpy.histogram()` can generate densities

One can directly get a density (for constant bin width) with the `density=True` kwarg:

```python
density, edges = np.histogram(x_samples, bins=10, density=True)
```

Check normalization:

```python
np.sum(density * np.diff(edges))
```

#### Comparison to the normal distribution
$$
\mathcal{N}(x; \mu, \sigma) = \frac{1}{\sqrt{ 2 \pi \sigma^2 }}e^{ - \frac{ (x - \mu)^2 } {2 \sigma^2} }
$$

```python
X = np.linspace(x_values.min(), x_values.max(), 100)

plt.step(x_values, p_k, where="mid", label="samples")

plt.plot(x_values, p_normal(x_values, mu=mu, sigma=sigma), marker="o", color="blue", linestyle="", alpha=0.4)
plt.plot(X, p_normal(X, mu=mu, sigma=sigma), color="blue", alpha=0.4, label="Normal")

plt.xlabel("x")
plt.ylabel("p(x)")
plt.legend();
```

## Simulating random diffusion

We will simulate the diffusion of a small particle such as a [virus](http://book.bionumbers.org/how-big-are-viruses/). Specifically, we pick the harmless [Brome Mosaic Virus](https://en.wikipedia.org/wiki/Brome_mosaic_virus):

### Parameters

[Brome Mosaic virus](https://bionumbers.hms.harvard.edu/bionumber.aspx?id=105893&ver=3&trm=virus+mass&org=) particle

* mass $m=7.6\times 10^{-21} \text{kg}$
* friction coefficient $\gamma=3.4 \times 10^{10}\,\text{s}^{-1}$,
* time step $\Delta t = 1\times10^{-9}\,\text{s}$,
* temperature $T=300\,\text{K}$
* starting position $x_0 = 0$

### Algorithm (random diffusion)

1. Generate a random step
   $$
   \Delta x = \Delta x_\text{random}
   $$
   with
   $$
   \Delta x_\text{random} \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{\Delta x_\text{random}^2}{2\sigma^2}\right)
   $$
   where $\sigma^2 = \frac{2 k_B T \Delta t}{m \gamma}$.
2. Save the step in `steps`.
3. If not reached max number of steps, go to 1.
4. Generate trajectory from steps
   $$
   x_{j+1} = x_j + \Delta x_{j+1}
   $$
   i.e.,
   $$
   x_{k} = x_0 + \sum_{j=1}^{k} \Delta x_j
   $$

Define parameters

```python
m = 7.6e-21      # kg
gamma = 3.4e10   # s^-1
dt = 10e-9       # s
T = 300          # K

kB = 1.380649e-23 # J/K

max_steps = 10000
x0 = 0           # m

sigma = np.sqrt(2*kB*T * dt / (m * gamma))  # m
```

```python
sigma
```

With $\Delta t = 10~\text{ns}$, the "typical" random step size $\sigma$ is on the order of 0.5 nm, i.e. with a virus particle size of ~50 nm, about 1/100 of the virus diameter.

Generate all steps:

   $$
   \Delta x = \Delta x_\text{random}
   $$
   with
   $$
   \Delta x_\text{random} \sim \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{\Delta x_\text{random}^2}{2\sigma^2}\right)
   $$

```python
steps = rng.normal(loc=0, scale=sigma, size=max_steps)
```

Generate trajectory:

   $$
   x_{k} = x_0 + \sum_{j=1}^{k} \Delta x_j
   $$

```python
x_trajectory = x0 + np.cumsum(steps)
```

```python
x_trajectory[:5]
```

Note: The first element *should* be 0: We need to include $x_0$ explicitly.
```python
x_trajectory = np.hstack([x0, x0 + np.cumsum(steps)])
```

Generate trajectory:

   $$
   x_{k} = x_0 + \sum_{j=0}^{k} \Delta x_j
   $$

with $\Delta x_0 = 0$.

```python
steps = rng.normal(loc=0, scale=sigma, size=max_steps+1)     # note size
steps[0] = 0                                                 # dx_0 = 0
```

```python
x_trajectory = x0 + np.cumsum(steps)
```

```python
x_trajectory.shape
```

```python
x_trajectory[:5]
```

### Plot random diffusion trajectory

```python
times = dt * np.arange(len(x_trajectory))
```

```python
plt.plot(times, x_trajectory)
plt.xlabel("time (s)")
plt.ylabel("position (m)");
```

### Generate multiple trajectories

Each simulation run is different (**stochastic**): What is the *average* behavior?

Run multiple repeats:

```python
N_repeats = 100
```

```python
steps = rng.normal(loc=0, scale=sigma, size=(N_repeats, max_steps+1))
steps[:, 0] = 0       # step 0 with dx=0
```

```python
steps.shape
```

```python
x_trajectories = x0 + np.cumsum(steps, axis=1)
```

```python
x_trajectories.shape
```

Plot the **ensemble of trajectories**

```python
times = dt * np.arange(x_trajectories.shape[1])
plt.plot(times, x_trajectories.T, alpha=0.5);
plt.xlabel("time (s)")
plt.ylabel("position (m)");
```

To show more clearly the distribution of trajectories, with the first one hightlighted:

```python
times = dt * np.arange(x_trajectories.shape[1])
plt.plot(times, x_trajectories.T, color="black", alpha=0.1);
plt.plot(times, x_trajectories[0], color="red", linewidth=3);
plt.xlabel("time (s)")
plt.ylabel("position (m)")
plt.savefig("diffusion_trajectories.png")
```

## Analysing stochastic trajectories

Given the **trajectory ensemble**, what is the **average** behavior (what are the underlying physical laws that are obscured by the stochasticity?)

#### Mean position

Average (mean) position over $N$ trajectories $x_j(t)$

$$
\bar x(t) = \langle x(t) \rangle = \frac{1}{N} \sum_{j=1}^N x_j(t)
$$

where the average is performed *at each time step* $k$ (for $t = k \Delta t$):

$$
\bar x_k = \langle x_k \rangle = \frac{1}{N} \sum_{j=1}^N x_{j,k}
$$

```python
x_mean = x_trajectories.mean(axis=0)
x_mean.shape
```

Plot the mean position $\bar{x}(t)$:

```python
plt.plot(times, x_mean)
plt.xlabel("time (s)")
plt.ylabel("average position (m)");
```

To put it into perspective: the mean position $\langle x(t) \rangle$ is close to 0 when compared to the trajectories themselves:

```python
plt.plot(times, x_trajectories.T, alpha=0.1, color="black")
plt.plot(times, x_mean, linewidth=3)
plt.xlabel("time (s)")
plt.ylabel("average position (m)");
```

This should not be surprising: the normal distribution has mean 0.

#### Distance from starting point

Distance from origin $x_0$ is

$$
d(t) = \sqrt{(x(t) - x_0)^2}
$$

with the squared distance

$$
d^2(t) = (x(t) - x_0)^2
$$

being the important quantity that measures how far the particle got away from $x_0$.

Averaging over many trajectories we obtain the **mean squared distance**

$$
\text{MSD}(t) = \langle d^2(t) \rangle = \langle (x(t) - x_0)^2 \rangle
$$

The **root mean square distance** is then simply
$$
\text{RMSD}(t) = \sqrt{\text{MSD}(t)} = \sqrt{\langle(x(t) - x_0)^2\rangle}
$$

For a single trajectory, the $d^2(t)$ is noisy:

```python
D2 = (x_trajectory - x0)**2
```

```python
plt.plot(times, D2)
plt.xlabel("time (s)")
plt.ylabel(r"$d^2(t)$ (m$^2$)");
```

Averaging $d^2(t)$ over many trajectories gives the MSD:

```python
MSD = np.mean((x_trajectories - x0)**2, axis=0)
```

```python
plt.plot(times, MSD)
plt.xlabel("time (s)")
plt.ylabel(r"$\mathrm{MSD}(t)$ (m$^2$)");
```

The MSD is approximately linear with time, as given by the Einstein diffusion equation (in 1D)

$$
\lim_{t\rightarrow\infty} \text{MSD}(t) =  2 D t
$$

With the  [Einstein relation](https://en.wikipedia.org/wiki/Einstein_relation_(kinetic_theory)) (also: a fluctation-dissipation theorem for diffusion coefficient $D$ and drag coefficient $\zeta$)

$$
D = \frac{kT}{\zeta} \quad\text{and}\quad \zeta = m\gamma
$$
we can plot the theoretical MSD.

An equivalent form uses the mobility $\mu = \zeta^{-1}$ ($\mathbf{v} = \mu \mathbf{F}$)

$$
D = \mu kT
$$

```python
D = kB * T / (m * gamma)
print(D)
```

```python
MSD_theory = 2 * D * times
```

Simulation and theoretical MSD plotted together:

```python
plt.plot(times, MSD, label="simulation")
plt.plot(times, MSD_theory, label="theory")
plt.xlabel("time (s)")
plt.ylabel(r"$\mathrm{MSD}(t)$ (m$^2$)")
plt.legend();
```

# Further exploration

- Increase the number of trajectories: Does the MSD get closer to the theorectical value?
- Increase the length of trajectories: Does the MSD get closer to the theorectical value?
- Estimate the distribution $p(x(t))$ (hint: use the histogram over the ensemble at fixed times $t$ and fix the bins; this distribution evolves over time); compare to the theoretical distribution (which is a Gaussian — why?)
- Obtain an estimate of $D$ from the trajectories.
- Have the particles move in a potential $U(x)$ and include the deterministic force,
- Extend the code to 2D. (Note: in $d$ dimensions, $\lim_{t\rightarrow+\infty} \langle (\mathbf{x}(t) - \mathbf{x}_0)^2\rangle = 2dDt$)

# Appendix

## Parameters for the Brome Mosaic Virus

Data for the Brome Mosaic Virus
* [mass](https://bionumbers.hms.harvard.edu/search.aspx?task=searchbytrmorg&log=y&trm=Brome%20Mosaic%20Virus) 4.6e3 kDa
* [D](https://bionumbers.hms.harvard.edu/bionumber.aspx?id=105894&ver=2&trm=Brome+Mosaic+Virus&org=) = 15.5 µm^2 s^-1 (T = 20ºC)

```python
Da = 1.660539066601e-27 # kg
T = 20 + 273.15 # K
kB = 1.380649e-23 # J/K

m_BMV = 4.6e3 * 1e3 * Da # kg
D_BMV = 15.5 * (1e-6)**2 # m^2/s
```

```python
print(f"mass = {m_BMV:.2g} kg")
print(f"T = {T:.1f} K")
```

$\gamma$ from [Einstein relation](https://en.wikipedia.org/wiki/Einstein_relation_(kinetic_theory)) (fluctuation-dissipation relationship between diffusion coefficient $D$ and drag coefficient $\zeta = m\gamma$)
\begin{align}
D &= \frac{kT}{\zeta} = \frac{kT}{m\gamma}\\
\gamma &= \frac{kT}{mD}
\end{align}

```python
gamma_BMV = kB*T / (m_BMV * D_BMV)
print(f"gamma = {gamma_BMV:.2g} s^-1")
```

## Finding $D$ by linear fit

We can use [scipy.optimize.curve_fit()](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html#scipy.optimize.curve_fit) as a "black box" fitting function (it minimizes the RMSD between fit function and data):

```python
# with scipy
from scipy import optimize

def diff_fit(t, D, x0=0):
    return 2 * D * t  + x0
```

```python
params, params_covariance = optimize.curve_fit(diff_fit, times, MSD, p0=[1])
D_fit, = params

print(f"D = {D:.2g}   D_fit = {D_fit:.2g}")
```

```python
MSD_fit = diff_fit(times, D_fit)

plt.plot(times, MSD, label="simulation")
plt.plot(times, MSD_theory, label=r"theory $D={0:.3g}\,\mathrm{{m}}^2\cdot\mathrm{{s}}^{{-1}}$".format(D))
plt.plot(times, MSD_fit, linestyle="--", linewidth=2,
         label=r"fit $D_\mathrm{{fit}}={0:.3g}\,\mathrm{{m}}^2\cdot\mathrm{{s}}^{{-1}}$".format(D_fit))
plt.xlabel("time (s)")
plt.ylabel(r"$\mathrm{MSD}(t)$ (m$^2$)")
plt.legend()
plt.savefig("diffusion_sim_theory_fit.png")
```

```python

```


# IYPT: Guitar String
Problem 11 of the 34th International Young Physicists' Tournament

## Analytical solution

using `sympy` library;

attempts to algorithmically find analytical solution for given parameters

```python
from sympy import *

# global time symbol
t = symbols('t')

# accepts parameters (optimal):
def analytical(
    beta    = symbols('beta'),
    gamma   = symbols('gamma'),
    omega_0 = symbols('omega_0'),
    m       = symbols('m'),
    F       = symbols('F', cls=Function),
    ):

    # second newton's law elements
    y        = symbols('y', cls=Function)
    ydprime  = y(t).diff(t, 2)
    damping  = 2 * beta * omega_0 * y(t).diff(t)
    system   = omega_0**2 * y(t) * (1 + gamma * y(t)**2)
    driving  = F(t) if callable(F) else F

    # putting it together
    lefthand = ydprime + damping + system
    ritehand = driving / m
    eq = Eq(lefthand, ritehand)
    print('input:')
    display(eq)

    # attempt to solve
    try:
        return dsolve(eq)
    except NotImplementedError:
        return None

# call with given parameters
F_m, omega, phi = symbols('F_m omega phi')
sol = analytical(
    gamma = 0,
    F = F_m * sin(omega * t + phi)
)

print('solution:')
display(sol)
```

    input:

$\displaystyle 2 \beta \omega_{0} \frac{d}{d t} y{\left(t \right)} + \omega_{0}^{2} y{\left(t \right)} + \frac{d^{2}}{d t^{2}} y{\left(t \right)} = \frac{F_{m} \sin{\left(\omega t + \phi \right)}}{m}$

    solution:

$\displaystyle y{\left(t \right)} = C_{1} e^{\omega_{0} t \left(- \beta - \sqrt{\beta^{2} - 1}\right)} + C_{2} e^{\omega_{0} t \left(- \beta + \sqrt{\beta^{2} - 1}\right)} - \frac{2 F_{m} \beta \omega \omega_{0} \cos{\left(\omega t + \phi \right)}}{m \left(4 \beta^{2} \omega^{2} \omega_{0}^{2} + \omega^{4} - 2 \omega^{2} \omega_{0}^{2} + \omega_{0}^{4}\right)} - \frac{F_{m} \omega^{2} \sin{\left(\omega t + \phi \right)}}{m \left(4 \beta^{2} \omega^{2} \omega_{0}^{2} + \omega^{4} - 2 \omega^{2} \omega_{0}^{2} + \omega_{0}^{4}\right)} + \frac{F_{m} \omega_{0}^{2} \sin{\left(\omega t + \phi \right)}}{m \left(4 \beta^{2} \omega^{2} \omega_{0}^{2} + \omega^{4} - 2 \omega^{2} \omega_{0}^{2} + \omega_{0}^{4}\right)}$

## Numerical solution

calculating performed with `scipy`, `numpy` libraries,

plotting via `matplotlib` library;

displays approximate numerical solution for given model

```python
from numpy import *
from scipy.integrate import odeint
from matplotlib import pyplot as plt

# formula parameters required;
# notation - as above so below
# duration - experiment duration
# sampling - samples per vibration
def numerical(
    beta,
    gamma,
    omega_0,
    m,
    F,
    y_0 = 0,
    duration = 100,
    sampling = 100,
    label = None,
    rtol = None,
    atol = None,
    ):

    # numerical model
    # y is list of the displacement function and its derivatives
    # t is respective time coordinate
    def model(y, t):
        system  = omega_0**2 * y[0] * (1 + gamma * y[0]**2)
        damping = 2 * beta * omega_0 * y[1]
        driving = F(t) / m
        return y[1], -system - damping + driving

    # initial conditions (x,y)
    initial = (0, y_0)

    # prepare samples
    total = duration * (omega_0 / (2 * pi)) * sampling
    space = linspace(0, duration, int(total))

    # run model
    sols  = odeint(model, initial, space, atol=atol, rtol=rtol)
    sol   = sols[0:,1]
    plt.plot(space, sol, label=label)
    return plt.legend(loc='best')

# define driving force
omega = 1
F_m   = 1
phi   = 0
F     = lambda t: F_m * sin(omega * t + phi)

# call with given parameters
sol = numerical(
    beta     = 1 * 10**-2,
    omega_0  = 1,
    gamma    = 0,
    m        = 1,
    F        = F,
    duration = 300,
    atol=10**-14,
    rtol=10**-13,
)
```

## Duffing oscillator

```python
display(numerical(
    beta     = 1. * 10**-3,
    omega_0  = 330 * 2 * pi,
    gamma    = .25 * 10**4,
    m        = 1.,
    F        = lambda t: F_m * sin((200*2*pi) * t + phi),
    sampling = 100,
    duration = .05,
    label    = '200 Hz'
))

display(numerical(
    beta     = 1. * 10**-3,
    omega_0  = 330 * 2 * pi,
    gamma    = .25 * 10**4,
    m        = 1.,
    F        = lambda t: F_m * sin((250*2*pi) * t + phi),
    sampling = 100,
    duration = .05,
    label    = '250 Hz',
))
```

```python
display(numerical(
    beta     = 1. * 10**-3,
    omega_0  = 330 * 2 * pi,
    gamma    = .25 * 10**4,
    m        = 1.,
    F        = lambda t: F_m * sin((300*2*pi) * t + phi),
    sampling = 100,
    duration = .05,
    label    = '300 Hz'
))

display(numerical(
    beta     = 1. * 10**-3,
    omega_0  = 330 * 2 * pi,
    gamma    = .25 * 10**4,
    m        = 1.,
    F        = lambda t: F_m * sin((330*2*pi) * t + phi),
    sampling = 100,
    duration = .05,
    label    = '330 Hz'
))
```


### Example 3: Diffusion

In this example we will look at the diffusion equation and how to handle second-order derivatives. For this we will introduce Devito's `.laplace` short-hand expression and demonstrate it using the examples from step 7 of the original tutorial.

So, the equation we are now trying to implement is

$$\frac{\partial u}{\partial t} = \nu \frac{\partial ^2 u}{\partial x^2} + \nu \frac{\partial ^2 u}{\partial y^2}$$

To discretize this equation we will use central differences and reorganizing the terms yields

\begin{align}
u_{i,j}^{n+1} = u_{i,j}^n &+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n - 2 u_{i,j}^n + u_{i-1,j}^n) \\
&+ \frac{\nu \Delta t}{\Delta y^2}(u_{i,j+1}^n-2 u_{i,j}^n + u_{i,j-1}^n)
\end{align}

As usual, we establish our baseline experiment by re-creating some of the original example runs. So let's start by defining some parameters.

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 31
ny = 31
nt = 17
nu = .05
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .25
dt = sigma * dx * dy / nu
```

We now set up the diffusion operator as a separate function, so that we can re-use if for several runs.

```python
def diffuse(u, nt):
    for n in range(nt + 1):
        un = u.copy()
        u[1:-1, 1:-1] = (un[1:-1,1:-1] +
                        nu * dt / dx**2 * (un[1:-1, 2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                        nu * dt / dy**2 * (un[2:,1: -1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))
        u[0, :] = 1
        u[-1, :] = 1
        u[:, 0] = 1
        u[:, -1] = 1
```

Now let's take this for a spin. In the next two cells we run the same diffusion operator for a varying nuper of timesteps to see our "hat function" dissipate to varying degrees.

```python
#NBVAL_IGNORE_OUTPUT

# Initialise u with hat function
u = np.empty((nx, ny))
init_hat(field=u, dx=dx, dy=dy, value=2.)
plot_field(u, zmax=2.5)

diffuse(u, nt=10)
plot_field(u, zmax=2.5)
```

```python
#NBVAL_IGNORE_OUTPUT

# Initialise u with hat function
u = np.empty((nx, ny))
init_hat(field=u, dx=dx, dy=dy, value=2.)

diffuse(u, nt=50)
plot_field(u, zmax=2.5)
```

Excellent. Now for the Devito part we need to note one important detail to our previous examples: we now have a second-order derivative. So, when creating our `TimeFunction` object we need to tell it about our spatial discretisation by setting `space_order=2`. Only then can we use the shorthand notation `u.dx2` and `u.dx2` to denote second derivatives.

```python
from devito import Grid, TimeFunction, Eq
from sympy import solve
from sympy.abc import a

# Initialize `u` for space order 2
grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid, space_order=2)

# Create an equation with second-order derivatives
eq = Eq(u.dt, a * (u.dx2 + u.dy2))
stencil = solve(eq, u.forward)[0]
eq_stencil = Eq(u.forward, stencil)

print(eq_stencil)
```

    Eq(u(t + dt, x, y), (a*dt*h_x**2*(-2*u(t, x, y) + u(t, x, y - h_y) + u(t, x, y + h_y)) + a*dt*h_y**2*(-2*u(t, x, y) + u(t, x - h_x, y) + u(t, x + h_x, y)) + h_x**2*h_y**2*u(t, x, y))/(h_x**2*h_y**2))

Now, there is another trick here! Note how the above formulation explicitly uses `u.dx2` and `u.dy2` to denote the laplace operator, which makes this equation dependent on the spatial dimension. We can instead use the notation `u.laplace` to denote all second derivatives in space, allowing us to reuse this code for 2D and 3D examples.

```python
eq = Eq(u.dt, a * u.laplace)
stencil = solve(eq, u.forward)[0]
eq_stencil = Eq(u.forward, stencil)

print(eq_stencil)
```

    Eq(u(t + dt, x, y), (a*dt*h_x**2*(-2*u(t, x, y) + u(t, x, y - h_y) + u(t, x, y + h_y)) + a*dt*h_y**2*(-2*u(t, x, y) + u(t, x - h_x, y) + u(t, x + h_x, y)) + h_x**2*h_y**2*u(t, x, y))/(h_x**2*h_y**2))

Great. Now all that is left is to pu it all together to build the operator and use it on our examples. For illustration purposes we will do this in one cell, including update equation and boundary conditions.

```python
#NBVAL_IGNORE_OUTPUT
from sympy import solve
from devito import Operator, Constant, Eq, INTERIOR

# Reset our data field and ICs
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)

# Create an operator with second-order derivatives
a = Constant(name='a')
eq = Eq(u.dt, a * u.laplace, region=INTERIOR)
stencil = solve(eq, u.forward)[0]
eq_stencil = Eq(u.forward, stencil)

# Create boundary condition expressions
x, y = grid.dimensions
t = grid.stepping_dim
bc = [Eq(u.indexed[t+1, 0, y], 1.)]  # left
bc += [Eq(u.indexed[t+1, nx-1, y], 1.)]  # right
bc += [Eq(u.indexed[t+1, x, ny-1], 1.)]  # top
bc += [Eq(u.indexed[t+1, x, 0], 1.)]  # bottom

op = Operator([eq_stencil] + bc)
op(time=10, dt=dt, a=nu)

plot_field(u.data[0], zmax=2.5)
```

And now let's use the same operator again to show the more diffused field. In fact, instead of resetting the previously used object `u`, we can also create a new `TimeFunction` object and tell our operator to use this.

```python
#NBVAL_IGNORE_OUTPUT
u2 = TimeFunction(name='u2', grid=grid, space_order=2)
init_hat(field=u2.data[0], dx=dx, dy=dy, value=2.)

op(u=u2, time=50, dt=dt, a=nu)

plot_field(u2.data[0], zmax=2.5)
```


```julia
using Random
using Distributions
using Statistics
using Printf
using Plots
```

```julia
# for readability, these are good settings to use
default(xtickfontsize=14,  ytickfontsize=14, ztickfontsize=14,
    guidefontsize=14, legendfontsize=12, lw=2,ms=8)
```

## Example 6.5
Taken from Rubinstein and Kroese.  Sample from the following density in $\mathbb{R}^2$ using the Gibbs sampler:
$$
f(x,y) = c_1(y) \exp \left(-\frac{1+y^2}{2} \left(x - \frac{4}{1+y^2}\right)^2\right)
$$
After some algebra, it can be shown that
\begin{align}
X\mid Y = y &\sim N\left(\tfrac{4}{1+y^2}, \tfrac{1}{1+y^2}\right)\\
Y\mid X = x &\sim N\left(\tfrac{4}{1+x^2}, \tfrac{1}{1+x^2}\right)
\end{align}
allowing us to sample basic Gaussians at each step.

```julia
X0 = ones(2);
T = 10^6; # total number of iterations
```

```julia
X = deepcopy(X0);
X_samples = [deepcopy(X)];
Random.seed!(100);
for _ in 1:T-1
    x = rand(Normal(4/(1+X[2]^2), 1/sqrt(1+X[2]^2)));
    y = rand(Normal(4/(1+x^2), 1/sqrt(1+x^2)));
    @. X  = [x,y];
    push!(X_samples, deepcopy(X))
end
```

```julia
histogram2d([X_[1] for X_ in X_samples],[X_[2] for X_ in X_samples], norm =:pdf)
xlabel!("x")
ylabel!("y")
```

```julia
@show mean([X_[1] for X_ in X_samples]);
```

```julia
plot(1:T, cumsum([X_[1] for X_ in X_samples])./(1:T), label="Running Avg.", legend=:bottomright, xscale=:log10)
xlabel!("Iterate")
ylabel!("Estimate")
```


# Exploring Multiple Dispatch

[Multiple dispatch](https://en.wikipedia.org/wiki/Multiple_dispatch) is a very powerful technique that can simplify code substantially. In short, it's about being able to define _generic_ functions that can work on a wide variety of types, perhaps using different code depending on the types used.

In this notebook, we'll explore how to do so-called automatic differentiation with multiple dispatch, using the Julia language.

## Taking Derivatives

Firstly, let's have a look at the problem. Suppose we have a function $f$ that does something unknown. We want to find its derivative, $f'$. How might we approach this problem?

Let's consider an example. Polynomial $p$ below is defined by $p(x) = 2x^3 + 3x^2 + 6x + 6$. We know from calculus that the derivative $p'(x) = 6x^2 + 6x + 6$. But how might we implement this in code? Recall the limit definition of a derivative: \begin{equation}
 f'(a) = \lim_{x\to a} \frac{f(x) - f(a)}{x-a}
\end{equation}

Then, to calculate the derivative at $a$, we might be inclined to choose a value of $x$ very close to $a$, and simply evaluate the above expression. How close? With floating point numbers, we might wish to choose $a + a \cdot \mathbf{u}$, where $\mathbf{u}$ is known as "[machine epsilon](https://en.wikipedia.org/wiki/Machine_epsilon)", and is a bound on the error due to rounding of floating point operations. (We will see later that this is not necessarily the best choice.)

We'll work with $64$-bit floating points, so let's first compute our value of $\mathbf{u}$ and define polynomial $p$:

```julia
const u = eps(Float64)
```

```julia
p(x) = 2x^3 + 3x^2 + 6x + 6
```

    p (generic function with 1 method)

Now let's create our differentiation function. We want this function to take in a function, and return a function that computes its derivative. In Julia, the `->` syntax denotes the creation of a function. Think of the below code as mathematically representing \begin{equation}
 f' = a \mapsto \frac{f(a + a \cdot \mathbf{u}) - f(a)}{a \cdot \mathbf{u}}
\end{equation} which is just a fancy way of saying \begin{equation}
 f'(a) = \frac{f(a + a \cdot \mathbf{u}) - f(a)}{a \cdot \mathbf{u}}
\end{equation}

```julia
der_u(f) = a -> (f(a + a * u) - f(a)) / (a * u)
```

    der_u (generic function with 1 method)

To test this function, we'll define `p′` (note that this is not an apostrophe, which would be a syntax error; this is a unicode character that looks similar), and evaluate it at several points. For instance, we expect $p'(0) = 6$, $p'(1) = 18$, since these are the exact result that calculus gives us.

```julia
p′ = der_u(p)
p′(1)
```

    WARNING: Base.SingleAsyncWork is deprecated, use Base.AsyncCondition instead.
      likely near /home/fengyang/.julia/v0.5/IJulia/src/kernel.jl:31
    in gc_protect_handle at /home/fengyang/.julia/v0.5/ZMQ/src/ZMQ.jl
    WARNING: Base.SingleAsyncWork is deprecated, use Base.AsyncCondition instead.
      likely near /home/fengyang/.julia/v0.5/IJulia/src/kernel.jl:31
    WARNING: Base.SingleAsyncWork is deprecated, use Base.AsyncCondition instead.
      likely near /home/fengyang/.julia/v0.5/IJulia/src/kernel.jl:31
    in gc_protect_handle at /home/fengyang/.julia/v0.5/ZMQ/src/ZMQ.jl
    WARNING: Base.SingleAsyncWork is deprecated, use Base.AsyncCondition instead.
      likely near /home/fengyang/.julia/v0.5/IJulia/src/kernel.jl:31
    in gc_protect_handle at /home/fengyang/.julia/v0.5/ZMQ/src/ZMQ.jl
    WARNING: Base.SingleAsyncWork is deprecated, use Base.AsyncCondition instead.
      likely near /home/fengyang/.julia/v0.5/IJulia/src/kernel.jl:31
    in gc_protect_handle at /home/fengyang/.julia/v0.5/ZMQ/src/ZMQ.jl

    16.0

Yikes. This is a complete disaster. Two off is bad enough, and returning a result that looks so innocent is even worse!

```julia
p′(0)
```

    NaN

Oof. If being off by $2$ were bad, being off by `NaN` must be even worse. Clearly we need to refine our approach. Firstly, we need to make sure that there's a minimum difference that we use (for the case of $0$). And next, we should be more conservative about our choice of epsilon. Machine epsilon is too small. How about its square root, which is a bigger number? We'll call this $\mathbf{v} = \sqrt{\mathbf{u}}$.

```julia
const v = sqrt(u)
der_v(f) = function(a)
    diff = a * v + v
    (f(a + diff) - f(a)) / diff
end
```

    der_v (generic function with 1 method)

```julia
p′ = der_v(p)
p′(1)
```

    18.00000023841858

```julia
p′(0)
```

    6.000000059604645

Phew. This is much better. It could be more exact, but it's quite good as is. Let's try it on some other functions. Let's start with $\mathrm{e}^x$, whose derivative is $\mathrm{e}^x$.

```julia
exp′ = der_v(exp)
exp′(0)
```

    1.0

```julia
exp′(1)
```

    2.7182818800210953

```julia
exp′(log(1000))
```

    1000.0000560290179

So far so good. How about $\sin(x)$, whose derivative is $\cos(x)$?

```julia
sin′ = der_v(sin)
sin′(0)
```

    1.0

```julia
sin′(acos(0.9))
```

    0.8999999961400434

Excellent! Everything is just fine and dandy. Now how about $\log(x)$ (base $\mathrm{e}$), whose derivative is $\frac{1}{x}$?

```julia
log′ = der_v(log)
log′(1)
```

    0.9999999850988391

```julia
log′(10)
```

    0.09999999674883756

```julia
log′(1/1000000)
```

    992622.6095255658

Hmm. The error near $0$ is a bit higher than we might prefer. We could fix that by reducing $\mathbf{v}$, but if we do that too much we end up in trouble. For now let's consider higher order derivatives. What about the second derivative of $\mathrm{e}^x$?

```julia
exp′′ = der_v(exp′)
exp′′(0)
```

    1.9999999701976776

```julia
exp′′(1)
```

    2.1408590227365494

```julia
exp′′(log(10))
```

    9.441047175715648

This is a disaster. The error, which was acceptable initially, just compounded itself. And the problem is not specific to $\mathrm{e}^x$.

```julia
sin′′ = der_v(sin′)
sin′′(π)
```

    0.008254849939365927

```julia
sin′′(π/2)
```

    -0.9078537644150019

```julia
log′′ = der_v(log′)
log′′(10)
```

    0.007438016716729511

You might imagine it to be a problem when the second deriviative of $\log$, which is $x\mapsto \frac{-1}{x^2}$, is computed to be a positive number of magnitude comparable to the negative number it's supposed to be. Third and fourth derivatives, as you might imagine, are even worse. The error is ridiculous.

```julia
exp′′′ = der_v(exp′′)
exp′′′(1)
```

    1.6777211679570556e7

```julia
sin′′′′ = der_v(der_v(sin′′))
sin′′′′(π)
```

    -0.7602858901604265

It is easy to not notice the scientific notation in the computation of the third derivative of $\mathrm{e}^x$ at $1$. The correct answer was $\mathrm{e}=2.718\dots$, and $1.6777$ would be far enough off already. But, look closely and it was actually computed to be $1.6777\times10^7$. Seven orders of magnitude off. For your entertainment, I have included the fifth derivative of $\mathrm{e}^x$ also.

```julia
exp′′′′′ = der_v(der_v(exp′′′))
exp′′′′′(1)
```

    -3.777892623345787e22

Oof. We need a better way. In an ideal world, we could compute exact derivatives. Instead of approximating a limit by picking small numbers, we could compute the limit _exactly_. Wouldn't that be neat. It would be great if it were possible, but we are working with dumb silicon machines here, which cannot possibly compute values as exact as a human could. One can dream though. And if one dreamt hard enough, and one knew enough linear algebra, then one might come up with dual numbers, the technique that makes this possible.

## Dual Numbers

A [dual number](https://en.wikipedia.org/wiki/Dual_number) is like a complex number in that it has two components. But instead of $\mathbb{R}[\sqrt{-1}]$, we will consider $\mathbb{R}[\varepsilon]$, where $\varepsilon$ is defined so that $\varepsilon^2=0$. The intuition behind this definition is that $\varepsilon$ is _really_ small. It's so small that it's smaller than **any** positive real number, but just a _little_ bigger than $0$. It's so small that if we multiply it by itself (multiplying really small by really small), that we can no longer see it.

Aha. Now what if instead of using $\mathbf{v} + a\mathbf{v}$ as our epsilon, which is small but not that small, we can use $\varepsilon$ as our epsilon? Fat chance, right? If even machine epsilon ($\mathbf{u}$) was too small to work, how could $\varepsilon$ possibly work?

Our first speed bump is that $\varepsilon$ is not something built-in. We need to define it. Let's worry about the details later and build a kind of `DualNumber` type:

```julia
immutable DualNumber{N <: Number} <: Number
    re::N
    ep::N
end
```

What the above definition says, in short, is that we wish to create a new `DualNumber` type with a single [type parameter](http://docs.julialang.org/en/release-0.4/manual/types/#man-parametric-types), which is a `Number`. In Julia, `Number` is an abstract type with many concrete implementations. Mathematically, we want `DualNumbers` to work with any field. Examples of common fields are $\mathbb{R}$ and $\mathbb{C}$. Furthermore, a `DualNumber` itself is a kind of number. We can create `DualNumber{Float64}` to deal with floating point type `DualNumber`s, and that's what we want for now. But we could do other cool stuff too. We'll look at that later.

The type itself contains two fields, a `re` field containing the real (big) part, and the `ep` field containing the epsilon (small) part. So something like `DualNumber(1.0, 2.0)` would be equivalent to $1 + 2\epsilon$.

Julia will by default print `DualNumber`s like `DualNumber{Float64}(1.0, 2.0)`. This is descriptive and useful, but for our eyes' sake we should print `DualNumber`s of floats the way we expect, as $a+b\varepsilon$, just like how complex numbers are usually printed. We can define the `Base.show` method on `DualNumber{T}` for all `T <: AbstractFloat`. `AbstractFloat` in Julia is an abstract type, again with many concrete implementations, in particular `Float16`, `Float32`, `Float64`, and `BigFloat`. We only want to define this show method on floats, because if we had `DualNumber`s of other types (such as complex numbers), it may not be printed nicely as $a+b\varepsilon$.

```julia
import Base: show

function show{T <: AbstractFloat}(io::IO, num::DualNumber{T})
    show(io, num.re)
    if signbit(num.ep)
        print(io, " − ")
        show(io, -num.ep)
    else
        print(io, " + ")
        show(io, num.ep)
    end
    print(io, "ɛ")
end
```

    show (generic function with 133 methods)

It would be convenient to have an $\varepsilon$ constant for testing and usability purposes. But what type should this constant be? Ideally, we want a field that can be embedded in any bigger field, so that our constant can be converted to any type of `DualNumber`. The smallest possible field is $\mathbb{Z}_2$. This is the field of booleans. So we want to define $\varepsilon$ as a `DualNumber{Bool}`. Sounds crazy, but it's mathematically sound.

```julia
const ɛ = DualNumber(false, true)
```

    DualNumber{Bool}(false,true)

Good. It works. But Julia still doesn't know how these numbers work; we've only taught it how to show them, and what $\varepsilon$ is. First, since these things are a vector space over their base field, we need to define addition and scalar multiplication. Scalar multiplication could be defined right now, but since we will be defining multiplication with these things soon enough, we can afford to wait on that briefly. The definition of addition is another example of multiple dispatch.

```julia
import Base: +

+{T<:Number}(x::DualNumber{T}, y::DualNumber{T}) = DualNumber{T}(x.re + y.re, x.ep + y.ep)

DualNumber(10.0, 17.0) + DualNumber(5.0, 9.0)
```

    15.0 + 26.0ɛ

It will also be useful to define some conversions between dual numbers and scalar types. This will let us write, say, `5 + 2.0ɛ` and get the right type back. Julia handles conversions and promotions using multiple dispatch. (Get used to it. Multiple dispatch is Julia's best feature, and good language design means heavily relying on your best feature! That's why C relies so much on raw pointer arithmetic.)

```julia
import Base: convert

convert{T<:Number}(::Type{DualNumber{T}}, x::DualNumber{T}) = x
convert{T<:Number}(::Type{DualNumber{T}}, x::DualNumber) =
    DualNumber{T}(convert(T, x.re), convert(T, x.ep))
convert{T<:Number}(::Type{DualNumber{T}}, x::Number) =
    DualNumber{T}(convert(T, x), zero(T))
```

    convert (generic function with 555 methods)

The above three methods describe ways to convert various kinds of things to `DualNumber`s. The first one says that to convert a dual number type to itself, nothing needs to be done. The second one tells Julia that to convert a `DualNumber` of one type to another, just convert each component. The third one tells Julia that to convert a regular number to a `DualNumber`, convert it to the base field and set the epsilon component to $0$. All these methods are mathematically straightforward and sound.

Conversions themselves are not enough to add dual numbers with scalars. We also need to tell Julia how to promote the operands to a common, compatible type. In our case, we want to convert the scalars to `DualNumber`s.

```julia
import Base: promote_rule

promote_rule{T<:Number, U<:Number}(::Type{DualNumber{T}}, ::Type{DualNumber{U}}) =
    DualNumber{promote_type(T, U)}
promote_rule{T<:Number, U<:Number}(::Type{DualNumber{T}}, ::Type{U}) =
    DualNumber{promote_type(T, U)}
```

    promote_rule (generic function with 99 methods)

The first rule here says that two `DualNumber` types should be promoted to a `DualNumber` type whose base field is one compatible with both scalar types. The second rule says that a `DualNumber` and a scalar should be promoted to a `DualNumber`, with base field again compatible with both scalar types. By now we have encoded all the properties of dual numbers as being $\mathbb{F}[\varepsilon]$ for some field $\mathbb{F}$, but we have yet to define the behaviour of this algebra under multiplication.

Note \begin{equation}
(a + b\varepsilon)(c + d\varepsilon) = ac + ad\varepsilon + bc\varepsilon + bd\varepsilon^2 = ac + (ad + bc)\varepsilon
\end{equation}
which you might notice resembles the Leibniz product rule. We can easily encode this:

```julia
import Base: *

*{T<:Number}(x::DualNumber{T}, y::DualNumber{T}) =
    DualNumber{T}(x.re * y.re, x.re * y.ep + x.ep * y.re)
```

    * (generic function with 140 methods)

As mentioned above, a vector space must support scalar multiplication. But the above definition, in addition to our promotion rules, has us covered. Have a look:

```julia
7(5.0 + 2ɛ)
```

    35.0 + 14.0ɛ

When you think about it, this is remarkably little code to implement an entire algebra on a new kind of number, especially when you consider that we have already handled tricky things like working with different types and conversions. This type business is a really good example of how powerful multiple dispatch is. We may now implement our derivative function:

```julia
der(f) = a -> f(DualNumber(a, one(a))).ep
```

    der (generic function with 1 method)

A result from linear algebra tells us that because of how our dual numbers are defined, when the real part is looked at in isolation, the operations of addition and multiplication return the same results as if we did them on real numbers. So there is no sense inspecting the real component of $f(a + \varepsilon)$ unless we also care about the value at that point. The epsilon component alone determines the derivative.

This may sound magical still, but rest assured that it is all mathematically sound. And it works:

```julia
p′ = der(p)
p′(0)
```

    6

```julia
p′(1)
```

    18

Exact derivatives. Isn't that neat. But before we get too excited, we should note that our dual numbers only work with addition and multiplication right now. We haven't implemented any other operations (though Julia was able to figure out how to do (integer) exponentiation, which is another testament to the power of multiple dispatch). We can't yet take the derivative of, say, $\mathrm{e}^x$ or $\sin(x)$ or $\cos(x)$ or $\log(x)$, for example. Nor have we implemented division. Heck, we haven't even implemented subtraction! None of that is very hard though. Observe:

```julia
import Base: exp, log, sin, cos, /, -

-{T<:Number}(x::DualNumber{T}) = DualNumber{T}(-x.re, -x.ep)
x::DualNumber - y::DualNumber = x + (-y)
exp(x::DualNumber) = DualNumber(exp(x.re), exp(x.re) * x.ep)
sin(x::DualNumber) = DualNumber(sin(x.re), cos(x.re) * x.ep)
cos(x::DualNumber) = DualNumber(cos(x.re), -sin(x.re) * x.ep)
log(x::DualNumber) = DualNumber(log(x.re), x.ep / x.re)
x::DualNumber / y::DualNumber = DualNumber(x.re / y.re, x.ep / y.re - x.re * y.ep / y.re^2)

```

    / (generic function with 49 methods)

Note that each of the functions we implemented is an application of the chain rule, with the exception of `/` which is an expanded form of the quotient rule. Convince yourself that these are mathematically correct. Then we can compute the derivative of any function, as long as it only calls the elementary functions that we've defined. Any other elementary functions needed can usually be quickly defined.

For instance, here is an implementation of exponentiation, which extends the integer exponentiation Julia inferred:

```julia
import Base: ^

x::DualNumber ^ y::DualNumber = exp(y * log(x))
```

    ^ (generic function with 47 methods)

And thus we can calculate the derivative of `x^(x^x)` like a piece of cake. Easier than [Sal Khan](https://www.khanacademy.org/math/differential-calculus/taking-derivatives/derivatives-inverse-functions/v/calculus-derivative-of-x-x-x) could do it! You can check [Wolfram Alpha](https://www.wolframalpha.com/input/?i=derivative+%28x^x^x%29+at+x%3D2) that this does indeed work.

```julia
der(x -> x^x^x)(2)
```

    107.11041244660137

We might expect that we could not find second derivatives in this way. The result of the `der` function, which relies on all the dual number arithmetic we developed, is too complex to even consider the possibility that it could deal with itself. Right?

```julia
der(der(x -> e^x))(5)
```

    148.4131591025766

```julia
e^5
```

    148.4131591025766

```julia
der(der(x -> sin(x)))(1)
```

    -0.8414709848078965

```julia
-sin(1)
```

    -0.8414709848078965

```julia
der(der(der(x -> log(x))))(10)
```

    0.002

```julia
2/10^3
```

    0.002

You really can't make this stuff up. Higher order derivatives, for free, and they're as exact as you want them. How? It turns out that the way we've written our code allows `DualNumber{DualNumber{Float64}}`. That is, we can make `DualNumber`s from `DualNumber`s, and get our second derivatives that way. The `der` function was general enough that it didn't care what type it was using, as long as it was a subtype of `Number`, which `DualNumber` is. Mathematically, this isn't quite kosher, since the `DualNumber`s are not a field, but it still works for derivatives. Some vector space properties may be lost in translation though, but we did not really use those anyways.

The only thing that would make this even better is if we could also compute exact complex derivatives. As you might have guessed, the answer is yes.

```julia
der(x -> x^2)(1 + 2im)
```

    2 + 4im

I hope that this illustrates two important principles: linear algebra is powerful, and multiple dispatch is powerful. When I saw this for the first time, it blew my mind.

## So what is multiple dispatch?

It might not have been clear what this magical fairy I was referring to actually _is_. What is multiple dispatch, and how did it help us achieve our result? I will try to briefly summarize below, but for more information, the [Julia documentation](http://docs.julialang.org/en/release-0.4/manual/methods/) gives a very good overview.

When we talk in mathematics about a concept like $+$, we are talking about an abstract operation that has certain properties. For example, we expect it to be associative and commutative. But how this operation actually works is different depending on what exactly we are adding. Adding two integers is not the same operation as adding two dual numbers, as you probably could guess. But from a more abstract level, we think of both as $+$.

Multiple dispatch allows the computer (and the person who implemented the methods in the first place) to handle the details of what exactly is the operation meant by $+$, while we relax and simply use it. The same function has multiple implementations depending on the types of its arguments. We may inspect this by looking at the native code generated for the addition (for my computer; your results may vary):

```julia
@code_native 8 + 8
```

    	.text
    Filename: int.jl
    Source line: 0
    	pushq	%rbp
    	movq	%rsp, %rbp
    Source line: 32
    	leaq	(%rdi,%rsi), %rax
    	popq	%rbp
    	retq
    	nopw	(%rax,%rax)

```julia
@code_native (1.0 + ɛ) + (2.0 - ɛ)
```

    	.text
    Filename: In[29]
    Source line: 0
    	pushq	%rbp
    	movq	%rsp, %rbp
    Source line: 3
    	movsd	(%rsi), %xmm0           # xmm0 = mem[0],zero
    	movsd	8(%rsi), %xmm1          # xmm1 = mem[0],zero
    	addsd	(%rdx), %xmm0
    	addsd	8(%rdx), %xmm1
    	movsd	%xmm1, 8(%rdi)
    	movsd	%xmm0, (%rdi)
    	movq	%rdi, %rax
    	popq	%rbp
    	retq
    	nopw	%cs:(%rax,%rax)

At the processor level, therefore, these two are very different operations. But the compiler handles those issues for us. We simply use $+$. So in some sense, multiple dispatch allows us to reason about code as we would reason about mathematics. It allows us to abstract away details and perform different operations using the same name. Then we can compose those operations into advanced functionality, and have the compiler generate the right sequence of instructions.

If small, building blocks like $+$ are specialized on types, that allows bigger, higher-level functions to not worry about them. That is how we were able to define polynomials like $p$, thinking that they would be applied to real numbers, and then evaluate them on `DualNumber`s so we could compute their derivative. That is how Julia was able to compute integer exponentiation for our `DualNumber`s even though we told it nothing about it. That is how things "just worked" when we tried to dualize `Complex` and `DualNumber`. That is magic.

Normally such a feature might make things slower. Julia's compiler is built in such a way that multiple dispatch comes at no runtime penalty. That's technologically incredible. Almost magic.

In fact, nothing about what we did above would scream "fast, efficient code" in most languages. It's also not the fastest way to do it in Julia. But even so, Julia 0.5 is able to optimize `der` in some very simple cases. For example, below it was able to constant fold the derivative of $x \mapsto x$ to $1$:

```julia
code_native(der(x -> x), (Int,))
```

    	.text
    Filename: In[34]
    Source line: 0
    	pushq	%rbp
    	movq	%rsp, %rbp
    	movl	$1, %eax
    Source line: 1
    	popq	%rbp
    	retq
    	nopl	(%rax,%rax)

Even though we've made no attempt to optimize our code, `der` is pretty fast. Check this out:

```julia
p(0)
p′(0)

@time p(5)
@time p′(5)
```

      0.000002 seconds (4 allocations: 160 bytes)
      0.000003 seconds (4 allocations: 160 bytes)

    186

Note that these times are both overestimates. My system clock cannot distingush anything that takes shorter than about 0.000005 seconds. More important is the number of bytes allocated, which is pretty reasonable for both.

## Bonus Content

Above I mentioned that `der` is able to take **exact** derivatives. That's a bold claim. I've done nothing to prove that so far, since we've been working with floating point numbers.

It turns out that I did not exaggerate. All we need is a type that can represent things symbolically! Enter... `SymbolicReal`:

```julia
immutable SymbolicReal <: Number
    expr
end

+(r::SymbolicReal, s::SymbolicReal) = SymbolicReal(:($(r.expr) + $(s.expr)))
*(r::SymbolicReal, s::SymbolicReal) = SymbolicReal(:($(r.expr) * $(s.expr)))
-(r::SymbolicReal, s::SymbolicReal) = SymbolicReal(:($(r.expr) - $(s.expr)))
/(r::SymbolicReal, s::SymbolicReal) = SymbolicReal(:($(r.expr) / $(s.expr)))
^(r::SymbolicReal, s::SymbolicReal) = exp(s * log(r))
sin(r::SymbolicReal) = SymbolicReal(:(sin($(r.expr))))
cos(r::SymbolicReal) = SymbolicReal(:(cos($(r.expr))))
exp(r::SymbolicReal) = SymbolicReal(:(exp($(r.expr))))
log(r::SymbolicReal) = SymbolicReal(:(log($(r.expr))))

convert(::Type{SymbolicReal}, r::SymbolicReal) = r
convert(::Type{SymbolicReal}, n::Number) = SymbolicReal(n)
promote_rule{T<:Number}(::Type{SymbolicReal}, ::Type{T}) = SymbolicReal
```

    promote_rule (generic function with 100 methods)

Here we bootstraped off Julia's expressions. Of course, we don't do any simplification or anything above. Expressions just keep growing and growing. But it really does work:

```julia
der(x -> x^3)(SymbolicReal(:(x)))
```

    SymbolicReal(:(x * (x * 1 + 1x) + 1 * (x * x)))

A human would simplify the above to $3x^2$, which is the exact derivative.

```julia
der(x -> x^x)(SymbolicReal(:(x)))
```

    SymbolicReal(:(exp(x * log(x)) * (x * (1 / x) + 1 * log(x))))

A human might simplify the above to $x^x (1 + \log(x))$, which is the exact derivative.

## DualNumbers.jl

Our implementation of dual numbers is rushed, hurried, and incomplete. For a solid implementation currently in use today, see [DualNumbers.jl](https://github.com/JuliaDiff/DualNumbers.jl/).

Automatic Differentiation with Multiple Dispatch.ipynb


# Mouvement Brownien - dynamique de Langevin

Ce notebook accompagne le TD du même nom.

### Modules utilisés

```python
import numpy as np
import tools.langevin as lgv
import matplotlib.pyplot as plt
import numpy.fft as fft
from tqdm import tqdm
```

## Définitions

On suit les notations du TD. Remarquez que l'on définit ici la température et on calcule $\sigma$ alors que l'inverse est fait dans le TD.

```python
gamma=1
temperature=10.0
sigma=np.sqrt(2.0*temperature*gamma)
mass=1.0
```

## Prise en main

Obtention d'une trajectoire de Langevin

$h$ est le pas de temps utilisé.
$n_{steps}$ est le nombre de pas de temps de la simulation.

Le temps total de simulation est alors $T=h*n_{steps}$.

```python
x0,v0=0.0,0.0

h=0.01
nsteps=100000

xtraj=np.zeros(nsteps)
vtraj=np.zeros(nsteps)
ttraj=np.zeros(nsteps)

x,v,step=x0,v0,0

ninit=2000
for step in tqdm(range (ninit)):
    x,v=lgv.langevin_step(x,v,h,gamma,sigma,mass)

for step in tqdm(range(nsteps)):
    x,v=lgv.langevin_step(x,v,h,gamma,sigma,mass)
    xtraj[step]=x
    vtraj[step]=v
    ttraj[step]=step*h
    step=step+1

```

    100%|██████████| 2000/2000 [00:00<00:00, 119953.78it/s]
    100%|██████████| 100000/100000 [00:00<00:00, 156349.16it/s]

Visualisation de la vitesse au cours du temps $v(t)$.

```python
plt.plot(ttraj[:],vtraj[:])
plt.show()
```

Visualisation de la position au cours du temps $x(t)$.

```python
plt.plot(ttraj[:],xtraj[:])
plt.show()
```

### Spectre de puissance et fonction de corrélation par le théorème de Wiener-Khintchine

Dans cette partie on utilise les librairies fft de python pour calculer la fonction d'autocorrélation des vitesses $\langle v(0)v(t) \rangle$ par le théorème de Wiener-Khintchine.

```python
varray=np.zeros(2*nsteps+1)
varray[0:nsteps]=vtraj
vfreq=fft.fft(varray)

spec=np.abs(vfreq)**2/nsteps

domega=2.*np.pi/h

freq=fft.fftfreq(2*nsteps+1)*domega
```

Visualisation du spectre de puissance

```python
plt.plot(freq,np.real(spec))
plt.plot(freq,temperature/mass/gamma/(1+(freq/gamma)**2)*2/h)

plt.xlim(-5,5)
plt.show()
```

Calcul par transformée de Fourier inverse de la fonction d'autocorrélation et visualisation

```python
correlation=fft.ifft(spec)
time=fft.fftfreq(2*nsteps+1)*h*(2*nsteps+1)
```

```python
plt.plot(time,np.real(correlation))
plt.plot(time,temperature/mass*np.exp(-np.abs(time)*gamma))
plt.xlim(-25,25)
plt.show()

```

### Déplacement carré moyen

Calcul et visualisation du déplacement carré moyen $\langle (x(t)-x(0))^2 \rangle$ où la moyenne est une moyenne temporelle.

```python
nmax=1000
msd=np.zeros(nmax)
time=np.arange(nmax)*h

for dh in tqdm(range(nmax)):
    xdiff=xtraj[dh:]-xtraj[:nsteps-dh]
    msd[dh]=np.mean(xdiff**2)
```

    100%|██████████| 1000/1000 [00:00<00:00, 7241.66it/s]

Visualisation en échelle linéaire et comparaison avec la droite asymptotique.
Le coefficient de diffusion est celui ontenu en TD.

```python
plt.plot(time,msd)
plt.plot(time,temperature/gamma*2*time)

plt.show()
```

Visualisation en échelle log-log. Remarquez que le régime diffusif est obtenu seulement à la fin de la simulation.

```python
plt.plot(time,msd)
plt.plot(time,temperature/gamma*2*time)
plt.xscale('log')
plt.yscale('log')
plt.show()
```

## Moyennes d'ensemble

### Réponse à une force constante

$f_0$ est la force constante appliquée au système. Ici $f_0$ est choisi grand pour permettre la visualisation. La vitesse qui en résulte est grande devant la vitesse quadratique moyenne donnée par la temperature. Cela n'a pas d'impact ici car la réponse du système est strictement linéaire. Dans le cas général, on serait dans ces conditions dans un régime non-linéaire où la réponse linéaire ne s'applique pas.

Notez, que l'on passe à la fonction langevin_step une fonction force(x). Cela permet de rajouter une force extérieure qui dépend de la position, par exemple un potentiel harmonique par

    k=1
    def force(x):
        return -k*x

nsamples est le nombre de trajectoires simulées

```python
f0=10
x0,v0=0.0,0.0

def force(x):
    return f0

nsteps=1000
nsamples=1000

h=0.01

xTraj=np.zeros([nsamples,nsteps])
vTraj=np.zeros([nsamples,nsteps])
tTraj=np.zeros([nsamples,nsteps])

for sample in tqdm(range(nsamples)):
    x,v=x0,v0

    for step in range (nsteps):
        xTraj[sample,step]=x
        vTraj[sample,step]=v
        tTraj[sample,step]=step*h
        x,v=lgv.langevin_step(x,v,h,gamma,sigma,mass,force)
```

    100%|██████████| 1000/1000 [00:07<00:00, 135.80it/s]

Moyennes sur toutes les trajectoires

```python
vTraj_aver=np.mean(vTraj,axis=0)
tTraj_aver=tTraj[0,:]
```

Visualisation de quelques trajeectoires et de la vitesse moyenne sur l'ensemble des trajectoires simulées.

```python
for i in range(0,nsamples,100):
    plt.plot(tTraj[i,:],vTraj[i,:],'b')

plt.plot(tTraj_aver,vTraj_aver,'r')

plt.show()
```

Comparaison de la vitesse moyenne observée et de la vitesse moyenne théorique.

```python
plt.plot(tTraj_aver,vTraj_aver,'r')
plt.plot(tTraj_aver,f0/gamma*(1-np.exp(-gamma*tTraj_aver)),'g')

plt.show()
```

### Etat transitoire

On part ici avec une vitesse v0 non nulle. La même remarque que pour l'application de la force est vraie ici aussi: v0 est en principe trop grand pour être dans le régime linéaire mais cela n'a pas d'impact sur ce modèle particulier.

```python
x0,v0=0.0,15

nsteps=1000
nsamples=1000

h=0.01

xTraj=np.zeros([nsamples,nsteps])
vTraj=np.zeros([nsamples,nsteps])
tTraj=np.zeros([nsamples,nsteps])

for sample in tqdm(range(nsamples)):
    x,v=x0,v0

    for step in range (nsteps):
        xTraj[sample,step]=x
        vTraj[sample,step]=v
        tTraj[sample,step]=step*h
        x,v=lgv.langevin_step(x,v,h,gamma,sigma,mass)
```

    100%|██████████| 1000/1000 [00:11<00:00, 88.17it/s]

Moyennes sur toutes les trajectoires

```python
vTraj_aver=np.mean(vTraj,axis=0)
tTraj_aver=tTraj[0,:]
```

Visualisation de quelques trajeectoires et de la vitesse moyenne sur l'ensemble des trajectoires simulées.

```python
for i in range(0,nsamples,100):
    plt.plot(tTraj[i,:],vTraj[i,:],'b')

plt.plot(tTraj_aver,vTraj_aver,'r')

plt.show()
```

Comparaison de la vitesse moyenne observée et de la vitesse moyenne théorique.

```python
plt.plot(tTraj_aver,vTraj_aver,'r')
plt.plot(tTraj_aver,v0*np.exp(-gamma*tTraj_aver),'g')

plt.show()
```

Calcul de la densité de probabilité des vitesses au dernier pas de temps et comparaison avec la distribution théorique.

```python
hist,bin_edges=np.histogram(vTraj[:,-1],bins=50,density=True)

v=(bin_edges[:-1]+bin_edges[1:])/2.0
dv=bin_edges[1]-bin_edges[0]

P=np.exp(-mass*v**2/(2.0*temperature))*np.sqrt(mass/(2.0*temperature*np.pi))
```

```python
plt.plot(v,hist)
plt.plot(v,P)
plt.show()
```

### Processus à l'équilibre

Ici, comme dans la partie prise en main, on commence chaque trajectoire par une partie d'équilibration de ninit pas de temps avant d'accumuler les données.

```python
nsteps=10000
nsamples=1000
ninit=2000

h=0.01

x0,v0=0.0,0.0

xTraj=np.zeros([nsamples,nsteps])
vTraj=np.zeros([nsamples,nsteps])
tTraj=np.zeros([nsamples,nsteps])

for sample in tqdm(range(nsamples)):
    x,v=x0,v0
    for step in range (ninit):
        x,v=lgv.langevin_step(x,v,h,gamma,sigma,mass)

    for step in range (nsteps):
        xTraj[sample,step]=x
        vTraj[sample,step]=v
        tTraj[sample,step]=step*h
        x,v=lgv.langevin_step(x,v,h,gamma,sigma,mass)
```

    100%|██████████| 1000/1000 [02:03<00:00,  8.08it/s]

### spectre de puissance et fonction d'autocorrélation de la vitesse

Même chose que dans la partie prise en main mais on effectue aussi une moyenne du spectre de puissance sur les trajectoires simulées. La comparaison avec les fonctions théoriques est bien meilleure.

```python
vArray=np.zeros([nsamples,2*nsteps+1])
vArray[:,0:nsteps]=vTraj
vFreq=fft.fft(vArray)

Spec=np.abs(vFreq)**2/nsteps

Spec_aver=np.mean(Spec,axis=0)

domega=2.*np.pi/h

Freq=fft.fftfreq(2*nsteps+1)*domega
```

Visualisation du spectre de puissance.

La formule théorique utilisée est:
\begin{equation}
S(\omega)=\frac{kT}{m\gamma}\times\frac{1}{1+\left(\frac{\omega}{\gamma}\right)^2}\times\frac{2}{h}
\end{equation}

Notez que dans toute la page la constante de Boltzmann vaut $k=1$.

```python
plt.plot(Freq,np.real(Spec_aver))
plt.plot(Freq,temperature/mass/gamma/(1+(Freq/gamma)**2)*2/h)

plt.xlim(-10,10)
plt.show()
```

Calcul et visualisation de la fonction d'autocorrélatiion des vitesses moyénnée sur le temps pour chaque trajectoire puis sur l'ensemble des trajectoires.

La fonction d'autocorrélation théorique est

\begin{equation}
    C(t)=\frac{kT}{m} e^{-\gamma t}
\end{equation}

```python
Correlation=fft.ifft(Spec_aver)
Time=fft.fftfreq(2*nsteps+1)*h*(2*nsteps+1)
```

```python
plt.plot(Time,np.real(Correlation))
plt.plot(Time,temperature/mass*np.exp(-np.abs(Time)*gamma))
plt.xlim(-25,25)
plt.show()

```

## A faire

Jouez! Changez h, x0, v0, nsteps, temperature etc.

Reprenez le transitoire avec $x_0=0$ et $v_0=0$ puis déterminer comme dans le TD la variance de $x(t)$ et la variance de $v(t)$

```python

```


```python
import sympy
sympy.init_printing()
## constants
stages = 3
```

```python
s = sympy.Symbol('s')
x = sympy.Symbol('x')
cf1, rf1, cc1, rc1 = sympy.symbols(r"C_f1, R_f1, C_c1, R_c1")
cf2, rf2, cc2, rc2 = sympy.symbols(r"C_f2, R_f2, C_c2, R_c2")
cf3, rf3, cc3, rc3 = sympy.symbols(r"C_f3, R_f3, C_c3, R_c3")
CauerVector = sympy.Matrix([[cc1, rc1],[cc2, rc2],[cc3, rc3]])

## enter FosterVector value manually.
FosterVector = sympy.ones(stages,2);FosterVector[1,1]=2;FosterVector[2,1]=3

ZfVector = sympy.zeros(stages,1)
Zfall = 0
```

```python
CauerVector
```

```python
FosterVector
```

```python
for i in range(3):
    ZfVector[i] = 1/(s*FosterVector[i,0] + 1/FosterVector[i,1])
    Zfall += ZfVector[i]
ZfVector, Zfall
```

```python
Zfall=Zfall.cancel()
```

```python
(pf, qf) = sympy.fraction(Zfall)
pf = sympy.Poly(pf,s)
qf = sympy.Poly(qf,s)
```

```python
cc3 = qf.nth(3)/pf.nth(2)
```

```python
Yfall = (1/Zfall - cc3*s).cancel()
(qf, pf) = sympy.fraction(Yfall)
qf = sympy.Poly(qf, s)
pf = sympy.Poly(pf, s)
```

```python
rc3 = pf.nth(2)/qf.nth(2)
```

```python
Zfall = (1/Yfall - rc3).cancel()
(pf, qf) = sympy.fraction(Zfall)
qf = sympy.Poly(qf, s)
pf = sympy.Poly(pf, s)
```

```python
cc2 = qf.nth(2)/pf.nth(1)
```

```python
Yfall = (1/Zfall - cc2 * s).cancel()
(qf, pf) = sympy.fraction(Yfall)
qf = sympy.Poly(qf, s)
pf = sympy.Poly(pf, s)
```

```python
rc2 = pf.nth(1)/qf.nth(1)
```

```python
Zfall = (1/Yfall  - rc2).cancel()
(pf, qf) = sympy.fraction(Zfall)
qf = sympy.Poly(qf, s)
pf = sympy.Poly(pf, s)
```

```python
cc1 = qf.nth(1)/pf.nth(0)
```

```python
Yfall = (1/Zfall - cc1 * s).cancel()
(qf, pf) = sympy.fraction(Yfall)
qf = sympy.Poly(qf, s)
pf = sympy.Poly(pf, s)
```

```python
rc1 = pf.nth(0)/qf.nth(0)
```

```python
rc3+rc2+rc1
```

```python

```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
from sympy import symbols
import numpy as np
```

```python
f = lambda x, y: x * y
```

```python
def find_gradient_2d(f):
        x, y = symbols('x y')
        f = f(x, y)
        df_dx = f.diff(x)
        df_dy = f.diff(y)
        return np.array([df_dx, df_dy])
```

```python
def eval_gradient_2d(grad, p):
    x = grad[0].subs('x', p[0]).subs('y', p[1])
    y = grad[1].subs('x', p[0]).subs('y', p[1])
    return np.array([x.evalf(), y.evalf()])
```

```python
def find_directional_deriv(grad, direction):
    x = grad[0] * direction[0]
    y = grad[1] * direction[1]
    return np.array([x, y])
```

```python
def eval_directional_deriv(dir_deriv, p):
    x = dir_deriv[0].subs('x', p[0]).subs('y', p[1])
    y = dir_deriv[1].subs('x', p[0]).subs('y', p[1])
    return np.array([x.evalf(), y.evalf()])
```

```python
grad = find_gradient_2d(f)
direction = np.array([1,0])
dir_deriv = find_directional_deriv(grad, direction)
dir_deriv
```

    array([y, 0], dtype=object)

```python
eval_gradient_2d(grad, [1,1])
```

    array([1.00000000000000, 1.00000000000000], dtype=object)

```python
X, Y = np.meshgrid(np.arange(1, 10, 0.25), np.arange(1, 10, 0.25))
G = [eval_gradient_2d(grad, [x, y]) for x in X for y in Y]
print(G)
U = [g[0] for g in G]
V = [g[1] for g in G]
print(U,V)

fig, ax = plt.subplots(figsize=(12,12))
q = ax.quiver(X, Y, U, V)
ax.quiverkey(q, X=0.3, Y=1.1, U=10, label='Quiver key, length = 10', labelpos='E')
```

```python

```

```python

```

Applied Math/Y2S2/.ipynb_checkpoints/Gradients & Directional Derivatives-checkpoint.ipynb

Applied Math/Y2S2/Gradients & Directional Derivatives.ipynb