```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import pearsonr
plt.rcParams['font.size'] = 13
plt.rcParams['axes.spines.right'] = False
plt.rcParams['ytick.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['xtick.top'] = False
```

## Information as reduced uncertainty

The foundation for information theory differs slightly from many other concepts in physics, as it is not derived out of empirical observations. Rather, Shannon (1948) started from an intuition of what properties an information measure should posses, and then showed that there only exist one measure with those properties. In short, he imagined a situation where the probabilities ($p_1, \ldots, p_N$) for $N$ outcomes/answers to an event/question are known beforehand, and soughed to quantify the information obtained ones the outcome/answer was learned.

For example, imagine a professor that wants to know how many students $x$ attended a specific lecture. The professor is assumed to know the distribution $p(x)$ over all possible number of attendants from previous experience, but the real number of attendants is unknown. The distribution $p(x)$ thus reflects current uncertainty, and ones the real number of attendees is learned, this uncertainty is decreased to zero. The basic idea is, therefore, to quantify the information learned by measuring how much the uncertainty has decreased.

```python
# Illustration of the uncertainty before and after
N = 16  # number of possible outcomes
mu = N/2.  # mean
sigma = N/4.  # standard deviation
x = np.arange(N)  # possible outcomes
p = np.exp(-(x-mu)**2/sigma**2)  # p(x)
p /= p.sum()  # Normalize

# One sample from p(x)
p_cum = np.cumsum(p)
outcome = np.argmax(np.random.rand() < p_cum)
y = np.zeros(N)
y[outcome] = 1.

# Plotting
plt.figure(figsize=(15, 3))
ax = plt.subplot(1, 2, 1)
ax.bar(x-0.4, p)
ax.set_xlabel('Number of attendants')
ax.set_ylabel('P(x)')
ax.set_title('Before')
ax = plt.subplot(1, 2, 2)
ax.bar(x, y)
ax.set_xlabel('Number of attendants');
ax.set_title('After');
```

Based on the idea above, Shannon (1948) proposed that a measure $H(p_1,\ldots,p_N)$ of uncertainty should posses the following three properties:
1. $H$ should be continuous in the $p_i$.
2. If all the $p_i$ are equal, $p_i=1/N$, then $H$ should be a monotonically increasing function of $N$.
3. If a choice can be broken down into two successive choices, the original $H$ should be a weighted sum of the individual values of $H$. For example: $H(\frac{1}{2}, \frac{1}{3}, \frac{1}{6}) = H(\frac{1}{2}, \frac{1}{2}) + \frac{1}{2}H(\frac{2}{3}, \frac{1}{3})$.

***
```
   -----|-----               -----|-----
   |    |    |               |         |
  1/2  2/6  1/6             1/2       1/2
                             |      ---|---
                             |      |     |
                             |     2/3   1/3
                             |      |     |
                            1/2    2/6   1/6
```
***
Shannon then moved on to shown that the only uncertainty measure that satisfies the above three properties is of the form:

$$
\begin{equation}
H=-\sum_i p_i \log(p_i),
\end{equation}
$$

where the base of the logarithm determines the information unit (usually base two which corresponds to bits). See Shannon (1948) or Bialek (2012) for the proof.

```python
# Uncertanities before and after
H_before = -np.sum(p*np.log2(p))
H_after = -np.sum(y[y>0]*np.log2(y[y>0]))

# Plotting
plt.figure(figsize=(15, 3))
ax = plt.subplot(1, 2, 1)
ax.bar(x, p)
ax.set_ylabel('P(x)')
ax.set_title('$H_\mathrm{before} = %2.1f$ bits' % H_before)
ax.set_xlabel('Number of attendants')
ax = plt.subplot(1, 2, 2)
ax.bar(x, y)
ax.set_title('$H_\mathrm{after} = %2.1f$ bits' % H_after)
ax.set_xlabel('Number of attendants');
```

## Entropy as a measure of uncertainty

Shannon (1948) chose to denote the uncertainty measure by $H$, and he referred to it as entropy due to its connection with statistical mechanics.
> Quantities of the form $H=-\sum_i p_i \log(p_i)$ play a central role in information theory as measures of **information, choice, and uncertainty**. The form of $H$ will be recognized as that of entropy as defined in certain formulations of statistical mechanics where $p_i$ is the probability of a system being in cell $i$ of its phase space. $H$ is then, for example, the $H$ in Boltzman's famous $H$ theorem. We shall call $H=-\sum_i p_i \log(p_i)$ the entropy of the set of probabilities $p_1,\ldots,p_n$.

Although fascinating, this connection might, however, not be enough to provide an intuitive picture of which factors that lead to high or low entropies. In short, we can answer this second question by noting that 1) the entropy is always non-negative, 2) it increases with the number of possible outcomes, and 3) it obtains its maximum value for any fixed number of outcomes when all are equally likely.

```python
# Entropies for various example distributions
N = 32
mu = N/2.
sigma = N/6.
x = np.arange(N)

# Distributions
p_equal = 1./N*np.ones(N)
p_normal = np.exp(-(x-mu)**2/sigma**2)
p_normal /= p_normal.sum()
p_random = np.random.rand(N)
p_random /= p_random.sum()
ps = [p_equal, p_normal, p_random]
p_max = np.hstack(ps).max()

# Plotting
plt.figure(figsize=(15, 3))
for idx, p in enumerate(ps, start=1):
    H = -np.sum(p*np.log2(p))
    ax = plt.subplot(1, len(ps), idx)
    ax.bar(x, p)
    ax.set_title('$H = %2.1f$ bits' % H)
    ax.set_ylim([0, p_max])
    if idx == 1:
        ax.set_ylabel('P(x)')
    elif idx == 2:
        ax.set_xlabel('Possible outcomes')

```

The entropy of a distribution, as presented above, can also be derived by searching for a minimum length code for denoting each outcome. That is, the entropy also represents a lower limit on how many bits one needs on average to encode each outcome. For example, imagine that $N=4$ and that the probabilities are: $p_1=0.5,\: p_2=0.25,\: p_3=0.125,\: p_4=0.125$. In this case, the minimum length codes would be:

| Outcome | Code |
|---------|:----:|
| 1       | 0    |
| 2       | 10   |
| 3       | 110  |
| 4       | 111  |

and the entropy (or average code length) $-0.5\log(0.5)-0.25\log(0.25)-2*0.125\log(0.125)=1.75$ bits. Bialek (2012) commented on this fact by writing:
>It is quite remarkable that the only way of quantifying how much we learn is to measure how much space is required to write it down.

Similarly, Bialek (2012) also provided the following link between entropy as a minimum length code and the amount of heat needed to heat up a room:
>Entropy is a very old idea. It arises in thermodynamics first as a way of keeping track of heat flows, so that a small amount of heat $dQ$ transferred at absolute temperature $T$ generates a change in entropy $dS=\frac{dQ}{T}$. Although there is no function $Q$ that measures the heat content of a system, there is a function $S$ that characterizes the (macroscopic) state of a system independent of the path to that state. Now we know that the entropy of a probability distribution also measures the amount of space needed to write down a description of the (microscopic) states drawn out of that distribution.

>Let us imagine, then, a thought experiment in which we measure (with some finite resolution) the positions and velocities of all gas molecules in a small room and types these numbers into a file on a computer. There are relatively efficient programs (gzip, or "compress" on a UNIX machine) that compress usch files to nearly their shortest possible length. If these programs really work as well as they can, then the length of the file tells us the entropy of the distribution out of which the numbers in the file are being drawn, but this is the entropy of the gas. Thus, if we heat up the room by 10 degreed and repeat the process, we will find that the resulting data file is longer. More profondly, if me measure the increase in the length of the file, we know the entropy change of the gas and hence the amount of heat that must be added to the room to increase the temperature. This connection between a rather abstract quantity (the length in bits of a computer file) and a very tangible physical quantity (the amount of heat added to a room) has long struck me as one of the more dramatic, if elementary, examples of the power of mathematics to unify descriptions of very disparate phenomena.

[Maxwell–Boltzmann distribution](https://en.wikipedia.org/wiki/Maxwell%E2%80%93Boltzmann_distribution)

## Mutual information

Most situation are not as easy the example with the professor, where the uncertainty was removed in total once the answer was obtained. That is, in practice we often face situation where the uncertainty in only partially decreased. For example, imagine a situation where a bright spot is flashed on one out of 8 equally likely horizontally placed locations {$x \in [0, 1,\ldots, 7]$}, and where our information about which location that was lit up comes from a light detector placed at one of the locations. The detector further has three states {$y \in [0, 1, 2]$}, and it responds with state 2 if the spot is flashed on the location where it is located, state 1 if the spot is flashed at either of the two neighboring locations, and state 3 otherwise. Assuming that the detector is placed at location 3, then its response to a flash at any of the eight locations is as depicted below.

```python
N = 8;  # Eight locations
placement = 3  # The detector's location
responses = np.zeros(N)  # Detector reponses at each location
responses[placement] = 2
responses[placement-1] = 1
responses[placement+1] = 1

# Plotting
plt.figure(figsize=(7.5, 3))
plt.bar(np.arange(N), responses)
plt.xlabel('Spot location')
plt.ylabel('Detector response');
```

If we now expand on the initial idea to define information as the entropy difference between before and after knowing the output of the detector, then we get:

$$
\begin{equation}
I(X;Y) = \sum_{i=0}^7 -p(x_i)\log p(x_i) - \sum_{j=0}^2 p(y_j) \sum_{i=0}^7 -p(x_i|y_j) \log p(x_i|y_j).
\end{equation}
$$

That is, from the initial uncertainty in flash spot location $\sum_{i=0}^7 -p(x_i)\log p(x_i)$, we subtract off the uncertainty that remains for each possible state of the detector $\sum_{i=0}^7 -p(x_i|y_j) \log p(x_i|y_j)$ weighted by its probability of occurrence $p(y_j)$. For the case described above, the relevant probability distributions and entropies are:

```python
# Probability distributions
px = 1./N * np.ones(N)
px_y0 = np.zeros(N) + np.float64((responses == 0)) / (responses == 0).sum()
px_y1 = np.zeros(N) + np.float64((responses == 1)) / (responses == 1).sum()
px_y2 = np.zeros(N) + np.float64((responses == 2)) / (responses == 2).sum()
py = 1./N * np.array([(responses==r).sum() for r in np.unique(responses)])
ps = [px, px_y0, px_y1, px_y2, py]
titles = ['$P(x)$', '$P(x|y=0)$', '$P(x|y=1)$', '$P(x|y=2)$', '$P(y)$']

# Plotting
Hs = []
plt.figure(figsize=(15, 3))
for idx, p in enumerate(ps, start=1):
    H = -np.sum(p[p>0]*np.log2(p[p>0]))
    Hs.append(H)
    ax = plt.subplot(1, len(ps), idx)
    ax.bar(np.arange(len(p)), p)
    ax.set_ylim([0, 1])
    ax.set_title(titles[idx-1] + ', $%2.1f$ bits' % H)
    if idx < len(ps):
        ax.set_xlabel('x')
    else:
        ax.set_xlabel('y')
    if idx > 1:
        ax.set_yticklabels([])
    else:
        ax.set_ylabel('Probability')

# Calculate and write out the mutual information
mi = Hs[0] - py[0]*Hs[1] - py[1]*Hs[2] - py[2]*Hs[3]
print('I=%3.2f - %3.2f*%3.2f - %3.2f*%3.2f - %3.2f*%3.2f=%3.2f' % (Hs[0], py[0], Hs[1], py[1], Hs[2], py[2], Hs[3], mi))
```

By further replacing the summation limits with $x\in X$ and $y\in Y$, respectively, we obtain the more general expression:

$$
\begin{equation}
I(X;Y) = \sum_{x\in X} -p(x)\log p(x) - \sum_{y\in Y} p(y) \sum_{x\in X} -p(x|y) \log p(x|y) = H(X) - H(X|Y),
\end{equation}
$$

where $H(X|Y)$ is the conditional entropy (i.e., the average uncertainty that remains ones $y$ is known) and $I$ the mutual information between $X$ and $Y$. Mutual information is thus a generalization of the initial idea that we can quantify what we learn as the difference in uncertainty before and after.

## Entropy, uncertainty or information

Shannon (1948) actually emphasized a different interpretation than the one presented above. As he was interested in the case where a source sends information over a noisy channel to a receiver, he interpreted the entropy $H(X)$ in $I(X;Y) = H(X) - H(X|Y)$ as the information produced by the source instead of an uncertainty. This interpretation can be understood by noting that the entropy can both be seen as an initial uncertainty or as an upper bound on the information learned when $H(X|Y)$ is zero (a duality that sometimes leads to confusion, especially if mutual information is abbreviated to information only). And in a source and receiver scenario, the upper limit obviously denotes the amount of information sent (produced) by the source. These different interpretations might seem unnecessary at first, but they help in interpreting the symmetry of the mutual information measure. Starting from the expression of mutual information as given above, one can reformulate it as:

$$
\begin{align}
I(X;Y) &= \sum_{x\in X} -p(x)\log p(x) - \sum_{y\in Y} p(y) \sum_{x\in X} -p(x|y) \log p(x|y) = H(X) - H(X|Y), \quad\quad (1) \\
 &=-\sum_{x\in X}\sum_{y\in Y} p(x, y)\log p(x) + \sum_{y\in Y} \sum_{x\in X} p(x,y) \log p(x|y), \\
 &= \sum_{y\in Y} \sum_{x\in X} p(x,y) \log \frac{p(x|y)}{p(x)}, \\
 &= \sum_{y\in Y} \sum_{x\in X} p(x,y) \log \frac{p(x,y)}{p(x)p(y)} = \dots = H(X) + H(Y) - H(X,Y), \\
 &= \quad \vdots \\
I(Y;X) &= \sum_{y\in Y} -p(y)\log p(y) - \sum_{x\in X} p(x) \sum_{y\in Y} -p(y|x) \log p(y|x) = H(Y) - H(Y|X), \quad\quad (2)
\end{align}
$$

Shannon interpreted these two descriptions as: (1) The information that was sent less the uncertainty of what was sent. (2) The amount of information received less the part which is due to noise. Observe that expression (2) two makes little sense for the detector example above if $H(Y)$ is interpreted as uncertainty, whereas it becomes clearer with the interpretation that Shannon's emphasized. From that point of view, expression (2) tells us that the mutual information is the information contained in the detector's response $H(Y)$ less the part that is due to noise $H(Y|X)$. However, as the detector is deterministic (no noise), we arrive at the conclusion that the mutual information should equal $H(Y)$ in our particular example, which it also does.

Additionally, we note that the mutual information has the following properties:
1. It is non-negative and equal to zero only when $x$ and $y$ are statistically independent, that is, when $p(x,y)=p(x)p(y)$.
2. It is bounded from above by either $H(X)$ or $H(Y)$, whichever is smaller.

## Mutual information as a general measure of correlation

As the mutual information is a measure of dependence between two random variables, it can also be understood in more familiar terms of correlations. To visualize this, imagine a joint distribution of two random variables ($X_1$ and $X_2$). Equation 1 and 2 above tells us that the mutual information can be obtained as either $H(X) - H(X|Y)$ or $H(Y) - H(Y|X)$. That is, the entropy of either marginal distribution less the conditional entropy. In more practical terms, this means that we subtract of the average uncertainty that remains ones either variable is known. And in even more practical terms, it corresponds to looking at individual rows or columns in the joint distribution, as these reflect the uncertainty that remains ones either variable is know. This is illustrated below where two 2D multivariate Gaussian distributions are plotted together with the mutual information between the two variables.

```python
# Generating one independent and one correlated gaussian distribution
N = 16
mu = (N-1) / 2.*np.ones([2, 1])
var = 9.
cov = 8.
cov_ind = np.array([[var, 0.], [0., var]])
cov_cor = np.array([[var, cov], [cov, var]])
[x1, x2,] = np.meshgrid(range(N), range(N))
p_ind = np.zeros([N, N])
p_cor = np.zeros([N, N])
for i in range(N**2):
    x_tmp = np.array([x1.ravel()[i]-mu[0], x2.ravel()[i]-mu[1]])
    p_ind.ravel()[i] = np.exp(-1/2 * np.dot(x_tmp.T, np.dot(np.linalg.inv(cov_ind), x_tmp)))
    p_cor.ravel()[i] = np.exp(-1/2 * np.dot(x_tmp.T, np.dot(np.linalg.inv(cov_cor), x_tmp)))
p_ind /= p_ind.sum()
p_cor /= p_cor.sum()

# Calculate I(X1;X2)
p1_ind = p_ind.sum(axis=1)
p2_ind = p_ind.sum(axis=0)
mi_ind = -np.sum(p1_ind*np.log2(p1_ind)) - np.sum(p2_ind*np.log2(p2_ind)) + np.sum(p_ind*np.log2(p_ind))
p1_cor = p_cor.sum(axis=1)
p2_cor = p_cor.sum(axis=0)
mi_cor = -np.sum(p1_cor*np.log2(p1_cor)) - np.sum(p2_cor*np.log2(p2_cor)) + np.sum(p_cor[p_cor>0]*np.log2(p_cor[p_cor>0]))

# Plotting
titles = ['Independent', 'Correlated']
p = [p_ind, p_cor]
mi = [mi_ind, mi_cor]
x_ticks = [0, 5, 10, 15]
fig = plt.figure(figsize=(15, 7.5))
for idx, p_tmp in enumerate(p):
    ax = fig.add_axes([0.1 + idx*0.5, 0.1, 0.25, 0.5])
    ax.imshow(p_tmp.reshape(N, N))
    ax.set_xticks(x_ticks)
    ax.set_xticklabels([])
    ax.set_xlabel('$x_1$')
    ax.set_yticks(x_ticks)
    ax.set_yticklabels([])
    ax.set_ylabel('$x_2$')
    ax.invert_yaxis()
    plt.draw()
    pos = ax.get_position()
    ax = fig.add_axes([pos.x0, 0.65, pos.x1-pos.x0, 0.1])
    ax.plot(range(N), p_tmp.sum(axis=1), 'o-')
    ax.set_xticks(x_ticks)
    ax.get_yaxis().set_visible(False)
    ax.spines['left'].set_visible(False)
    ax.set_title(titles[idx] + ', $I(X_1;X_2) = %3.2f$ bits' % mi[idx])
    ax = fig.add_axes([pos.x1 + 0.03, 0.1, 0.1/2, 0.5])
    ax.plot(p_tmp.sum(axis=0), range(N), 'o-')
    ax.set_yticks(x_ticks)
    ax.get_xaxis().set_visible(False)
    ax.spines['bottom'].set_visible(False)

print('H(X1): %3.2f bits' % -np.sum(p1_cor*np.log2(p1_cor)))
print('H(X2): %3.2f bits' % -np.sum(p2_cor*np.log2(p2_cor)))
print('H(X1,X2)_ind: %3.2f bits' % -np.sum(p_ind*np.log2(p_ind)))
print('H(X1,X2)_cor: %3.2f bits' % -np.sum(p_cor[p_cor>0]*np.log2(p_cor[p_cor>0])))
```

Another way of understanding why mutual information measures correlation is to look at the expression $I(X;Y) = H(X) + H(Y) - H(X,Y)$, from which we observe that the joint entropy $H(X,Y)$ is subtracted from the sum of the individual entropies. As entropy increases with uncertainty (or possible outcomes), we can infer that a less spread out joint distribution will cause a smaller subtraction. Importantly, however, the shape of the joint distribution does not matter, only how concentrated the probability mass is to a small number of outcomes. This is an important distinction that makes mutual information a general measure of correlation, in contrast to the commonly used correlation coefficients (Pearson's r), which only captures linear correlations. The example below highlight this by calculating the mutual information and the correlation coefficient for both a linear and quadratic relationship between $x$ and $y$.

```python
# Generate y responses as y = f(x) for 16 x values with f(x) being either f(x)=x or f(x) = -x^2
x = np.arange(-3.75, 4, 0.5)
y = [x, -x**2]

# Entropies, mutual information, correlation coefficients
Hx = [np.log2(x.size), np.log2(x.size)]  # Assume each x-value is equally likely
Hy = [np.log2(np.unique(y_tmp).size) for y_tmp in y]
mi = Hy  # H(Y|X) = 0 as there is no noise, thus I = H(Y)
r = [pearsonr(x, y_tmp)[0] for y_tmp in y]

# Plotting
fig = plt.figure(figsize=(15, 3))
for i in range(len(y)):
    ax = plt.subplot(1, len(y), i+1)
    ax.plot(x, y[i], 'o')
    ax.set_xlabel('$x$')
    ax.set_ylabel('$y$')
    info = '$r: %2.1f$\n$H(X): %2.1f$ bits\n$H(Y): %2.1f$ bits\n$I(X;Y): %2.1f$ bits' % (r[i], Hx[i], Hy[i], mi[i])
    ax.text(x[2]-i*x[2], y[i].max()-i*5, info, va='top', ha='center')
```

The mutual information retains its maximum value in both cases (remember that it is bounded from above by min[H(x), H(y)]), whereas the correlation coefficient indicates maximal correlation for the linear $f$ and no correlation for the quadratic $f$. Additionally, the quadratic example provides a nice description of how the mutual information can be interpreted: If we learn 3 bits of information by observing $y$, then our uncertainty about $x$ is one bit $H(X) - I(X;Y)$. This, in turn, corresponds to a choice between two equally likely alternatives, a condition that simply reflects that there are two different $x$-values mapping onto the same $y$-value.

InformationTheory/Part 1, entropy, uncertainty and information.ipynb


# Particle in one-dimensional potential well

## Period of oscillations  in potential well

Dynamics of a particle of mass $m$ moving in one dimension $OX$ is described the Newton equation

$$m\ddot x =m\dot v = F(x) = -U'(x),$$

where $F(x)$ is a force acting on tha particle and $U(x)$ is potential energy of the particle. We assume that there are no dissipative forces. Therefore, as we know from the previous section, the total energy $E$ of the particle is conserved, i.e.,

\begin{equation}
\label{eq:cons_E1d}
\frac{mv^2}{2} + U(x) = E = const.
\end{equation}

If we treat the velocity $v=\frac{dx}{dt}$ as an  independent variable, it is implicit equation of the orbit (the phase curve) in **phase space**  $(x,v)$ of the  particle with  energy $E$. This equation can be rewritten in the form

$$\frac{m}{2}\left(\frac{dx}{dt}\right)^2+U(x)=E$$

It is a first-order differential equation and can be solved by the method of variable separation. The result reads

\begin{equation}
\label{eq:t_cons_E1d}
 t=\sqrt{\frac{m}{2}} \; \int_{a}^{b}{\frac{dx}{\sqrt{E-U(x)}}}
\end{equation}

Here, $a$ is the initial position $x(0)=a$ and $b$ is the final position $x(t)=b$ of the particle. The time $t$ is time for moving the particle from the point $a$ to the point $b$ under the condition that $E \ge U(x)$ for all $x \in (a, b)$.

This is a useful formula which allows to calculate period of oscillations of the particle in a potential well.

In this section we will consider a class of potentials in the form

\begin{equation}
\label{eq:Uxn}
U(x)=A |x|^n,
\end{equation}

where $n$ is a positive real number.

These potentials are similar: they are bounded from below,   have only one minimum at $x=0$ and tends to infinity when $x\to \pm \infty$. In such potential the particle motion is bounded and the particle oscillates between two symmetrical positions $x_0$ and $-x_0$ which in general depend on the total energy $E$ and are determined by the equation

$$U(\pm x_0) = E$$

Because of symmetry, the period $T$ of oscillations can be determined by the equation

\begin{equation}
\label{eq:T}
 T=4 \; \sqrt{\frac{m}{2}} \; \int_{0}^{x_0}{\frac{dx}{\sqrt{E-U(x)}}}
\end{equation}

This class of potentials is simple but nevertheless analysis of $T$ in dependence of the index $n$ and the total energy $E$ is very interesting and instructive.

We will use computer algebra and numerical methods to investigate properties of motion is such potential wells.

```python
load('cas_utils.sage')
```

```python
t = var('t')
m = var('m')
A = var('A')
assume(A > 0)
assume(m > 0)
y = function('y')(t)
de = m*diff(y,t,2) + 2*A*y == 0
showmath( desolve(de, y,ivar=t) )
```

It is an analytical solution of the Newton equation in the case when $n=2$ (a harmonic potential).

## Particle in potential $x^2$

For $n=2$ the system is  a harmonic oscillator:

$$U(x)=A x^2.$$

```python
#reset()
var('m A x E')
forget()
assume(A > 0)
assume(E > 0)
assume(E,'real')
```

To obtain the integration limit $x_0$  in the formula for the period of oscillations, we must solve the equation:

$$U(x)=E$$

So for the $Ax^2$ potential, we have:

```python
U(A,x) = A*x^2
xextr = solve (U(A=A,x=x)==E,x)
showmath(xextr)
```

These formulas describe the values of the oscillator's extremal positions for a given energy. Let's put them into the formula for $T$:
:

```python
period = 2*sqrt(m/2)*integrate( 1/sqrt(E-U(A,x)),(x,x.subs(xextr[0]),x.subs(xextr[1])))
period = period.canonicalize_radical()
showmath(period)
```

We see that the period $T$ does not depend on energy of the oscillator. It means that it does not depend on the initial conditions because the total energy of the particle depends on them. In turn, it means that it does not depend on the distance  between the points $-x_0$ and $x_0$. It seems to be unusual behavior: time to travel  from $-1$ to $1$ and back is the same as time to travel from $-10000$ to $10000$ and back. In the second case the distance is much,  much longer but time is the same. This is an exceptional property valid only for the harmonic potential!

## Particle in $|x|^n$ potential

If $n\neq2$, the general formula for the period can be written as:

$$T=4 \sqrt{\frac{m}{2}} \; \int_0^{x_0}\frac{dx}{\sqrt{E- A x^n}}$$

or in the equivalent form:

$$T=4 \sqrt{\frac{m}{2}}\frac{1}{\sqrt{E}}\int_0^{x_0}\frac{dx}{\sqrt{1-Ax^n/E}}$$

This  integral can be transformed  to a dimensionless form  by substitution

$$\frac{A}{E}x^n=y^n.$$

It  is in fact a  linear relationship between $x$ and $y$:

$$\left(\frac{A}{E}\right)^{\frac{1}{n}}x=y.$$

Therefore, we can change the integration variable to $y$.  To do this,  we use SAGE to transform the expression under integral  in the following way:

```python
var('dx dy A E x y')
var('n',domain='integer')
assume(n>=0)
assume(A>0)
assume(E>0)
ex1 = dx/sqrt(1-A/E*x^n)
showmath(ex1)
```

and we substitute:

```python
ex2 = ex1.subs({x:(E/A)^(1/n)*y,dx:dy*(E/A)^(1/n)})
showmath( ex2.canonicalize_radical().full_simplify() )
```

Let's take out the expression that depends on the parameters $A$ and $E$:

```python
expr2 = (ex2/dy*sqrt(-y^n + 1)).full_simplify()
showmath( expr2.canonicalize_radical() )
```

```python
prefactor = expr2*sqrt(m/2)*4*1/sqrt(E)
showmath( prefactor.canonicalize_radical() )
```

Finally, we obtain:

$$T=4 \sqrt{\frac{m}{2}}\frac{1}{A^{1/n}} E^{\frac{1}{n}-\frac{1}{2}}\int_0^{y_0}\frac{dy}{\sqrt{1- y^n}}$$

For $n=2$, dependence on $E$ disappears, as we already have seen in the previous case.

We still need to calculate the upper limit  $y_0$ of integration.  In the integral, the upper limit is the position in which the total energy is the potential energy:

$$U(x)=E$$

In this case $$Ax^n=E.$$

By changing the variables we get:

```python
solve( (A*x^n == E).subs({x:(E/A)^(1/n)*y}), y)
```

That is, the integration limit is $y_0=1.$

Therefore the period of oscillations is given by the relation:

$$T=4 \sqrt{\frac{m}{2}}\frac{1}{A^{1/n}} E^{\frac{1}{n}-\frac{1}{2}}\int_0^{1}\frac{dy}{\sqrt{1- y^n}}$$

We note that only for the case $n=2$, the period $T$ does not depend on E (i.e. on initial conditions, i.e. on the distance). In other cases it depends on the total energy $E$, i.e. it depends on initial conditions, i.e. it depends on the distance  between the points $-x_0$ and $x_0$.

The above equation shows how much time the particle needs to travel the distance for one oscillation in dependence on $E$ and in consequence on the distance:   If energy $E$ is higher then the distance $4x_0$ is longer.

The scaled integral can be expressed by the beta function of Euler http://en.wikipedia.org/wiki/Beta_function.
We can calculate it:

```python
var('a')
assume(a,'integer')
assume(a>0)
print( assumptions() )
```

```python
integrate(1/sqrt(1-x^(a)),(x,0,1) )
```

We get a formula containing the beta function. It can be evaluated numerically for any values of the $a$ parameter.

```python
(beta(1/2,1/a)/a).subs({a:2}).n()
```

Let's examine this formula numerically. You can use the $\beta$ function, or numerically estimate the integral. This second approach allows you to explore any potential, not just $U(x)=ax^n$.

```python
def beta2(a,b):
    return gamma(a)*gamma(b)/gamma(a+b)

a_list = srange(0.02,5,0.1)
a_list2 = [1/4,1/3,1/2,1,2,3,4,5]

integr_list = [ integral_numerical( 1/sqrt(1-x^a_) ,0,1, algorithm='qng',rule=2)[0] \
               for a_ in a_list ]
integr_list_analytical = [ beta2(1/2, 1/a_)/a_ for a_ in a_list2 ]
```

we obtain some analytically simple formulas:

```python
showmath( integr_list_analytical )
```

Not  we can compare those analytical numbers with numerical results, for example on the plot:

```python
plt_num = list_plot(zip( a_list,integr_list), plotjoined=True )
plt_anal = list_plot(zip( a_list2,integr_list_analytical),color='red')
(plt_num + plt_anal).show(ymin=0,figsize=(6,2))
```

Having an analytical solution, you can examine the asymptotics for large $n$:

```python
var('x')
asympt = limit( beta2(1/2, 1/x)/x,x=oo )
asympt
```

```python
plt_asympt = plot(asympt,(x,0,5),linestyle='dashed',color='gray')
```

Let's add a few points for which the integral takes exact values

```python
l = zip(a_list2[:5],integr_list_analytical[:5])
showmath(l)
```

```python
def plot_point_labels(l):
    p=[]
    for x,y in l:
        p.append(  text( "$("+latex(x)+", "+latex(y)+")$" ,(x+0.1,y+0.2) , fontsize=14,horizontal_alignment='left',color='gray') )
        p.append( point ( (x,y),size=75,color='red' ) )
    return sum(p)
```

```python
some_points = plot_point_labels(l)
```

```python
plt_all = plt_num+plt_anal+plt_asympt+some_points
plt_all.show(figsize=(6,3),ymin=0,ymax=7)
```

## Numerical convergence

The integral
$$\int_0^1  \frac{dx}{\sqrt{1-x^n}}$$ seems to be divergent for $n:$

```python
showmath( numerical_integral( 1/sqrt(1-x^(0.25)) , 0, 1)  )
```

However, choosing the right algorithm gives the correct result:

```python
a_ = 1/4. # exponent in integral
integral_numerical( 1/sqrt(1-abs(x)^a_) , 0, 1, algorithm='qags')
```

lets check it out with an exact formula:

```python
(beta(1/2,1/a)/a).subs({a:a_}).n()
```

Indeed, we  see that carefull numerical integration gives finite result.

## The dependence of  period on  energy for different $n$.

```python
var('E x n')
def draw_E(n,figsize=(6,2.5)):
    p = []
    p2 = []
    p.append( plot(abs(x)^n,(x,-2,2),\
                   ymin=0,ymax=4,legend_label=r"$U(x)=|x|^{%s}$" % n )   )
    p.append( plot( (x)^2,(x,-2,2),\
                   color='gray',legend_label=r"$U(x)=x^{2}$",\
                   axes_labels=[r"$x$",r"$U(x)$"]  ))

    p2.append(  plot(  4/sqrt(2)*(beta(1/2, 1/n)/n)* E^(1/n-1/2),\
                     (E,0.00,3),ymin=0,ymax=7,axes_labels=[r"$E$",r"$T$"]  ) )
    p2.append(  plot(  4/sqrt(2)*(beta(1/2, 1/2)/2),\
                     (E,0.00,3) ,color='gray' ) )

    show( sum(p), figsize=figsize )
    show( sum(p2), figsize=figsize )

```

```python
@interact
def _(n=slider([1/4,1/2,2/3,3/4,1,3/2,2,3])):
    draw_E(n)
```

```python
import os
if 'PDF' in os.environ.keys():
    draw_E(1/2)
    draw_E(1)
    draw_E(4)
```

We can plot the dependence of period $T$ on energy (i.e. amplitude) $T(E)$ for different values of $n$. In figure belowe we see that if $n>2$ then oscillations are faster as energy grows. On the other hand if $n<1$, oscillations are getting slower with growing energy.

Another interesting observation is that for potentials with $n>>1$ and $n<<1$ oscillations will become arbitrarily slow and fast, respectively when $E\to0$. For $n>1$ the potential well is *shallow* at the bottom and *steep* far away from the minimum and for $n<1$ the opposite is true.

```python
n_s  = [1/3,2/3, 1, 2, 3]
plot( [4/sqrt(2)*(beta(1/2, 1/n_)/n_)* E^(1/n_-1/2) \
                 for n_ in  n_s],\
    (E,0.00,2),axes_labels=[r"$E$",r"$T$"],\
     legend_label=['$n='+str(latex(n_))+'$' for n_ in n_s],\
     ymax=7, figsize=(6,3) )

```

## Numerical integration of equations of motion

Here we will investigate numerically period  $T$  and compare with the analytical formula.
First, let's have a closer look how the potential and force behave for different index $n$:

```python
def plot_Uf(n_):
    U(x) = abs(x)^(n_)
    plt = plot( [U(x),-diff( U(x),x)],(x,-1,1),\
               detect_poles='show',ymin=-3,ymax=3,
              legend_label=[r"$U(x)$",'$f(x)$'])
    plt.axes_labels([r"$x$",r"$U(x)=|x|^%s$"%latex(n_)])
    show(plt,figsize=(6,3))
```

```python
@interact
def _(n_=slider(0.1,2,0.1)):
    plot_Uf(n_)
```

```python
if 'PDF' in os.environ.keys():
    plot_Uf(1)
    plot_Uf(2)
    plot_Uf(1/2)
```

We can see that for $n \ge 1$ the force and potential are continuous. If $n=1$ then force has finite jump (discontinuity). Both those cases should not be a problem for numerical integration.

However for $n<1$ we have infinite force at $x=0$.

#### Problems with numerical integration

There is a possibility that  if the ODE integrator comes too close, it will blow out!

We can fix this problem by softening the  potential singularity by adding small number:
$$ |x| \to |x| + \epsilon. $$

```python
var('x',domain='real')
var('v t')
eps = 1e-6
U(x) = (abs(x)+eps)^(1/2)
showmath( U.diff(x).expand().simplify() )
```

to make sure that Sage will not leave $x/|x|$ unsimplified we can do:

```python
w0 = SR.wild(0)
w1 = SR.wild(1)
f = -U.diff(x).subs({w0*w1/abs(w1):w0*sign(w1)})
```

```python
showmath( f(x) )
```

```python
ode_pot = [v,f(x)]

t_lst = srange(0,10,0.01)
sol = desolve_odeint(ode_pot,[1,.0],t_lst,[x,v])
```

```python
p = line(zip(t_lst, sol[:,0])) + line(zip(t_lst, sol[:,1]), color='red')
p.axes_labels(['$t$','$x(t),v(t)$'])
p + plot(1,(x,0,10),linestyle='dashed',color='gray')
```

We can evaluate the period $T$ from the trajectory obtained via numerical solutions. For this purpose one might need an interpolation of numerical table returned by `desolve_odeint`:

```python
import numpy as np
def find_period(x,t):
    zero_list=[]
    x0 = x[0]
    for i in range(1,len(x)):
        if x[i]*x[i-1] < 0:
            zero_list.append( - (t[i-1]*x[i] - t[i]*x[i-1])/(x[i-1] - x[i]) )
    lnp = np.array(zero_list)
    return 2*(  (lnp-np.roll(lnp,1))[1:] ).mean()
```

```python
var('x1 x2 t1 t2 a b ')
showmath(  (-b/a).subs(  solve([a*t1+b==x1,a*t2+b==x2],[a,b], solution_dict=True)[0] ) )
```

We find numerically a period of trajectory:

```python
T = find_period( sol[:,0],t_lst)
T
```

```python
assert abs(T-7.54250)<1e-4
```

Exact results  for comparison:

```python
# for n=2 2*pi/sqrt(2)==(2*pi/sqrt(2)).n()
table( [["n","T"]]+[ [n_,((4/sqrt(2)*(beta(1/2, 1/n_)/n_)* E^(1/n_-1/2)).subs({E:1})).n()]
                    for n_ in [1/4,1/3,1/2,2/3,1,2,3,4,5] ] )

```

## Using the formula for the period to reproduce the trajectory of movement

We take $m=1$ and $A=1$ (then $x=E$), then we can reproduce the trajectory reversing the formula for $T(E)$.

```python
var('x')
U(A,x) = A*x^2
A = 1/2
E = 1
m = 1.
x1=0.1
showmath( solve(E-U(A,x), x) )
```

```python
t_lst  = [ (sqrt(m/2.)*integrate( 1/sqrt(E-U(A,x)),(x,0,x1)).n(),x1)  \
          for x1 in srange(0,sqrt(2.)+1e-10,1e-2)]
```

```python
point(t_lst ,color='red')+\
 plot(sqrt(2)*sin(x),(x,0,pi),figsize=(6,2))
```

Interestingly, if we known the dependence of $T(E)$ then we can calculate exactly the potential!

\newpage


# Examples of  how to use SymPy

SymPy is a Python package for symbolic manipulation. In this notebook we look at a few examples of useful operations, such as differentiation, integration, simplification and solving polynomials.

SymPy suppports many more operations. The documentation can be found at: https://docs.sympy.org/latest/index.html

```python
from sympy import *
```

You have to tell SymPy which variables are to be treated as symbols. In this notebook we'll make `x`, `y` and `z` symbols.

```python
x, y, z = symbols('x y z')
```

## Defining equations and displaying them

We can now write expressions, and assign them to a variable.

```python
a = (x + x**2)/(x*sin(y)**2 + x*cos(y)**2)
a
```

$\displaystyle \frac{x^{2} + x}{x \sin^{2}{\left(y \right)} + x \cos^{2}{\left(y \right)}}$

```python
b = x**2 + 1
b
```

$\displaystyle x^{2} + 1$

Simple simplifications are automatically applied

```python
b - 1
```

$\displaystyle x^{2}$

If you define multiple expressions in a cell you can display them with the `display()` command

```python
a = x**2 + x
display(a)

b = 2*x + 3
display(b)

display(a+b)
```

$\displaystyle x^{2} + x$

$\displaystyle 2 x + 3$

$\displaystyle x^{2} + 3 x + 3$

## Simplification

In simple cases SymPy will automatically simplify. More complicated expressions can simplified using the `simplify()` command

```python
a = (x-2)*(x+3)/((x-2)*(x-9))
display(a)

a = (x**2 +x -6)/(x**2 - 11*x+18)
display(a)
simplify(a)
```

$\displaystyle \frac{x + 3}{x - 9}$

$\displaystyle \frac{x^{2} + x - 6}{x^{2} - 11 x + 18}$

$\displaystyle \frac{x + 3}{x - 9}$

`simplify()` will also use trigonometric identities

```python
a = (x + x**2)/(x*sin(y)**2 + x*cos(y)**2)
display(a)
display(simplify(a))
```

$\displaystyle \frac{x^{2} + x}{x \sin^{2}{\left(y \right)} + x \cos^{2}{\left(y \right)}}$

$\displaystyle x + 1$

## Differentiation
Expressions can be differentiated using the `diff()` command. The first argument is the expression and the second argument is the variable you want to differential with respect too.

```python
a = x**2 + 3*x + 4*y**2
display(diff(a,x))
display(diff(a,y))
```

$\displaystyle 2 x + 3$

$\displaystyle 8 y$

You can also use trigonometric functions, such as `sin()`, `cos()`, `sinh()`, and their inverses `asin()`, `acos()`, `asinh()` etc

```python
diff(asinh(x),x)
```

$\displaystyle \frac{1}{\sqrt{x^{2} + 1}}$

It's easy to get complicated expressions. It is often useful to simplify the result.

```python
res = diff((x**2 +x -6)/(x**2 - 11*x+18))
display(res)
display(simplify(res))
```

$\displaystyle \frac{\left(11 - 2 x\right) \left(x^{2} + x - 6\right)}{\left(x^{2} - 11 x + 18\right)^{2}} + \frac{2 x + 1}{x^{2} - 11 x + 18}$

$\displaystyle - \frac{12}{x^{2} - 18 x + 81}$

## Integration

You can also integrate using the `integrate()` command.

```python
a = x/(x**2+2*x+1)
display(a)

integrate(a, x)
```

$\displaystyle \frac{x}{x^{2} + 2 x + 1}$

$\displaystyle \log{\left(x + 1 \right)} + \frac{1}{x + 1}$

You can also do definite integration:

```python
b = integrate(sin(x), (x,0,3))
display(b)
```

$\displaystyle 1 - \cos{\left(3 \right)}$

SymPy will give the exact result. You can numerically evaluate it using `N()`. You can pass a second argument for the precision you wish to evaluate the result to.

```python
display(N(b))
display(N(b, 50))
```

$\displaystyle 1.98999249660045$

$\displaystyle 1.9899924966004454572715727947312613023936790966156$

If SymPy is not sure how to perform the integral it will just return it an unevaluated expression:

```python
integrate(cos(x)/(x**2+1),(x,1,5))
```

$\displaystyle \int\limits_{1}^{5} \frac{\cos{\left(x \right)}}{x^{2} + 1}\, dx$

## Solving equations

SymPy can also solve some equations, such as polynomials.

```python
solveset((x-2)*(x+3), x)
```

$\displaystyle \left\{-3, 2\right\}$

```python
solveset(x**2 +x -1, x)
```

$\displaystyle \left\{- \frac{1}{2} + \frac{\sqrt{5}}{2}, - \frac{\sqrt{5}}{2} - \frac{1}{2}\right\}$

Here a polynomial we encountered whilst looking for period-2 orbits related to root finding:

$$ 16 x_p^9 - x_p(3 x_p^2 - 1)\left[12x_p^6 - (3x_p^2 -1)^2\right] = 0$$

Let's find the roots

```python
solveset(16*x**9 - x*(3*x**2 - 1)*(12*x**6 - (3*x**2-1)**2),x)
```

$\displaystyle \left\{-1, 0, 1, - \frac{\sqrt{5}}{5}, \frac{\sqrt{5}}{5}, - \frac{\sqrt{2} \cos{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2} - \frac{\sqrt{2} i \sin{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2}, - \frac{\sqrt{2} \cos{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2} + \frac{\sqrt{2} i \sin{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2}, \frac{\sqrt{2} \cos{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2} - \frac{\sqrt{2} i \sin{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2}, \frac{\sqrt{2} \cos{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2} + \frac{\sqrt{2} i \sin{\left(\frac{\operatorname{atan}{\left(\frac{\sqrt{7}}{3} \right)}}{2} \right)}}{2}\right\}$

## Derive the weights for Simpson's method

Recall that Lagrange Polynomial through points $f(a), f(b), f(c)$ is given by

$$P_2(x) = f(a)\frac{(x-b)(x-c)}{(a-b)(a-c)} + f(b)\frac{(x-a)(x-c)}{(b-a)(b-c)} + f(c)\frac{(x-a)(x-b)}{(c-a)(c-b)}$$

We integrate this to get Simpson's method:

$$ \int^b_a P_2(x) \, dx = \frac{b-a}{6}\left[f(a) + 4 f\left(\frac{a+b}{2}\right) + f(b)\right]$$

Let's do this integration with SymPy

```python
a, b, c = symbols('a b c')
f = Function('f')
```

```python
c = (a+b)/2

P2 = f(a)*(x-b)*(x-c)/((a-b)*(a-c)) + f(b)*(x-a)*(x-c)/((b-a)*(b-c)) + f(c)*(x-a)*(x-b)/((c-a)*(c-b))
```

```python
integral = integrate(P2,(x,a,b))
display(integral)
```

$\displaystyle - \frac{a^{3} \left(2 f{\left(a \right)} + 2 f{\left(b \right)} - 4 f{\left(\frac{a}{2} + \frac{b}{2} \right)}\right)}{3 a^{2} - 6 a b + 3 b^{2}} + \frac{a^{2} \left(a f{\left(a \right)} + 3 a f{\left(b \right)} - 4 a f{\left(\frac{a}{2} + \frac{b}{2} \right)} + 3 b f{\left(a \right)} + b f{\left(b \right)} - 4 b f{\left(\frac{a}{2} + \frac{b}{2} \right)}\right)}{2 a^{2} - 4 a b + 2 b^{2}} - \frac{a \left(a^{2} f{\left(b \right)} + a b f{\left(a \right)} + a b f{\left(b \right)} - 4 a b f{\left(\frac{a}{2} + \frac{b}{2} \right)} + b^{2} f{\left(a \right)}\right)}{a^{2} - 2 a b + b^{2}} + \frac{b^{3} \left(2 f{\left(a \right)} + 2 f{\left(b \right)} - 4 f{\left(\frac{a}{2} + \frac{b}{2} \right)}\right)}{3 a^{2} - 6 a b + 3 b^{2}} - \frac{b^{2} \left(a f{\left(a \right)} + 3 a f{\left(b \right)} - 4 a f{\left(\frac{a}{2} + \frac{b}{2} \right)} + 3 b f{\left(a \right)} + b f{\left(b \right)} - 4 b f{\left(\frac{a}{2} + \frac{b}{2} \right)}\right)}{2 a^{2} - 4 a b + 2 b^{2}} + \frac{b \left(a^{2} f{\left(b \right)} + a b f{\left(a \right)} + a b f{\left(b \right)} - 4 a b f{\left(\frac{a}{2} + \frac{b}{2} \right)} + b^{2} f{\left(a \right)}\right)}{a^{2} - 2 a b + b^{2}}$

```python
res = simplify(integral)
display(res)
```

$\displaystyle - \frac{a f{\left(a \right)}}{6} - \frac{a f{\left(b \right)}}{6} - \frac{2 a f{\left(\frac{a}{2} + \frac{b}{2} \right)}}{3} + \frac{b f{\left(a \right)}}{6} + \frac{b f{\left(b \right)}}{6} + \frac{2 b f{\left(\frac{a}{2} + \frac{b}{2} \right)}}{3}$

```python
factor(res)
```

$\displaystyle - \frac{\left(a - b\right) \left(f{\left(a \right)} + f{\left(b \right)} + 4 f{\left(\frac{a}{2} + \frac{b}{2} \right)}\right)}{6}$

```python

```


# Exercise 1

```python
import matplotlib.pyplot as plt
from ipywidgets import interactive

import pandas as pd
import numpy as np
import sympy as sp
from scipy.stats import norm

from pylab import rcParams
rcParams['figure.figsize'] = 12, 8
```

## 1.1 Distribution of a random variable X
Make sure you understand what the probability and density functions are and how they are related to the cumulative distribution function.
Let X denotes a continuous random variable, $X\in[0, +\infty]$, with a probability density function (pdf) $f(x)$ and cumulative distribution function (cdf) $F(x)$.
* Please write down the relationship between $F(x)$ and $f(x)$.

```python
x,t = sp.symbols('x t')
f = sp.Function('f')(x)  # PDF
F = sp.Function('F')(x)  # CDF
```

```python
equation_CDF = sp.Eq(F,sp.Integral(f,(t, -sp.oo, x)))
equation_CDF
```

$\displaystyle F{\left(x \right)} = \int\limits_{-\infty}^{x} f{\left(x \right)}\, dt$

* Please write down the mean and variance of the random variable $X$

```python
x_mean = sp.symbols('\overline{x}')
sigma = sp.symbols('sigma') ## Standard deviation
n = sp.symbols('n')
```

### Mean value:

```python
equation_mean_x = sp.Eq(x_mean,1/n*sp.Sum(x,(x, 1,n)))
equation_mean_x
```

$\displaystyle \overline{x} = \frac{\sum_{x=1}^{n} x}{n}$

### Variance

```python
equation_variance_x = sp.Eq(sigma**2,1/(n-1)*sp.Sum((x-x_mean)**2,(x, 1,n)))
equation_variance_x
```

$\displaystyle \sigma^{2} = \frac{\sum_{x=1}^{n} \left(- \overline{x} + x\right)^{2}}{n - 1}$

## 1.2	Empirical distribution $F(x)$ of random variable X

```python
N=10000
MU=10
SIGMA=30
x_sample = np.random.normal(loc=MU, scale=SIGMA, size=N)
```

Compute the empirical distribution of $X$, $F_1(x)$, and plot it in a figure.

```python
fig,ax = plt.subplots()

values = np.sort(x_sample)
density = np.arange(N)/N

ax.plot(values,density,'-');
ax.set_title('Empirical distribution $F(x)$');
ax.set_xlabel('x')
ax.set_ylabel('P(X<x)');
```

## Calculating empirical probability density function $f(x)$
Empirical probability density function $f(x)$ is calculated by conducting numerical derivation of the empirical distribution $F(x)$

```python
smothing=400
x_ = np.linspace(np.min(values), np.max(values),int(N/smothing))
cdf_empirical = np.interp(x_, values,density)
```

Reduce the number of points to get more stable derivatives.

```python
fig,ax = plt.subplots()
ax.plot(values,density,'-');
ax.plot(x_,cdf_empirical,'.', label='Reduced number of points');
ax.legend()

ax.set_title('Empirical distribution $F(x)$');
ax.set_xlabel('x')
ax.set_ylabel('P(X<x)');
```

```python
pdf_empirical = np.gradient(cdf_empirical,x_)
```

```python
fig,ax = plt.subplots()
fig.set_dpi(200)
ax.hist(values, bins=100, density=True, zorder=-10, alpha=0.5);
ax.plot(x_,pdf_empirical,'g-', label='pdf (empirical)')

pdf_nominal=norm.pdf(values,loc=MU, scale=SIGMA)
ax.plot(values,pdf_nominal,'r--', label='pdf (nominal)')

ax.set_title('Empirical $f(x)$ with histogram an numerical derivation')
ax.legend();
```

## But how can $F(x)$ be created this way?
Create an array of random samples from a discrete uniform distribution over [0,2].

```python
N=10000
x_max=2
x_sample = np.random.randint(low=0,high=x_max+1,size=N)

values = np.sort(x_sample)
density = np.arange(N)/N

x_=np.arange(0,x_max+1,1)
cdf_empirical = np.interp(x_, values,density,)

```

```python
fig,ax=plt.subplots()
ax.plot(values,density,'-');
ax.plot(x_,cdf_empirical,'--', label='slope')
ax.legend()
ax.set_title('Empirical distribution $F(x)$');
ax.set_xlabel('x')
ax.set_ylabel('P(X<x)');
ax.set_xticks(np.arange(0,x_max+1,1))
ax.grid()
```

```python
(1-cdf_empirical[0])/x_max
```

    0.3335

## 1.3	Quantiles of X
The concept of quantile is important in reliability analysis and extreme predictions. The quantile of a distribution could be defined in different ways. For the random variable $X$, the quantile as a number of $x_{\alpha}$ is defined as,
$$F(x_{\alpha})=P(X<x_{\alpha})=\alpha$$

```python
norm.ppf(0.05,loc=MU, scale=SIGMA)
```

    -39.345608808544185

```python
norm.ppf(0.95,loc=MU, scale=SIGMA)
```

    59.34560880854416

### Numerical quantiles:

```python
np.quantile(x,0.05)
```

$\displaystyle 1.0 x$

```python
np.quantile(x,0.95)
```

$\displaystyle 1.0 x$

## 2	Distributions of two or more random variables
Let $X$, $Y$ denote two independent random variables, while $X, Y\in[0, +\infty]$. The joint pdf and cdf are represented by $f_{X,Y}(x,y)$ and $F_{X,Y}(x,y)$, respectively.

* Please write down the relation between the joint pdf and cdf of X and Y.

```python
x,y,zeta_1,zeta_2 = sp.symbols('x y zeta_1 zeta_2')
f_x_y = sp.Function('f_x,y')(x,y)  # PDF
F_x_y = sp.Function('F_x,y')(x,y)  # CDF

```

```python
equation_CDF2 = sp.Eq(F_x_y,sp.Integral(f_x_y,(zeta_2, -sp.oo, y),(zeta_1, -sp.oo, x)))
equation_CDF2
```

$\displaystyle \operatorname{F_{x,y}}{\left(x,y \right)} = \int\limits_{-\infty}^{x}\int\limits_{-\infty}^{y} \operatorname{f_{x,y}}{\left(x,y \right)}\, d\zeta_{2}\, d\zeta_{1}$

* Please write down the marginal distributions of X and Y, respectively.

```python
F_x = sp.Function('F_x')(x)  # CDF
sp.Eq(F_x,sp.Integral(f_x_y,(zeta_2, -sp.oo, sp.oo),(zeta_1, -sp.oo, x)))
```

$\displaystyle \operatorname{F_{x}}{\left(x \right)} = \int\limits_{-\infty}^{x}\int\limits_{-\infty}^{\infty} \operatorname{f_{x,y}}{\left(x,y \right)}\, d\zeta_{2}\, d\zeta_{1}$

```python
F_y = sp.Function('F_y')(y)  # CDF
sp.Eq(F_y,sp.Integral(f_x_y,(zeta_2, -sp.oo, y),(zeta_1, -sp.oo, sp.oo)))
```

$\displaystyle \operatorname{F_{y}}{\left(y \right)} = \int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{y} \operatorname{f_{x,y}}{\left(x,y \right)}\, d\zeta_{2}\, d\zeta_{1}$

```python
from sympy.stats import P, E, variance, Die, Normal
```

* Please write down the expected value and variance of X and Y, respectively.
# ???

## Common used distribution in maritime industry
In the maritime industry, a few distributions are commonly used for design and safety analysis. Without going too much into the details, the normal distribution is definitely the most popular distribution type. When values of observed data are too scattered, the lognormal distribution could be used to describe the data distribution. In the engineering applications, ship responses, for example motions, stress signals etc., are often assumed to be Gaussian for convenience even though it is not the real case. Then the local maxima (minima) of the response are often Rayleigh distributed. Further, the Weibull distribution is often used to describe the stress range distribution for long-term fatigue analysis, while Gumbel distribution to fit the yearly maxima for extreme predictions.
After this course, you are supposed to know the exact forms of some distributions, for example normal and lognormal. For others, you have to know at least how to compute the probability using the Matlab commands. Here the stress signals measured in a 4400TEU container ship will be used in the following exercise. You could download the data from the course website.

### Weibull distribution
[Wikipedia](https://en.wikipedia.org/wiki/Weibull_distribution)

```python
from scipy.stats import exponweib
```

The Weibull Minimum Extreme Value distribution, from extreme value theory, is also often simply called the Weibull distribution.

```python
def f(lamda=1.0,k=2.0, MU=0.0, SIGMA=1.0):

    N=200
    x=np.linspace(0,6,N)
    pdf = exponweib.pdf(x,lamda,k,loc=MU, scale=SIGMA)

    fig,ax = plt.subplots()
    ax.plot(x,pdf);
    ax.set_xlabel('x')
    ax.set_ylabel('P(x)')
    ax.set_title('Weibull distribution ($\lambda$:%0.2f, $k$:%0.2f)' % (lamda,k));
    ax.set_xlim(0,6)
    ax.set_ylim(0,3)

interactive_plot = interactive(f, lamda=(1.0, 3.0, 0.2), k=(0.2, 5, 0.2), MU=(0,4,0.2), SIGMA=(0.5,3.0,0.2))
interactive_plot
```

    interactive(children=(FloatSlider(value=1.0, description='lamda', max=3.0, min=1.0, step=0.2), FloatSlider(val…

```python

```

```python

```

```python

```

```python

```


```python
import sympy as sp
```

```python
x,y = sp.symbols('x y')
f = sp.Function('f')(x)
g = sp.Function('f')(x, y)
f
g
```

$\displaystyle f{\left(x,y \right)}$

```python
f.diff(x)
```

$\displaystyle \frac{d}{d x} f{\left(x \right)}$

```python
g.diff(x)
```

$\displaystyle \frac{\partial}{\partial x} f{\left(x,y \right)}$

```python
ode = sp.Eq(x*f.diff(x,x)+f.diff(x), x**3)
display(ode)
sol = sp.dsolve(ode)
sol
```

$\displaystyle x \frac{d^{2}}{d x^{2}} f{\left(x \right)} + \frac{d}{d x} f{\left(x \right)} = x^{3}$

$\displaystyle f{\left(x \right)} = C_{1} + C_{2} \log{\left(x \right)} + \frac{x^{4}}{16}$

```python
ivp = sp.dsolve(ode, ics={f.subs(x,1):0, f.diff().subs(x,2):1})
ivp
```

$\displaystyle f{\left(x \right)} = \frac{x^{4}}{16} - 2 \log{\left(x \right)} - \frac{1}{16}$

```python
ivp.subs(x,1)

```

$\displaystyle f{\left(1 \right)} = 0$

```python
ivp.rhs.diff().subs(x,2)
```

$\displaystyle 1$

```python
(x*ivp.rhs.diff(x,x) + ivp.rhs.diff()).simplify()
```

$\displaystyle x^{3}$

```python

```


# Systems of Nonlinear Equations
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

# Example 1

A system of nonlinear equations consists of several nonlinear functions - as many as there are unknowns. Solving a system of nonlinear equations means funding those points where the functions intersect each other. Consider for example the following system of equations
\begin{equation}
y = 4x - 0.5 x^3
\end{equation}
\begin{equation}
y = \sin(x)e^{-x}
\end{equation}

The first step is to write these in residual form
\begin{equation}
f_1 = y - 4x + 0.5 x^3,\\
f_2 = y -  \sin(x)e^{-x}
\end{equation}

```python
import numpy as np
from numpy import cos, sin, pi, exp
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
from scipy.optimize import fsolve
```

```python
y1 = lambda x: 4 * x - 0.5 * x**3
y2 = lambda x: sin(x)*exp(-x)
x = np.linspace(-3.5,4,100)
plt.ylim(-8,6)
plt.plot(x,y1(x), 'k')
plt.plot(x,y2(x), 'r')
plt.grid()
plt.savefig('example1.pdf')
```

```python
def newton_solver(F, J, x, tol):
    F_value = F(x)
    err = np.linalg.norm(F_value, ord=2)  # l2 norm of vector
    err = tol + 100
    niter = 0
    while abs(err) > tol and niter < 100:
        J_value = J(x)
        delta = np.linalg.solve(J_value, -F_value)
        x = x + delta
        F_value = F(x)
        err = np.linalg.norm(F_value, ord=2)
        niter += 1

    # Here, either a solution is found, or too many iterations
    if abs(err) > tol:
        niter = -1
    return x, niter, err
```

```python
def F(xval):
    x = xval[0]
    y = xval[1]
    f1 = 0.5 * x**3 + y - 4*x
    f2 = y - sin(x)*exp(-x)
    return np.array([f1,f2])

def J(xval):
    x = xval[0]
    y = xval[1]
    return np.array([[1.5 * x**2 - 4, 1],
                     [-cos(x)*exp(-x)+sin(x)*exp(-x), 1]])

def Jdiff(F,xval):
    n = len(xval)
    J = np.zeros((n,n))
    col = 0
    for x0 in xval:
        delta = np.zeros(n)
        delta[col] = 1e-4 * x0 + 1e-12
        diff = (F(xval + delta) - F(xval))/delta[col]
        J[:,col] = diff
        col += 1
    return J
```

```python
tol = 1e-8
xguess = np.array([-2,-4])
x, n, err = newton_solver(F, J, xguess, tol)
print (n, x)
print ('Error Norm =',err)
```

    4 [-1.46110592 -4.28481694]
    Error Norm = 1.523439979143842e-11

```python
fsolve(F,(-2,-4))
```

    array([-1.46110592, -4.28481694])

# Example 2
Find the roots of the following system of equations
\begin{equation}
x^2 + y^2 = 1, \\
y = x^3 - x + 1
\end{equation}
First we assign $x_1 \equiv x$ and $x_2 \equiv y$ and rewrite the system in residual form
\begin{equation}
f_1(x_1,x_2) = x_1^2 + x_2^2 - 1, \\
f_2(x_1,x_2) = x_1^3 - x_1 - x_2 + 1
\end{equation}

```python
x = np.linspace(-1,1)
y1 = lambda x: x**3 - x + 1
y2 = lambda x: np.sqrt(1 - x**2)
plt.plot(x,y1(x), 'k')
plt.plot(x,y2(x), 'r')
plt.grid()
```

```python
def F(xval):
    x1 = xval[0]
    x2 = xval[1]
    f1 = x1**2 + x2**2 - 1
    f2 = x1**3 - x1 + 1 - x2
    return np.array([f1,f2])

def J(xval):
    x1 = xval[0]
    x2 = xval[1]
    return np.array([[2.0*x1,       2.0*x2],
                     [3.0*x1**2 -1, -1]])
```

```python
tol = 1e-8
xguess = np.array([0.5,0.5])
x, n, err = newton_solver(F, J, xguess, tol)
print (n, x)
print ('Error Norm =',err)
```

    6 [0.74419654 0.66796071]
    Error Norm = 4.965068306494546e-16

```python
fsolve(F,(0.5,0.5))
```

    array([0.74419654, 0.66796071])

```python
import urllib
import requests
from IPython.core.display import HTML
def css_styling():
    styles = requests.get("https://raw.githubusercontent.com/saadtony/NumericalMethods/master/styles/custom.css")
    return HTML(styles.text)
css_styling()

```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


Traçar um esboço do gráfico e obter uma equação da parábola que satisfaça as condições dadas.

24. Foco:$F(3,-1)$; diretriz: $2x - 1 = 0$

Arrumando a equação da diretriz

$d: x = \frac{1}{2}$

Fazendo um esboço é possível perceber que a parábola é paralela ao eixo $x$, logo sua equação é dada por $(y-k)^2 = 2p(x-h)$

Sabendo que a distância da diretriz até o foco é $p$, podemos calcular sua distância para achar $\frac{p}{2}$ usando o ponto$P(\frac{1}{2},-1)$ da diretriz

$p = \sqrt{(3-\frac{1}{2})^2 + (-1-(-1))^2}$

$p = \sqrt{(\frac{5}{2})^2 + 0}$

$p = \pm \sqrt{\frac{25}{4}}$

$p =  \frac{5}{2}$

$\frac{p}{2} = \frac{5}{4}$

Somando $\frac{p}{2}$ no eixo $x$ da diretriz, obtemos as coordenadas do vértice

$V(\frac{7}{4},-1)$

Substituindo agora os pontos dos vértice e o valor de $p$ na fórmula, temos que

$(y-(-1))^2 = 2 \cdot \frac{5}{2} \cdot (x-\frac{7}{4}) $

$(y+1)^2 = 5(x-\frac{7}{4})$

$y^2 + 2y + 1 = 5x - \frac{35}{4}$

$y^2 + 2y - 5x + 1 + \frac{35}{4}$

Tirando o mmc entre e somando $1 + \frac{35}{4}$, temos que

$y^2 + 2y - 5x + \frac{39}{4} = 0$

Multiplicando por $4$

$4y^2 + 8y - 25x + 39 = 0$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((y+1)**2, 5*(x-7/4)), (x,-20,20), (y,-20,20),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/24.ipynb