```python
%run common-imports.ipynb
```

#  Gradient Descent

an effective way to optimize the loss function, and thus make the algorithm learn an effective model for estimations on a dataset $\mathscr{D}$ comprised of predictors and response. will do the process of learning an effective model through  gradient descent, by working out the steps. For simplicity, we will consider the case of ordinary least squares regression (simple linear regression).

```python
raw_data = pd.read_csv("../datasets/old-faithful-geyser.csv", sep='\t')
scaler = StandardScaler()
scaled = scaler.fit_transform(raw_data)
data = pd.DataFrame(data={'eruptions':scaled[:, 0], 'waiting': scaled[:,1]})
data.describe()
```

## Data Visualization

first visualize a scatter plot of the data and look at the histogram of the features. Finally, let us pull of this together with a kernel density plot, into a single plot with subplots.

```python
plt.scatter(data['eruptions'], data['waiting'], alpha=0.5, s=150, color='salmon')
plt.title(r'\textbf{Scatter-plot of $waiting$ vs $eruptions$}')
plt.xlabel(r'eruptions $\longrightarrow$');
plt.ylabel(r'waiting $\longrightarrow$');
plt.show();
data.hist(bins=50, alpha = 0.4, color='salmon', xrot=90, figsize=(10,5));
plt.tight_layout()
```

```python
sns.set_palette("Reds")

sample = data
g = sns.pairplot(sample, diag_kind='kde',
                   plot_kws = { 'alpha': 0.20, 's': 80, 'edgecolor': 'k', 'color':'salmon'},
                   size=5, );
g.map_diag(sns.kdeplot, color='salmon',  shade=True);
g.map_upper(plt.scatter, color='salmon', alpha=0.5);
g.map_lower(sns.kdeplot, shade=False, shade_lowest=False, cbar=True);

plt.tight_layout()
```

## Gradient Descent from first principles

Recall that a step of learning is given by the equation:
$$\begin{align}\boldsymbol{\beta_\text{next}} &= \boldsymbol{\beta} - \alpha \nabla \mathscr {L(\mathbf\beta)} \\
\text{where} &: \nonumber\\
\boldsymbol{\beta} &: \text{the parameter vector} \begin{pmatrix} \beta_0\\ \beta_1 \end{pmatrix}\nonumber\\
\boldsymbol{\beta_\text{next}} &: \text{the next value of the parameter vector} \nonumber\\
\alpha &: \text{the learning rate} \nonumber\\
\mathscr{L(\mathbf\beta)} &: \text{the loss function}\nonumber\\
\end{align}$$

Let us consider an ordinary least squares loss function, so that we define the loss as simply the sum-squared errors.

$$\begin{align}\mathscr{L} &= \sum_{i=1}^n (y_i - \widehat{y}_i)^2\\
&= \sum_{i=1}^n (y_i - \beta_0 -\beta_1 x_i)^2
\end{align}
$$

The expression for the gradient of the loss is, therefore:

$$
\begin{equation}
\nabla \mathscr {L}(\mathbf\beta) = \begin{pmatrix} \frac{\partial L}{\partial\beta_0} \\ \frac{\partial L}{\partial\beta_1} \end{pmatrix}
\end{equation}
$$

Let us work out each of the component derivatives:

$$\begin{align}\frac{\partial \mathscr L}{\partial \beta_0} &= - \sum_{i=1}^n (y_i -\beta_0 - \beta_1 x_i)\\
\frac{\partial \mathscr L}{\partial \beta_1} &= - \sum_{i=1}^n x_i(y_i -\beta_0 - \beta_1 x_i)\\
\end{align}$$

Therefore, the gradient descent step can be expressed as:

$$\begin{align}
\beta_{0, next} &= \beta_0 + \alpha\sum_{i=1}^n (y_i -\beta_0 - \beta_1 x_i) \\
\beta_{1, next} &= \beta_1 + \alpha\sum_{i=1}^n x_i(y_i -\beta_0 - \beta_1 x_i)
\end{align}
$$

We are going to perform a batch gradient descent, i.e. use all of the training data to compute loss in each step. Let us start with a small learning rate, say $\alpha = 10^{-5}$. We need a stopping criterion for our learning process.

For simplicity, let us stop after say 200 epochs, i.e. running the gradient descent for a two hundred steps.

**Definition: EPOCH**
While training, an epoch is a complete cycle through the entire dataset. In other words, each datum in the dataset should have contributed to the learning in that cycle.

For simple or batch gradient descent, we use all of the data in each cycle; therefore, one step of gradient descent maps to an entire epoch. Later we will learn about other variants of the gradient descent process; in particular, we will learn about stochastic gradient descent, mini-batch gradient descent. We will also learn about various momentum based variants. But that is a topic for a later day.

```python
α = pow(10, -4)
β_0, β_1 =  4, 4 #np.random.normal(0,5,2) # initialize the parameters to some random values.
epochs = 200
X, Y = data['eruptions'].values, data['waiting'].values
α, β_0, β_1, epochs
```

```python
intermediates = pd.DataFrame(columns=['epoch', 'β0', 'β1', 'loss'])

for epoch in range(epochs+1):

    # compute the gradients
    dβ_0    = -sum([yi - β_0 - β_1*xi for xi, yi in zip (X, Y)])
    dβ_1    = -sum([xi*(yi - β_0 - β_1*xi) for xi, yi in zip (X, Y)])

    # gradient descent step
    β_0     = (β_0 - α*dβ_0)
    β_1     = (β_1 - α*dβ_1)

    # update the loss function
    loss    = sum([ (yi -β_0 -β_1*xi)**2 for xi, yi in zip (X, Y)])

    # store the values for later visualization
    intermediates.loc[intermediates.shape[0]] = [epoch, β_0, β_1, loss]
```

Let us preview some of the rows of data.

```python
pd.set_option("display.precision", 3)
intermediates
```

## Parameter optimization with epochs

Let us now see how the learning of the parameters $\beta_0$ and $\beta_1$ happens with each epoch.

```python
plt.figure(figsize=(10,10))
plt.plot(intermediates.epoch, intermediates.β0, linewidth=2, alpha=1, color='maroon');
plt.xlabel(r'$Epoch\longrightarrow$');
plt.ylabel(r'$\beta_0\longrightarrow$');
plt.title(r"""$\beta_0\ vs\ epochs$""");
```

```python
plt.figure(figsize=(10,10))
plt.plot(intermediates.epoch, intermediates.β1, linewidth=2, alpha=1, color='maroon');
plt.xlabel(r'$Epoch\longrightarrow$');
plt.ylabel(r'$\beta_1\longrightarrow$');
plt.title(r"""$\beta_1\ vs\ epochs$""");
```

Also, let us see how the loss (sum squared errors) decreased with the epochs.

```python
plt.figure(figsize=(20,10))
plt.plot(intermediates.epoch, intermediates.loss, linewidth=2, alpha=1, color='maroon');
plt.xlabel(r'$Epoch\longrightarrow$');
plt.ylabel(r'$\mathcal L\longrightarrow$');
plt.title(r"""$\mathcal L\ vs\ epochs$""");
```

## Loss contour plots in the parameter space

Let us start by creating a grid mesh of points in the hypothesis space, each point corresponding to a particular hypothesis's parameter value $\boldsymbol\beta = \begin{pmatrix}\beta_0 \\ \beta_1 \end{pmatrix}$. Next, let us compute the value of the loss function, $\mathscr{L}$, as the sum-squared errors at each of these points.

### Loss Contour plot

```python
def loss(β0:float, β1:float) -> float:
     return sum ([(yi - β0 -β1*xi)**2 for xi,yi in zip(X, Y)])

# Create the mesh grid of values.
b0     = np.linspace(-4, 4, 1000)
b1     = np.linspace(-4,4,1000)
B0, B1 = np.meshgrid(b0, b1)
rss    = loss(B0, B1)
```

look at the contour surfaces (curves of equal loss or errors). Because we had standardized the data, the contour plot is elliptic will low eccentricity, i.e. almost circular; however, because we have drawn the plot in a 2:1 ratio, it looks more elliptical than it is. Change the plot size below to a square (20,20) for the figure size, and see what happens.

```python
levels=[0, 50, 100, 200, 400, 800, 1600, 3200, 6400, 10_000]
plt.figure(figsize=(20,10))
contour_plot = plt.contour(B0, B1, rss, levels, colors='black', linestyles='dashed', linewidths=1, )
plt.clabel(contour_plot, inline=1, fontsize=10)
contour_plot = plt.contourf(B0, B1, rss,alpha=0.7)
plt.xlabel(r'$\beta_0\longrightarrow$');
plt.ylabel(r'$\beta_1\longrightarrow$');
plt.title(r'\textbf{Loss function contour plot}');
```

### Loss surface and the contour plots in the parameter plane ($\boldsymbol \beta$-hyperplane)

see the relationship between the loss surface, and its projection on the $\boldsymbol \beta$-hyperplane.

**Note: in order to show the loss surface above the contour plot, we have artificially added 2000 to the residual loss, in order to lift the loss surface. To see the correct loss surface, reset the `loss_lift` to zero**

```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(figsize=(20,30))
ax = fig.add_subplot(111, projection='3d')
loss_lift = 2000 # artificially lifing the loss surface to better show the contour plot below it.

#Plot the loss surface
ax.plot_surface(B0, B1, loss_lift + (rss),
                cmap='viridis', alpha=0.9);

levels=[0, 100, 200, 400, 800, 1600, 3200, 6400, 10_000]

# Plot the contours
ax.contourf(B0, B1, rss, levels, offset = 0, alpha=0.5)
plt.xlabel(r'$\beta_0\longrightarrow$');
plt.ylabel(r'$\beta_1\longrightarrow$');
ax.set_zlabel(r'$Loss\longrightarrow$');

# Plot the learning journey
ax.plot(intermediates.β0, intermediates.β1, intermediates.loss + loss_lift,  color='red',zdir='z', linewidth=6)
ax.scatter(intermediates.β0, intermediates.β1, s=20,  color='salmon',zdir='z')
plt.title(r'\textbf {The loss surface and its contour plot}');
plt.tight_layout()
```

## GRADIENT DESCENT ANIMATION

create an animation to show the progression of learning with gradient descent. In the video below, the process of gradient descent is shown as the gradual optimization takes place.

*Caution: This code below to generate the animation can be computationally very intensive and take quite some time to finish. The value of the loop has deliberately been set to zero, so that you don't accidentally run it.*

If you want to run it, uncomment the line of code below that sets the variable `iterations`

```python
from celluloid import Camera
fig = plt.figure(figsize=(20,30))
camera = Camera(fig)

from mpl_toolkits.mplot3d import Axes3D
ax = fig.add_subplot(111, projection='3d')
loss_lift = 2000

iterations = 0                          # for safety, that this code does not accidentally get run.
#iterations = intermediates.shape[0]+1  # uncomment this if you want to generate a new animation video.
# Plot the learning journey
for i in range(iterations):
    ax.plot_surface(B0, B1, loss_lift + (rss),
                cmap='viridis', alpha=0.9);

    levels=[0, 100, 200, 400, 800, 1600, 3200, 6400, 10_000]

    ax.contourf(B0, B1, rss, levels, offset = 0, alpha=0.5)
    plt.xlabel(r'$\beta_0\longrightarrow$');
    plt.ylabel(r'$\beta_1\longrightarrow$');
    ax.set_zlabel(r'$Loss\longrightarrow$');
    subset = intermediates.head(i+1)
    ax.plot(subset.β0, subset.β1, subset.loss + loss_lift,  color='red',zdir='z', linewidth=6)
    ax.plot(subset.β0, subset.β1, linewidth=6,  color='salmon',zdir='z')
    plt.title(r'\textbf {The loss surface and its contour plot}');
    plt.tight_layout()
    camera.snap()

animation = camera.animate()
animation.save('animation-gradient-descent.mp4')
```

```python

```


# Barotropic quasigeostrophic model

### Model equations
The QGModel class solves the two-dimensional equation

\begin{equation}\tag{1}
q_t + \beta \psi_x + J(\psi,q) =  \nu \triangle q -\nu_4 \triangle^2 q - \mu q  \,,
\end{equation}

where the vorticity, $q$, is related to the streamfunction via the inversion relation

\begin{equation}\tag{2}
q = \triangle \psi\, .
\end{equation}

### Power integrals

The kinetic energy, $K_e \equiv \langle |\nabla \psi|^2 \rangle/2$, satisfies

\begin{equation}\tag{3}
\dot K_e = \underbrace{-\nu \langle|\nabla \psi|^2\rangle + \nu_4 \langle q \triangle q\rangle - \mu \langle |\nabla \psi|^2 \rangle}_{\equiv\,\,\varepsilon_\psi}\,.
\end{equation}

The enstrophy, $S_e \equiv \langle q^2\rangle/2$, satisfies

\begin{equation}\tag{4}
\dot S_e = \underbrace{-\beta\langle q \psi_x\rangle}_{\equiv\,\,\Gamma_q}\,\,\, \underbrace{-\nu \langle|\nabla q|^2\rangle - \nu_4 \langle (\triangle q)^2\rangle - \mu \langle q^2 \rangle}_{\equiv\,\, \chi_q}\,.
\end{equation}


# Barrier Options

```python
import pyvacon.analytics as analytics
import datetime as dt
import pyvacon.tools.converter as converter
import pyvacon.tools.enums as enums
import pyvacon.marketdata.testdata as mkt_testdata
import pyvacon.instruments.testdata as ins_testdata
import math
from scipy.stats import norm
import pyvacon.marketdata.plot as mkt_plot #import module for plotting functionality
#the next lin is a jupyter internal command to show the matplotlib graphs within the notebook
%matplotlib inline
```

```python
def exp(x):
    return math.exp(x)
def cdf(x):
    return norm.cdf(x)
def log(x):
    return math.log(x)
def sqrt(x):
    return math.sqrt(x)
```

## Definition of Barrier Options

Barrier options are options where the payoff depends on whether the underlying's spot price reaches a certain level during a certain period of time. Barrier options can be classified in know-out options and knock-in options. A knock-in option comes into existence only when the underlying's spot price reaches the defined barrier; a knock-out option ceases to exist if the underlying's spot prices reaches the defined barrier. The different barrier options including their payoff profile are presented in this notebook. For a detailed description please refer to Hull, *Options, futures, and other derivatives, 8th Edition,* 2012, pp. 579-581.

The following code defines the valuation formula for barrier options assuming a non-dividend paying stock.

```python
def BarrierOptionPricer(_Type, S0, K, H, r, q, sigma, T, t=0):
    _lambda = (r-q+sigma**2/2)/sigma**2
    y = (log(H**2/(S0*K)))/(sigma*sqrt(T-t))+_lambda*sigma*sqrt(T-t)
    x1 = (log(S0/H))/(sigma*sqrt(T-t))+_lambda*sigma*sqrt(T-t)
    y1 = (log(H/S0))/(sigma*sqrt(T-t))+_lambda*sigma*sqrt(T-t)
    d1= (log(S0/K)+(r+sigma**2/2)*(T-t))/(sigma*sqrt(T-t))
    d2 = d1-sigma*sqrt(T-t)
    p = -1*(S0*cdf(-1*d1)-K*exp(-r*(T-t))*cdf(-1*d2))
    c = 1*(S0*cdf(1*d1)-K*exp(-r*(T-t))*cdf(1*d2))
    cdi = S0*exp(-q*(T-t))*(H/S0)**(2*_lambda)*cdf(y)-K*exp(-r*(T-t))*(H/S0)**(2*_lambda-2)*cdf(y-sigma*sqrt(T-t))
    cdo = S0*cdf(x1)*exp(-q*(T-t))-K*exp(-r*(T-t))*cdf(x1-sigma*sqrt(T-t))-S0*exp(-q*(T-t))*(H/S0)**(2*_lambda)*cdf(y1)+K*exp(-r*(T-t))*(H/S0)**(2*_lambda-2)*cdf(y1-sigma*sqrt(T-t))
    cui = S0*cdf(x1)*exp(-q*(T-t))-K*exp(-r*(T-t))*cdf(x1-sigma*sqrt(T-t))-S0*exp(-q*(T-t))*(H/S0)**(2*_lambda)*(cdf(-y)-cdf(-y1))+K*exp(-r*(T-t))*(H/S0)**(2*_lambda-2)*(cdf(-y+sigma*sqrt(T-t))-cdf(-y1+sigma*sqrt(T-t)))
    pui = -S0*exp(-q*(T-t))*(H/S0)**(2*_lambda)*cdf(-y)+K*exp(-r*(T-t))*(H/S0)**(2*_lambda-2)*cdf(-y+sigma*sqrt(T-t))
    puo = -S0*cdf(-x1)*exp(-q*(T-t))+K*exp(-r*(T-t))*cdf(-x1+sigma*sqrt(T-t))+S0*exp(-q*(T-t))*(H/S0)**(2*_lambda)*cdf(-y1)-K*exp(-r*(T-t))*(H/S0)**(2*_lambda-2)*cdf(-y1+sigma*math.sqrt(T-t))
    pdi = -S0*cdf(-x1)*exp(-q*(T-t))+K*exp(-r*(T-t))*cdf(-x1+sigma*sqrt(T-t))+S0*exp(-q*(T-t))*(H/S0)**(2*_lambda)*(cdf(y)-cdf(y1))-K*exp(-r*(T-t))*(H/S0)**(2*_lambda-2)*(cdf(y-sigma*sqrt(T-t))-cdf(y1-sigma*sqrt(T-t)))
    if _Type =='cdi' and H<K and S0>H:
        return cdi
    if _Type =='cdi' and H>=K and S0>H:
        return c-cdo
    if _Type =='cdi' and S0<=H:
        return c
    if _Type =='cdo' and H<K and S0>H:
        return c-cdi
    if _Type =='cdo' and H<K and S0<=H:
        return 0
    if _Type =='cdo' and H>=K and S0>H:
        return cdo
    if _Type =='cdo' and H>=K and S0<=H:
        return 0
    if _Type =='cui' and H>K:
        return cui
    if _Type =='cui' and H<=K:
        return c
    if _Type =='cuo' and H>K and S0<H:
        return c-cui
    if _Type =='cuo' and H>K and S0>=H:
        return 0
    if _Type =='cuo' and H<=K:
        return 0.0
    if _Type =='pui' and H>=K and S0<H:
        return pui
    if _Type =='pui' and H<K and S0<H:
        return p-puo
    if _Type =='pui' and S0>=H:
        return p
    if _Type =='puo':
        if S0>=H:
            return 0
        else:
            if _Type =='puo' and H>=K:
                return p-pui
            if _Type =='puo' and H<K:
                return puo
    if _Type =='pdi' and H>=K:
        return p
    if _Type =='pdi' and H<K:
        return pdi
    if _Type =='pdo' and H>=K:
        return 0
    if _Type =='pdo' and H<K and S0>H:
        return p-pdi
    if _Type =='pdo' and H<K and S0<=H:
        return 0
    if _Type =='c':
        return c
    if _Type =='p':
        return p
```

```python
spots = analytics.vectorDouble()

S0 = 30
n=0.1
while n <=100:
    spots.append(n)
    n=n+0.1

K = 50
H1 = 40
H2 = 60
r = 0.05
q = 0
sigma = 0.3
T = 1
t = 0
```

### Barrier call options

#### Down-and-in call

A down-and-in call is a call option which comes into existence if the stock price hits a barrier which is below the initial asset price.

If the barrier $H$ is less than or equal to the strike price $K$, the formula to price a down-and-in call is defined as

$$c_{di}=S_0e^{-qT}(H/S_0)^{2\lambda}N(y)-Ke^{-rT}(H/S_0)^{2\lambda-2}N(y-\sigma\sqrt{T}),$$

where

\begin{align}
\lambda &= \frac{r-q+\sigma^2/2}{\sigma^2} \\
y &= \frac{\ln[H^2/(S_0K)]}{\sigma\sqrt{T}}+\lambda\sigma\sqrt{T}. \\
\end{align}

$S_0$ is the underlying's spot price, $K$ is the strike price, $H$ is the barrier level, $\sigma$ is the underlying's volatility, $r$ is the risk-free interest rate, $q$ is the borrowing rate, and $T$ is the time to maturity. $N(x)$ is the cumulative probability distribution function for a standardized normal distribution.

If the barrier is greater than or equal to the strike price, the formula for the down-and-in call is

$$c_{di}=c-c_{do}.$$

```python
# Assumption that H has not been reached yet. If H is reached, product becomes normal plain vanilla call.

cdi_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    cdi_price1.append(BarrierOptionPricer('cdi', spots[s], K, H1, r, q, sigma, T, t))
vanilla_call1 = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_call1.append(BarrierOptionPricer('c', spots[s], K, H1, r, q, sigma, T, t))

cdi_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    cdi_price2.append(BarrierOptionPricer('cdi', spots[s], K, H2, r, q, sigma, T, t))

fig, (cdi1, cdi2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
cdi1.plot(spots, cdi_price1, 'k', label='Down-and-in call')
cdi1.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cdi1.set_title('Down-and-in call H<K')
cdi1.set_xlabel('Spot')
cdi1.set_ylabel('Price')
cdi1.axvline(x=K, label='Strike', ls= '--', c='g')
cdi1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = cdi1.legend(loc='best', shadow=True, fontsize='medium')

#fig, cdi2 = mkt_plot.plt.subplots()
cdi2.plot(spots, cdi_price2, 'k', label='Down-and-in call')
cdi2.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cdi2.set_title('Down-and-in call H>K')
cdi2.set_xlabel('Spot')
cdi2.set_ylabel('Price')
cdi2.axvline(x=K, label='Strike', ls= '--', c='g')
cdi2.axvline(x=H2, label='Barrier', ls=':', c='r')
legend = cdi2.legend(loc='best', shadow=True, fontsize='medium')
```

#### Down-and-out call

A down-and-out call in a call option that ceases to exists when the stock price hits a barrier which is below the initial asset price.

If $H \leq K$, the formula for the down-and-out call is

$$c_{do}=c-c_{di},$$

if $H \geq K$, the formula is

$$c_{do}=S_0N(x_1)e^{-qT}-Ke^{-rT}N(x_1-\sigma\sqrt{T})-S_0e^{-qT}(H/S_0)^{2\lambda}N(y_1)+Ke^{-rT}(H/S_0)^{2\lambda-2}N(y_1-\sigma\sqrt{T})$$

where

\begin{align}
x_1 &=\frac{\ln(S_0/H}{\sigma\sqrt{T}}+\lambda\sigma\sqrt{T} \\
y_1 &=\frac{\ln(H/S_0}{\sigma\sqrt{T}}+\lambda\sigma\sqrt{T}. \\
\end{align}

```python
vanilla_call1 = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_call1.append(BarrierOptionPricer('c', spots[s], K, H1, r, q, sigma, T, t))

cdo_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    cdo_price1.append(BarrierOptionPricer('cdo', spots[s], K, H1, r, q, sigma, T, t))

cdo_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    cdo_price2.append(BarrierOptionPricer('cdo', spots[s], K, H2, r, q, sigma, T, t))

fig, (cdo1, cdo2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
cdo1.plot(spots, cdo_price1, 'k', label='Down-and-out call')
cdo1.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cdo1.set_title('Down-and-out call H<K')
cdo1.set_xlabel('Spot')
cdo1.set_ylabel('Price')
cdo1.axvline(x=K, label='Strike', ls= '--', c='g')
cdo1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = cdo1.legend(loc='best', shadow=True, fontsize='medium')

#fig, cdo2 = mkt_plot.plt.subplots()
cdo2.plot(spots, cdo_price2, 'k', label='Down-and-out call')
cdo2.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cdo2.set_title('Down-and-out call H>K')
cdo2.set_xlabel('Spot')
cdo2.set_ylabel('Price')
cdo2.axvline(x=K, label='Strike', ls= '--', c='g')
cdo2.axvline(x=H2, label='Barrier', ls=':', c='r')
legend = cdo2.legend(loc='best', shadow=True, fontsize='medium')
```

#### Up-and-in call

An up-and-in call is a call option which comes into existence if the spots hits a barrier which is above the initial asset price.

In the case of $H \leq K$ the value of the up-and-in call $c_{ui}$ is $c$.

When $H > K$ the formula for the up-and-in call is defined as

$$c_{ui}=S_0N(x_1)e^{-qT}-Ke^{-rT}N(x_1-\sigma\sqrt{T})-S_0e^{-qT}(H/S_0)^{2\lambda}[N(-y)-N(-y_1)]+Ke^{-rT}(H/S_0)^{2\lambda-2}[N(-y+\sigma\sqrt{T})-N(-y_1+\sigma\sqrt{T})].$$

```python
# Assumption that H has not been reached yet. If the barrier is hit, the it is a plain vanilla call.

vanilla_call1 = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_call1.append(BarrierOptionPricer('c', spots[s], K, H1, r, q, sigma, T, t))

cui_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    cui_price1.append(BarrierOptionPricer('cui', spots[s], K, H1, r, q, sigma, T, t))

cui_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    cui_price2.append(BarrierOptionPricer('cui', spots[s], K, 80, r, q, sigma, T, t))

fig, (cui1, cui2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
cui1.plot(spots, cui_price1, 'k', label='Up-and-in call')
cui1.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cui1.set_title('Up-and-in call H<K')
cui1.set_xlabel('Spot')
cui1.set_ylabel('Price')
cui1.axvline(x=K, label='Strike', ls= '--', c='g')
cui1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = cui1.legend(loc='best', shadow=True, fontsize='medium')

#fig, cui2 = mkt_plot.plt.subplots()
cui2.plot(spots, cui_price2, 'k', label='Up-and-in call')
cui2.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cui2.set_title('Up-and-in call H>K')
cui2.set_xlabel('Spot')
cui2.set_ylabel('Price')
cui2.axvline(x=K, label='Strike', ls= '--', c='g')
cui2.axvline(x=80, label='Barrier', ls=':', c='r')
legend = cui2.legend(loc='best', shadow=True, fontsize='medium')
```

#### Up-and-out call

An up-and-out call is a call option which ceases to exist when the stock price hits a barrier which is above the initial asset price.

When $H \leq K$, the value of the up-and-out call is zero.

When $H > K$, formula for the up-and-out call is defined as

$$c_{uo}=c-c_{ui}.$$

```python
vanilla_call1 = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_call1.append(BarrierOptionPricer('c', spots[s], K, H1, r, q, sigma, T, t))

cuo_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    cuo_price1.append(BarrierOptionPricer('cuo', spots[s], K, H1, r, q, sigma, T, t))

cuo_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    cuo_price2.append(BarrierOptionPricer('cuo', spots[s], K, H2, r, q, sigma, T, t))

fig, (cuo1, cuo2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
cuo1.plot(spots, cuo_price1, 'k', label='Up-and-out call')
#cuo1.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cuo1.set_title('Up-and-out call H<K')
cuo1.set_xlabel('Spot')
cuo1.set_ylabel('Price')
cuo1.axvline(x=K, label='Strike', ls= '--', c='g')
cuo1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = cuo1.legend(loc='best', shadow=True, fontsize='medium')

#fig, cuo2 = mkt_plot.plt.subplots()
cuo2.plot(spots, cuo_price2, 'k', label='Up-and-out call')
#cuo2.plot(spots, vanilla_call1, 'y:', label='Plain vanilla call')
cuo2.set_title('Up-and-out call H>K')
cuo2.set_xlabel('Spot')
cuo2.set_ylabel('Price')
cuo2.axvline(x=K, label='Strike', ls= '--', c='g')
cuo2.axvline(x=H2, label='Barrier', ls=':', c='r')
legend = cuo2.legend(loc='best', shadow=True, fontsize='medium')
```

### Barrier put options

#### Down-and-in put

A down-and-in put is a put option which comes into existence if the spot price hits a barrier which is below the initial asset price.

When the barrier is greater than or equal to the strike price, the value of the down-and-in put is equal to a plain vanilla put $p$. If the barrier is less than the strike price, the formula for the down-and-in put is defined as

$$p_{di}=-S_0N(-x_1)e^{-qT}+Ke^{-rT}N(-x_1+\sigma\sqrt{T})+S_0e^{-qT}(H/S_0)^{2\lambda}[N(y)-N(y_1)]-Ke^{-rT}(H/S_0)^{2\lambda-2}[N(y-\sigma\sqrt{T}-N(y_1-\sigma\sqrt{T}].$$

```python
# H<K: As soon as the barrier is hit, the down-and-in put becomes a plain vanilla put.

vanilla_put = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_put.append(BarrierOptionPricer('p', spots[s], K, H1, r, q, sigma, T, t))

pdi_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    pdi_price1.append(BarrierOptionPricer('pdi', spots[s], K, 30, r, q, sigma, T, t))

pdi_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    pdi_price2.append(BarrierOptionPricer('pdi', spots[s], K, H2, r, q, sigma, T, t))

fig, (pdi1, pdi2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
pdi1.plot(spots, pdi_price1, 'k', label='Down-and-in put')
pdi1.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
pdi1.set_title('Down-and-in put H<K')
pdi1.set_xlabel('Spot')
pdi1.set_ylabel('Price')
pdi1.axvline(x=K, label='Strike', ls= '--', c='g')
pdi1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = pdi1.legend(loc='best', shadow=True, fontsize='medium')

#fig, pdi2 = mkt_plot.plt.subplots()
pdi2.plot(spots, pdi_price2, 'k', label='Down-and-in put')
pdi2.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
pdi2.set_title('Down-and-in put H>K')
pdi2.set_xlabel('Spot')
pdi2.set_ylabel('Price')
pdi2.axvline(x=K, label='Strike', ls= '--', c='g')
pdi2.axvline(x=H2, label='Barrier', ls=':', c='r')
legend = pdi2.legend(loc='best', shadow=True, fontsize='medium')
```

#### Down-and-out put

A down-and-out put is a put option which ceases to exists when the spot price hits a barrier which is below the initial asset price.

When the barrier is greater than or equal to the strike price, the value of the down-and-out put is zero. If the barrier is less than the strike price, the formula for the down-and-out put is defined as

$$p_{do} = p - p_{di}.$$

```python
vanilla_put = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_put.append(BarrierOptionPricer('p', spots[s], K, H1, r, q, sigma, T, t))

pdo_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    pdo_price1.append(BarrierOptionPricer('pdo', spots[s], K, H1, r, q, sigma, T, t))

pdo_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    pdo_price2.append(BarrierOptionPricer('pdo', spots[s], K, H2, r, q, sigma, T, t))

fig, (pdo1, pdo2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
pdo1.plot(spots, pdo_price1, 'k', label='Down-and-out put')
#pdo1.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
pdo1.set_title('Down-and-out put H<K')
pdo1.set_xlabel('Spot')
pdo1.set_ylabel('Price')
pdo1.axvline(x=K, label='Strike', ls= '--', c='g')
pdo1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = pdo1.legend(loc='best', shadow=True, fontsize='medium')

#fig, pdo2 = mkt_plot.plt.subplots()
pdo2.plot(spots, pdo_price2, 'k', label='Down-and-out put')
#pdo2.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
pdo2.set_title('Down-and-out put H>K')
pdo2.set_xlabel('Spot')
pdo2.set_ylabel('Price')
pdo2.axvline(x=K, label='Strike', ls= '--', c='g')
pdo2.axvline(x=H2, label='Barrier', ls=':', c='r')
legend = pdo2.legend(loc='best', shadow=True, fontsize='medium')
```

#### Up-and-in put

An up-and-in put is a put option that comes into existence if the spot price hits a barrier which is above the initial asset price.

When $H \geq K$, the formula for the up-and-in put is defined as

$$ p_{ui}=S_0e^{-qT}(H/S_0)^{2\lambda}N(-y)+Ke^{-rT}(H/S_0)^{2\lambda-2}N(-y+\sigma\sqrt{T})$$

when $H<K$ the formula is

$$ p_{ui}=p-p_{uo}.$$

```python
vanilla_put = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_put.append(BarrierOptionPricer('p', spots[s], K, H1, r, q, sigma, T, t))

pui_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    pui_price1.append(BarrierOptionPricer('pui', spots[s], K, H1, r, q, sigma, T, t))

pui_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    pui_price2.append(BarrierOptionPricer('pui', spots[s], K, H2, r, q, sigma, T, t))

fig, (pui1, pui2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
pui1.plot(spots, pui_price1, 'k', label='Up-and-in put')
pui1.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
pui1.set_title('Up-and-in put H<K')
pui1.set_xlabel('Spot')
pui1.set_ylabel('Price')
pui1.axvline(x=K, label='Strike', ls= '--', c='g')
pui1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = pui1.legend(loc='best', shadow=True, fontsize='medium')

#fig, pui2 = mkt_plot.plt.subplots()
pui2.plot(spots, pui_price2, 'k', label='Up-and-in put')
pui2.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
pui2.set_title('Up-and-in H>K')
pui2.set_xlabel('Spot')
pui2.set_ylabel('Price')
pui2.axvline(x=K, label='Strike', ls= '--', c='g')
pui2.axvline(x=H2, label='Barrier', ls=':', c='r')
legend = pui2.legend(loc='best', shadow=True, fontsize='medium')
```

#### Up-and-out put

An up-and-out put is a put option which ceases to exists when the spot price hits a barrier which is above the initial asset price.

When r $H \geq K$, the formula for the up-and-out put is defined as

$$ p_{uo}=p-p_{ui},$$

when $H<K$ the formula is

$$p_{uo}=-S_0N(-x_1)e^{-qT}+Ke^{-rT}N(-x_1+\sigma\sqrt{T})+S_0e^{-qT}(H/S_0)N(-y_1)-Ke^{-rT}(H/S_0)^{2\lambda-2}N(-y_1+\sigma\sqrt{T}).$$

```python
vanilla_put = analytics.vectorDouble()
for s in range(len(spots)):
    vanilla_put.append(BarrierOptionPricer('p', spots[s], K, H1, r, q, sigma, T, t))

puo_price1 = analytics.vectorDouble()
for s in range(len(spots)):
    puo_price1.append(BarrierOptionPricer('puo', spots[s], K, H1, r, q, sigma, T, t))

puo_price2 = analytics.vectorDouble()
for s in range(len(spots)):
    puo_price2.append(BarrierOptionPricer('puo', spots[s], K, H2, r, q, sigma, T, t))

fig, (puo1, puo2) = mkt_plot.plt.subplots(1,2, figsize=(12,4),dpi=100,num=1)
puo1.plot(spots, puo_price1, 'k', label='Up-and-out put')
puo1.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
puo1.set_title('Up-and-out put H<K')
puo1.set_xlabel('Spot')
puo1.set_ylabel('Price')
puo1.axvline(x=K, label='Strike', ls= '--', c='g')
puo1.axvline(x=H1, label='Barrier', ls=':', c='r')
legend = puo1.legend(loc='best', shadow=True, fontsize='medium')

#fig, puo2 = mkt_plot.plt.subplots()
puo2.plot(spots, puo_price2, 'k', label='Up-and-out put')
puo2.plot(spots, vanilla_put, 'y:', label='Plain vanilla put')
puo2.set_title('Up-and-out H>K')
puo2.set_xlabel('Spot')
puo2.set_ylabel('Price')
puo2.axvline(x=K, label='Strike', ls= '--', c='g')
puo2.axvline(x=H2, label='Barrier', ls=':', c='r')
legend = puo2.legend(loc='best', shadow=True, fontsize='medium')
```

---


```python
import itertools as it
from kanren import isvar, membero, var, run, eq
from kanren.core import success, fail, condeseq
from sympy.ntheory.generate import prime, isprime

# xが素数かどうかを判定する
def check_prime(x):
    if isvar(x):
        return condeseq([eq(x, p)] for p in map(prime, it.count(1)))
    else:
        return success if isprime(x) else fail

x = var()

list_nums = (23, 4, 27, 17, 13, 10, 21, 29, 3, 32, 11, 19)
print('List of primes in the list:')
print(set(run(0, x, (membero, x, list_nums), (check_prime, x))))
```

```python
print('List of first 7 prime numbers:')
print(run(7, x, (check_prime, x)))
```


# Input Driven HMM

This notebook is a simple example of an HMM with exogenous inputs.  The inputs modulate the probability of discrete state transitions via a multiclass logistic regression. Let $z_t \in \{1, \ldots, K\}$ denote the discrete latent state at time $t$ and $u_t \in \mathbb{R}^U$ be the exogenous input at time~$t$.  The transition probability is given by,

$$
\begin{align}
\Pr(z_t = k \mid z_{t-1} = j, u_t) =
\frac{\exp\{\log P_{j,k} + w_k^\mathsf{T} u_t\}}
{\sum_{k'=1}^K \exp\{\log P_{j,k'} + w_{k'}^\mathsf{T} u_t\}}.
\end{align}
$$

The parameters of the transition model are $P \in \mathbb{R}_+^{K \times K}$, a baseline set of (unnormalized) transition weights, and $W \in \mathbb{R}^{K \times U}$, a set of input weights.

```python
import numpy as np
import numpy.random as npr
import matplotlib.pyplot as plt
import ssm
from ssm.util import one_hot, find_permutation

%matplotlib inline

npr.seed(0)
```

```python
# Set the parameters of the HMM
T = 1000 # number of time bins
K = 2    # number of discrete states
D = 1    # data dimension
M = 1    # input dimension
C = 3    # number of output types/categories

# Make an HMM
true_hmm = ssm.HMM(K, D, M,
               observations="categorical", observation_kwargs=dict(C=C),
               transitions="inputdriven")

# Optionally, turn up the input weights to exaggerate the effect
# true_hmm.transitions.Ws *= 3

# Create an exogenous input
inpt = np.sin(2 * np.pi * np.arange(T) / 50)[:, None] + 1e-1 * npr.randn(T, M)

# Sample some data from the HMM
z, y = true_hmm.sample(T, input=inpt)

# Compute the true log probability of the data, summing out the discrete states
true_lp = true_hmm.log_probability(y, inputs=inpt)
```

```python
# Plot the data
plt.figure(figsize=(8, 5))

plt.subplot(311)
plt.plot(inpt)
plt.xticks([])
plt.xlim(0, T)
plt.ylabel("input")

plt.subplot(312)
plt.imshow(z[None, :], aspect="auto")
plt.xticks([])
plt.xlim(0, T)
plt.ylabel("discrete\nstate")
plt.yticks([])

plt.subplot(313)
plt.imshow(one_hot(y[:,0], C).T, aspect="auto")
plt.xlim(0, T)
plt.ylabel("observation")
plt.yticks(np.arange(C))
```

```python
# Now create a new HMM and fit it to the data with EM
N_iters = 50
hmm = ssm.HMM(K, D, M,
          observations="categorical", observation_kwargs=dict(C=C),
          transitions="inputdriven")

# Fit
hmm_lps = hmm.fit(y, inputs=inpt, method="em", num_em_iters=N_iters)
```

    HBox(children=(IntProgress(value=0, max=50), HTML(value='')))

```python
# Find a permutation of the states that best matches the true and inferred states
hmm.permute(find_permutation(z, hmm.most_likely_states(y, input=inpt)))
z_inf = hmm.most_likely_states(y, input=inpt)
```

```python
# Plot the log probabilities of the true and fit models
plt.plot(hmm_lps, label="EM")
plt.plot([0, N_iters], true_lp * np.ones(2), ':k', label="True")
plt.legend(loc="lower right")
plt.xlabel("EM Iteration")
plt.xlim(0, N_iters)
plt.ylabel("Log Probability")
```

```python
# Plot the true and inferred states
plt.figure(figsize=(8, 3.5))

plt.subplot(211)
plt.imshow(z[None, :], aspect="auto")
plt.xticks([])
plt.xlim(0, T)
plt.ylabel("true\nstate")
plt.yticks([])

plt.subplot(212)
plt.imshow(z[None, :], aspect="auto")
plt.xlim(0, T)
plt.ylabel("inferred\nstate")
plt.yticks([])
```

```python
# Plot the true and inferred input effects
plt.figure(figsize=(8, 4))

vlim = max(abs(true_hmm.transitions.log_Ps).max(),
           abs(true_hmm.transitions.Ws).max(),
           abs(hmm.transitions.log_Ps).max(),
           abs(hmm.transitions.Ws).max())

plt.subplot(141)
plt.imshow(true_hmm.transitions.log_Ps, vmin=-vlim, vmax=vlim, cmap="RdBu", aspect=1)
plt.xticks(np.arange(K))
plt.yticks(np.arange(K))
plt.title("True\nBaseline Weights")

plt.subplot(142)
plt.imshow(true_hmm.transitions.Ws, vmin=-vlim, vmax=vlim, cmap="RdBu", aspect=M/K)
plt.xticks(np.arange(M))
plt.yticks(np.arange(K))
plt.title("True\nInput Weights")

plt.subplot(143)
plt.imshow(hmm.transitions.log_Ps, vmin=-vlim, vmax=vlim, cmap="RdBu", aspect=1)
plt.xticks(np.arange(K))
plt.yticks(np.arange(K))
plt.title("Inferred\nBaseline Weights")

plt.subplot(144)
plt.imshow(hmm.transitions.Ws, vmin=-vlim, vmax=vlim, cmap="RdBu", aspect=M/K)
plt.xticks(np.arange(M))
plt.yticks(np.arange(K))
plt.title("Inferred\nInput Weights")
```


# Пример с $\Delta$-кинематикой

## Прямая задача

```python
from sympy import *

def rz(a):
    return Matrix([
        [cos(a), -sin(a), 0, 0],
        [sin(a), cos(a), 0, 0],
        [0, 0, 1, 0],
        [0, 0, 0, 1]
    ])

def ry(a):
    return Matrix([
        [cos(a), 0, sin(a), 0],
        [0, 1, 0, 0],
        [-sin(a), 0, cos(a), 0],
        [0, 0, 0, 1]
    ])

def rx(a):
    return Matrix([
        [1, 0, 0, 0],
        [0, cos(a), -sin(a), 0],
        [0, sin(a), cos(a), 0],
        [0, 0, 0, 1]
    ])

def trs(x, y, z):
    return Matrix([
        [1, 0, 0, x],
        [0, 1, 0, y],
        [0, 0, 1, z],
        [0, 0, 0, 1]
    ])

def vec(x, y, z):
    return Matrix([
        [x],
        [y],
        [z],
        [1]
    ])
```

Дельта-принтер:

[источник изображения](https://www.3dprintersonlinestore.com/delta-3d-printer-kit)

Возьмем $0$ в центре, $X$ направлено вправо, $Y$ - вглубь, $Z$ - вверх.

Для описания положения "ползунов" будем отчситывать их высоту относительно основания. Первый ползун - в глубине, второй - слева, третий - справа.

```python
q1, q2, q3 = symbols("q_1, q_2, q_3")
```

Обратим внимание, что плоский перенос тяг вдоль оси, проходящей через центр и ползун не будет влиять на решение кинематики.
За счет этого мысленно сместим тяги в центр подвижной платформы, и определим проекцию расстояния от крепления тяги до центра на горизонтальную плоскость.
Это расстояние, например, в прошивке Marlin называется `delta radius` и обозначается $R$:

```python
delta_radius = symbols("R")
```

Длину тяг (`diagonal rod`) обозначим как $L$:

```python
diagonal_rod = symbols("L")
```

Тогда, положение ползунов будет следующим:

```python
slider1 = vec(0, delta_radius, q1)
slider2 = rz(2*pi/3) * vec(0, delta_radius, q2)
slider3 = rz(4*pi/3) * vec(0, delta_radius, q3)
```

Положение центра подвижной платформы зададим как вектор-столбец:
$$
p =
\begin{bmatrix}
    x \\
    y \\
    z \\
    1
\end{bmatrix}
$$

```python
x, y, z = symbols("x, y, z")
p = vec(x, y, z)
```

Тогда, "очевидно" что расстояние от центра платформы до каждого ползуна должно равняться длине тяги, а квадрат расстояния - квадрату длины тяги:

```python
def distance_sq(a, b):
    return (a[0] - b[0]) ** 2 +\
        (a[1] - b[1]) ** 2 +\
        (a[2] - b[2]) ** 2
```

```python
condition1 = Eq(distance_sq(p, slider1), diagonal_rod**2)
condition2 = Eq(distance_sq(p, slider2), diagonal_rod**2)
condition3 = Eq(distance_sq(p, slider3), diagonal_rod**2)
```

```python
import time
start = time.time()
fk_solution = solve(
    [
        condition1,
        condition2,
        condition3
    ],
    [
        x,
        y,
        z
    ],
)
end = time.time()
print("Прошло секунд:", end - start)
```

Должно быть два решения:

```python
len(fk_solution)
```

Рассмотрим первое:

```python
fk_solution[0][0]
```

```python
fk_solution[0][1]
```

```python
fk_solution[0][2]
```

Теперь нарисуем какой-нибудь принтер и посмотрим на его движение:

```python
from matplotlib import pyplot as plt
from matplotlib import animation
import numpy as np
from IPython.display import HTML
%matplotlib notebook
```

```python
def fk(r, l, pos1, pos2, pos3):
    x = fk_solution[0][0].evalf(subs={
        delta_radius: r,
        diagonal_rod: l,
        q1: pos1,
        q2: pos2,
        q3: pos3
    })
    y = fk_solution[0][1].evalf(subs={
        delta_radius: r,
        diagonal_rod: l,
        q1: pos1,
        q2: pos2,
        q3: pos3
    })
    z = fk_solution[0][2].evalf(subs={
        delta_radius: r,
        diagonal_rod: l,
        q1: pos1,
        q2: pos2,
        q3: pos3
    })
    return (x, y, z)
```

```python
def generalized(t, total):
    omega = t / total * 2 * np.pi
    q1 = 350 + 25 * np.sin(omega)
    q2 = 350 + 25 * np.sin(omega + 2/3*np.pi)
    q3 = 350 + 25 * np.sin(omega + 4/3*np.pi)
    return (q1, q2, q3)
```

```python
delta_radius_ex = 124
diagonal_rod_ex = 337

fig = plt.figure()
ax = fig.add_subplot(projection="3d")

ax.set_xlim([-delta_radius_ex, delta_radius_ex])
ax.set_ylim([-delta_radius_ex, delta_radius_ex])
ax.set_zlim([0, 2 * delta_radius_ex])
ax.plot([0]*2, [delta_radius_ex] * 2, [0, 400], color="#000000")
ax.plot([-np.sqrt(3)/2*delta_radius_ex]*2, [-delta_radius_ex/2]*2, [0, 400], color="#000000")
ax.plot([np.sqrt(3)/2*delta_radius_ex]*2, [-delta_radius_ex/2]*2, [0, 400], color="#000000")
rod1, = ax.plot([], [], [], color="#ff0000")
rod2, = ax.plot([], [], [], color="#00ff00")
rod3, = ax.plot([], [], [], color="#0000ff")

total = 36

def animate(frame):
    q1, q2, q3 = generalized(frame, total)
    x, y, z = fk(delta_radius_ex, diagonal_rod_ex, q1, q2, q3)
    rod1.set_data_3d([0, x], [delta_radius_ex, y], [q1, z])
    rod2.set_data_3d([-np.sqrt(3)/2*delta_radius_ex, x], [-delta_radius_ex/2, y], [q2, z])
    rod3.set_data_3d([np.sqrt(3)/2*delta_radius_ex, x], [-delta_radius_ex/2, y], [q3, z])

ani = animation.FuncAnimation(
    fig,
    animate,
    frames=total,
    interval=1000.0/25
)
```

## Обратная задача

```python
start = time.time()
ik_solution = solve(
    [
        condition1,
        condition2,
        condition3
    ],
    [
        q1,
        q2,
        q3
    ],
)
end = time.time()
print("Прошло секунд:", end - start)
```

Решений должно быть $2^3$:

```python
len(ik_solution)
```

Рассмотрим восьмое:

```python
ik_solution[7][0]
```

```python
ik_solution[7][1]
```

```python
ik_solution[7][2]
```

Помоделируем наше решение:

```python
def ik(r, l, pos1, pos2, pos3):
    q1 = ik_solution[7][0].evalf(subs={
        delta_radius: r,
        diagonal_rod: l,
        x: pos1,
        y: pos2,
        z: pos3
    })
    q2 = ik_solution[7][1].evalf(subs={
        delta_radius: r,
        diagonal_rod: l,
        x: pos1,
        y: pos2,
        z: pos3
    })
    q3 = ik_solution[7][2].evalf(subs={
        delta_radius: r,
        diagonal_rod: l,
        x: pos1,
        y: pos2,
        z: pos3
    })
    return (q1, q2, q3)
```

```python
def position(t, total):
    omega = t / total
    r = delta_radius_ex / 2
    h = 50
    if omega < 0.25:
        tau = omega/0.25
        return (
            -r + r*tau*2,
            -r,
            h
        )
    elif omega < 0.5:
        tau = (omega - 0.25)/0.25
        return (
            r,
            -r + r*tau*2,
            h
        )
    elif omega < 0.75:
        tau = (omega - 0.5)/0.25
        return (
            r - r*tau*2,
            r,
            h
        )
    else:
        tau = (omega - 0.75)/0.25
        return (
            -r,
            r - r*tau*2,
            h
        )
```

```python
fig = plt.figure()
bx = fig.add_subplot(projection="3d")

bx.set_xlim([-delta_radius_ex, delta_radius_ex])
bx.set_ylim([-delta_radius_ex, delta_radius_ex])
bx.set_zlim([0, 2 * delta_radius_ex])
bx.plot([0]*2, [delta_radius_ex] * 2, [0, 400], color="#000000")
bx.plot([-np.sqrt(3)/2*delta_radius_ex]*2, [-delta_radius_ex/2]*2, [0, 400], color="#000000")
bx.plot([np.sqrt(3)/2*delta_radius_ex]*2, [-delta_radius_ex/2]*2, [0, 400], color="#000000")
rod1, = bx.plot([], [], [], color="#ff0000")
rod2, = bx.plot([], [], [], color="#00ff00")
rod3, = bx.plot([], [], [], color="#0000ff")

total = 100

def animate2(frame):
    x, y, z = position(frame, total)
    q1, q2, q3 = ik(delta_radius_ex, diagonal_rod_ex, x, y, z)
    rod1.set_data_3d([0, x], [delta_radius_ex, y], [q1, z])
    rod2.set_data_3d([-np.sqrt(3)/2*delta_radius_ex, x], [-delta_radius_ex/2, y], [q2, z])
    rod3.set_data_3d([np.sqrt(3)/2*delta_radius_ex, x], [-delta_radius_ex/2, y], [q3, z])

animate2(0)
ani2 = animation.FuncAnimation(
    fig,
    animate2,
    frames=total,
    interval=1000.0/25
)
```


## Exercise 1 (10 points): Linear Discriminant Analysis [Pen and Paper]

Not every feature dimension is equally useful for classification. Usually, some are more important to distinguish between classes than others. To evaluate the different dimensions, we can use linear discriminant analysis (LDA) which constructs a new feature space where the axes are designed to discriminate the classes as much as possible. In addition, LDA also reduces the
dimensions of our feature space which saves computational costs and can help to overcome overfitting.

In this exercise, we look at a two-dimensional example of a two-class problem (with the classes 𝜔1 and 𝜔2) and try to understand the basic concepts behind LDA. The next assignment then focusses on a more realistic example where LDA is used as a preprocessing step in a classification problem.

The general idea of LDA is to maximize the criterion function

\begin{equation} J(w) = \frac{\tilde{\mu}_1 - \tilde{\mu}_2}{\tilde{\sigma}_1^2 + \tilde{\sigma}_2^2}  =
\frac{w^TS_Bw}{w^TS_Ww}\end{equation}

$𝑤$ is the normalized vector representing the axes of the new feature space. $\tilde{\mu}_1, \tilde{\mu}_2$ and $\tilde{\sigma}_1^2, \tilde{\sigma}_2^2$ are the means and variances, respectively, of the two classes which are projected to the new feature space, i.e. projected on the vector $w$. Remember that an arbitrary point $x$ is projected on $w$ via

\begin{equation} x = w^Tx \end{equation}

$\tilde{x}$ denotes the one-dimensional value relative to the new coordinate system spanned by $w$. To maximize $J(w)$, we need the between-class scatter matrix

\begin{equation} S_B = (\mu_1 - \mu_2)(\mu_1 - \mu_2)^T \end{equation}

where $\mu_i = (\mu_{ix}, \mu_{iy}$ denotes the mean vectors of the two classes (column vectors). For our dataset, they are defined as (the complete dataset was centred around the origin)
$$ \mu_1 =
\begin{pmatrix}
-1.7\\
-1.20\\
\end{pmatrix}
\hspace{0.5cm}and\hspace{0.5cm}
\mu_2 =
\begin{pmatrix}
1.17\\
1.20\\
\end{pmatrix}
$$

Finally, for the within-scatter matrix 𝑆𝑊 we use the combined covariance matrix which simply sums up the covariance matrices of the two classes: $S_W = \sum = {\sum}_1 + {\sum}_2$.
1. We first want to take a look at the between-class scatter matrix $S_B$.

    a) Figure 2 shows the dataset we are using throughout this exercise and labels the horizontal and vertical distances between the mean vectors.

 \begin{equation} dx = (\mu_{1x} − \mu_{2x}) = −2.33 \hspace{0.5cm}and\hspace{0.5cm} dy = (\mu_{1y} − \mu_{2y}) = −2.40\end{equation}
Calculate all values of the matrix $S_B$ (Equation 3) without using the definitions of Equation 4 (but you can use it to check your result). Hint: start by deriving the general form of $S_B$ and then plug in $dx$ and $dy$.

    b) Building upon your previous results, what is the rank of $𝑆_𝐵$

2. Figure 3, Figure 4 and Figure 5 show the situation of the weight vectors $w_1$, $w_2$ and $w_3$, respectively. The vector $w_1$ of Figure 3 maximizes the distance between the projected means, $w_2 $ of Figure 4 minimizes it and $w_3$ of Figure 5 finally maximizes the criterion function of Equation 1. In the next steps, we want to calculate the corresponding missing values of Table 1.

    a) Start by mapping each line of Table 1 to the corresponding figure based on the values of the projected variances $\omega_1^2 \hspace{0.5cm}and\hspace{0.5cm} \omega_2^2$.

    b) Calculate the squared differences between the projected means $(\tilde{\mu}_1 - \tilde{\mu}_2)^2$. Hint: you can name one value directly, another can be calculated based on the definitions of Equation 4 and for yet another you need to project the mean values $\mu_1$ and $\mu_2$ first onto the new axis $w$ (cf. Equation 2).

    c) Find the missing vectors $w_1 $and $w_2$ (normalized).

    d) Finally, we arrive at the point where we can actually compute the values of the criterion function $J(w)$ of Equation 1. Calculate the missing values in the last column of Table 1. If done correctly, Figure 5 should correspond to the line where $J(w)$ is maximal.

3. Explain why it is not enough to maximize only the squared distance between the projected means when we want to achieve a good class discrimination (Figure 3 vs. Figure 5).

4. We can use the result of LDA to construct a new coordinate system with good classdiscrimination capabilities. To achieve this, we need to solve the regular eigenvalue problem
\begin{equation}S_W^-1 S_Bw = \lambda w. \end{equation}

Asking a computer system to calculate the eigenvalues $\lambda_i$ and corresponding eigenvectors $\mu_i$ for the matrix $S_W^{-1}S_B$ rewards us with

 \begin{equation}\lambda_1 = 14.15, \lambda_2 = 0 \hspace{0.5cm}and\hspace{0.5cm} \
\mu_1 =
\begin{pmatrix}
-0.97\\
-0.25\\
\end{pmatrix},
\mu_2 =
\begin{pmatrix}
-0.72\\
0.70\\
\end{pmatrix}
\end{equation}

As you can see, the result is already ascendingly sorted by the magnitude of the eigenvalues and the eigenvectors are normalized, i.e. $‖u_i‖ = 1$. Also, note that $u_1 = w_3$ which is not a coincidence since the vector in Figure 5 was chosen to maximize Equation 1.

    a) Why is $\lambda_2 = 0$? Is this only related to the specific dataset here or can you maybe infer a general rule? Hint: argue with what you found out about the rank of $𝑆_𝐵$ previously.

    b) We can use the first eigenvector and Equation 2 to project every data point onto the line of the new coordinate system. The result of this process can be seen graphically in Figure 1. Calculate for the two points

 $$ x_1 =
\begin{pmatrix}
-0.42\\
-0.38\\
\end{pmatrix} \in \omega_1
\hspace{0.5cm}and\hspace{0.5cm}
x_2 =
\begin{pmatrix}
1.88\\
1.08\\
\end{pmatrix} \in \omega_2
$$
 the projection onto $u_1$ numerically, i.e. the values $\tilde{x}_1$ and $\tilde{x}_2$.

    c) Based on the definition of the first eigenvector $u_1$ and the calculation you just performed, which of the original features, i.e. the axes 𝑥 and 𝑦, is more important for class discrimination?

    d) Which threshold $w_0 \in \mathbb{R}$ would you choose to classify new data points $x_i$ based on their projected value $\tilde{x}_i$?


# Numerical Integration

```python
from itertools import combinations
```

```python
import sympy
from sympy import Function, integrate, Product, Sum, Symbol, symbols
from sympy.abc import a,b,h,i,k,m,n,x
from sympy import Rational as Rat
```

```python
def lagrange_basis_polys(N,x,xpts=None):
    """
    lagrange_basis_polynomials(N,x,xpts)
    returns the Lagrange basis polynomials as a list

    INPUTS/PARAMETERS
    -----------------
    <int> N - N > 0.  Note that there are N+1 points total
    <sympy.Symbol> x
    <list> xpts
    """
    assert N > 0
    if xpts != None:
        assert len(xpts) == N + 1

    if xpts == None:
        print "I'll generate symbolic sympy symbols for you for xpts"
        xpts = symbols('x0:'+str(N+1))

    basis_polys = []
    for i in range(N+1):
        tmpprod = Rat(1)
        for k in [k for k in range(N+1) if k != i]:
            tmpprod = tmpprod * (x - xpts[k])/(xpts[i]-xpts[k])
        basis_polys.append(tmpprod)
    return basis_polys

def lagrange_interp(N,x,xpts=None,ypts=None):
    """
    lagrange_interp(N,x,xpts,ypts)
    Lagrange interpolation formula
    """
    if xpts != None and ypts != None:
        assert len(xpts) == len(ypts)

    if xpts == None:
        print "I'll generate symbolic sympy symbols for you for xpts"
        xpts = symbols('x0:'+str(N+1))
    if ypts == None:
        print "I'll generate symbolic sympy symbols for you for xpts"
        ypts = symbols('y0:'+str(N+1))

    basis = lagrange_basis_polys(N,x,xpts)
    p_N = sum( [ypts[i]*basis[i] for i in range(N+1)] )
    return p_N
```

```python
xpts = symbols('x0:'+str(1+1))
ypts = symbols('y0:'+str(1+1))
p_1x = lagrange_interp(1,x,xpts,ypts)
```

Below is, mathematically, $f_{-h} := p_1(x)$ with $(x_0,y_0) = (x_0-h, f(x_0-h)), (x_1,y_1) = (x_0,f(x_0))$ and
$\int_{x_0-h}^{x_0} f_{-h}$

```python
x_0 = Symbol('x_0',real=True)
f = Function('f')
f_minush = p_1x.subs({xpts[0]:x_0-h,xpts[1]:x_0, ypts[0]:f(x_0-h), ypts[1]:f(x_0) })
integrate( f_minush, (x,x_0-h,x_0 ) )
```

    -x_0**2*(-f(x_0) + f(-h + x_0))/(2*h) + x_0*(h*f(x_0) - x_0*f(x_0) + x_0*f(-h + x_0))/h + (-h + x_0)**2*(-f(x_0) + f(-h + x_0))/(2*h) - (-h + x_0)*(h*f(x_0) - x_0*f(x_0) + x_0*f(-h + x_0))/h

Then, we can use `sympy` to calculate, symbolically, $f_{h} := p_1(x)$ with $(x_0,y_0) = (x_0, f(x_0)), (x_1,y_1) = (x_0+h,f(x_0+h))$ and
$\int_{x_0}^{x_0+h} f_{h}$

```python
f_h = p_1x.subs({xpts[0]:x_0,xpts[1]:x_0+h, ypts[0]:f(x_0), ypts[1]:f(x_0+h) })
integrate( f_h, (x,x_0,x_0+h ) )
```

    -x_0**2*(-f(x_0) + f(h + x_0))/(2*h) - x_0*(h*f(x_0) + x_0*f(x_0) - x_0*f(h + x_0))/h + (h + x_0)**2*(-f(x_0) + f(h + x_0))/(2*h) + (h + x_0)*(h*f(x_0) + x_0*f(x_0) - x_0*f(h + x_0))/h

```python
( integrate( f_minush, (x,x_0-h,x_0 ) ) + integrate( f_h, (x,x_0,x_0+h ) )  ).simplify()
```

    h*(2*f(x_0) + f(-h + x_0) + f(h + x_0))/2

Success! Trapezoid rule was rederived (stop using pen/pencil and paper or chalkboard; computers can do computations faster and without mistakes)

For a second order polynomial, $p_{N=2}(x)$,

```python
xpts = symbols('x0:'+str(2+1))
ypts = symbols('y0:'+str(2+1))
p_2x = lagrange_interp(2,x,xpts,ypts)
```

```python
f2_h = p_2x.subs({xpts[0]:x_0-h,xpts[1]:x_0,xpts[2]:x_0+h,ypts[0]:f(x_0-h), ypts[1]:f(x_0),ypts[2]:f(x_0+h) })
```

```python
integrate( f2_h,(x,x_0-h,x_0+h)).simplify()
```

    h*(4*f(x_0) + f(-h + x_0) + f(h + x_0))/3

# Legendre Polynomials

I don't find the `sympy` documentation very satisfying (other than listing the argument syntax, no examples of usage, nor further explanation, beyond the barebones argument syntax, is given).  So what I've done here is to try to show what I've done.

```python
from sympy.polys.orthopolys import legendre_poly
```

```python
print "n \t \t \t \t P_n(x) \n"
for i in range(11):
    print str(i) + "\t \t \t \t " , legendre_poly(i,x)
```

    n 	 	 	 	 P_n(x)

    0	 	 	 	  1
    1	 	 	 	  x
    2	 	 	 	  3*x**2/2 - 1/2
    3	 	 	 	  5*x**3/2 - 3*x/2
    4	 	 	 	  35*x**4/8 - 15*x**2/4 + 3/8
    5	 	 	 	  63*x**5/8 - 35*x**3/4 + 15*x/8
    6	 	 	 	  231*x**6/16 - 315*x**4/16 + 105*x**2/16 - 5/16
    7	 	 	 	  429*x**7/16 - 693*x**5/16 + 315*x**3/16 - 35*x/16
    8	 	 	 	  6435*x**8/128 - 3003*x**6/32 + 3465*x**4/64 - 315*x**2/32 + 35/128
    9	 	 	 	  12155*x**9/128 - 6435*x**7/32 + 9009*x**5/64 - 1155*x**3/32 + 315*x/128
    10	 	 	 	  46189*x**10/256 - 109395*x**8/256 + 45045*x**6/128 - 15015*x**4/128 + 3465*x**2/256 - 63/256

```python
sympy.latex(legendre_poly(2,x))
```

    '\\frac{3 x^{2}}{2} - \\frac{1}{2}'

```python
sympy.N( sympy.integrate(1/(2+x**2),(x,0,3)) )
```

    0.799232657543987

```python

```


```python
import sys
import sympy
from sympy import *
from sympy.abc import x, n, z, t, k
from sympy.core.cache import *

clear_cache()

init_printing(use_latex='mathjax') # for nice printing, a-la' TeX

sys.setrecursionlimit(100000)
```

```python
from itertools import *
```

# Difference analysis of Polynomial sequences

Let $\{a_{n}\}_{n\in\mathbb{N}}$ be a sequence of Natural numbers. Assume that each coefficient $a_{n}$ can be written as a polynomial $\mathcal{A}_{k}$, of order $k$, in the variable $n$, namely:
$$ a_{n} = \hat{a}_{0} + \hat{a}_{1}n + \ldots + \hat{a}_{k}n^{k} = \mathcal{A}_{k}(n)$$
where $\hat{a}_{i}\in\mathbb{Q}$ and $\hat{a}_{k}\neq0$.

```python
def first_sequence_poly(n): return 1+n+2*n**2
```

```python
first_sequence_exploded = [first_sequence_poly(i) for i in range(14)]
first_sequence_exploded
```

$$\left [ 1, \quad 4, \quad 11, \quad 22, \quad 37, \quad 56, \quad 79, \quad 106, \quad 137, \quad 172, \quad 211, \quad 254, \quad 301, \quad 352\right ]$$

```python
def evaluate_poly_sym(polynomial):
    return lambda n: Poly(polynomial(n), polynomial.gen)

def difference(polynomial):
    return lambda n: (polynomial(n+1) - polynomial(n)).simplify()

def difs(sequence):

    if not sequence: return []
    elif len(sequence) is 1: return []
    else:
        n, m, *rest = sequence
        return [m-n] + difs([m]+rest)

def constant(l):
    if not l: return True
    else:
        car, *cdr = l
        return all(map(lambda a: car == a, cdr))

def difLists(lists):
    if not lists: return []
    else:
        l, *rest = lists
        if constant(l): return lists
        else: return difLists([difs(l)]+lists)

def firstDifs(sequence):
    return reversed(list(map(lambda l: l[0], difLists([sequence]))))

def genDifs(sequence):
    return reversed(list(map(lambda l: l[-1], difLists([sequence]))))

def nextD(lasts):
    new_coeff = list(accumulate(lasts))[-1]
    new_lasts = [new_coeff]
    for i in range(1,len(lasts)): new_lasts.append(new_lasts[i-1]-lasts[i-1])
    assert new_lasts[-1] == lasts[-1]
    return new_lasts

def next_element(sequence):
    return nextD(list(genDifs(sequence)))[0]

def continue_sequence(sequence):
    lasts = list(genDifs(sequence))
    while True:
        lasts = nextD(lasts)
        yield lasts[0]
```

```python
f=first_sequence_poly(n+1)
f
```

$$n + 2 \left(n + 1\right)^{2} + 2$$

```python
differentiated_poly = difference(first_sequence_poly)(n) # keep it simbolically
mapped_difference = map(lambda i: differentiated_poly.subs(n,i), range(13))
list(mapped_difference)
```

$$\left [ 3, \quad 7, \quad 11, \quad 15, \quad 19, \quad 23, \quad 27, \quad 31, \quad 35, \quad 39, \quad 43, \quad 47, \quad 51\right ]$$

```python
difs(first_sequence_exploded)
```

$$\left [ 3, \quad 7, \quad 11, \quad 15, \quad 19, \quad 23, \quad 27, \quad 31, \quad 35, \quad 39, \quad 43, \quad 47, \quad 51\right ]$$

```python
second_sequence = [-12, -11, 6, 45, 112, 213, 354, 541, 780, 1077]
```

```python
first_difference_sequence = difs(second_sequence)
first_difference_sequence
```

$$\left [ 1, \quad 17, \quad 39, \quad 67, \quad 101, \quad 141, \quad 187, \quad 239, \quad 297\right ]$$

```python
second_difference_sequence = difs(first_difference_sequence)
second_difference_sequence
```

$$\left [ 16, \quad 22, \quad 28, \quad 34, \quad 40, \quad 46, \quad 52, \quad 58\right ]$$

```python
third_difference_sequence = difs(second_difference_sequence)
third_difference_sequence
```

$$\left [ 6, \quad 6, \quad 6, \quad 6, \quad 6, \quad 6, \quad 6\right ]$$

```python
difLists([second_sequence])
```

$$\left [ \left [ 6, \quad 6, \quad 6, \quad 6, \quad 6, \quad 6, \quad 6\right ], \quad \left [ 16, \quad 22, \quad 28, \quad 34, \quad 40, \quad 46, \quad 52, \quad 58\right ], \quad \left [ 1, \quad 17, \quad 39, \quad 67, \quad 101, \quad 141, \quad 187, \quad 239, \quad 297\right ], \quad \left [ -12, \quad -11, \quad 6, \quad 45, \quad 112, \quad 213, \quad 354, \quad 541, \quad 780, \quad 1077\right ]\right ]$$

```python
list(firstDifs(second_sequence))
```

$$\left [ -12, \quad 1, \quad 16, \quad 6\right ]$$

```python
lasts = list(genDifs(second_sequence))
lasts
```

$$\left [ 1077, \quad 297, \quad 58, \quad 6\right ]$$

```python
nextD(lasts)
```

$$\left [ 1438, \quad 361, \quad 64, \quad 6\right ]$$

```python
next_element(second_sequence)
```

$$1438$$

```python
take(continue_sequence(second_sequence),20)
```

$$\left [ 1438, \quad 1869, \quad 2376, \quad 2965, \quad 3642, \quad 4413, \quad 5284, \quad 6261, \quad 7350, \quad 8557, \quad 9888, \quad 11349, \quad 12946, \quad 14685, \quad 16572, \quad 18613, \quad 20814, \quad 23181, \quad 25720, \quad 28437\right ]$$

```python
fibs = [fibonacci(i) for i in range(20)]
fibs
```

$$\left [ 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13, \quad 21, \quad 34, \quad 55, \quad 89, \quad 144, \quad 233, \quad 377, \quad 610, \quad 987, \quad 1597, \quad 2584, \quad 4181\right ]$$

```python
take(continue_sequence(fibs),20)
```

$$\left [ 13530, \quad 142065, \quad 1357325, \quad 9618830, \quad 54023390, \quad 254254950, \quad 1041111056, \quad 3806846406, \quad 12666486037, \quad 38897569993, \quad 111464146280, \quad 300666911313, \quad 768860983418, \quad 1874805032081, \quad 4380558827899, \quad 9848298629580, \quad 21378856330309, \quad 44949237172189, \quad 91775250522398, \quad 182390568524187\right ]$$

```python
difLists([fibs])
```

$$\left [ \left [ 4181\right ], \quad \left [ -2584, \quad 1597\right ], \quad \left [ 1597, \quad -987, \quad 610\right ], \quad \left [ -987, \quad 610, \quad -377, \quad 233\right ], \quad \left [ 610, \quad -377, \quad 233, \quad -144, \quad 89\right ], \quad \left [ -377, \quad 233, \quad -144, \quad 89, \quad -55, \quad 34\right ], \quad \left [ 233, \quad -144, \quad 89, \quad -55, \quad 34, \quad -21, \quad 13\right ], \quad \left [ -144, \quad 89, \quad -55, \quad 34, \quad -21, \quad 13, \quad -8, \quad 5\right ], \quad \left [ 89, \quad -55, \quad 34, \quad -21, \quad 13, \quad -8, \quad 5, \quad -3, \quad 2\right ], \quad \left [ -55, \quad 34, \quad -21, \quad 13, \quad -8, \quad 5, \quad -3, \quad 2, \quad -1, \quad 1\right ], \quad \left [ 34, \quad -21, \quad 13, \quad -8, \quad 5, \quad -3, \quad 2, \quad -1, \quad 1, \quad 0, \quad 1\right ], \quad \left [ -21, \quad 13, \quad -8, \quad 5, \quad -3, \quad 2, \quad -1, \quad 1, \quad 0, \quad 1, \quad 1, \quad 2\right ], \quad \left [ 13, \quad -8, \quad 5, \quad -3, \quad 2, \quad -1, \quad 1, \quad 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5\right ], \quad \left [ -8, \quad 5, \quad -3, \quad 2, \quad -1, \quad 1, \quad 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13\right ], \quad \left [ 5, \quad -3, \quad 2, \quad -1, \quad 1, \quad 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13, \quad 21, \quad 34\right ], \quad \left [ -3, \quad 2, \quad -1, \quad 1, \quad 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13, \quad 21, \quad 34, \quad 55, \quad 89\right ], \quad \left [ 2, \quad -1, \quad 1, \quad 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13, \quad 21, \quad 34, \quad 55, \quad 89, \quad 144, \quad 233\right ], \quad \left [ -1, \quad 1, \quad 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13, \quad 21, \quad 34, \quad 55, \quad 89, \quad 144, \quad 233, \quad 377, \quad 610\right ], \quad \left [ 1, \quad 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13, \quad 21, \quad 34, \quad 55, \quad 89, \quad 144, \quad 233, \quad 377, \quad 610, \quad 987, \quad 1597\right ], \quad \left [ 0, \quad 1, \quad 1, \quad 2, \quad 3, \quad 5, \quad 8, \quad 13, \quad 21, \quad 34, \quad 55, \quad 89, \quad 144, \quad 233, \quad 377, \quad 610, \quad 987, \quad 1597, \quad 2584, \quad 4181\right ]\right ]$$

```python

```


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Cumulative Distribution Functions

A random process can be characterized by the statistical properties of its amplitude values. [Cumulative distribution functions](https://en.wikipedia.org/wiki/Cumulative_distribution_function) (CDFs) are one possibility to do so.

### Univariate Cumulative Distribution Function

The univariate CDF $P_x(\theta, k)$ of a continuous-amplitude real-valued random signal $x[k]$ is defined as

\begin{equation}
P_x(\theta, k) := \Pr \{ x[k] \leq \theta\}
\end{equation}

where $\Pr \{ \cdot \}$ denotes the probability that the given condition holds. The univariate CDF quantifies the probability that for the entire ensemble and for a fixed time index $k$ the amplitude $x[k]$ is smaller or equal to $\theta$. The term '*univariate*' reflects the fact that only one random process is considered.

The CDF shows the following properties which can be concluded directly from its definition

\begin{equation}
\lim_{\theta \to -\infty} P_x(\theta, k) = 0
\end{equation}

and

\begin{equation}
\lim_{\theta \to \infty} P_x(\theta, k) = 1
\end{equation}

The former property results from the fact that all amplitude values $x[k]$ are larger than $- \infty$, the latter from the fact that all amplitude values lie within $- \infty$ and $\infty$. The univariate CDF $P_x(\theta, k)$ is furthermore a non-decreasing function.

The probability that $\theta_1 < x[k] \leq \theta_2$ is given as

\begin{equation}
\Pr \{\theta_1 < x[k] \leq \theta_2\} = P_x(\theta_2, k) - P_x(\theta_1, k)
\end{equation}

Hence, the probability that a continuous-amplitude random signal takes a specific value $x[k]=\theta$ is zero when calculated by means of the CDF. This motivates the definition of probability density functions introduced later.

### Bivariate Cumulative Distribution Function

The statistical dependencies between two signals are frequently of interest in statistical signal processing. The bivariate or joint CDF $P_{xy}(\theta_x, \theta_y, k_x, k_y)$ of two continuous-amplitude real-valued random signals $x[k]$ and $y[k]$ is defined as

\begin{equation}
P_{xy}(\theta_x, \theta_y, k_x, k_y) := \Pr \{ x[k_x] \leq \theta_x  \wedge y[k_y] \leq \theta_y  \}
\end{equation}

The joint CDF quantifies the probability for the entire ensemble of sample functions that for a fixed $k_x$ the amplitude value $x[k_x]$ is smaller or equal to $\theta_x$ and that for a fixed $k_y$ the amplitude value $y[k_y]$ is smaller or equal to $\theta_y$. The term '*bivariate*' reflects the fact that two random processes are considered. The bivariate CDF can also be used to characterize the statistical properties of one random signal $x[k]$ at two different time-instants $k_x$ and $k_y$ by setting $y[k] = x[k]$

\begin{equation}
P_{xx}(\theta_1, \theta_2, k_1, k_2) := \Pr \{ x[k_1] \leq \theta_1  \wedge y[k_2] \leq \theta_2  \}
\end{equation}

The definition of the bivariate CDF can be extended straightforward to the case of more than two random variables. The resulting CDF is termed as multivariate CDF.

## Probability Density Functions

[Probability density functions](https://en.wikipedia.org/wiki/Probability_density_function) (PDFs) describe the probability for one or multiple random signals to take on a specific value. Again the univariate case is discussed first.

### Univariate Probability Density Function

The univariate PDF $p_x(\theta, k)$ of a continuous-amplitude real-valued random signal $x[k]$ is defined as the derivative of the univariate CDF

\begin{equation}
p_x(\theta, k) = \frac{\partial}{\partial \theta} P_x(\theta, k)
\end{equation}

Due to the properties of the CDF and the definition of the PDF, it shows the following properties

\begin{equation}
p_x(\theta, k) \geq 0
\end{equation}

and

\begin{equation}
\int\limits_{-\infty}^{\infty} p_x(\theta, k) \, \mathrm{d}\theta = P_x(\infty, k) = 1
\end{equation}

The univariate PDF has only positive values and the area below the PDF is equal to one.

Due to the definition of the PDF as derivative of the CDF, the CDF can be computed from the PDF by integration

\begin{equation}
P_x(\theta, k) = \int\limits_{-\infty}^{\theta} p_x(\theta, k) \, \mathrm{d}\theta
\end{equation}

#### Example - Estimate of an univariate PDF by the histogram

In the process of calculating a [histogram](https://en.wikipedia.org/wiki/Histogram), the entire range of amplitude values of a random signal is split into a series of intervals (bins). For a given random signal the number of samples is counted which fall into one of these intervals. This is repeated for all intervals. The counts are finally normalized with respect to the total number of samples. This process constitutes a numerical estimation of the PDF of a random process.

In the following example the histogram of an ensemble of random signals is computed for each time index $k$. The CDF is computed by taking the cumulative sum over the histogram bins. This constitutes a numerical approximation of above integral

\begin{equation}
\int\limits_{-\infty}^{\theta} p_x(\theta, k) \approx \sum_{i=0}^{N} p_x(\theta_i, k) \, \Delta\theta_i
\end{equation}

where $p_x(\theta_i, k)$ denotes the $i$-th bin of the PDF and $\Delta\theta_i$ its width.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

K = 32  # number of temporal samples
N = 10000  # number of sample functions
bins = 100  # number of bins for the histogram

# draw sample functions from a random process
np.random.seed(2)
x = np.random.normal(size=(N, K))
x += np.tile(np.cos(2*np.pi/K*np.arange(K)), [N, 1])

# compute the histogram
px = np.zeros((bins, K))
for k in range(K):
    px[:, k], edges = np.histogram(x[:, k], bins=bins, range=(-4,4), density=True)

# compute the CDF
Px = np.cumsum(px, axis=0) * 8/bins

# plot the PDF
plt.figure(figsize=(10,6))
plt.pcolor(np.arange(K), edges, px)
plt.title(r'Estimated PDF $\hat{p}_x(\theta, k)$')
plt.xlabel(r'$k$')
plt.ylabel(r'$\theta$')
plt.colorbar()
plt.autoscale(tight=True)

# plot the CDF
plt.figure(figsize=(10,6))
plt.pcolor(np.arange(K), edges, Px, vmin=0, vmax=1)
plt.title(r'Estimated CDF $\hat{P}_x(\theta, k)$')
plt.xlabel(r'$k$')
plt.ylabel(r'$\theta$')
plt.colorbar()
plt.autoscale(tight=True)
```

**Exercise**

* Change the number of sample functions `N` or/and the number of `bins` and rerun the examples. What changes? Why?

In numerical simulations of random processes only a finite number of sample functions and temporal samples can be considered. This holds also for the number of intervals (bins) used for the histogram. As a result, numerical approximations of the CDF/PDF will be subject to statistical uncertainties that typically will become smaller if the number of sample functions `N` is increased.

### Bivariate Probability Density Function

The bivariate or joint PDF $p_{xy}(\theta_x, \theta_y, k_x, k_y)$ of two continuous-amplitude real-valued random signals $x[k]$ and $y[k]$ is defined as

\begin{equation}
p_{xy}(\theta_x, \theta_y, k_x, k_y) := \frac{\partial^2}{\partial \theta_x \partial \theta_y} P_{xy}(\theta_x, \theta_y, k_x, k_y)
\end{equation}

The bivariate PDF quantifies the joint probability that $x[k]$ takes the value $\theta_x$ and that $y[k]$ takes the value $\theta_y$ for the entire ensemble of sample functions.

If $x[k] = y[k]$ the bivariate PDF $p_{xx}(\theta_1, \theta_2, k_1, k_2)$ describes the probability that the random signal $x[k]$ takes the value $\theta_1$ at time instance $k_1$ and the value $\theta_2$ at time instance $k_2$. Hence, $p_{xx}(\theta_1, \theta_2, k_1, k_2)$ provides insights into the temporal dependencies of a random signal $x[k]$.

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016*.


# 高校数学でわかる線形代数 (:41)

```python
# 3行の拡大係数行列
from sympy import * # NumPyでは、xやy等の変数を含んだ行列計算は出来ないのでsympy
k = Symbol('k')
A = Matrix([[k,0,0],[0,1,0],[0,0,1]])
B = Matrix([[0,1,0,50],[1,1,0,150],[0,0,1,70]])
A*B
```

    Matrix([
    [0, k, 0, 50*k],
    [1, 1, 0,  150],
    [0, 0, 1,   70]])

```python
# 1行目をk倍して3行目に足すのはどうでしょうか?
C = Matrix([[1,0,0],[0,1,0],[k,0,1]])
D = Matrix([[1,2,0,50],[1,1,0,150],[2,3,1,70]])
C*D
```

    Matrix([
    [    1,       2, 0,        50],
    [    1,       1, 0,       150],
    [k + 2, 2*k + 3, 1, 50*k + 70]])

## 行の基本変形を行列の掛け算に置き換える(:42)

```python
# 3行目-2行目
E = Matrix([[1,0,0],[0,1,0],[0,-1,1]])
F = Matrix([[1,2,1,270],[1,1,0,150],[1,1,1,220]])
L = E*F
L
```

    Matrix([
    [1, 2, 1, 270],
    [1, 1, 0, 150],
    [0, 0, 1,  70]])

```python
# 1行目-3行目
G = Matrix([[1,0,-1],[0,1,0],[0,0,1]])

M = G*L
M
```

    Matrix([
    [1, 2, 0, 200],
    [1, 1, 0, 150],
    [0, 0, 1,  70]])

```python
# 1行目-2行目
H = Matrix([[1,-1,0],[0,1,0],[0,0,1]])

N = H*M
N
```

    Matrix([
    [0, 1, 0,  50],
    [1, 1, 0, 150],
    [0, 0, 1,  70]])

```python
# 2行目-1行目
I = Matrix([[1,0,0],[-1,1,0],[0,0,1]])

O = I*N
O
```

    Matrix([
    [0, 1, 0,  50],
    [1, 0, 0, 100],
    [0, 0, 1,  70]])

```python
# 1行目と2行目の交換
J = Matrix([[0,1,0],[1,0,0],[0,0,1]])

P = J*O
P
```

    Matrix([
    [1, 0, 0, 100],
    [0, 1, 0,  50],
    [0, 0, 1,  70]])

# 複数の行の基本変形を1つの行列にまとめよう
このように，「行列のかけ算」で「行の基本変形」が行えるということは，
このすべてのかけ算を行列 $\boldsymbol{ A }$ に左からかけると，
ここで見たように，拡大係数行列の左側部分の小行列は単位行列になることを意味しています．
実際にここで使った行列のかけ算をしてみると

```python
Q = J*I*H*G*E*F
Q
```

    Matrix([
    [1, 0, 0, 100],
    [0, 1, 0,  50],
    [0, 0, 1,  70]])

となり，拡大係数行列の左側部分は単位行列になっています．
この左側の $5$ つの行列のかけ算を求めてみましょう．

```python
R = J*I*H*G*E
R
```

    Matrix([
    [-1,  1,  1],
    [ 1,  0, -1],
    [ 0, -1,  1]])

この行列を $\boldsymbol{A'}$ と書くことにして，$\boldsymbol{A'}$ を $\boldsymbol{A}$ にかけると単位行列 $\boldsymbol{E}$ になるので，この関係は

```python

```


# Exercise from PS: Adverse selection with heterogenous ability

## The setup:

* Continuum of risk-neutral entrepreneurs (same initial $A$)

* Investment cost of cost $I$ (where $I>A$) yields $R$ in return w. probability $p$. Otherwise $0$ return.

* Agents differ in the probability of success $p$. Is distributed on $[0,1]$ according to the pdf function $f(p_i)$. (As usual $F(p)$ is the share of agents w. $p_i\leq p$. $f(p)$ is the marginal change in that probability around $p_i=p$.)

* Total supply of loanable funds is simply presented by the increasing function $S(r)$.

* This $S(r)$ is supplied by a competetive market (e.g. banks).

* Assume perfect information.

* Compared to before: No assumption of private benefits from not behaving.

## Q1: Individual rationality constraint, given that the agent has the choice between investment $I$ or depositing $A$.

This is straightforwardly given by:

$$\begin{align}
    p(R-R_b(p)) \geq (1+r)A \tag{IR}
\end{align}$$

Note that we assume that the payment $R_b$ may depend on $p$, as we assume **perfect information** for now. Note here:

* The LHS denotes profits from the investments after paying $R_b(p)$ on loan if successful. Otherwise zero.

* The RHS denotes the value of depositing $A$ at interest rate $r$.

## Q2: Write equations determining the riskless rate $r$, the identity of the marginal investor, and the zero (marginal) profit for the bank.

Let us start by imposing **zero profits** for the banks:

$$ \begin{align}
pR_b(p) = (1+r)(I-A), && \forall p \tag{ZP}
\end{align}$$

where $pR_b(p)$ is the payment if the project is successful, and the cost is the alternative cost of the loan: $(1+r)(I-A)$. Denote for now $B\equiv I-A$ the loan given by the bank. Note once again that the perfect information assumption implies that the zero profits hold for all $p$.

The **marginal investor** is given by the $\bar{p}$, where the IR constraint exactly holds. Imposing the zero profits condition we then have:

$$\begin{align}
  \bar{p}R-\underbrace{(1+r)(I-A)}_{\equiv \bar{p}R_b(\bar{p})\text{ from (ZP)}}=(1+r)A && \Rightarrow && \bar{p}(r) = \dfrac{1+r}{R}I
\end{align}$$

Finally the **riskless rate (r)** (it is riskless, due to the perfect information assumption) is determined by the equilibrium condition; i.e. that demand equals supply. With the threshold $\bar{p}$ all entrepreneurs who have a larger probability of success than the level $\bar{p}$, where profits are zero and (IR) holds with equality.

Recall that $f(p)$ denoted the distribution of the entrepreneurs. Thus demand for credit is extended to all entrepreneurs with $p\geq \bar{p}$, or:

$$\begin{align}
    D(r) = \int_{\bar{p}(r)}^1(I-A)f(p)\mbox{ }dp
\end{align}$$

Supply is given by the *aggregate supply function* $S(r)$, as well as the initial wealth $(A)$ from the entrepreneurs who do not obtain credit to carry out their investment. Define this new *total* supply function $\tilde{S}(r)$:

$$\begin{align}
    \tilde{S}(r) = S(r) + \int_0^{\bar{p}(r)} A f(p)\mbox{ }dp
\end{align}$$

## Q3: Confirm that only projects with positive net present value are carried out.

This is given, as banks have a zero profits condition, and entrepreneurs have the IR condition.

## Q4: Assume p is not observable. Compute equilibrium lending contract, the idendity of the marginal investor and show that there is overinvestment.

Now that the banks cannot observe $p$, they cannot make the contracts contingent on $p$. Thus they can only propose one contract, offering $R_b^*$ if the project is successful. Let $p^*$ denote the marginal investor now. From the IR constraint (with equality) we get:

$$\begin{align}
    p^*(R-R^*_b)=(1+r)A \tag{IR*}
\end{align}$$

Zero profits for the bank:

$$\begin{align}
    \int_{p^*}^1 p R_b^*f(p)\mbox{ }dp = \left(1- F(p^*)\right)(1+r)(I-A).
\end{align}$$

The LHS gives the payout to successful projects. The RHS is the alternative cost of the loan contracts. Note that $1-F(p^*)$ is the share of entrepreneurs with $p\geq p^*$.

Note that we can define this as the **conditional expected value**:

$$\begin{align}
    \dfrac{\int_{p^*}^1 p R_b^*f(p)\mbox{ }dp}{1- F(p^*)} \equiv R_b^* \mathbb{E}\left[p|p\geq p^*\right] = (1+r)(I-A).  \tag{ZP*}
\end{align}$$

Combining the zero profits, and the marginal investor identity we then have:

$$\begin{align}
        (1+r)A&= p^*R-p^*R_b^* \\
              &= p^*R-R_b^* \mathbb{E}\left[p|p\geq p^*\right] \dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]} \\
              &= p^*R-(1+r)*(I-A)*\dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]}
\end{align}$$

where the last line uses $(IR^*)$.

Rearranging this expression we get:

$$\begin{align}
    p^*R = I (1+r)\dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]}+A(1+r)\left[1-\dfrac{p^*}{\mathbb{E}\left[p|p\geq p^*\right]}\right]\leq (1+r)I \tag{Marginal*}
\end{align}$$

The RHS is a linear combination of $(1+r)I$ and $(1+r)A$. Compared to this the marginal investor was defined by $\bar{p}R=(1+r)I$ previously. As $I>A$ this establishes the inequality in (Marginal$^*$). Finally this entails directly that $p^*\leq \bar{p}$.

This inequality entails **overinvestment**:

* When $p$ is observed by the bank, the marginal investor is identified, such that *marginal* profits are zero.

* When $p$ is not observed by the bank, the marginal investor is identified, such that *average* profits are zero.

* Thus credit is extended to types that are unprofitable, $p^*\leq \bar{p}$.