# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Auto-Correlation Function

[Correlation](https://en.wikipedia.org/wiki/Correlation_and_dependence) is a statistical measure for the dependencies between random processes or between the samples of one random process. The [auto-correlation function](https://en.wikipedia.org/wiki/Autocorrelation) (ACF) characterizes the temporal dependencies within one random signal $x[k]$. It is an important measure for the analysis of signals in communications engineering, source coding and system identification.

### Definition

For a continuous-amplitude real-valued random signal $x[k]$ the ACF is defined by the [second-order ensemble avarage](ensemble_averages.ipynb#Second-Order-Ensemble-Averages) of the signal at two different time-instants $k_1$ and $k_2$

\begin{equation}
\varphi_{xx}[k_1, k_2] = E\{ x[k_1] \cdot x[k_2] \}
\end{equation}

Under the assumption of [wide-sense stationarity](stationary_ergodic.ipynb#Wide-Sense-Stationary-Random-Processes) (WSS) the ACF does only depend on the difference $\kappa = k_1 - k_2$ between the considered sample indexes

\begin{equation}
\varphi_{xx}[\kappa] = E\{x[k] \cdot x[k-\kappa] \}
\end{equation}

where $\kappa$ is commonly chosen as sample index instead of $k$ in order to indicate that it denotes a shift/lag. The ACF quantifies the similarity of a signal with a shifted version of itself. It has high values for high similarity and low values for low similarity.

If the process is additionally [wide-sense ergodic](stationary_ergodic.ipynb#Wide-Sense-Ergodic-Random-Processes), the ACF can be computed by averaging along one sample function

\begin{equation}
\varphi_{xx}[\kappa] = \lim_{K \to \infty} \frac{1}{2K + 1} \sum_{k=-K}^{K} x[k] \cdot x[k-\kappa]
\end{equation}

Note that the normalization on the left side of the sum is discarded in some definitions of the ACF. Above summation resembles strongly the definition of the [discrete convolution](https://en.wikipedia.org/wiki/Convolution#Discrete_convolution). For a random signal $x_N[k] = \text{rect}_N[k] \cdot x[k]$ of finite length $N$ and by exploiting the properties of a wide-sense ergodic random process one yields

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{N} \sum_{k=0}^{N-1} x_N[k] \cdot x_N[k-\kappa] = \frac{1}{N} \; x_N[k] * x_N[-k]
\end{equation}

where the ACF $\varphi_{xx}[\kappa] = 0$ for $|\kappa| > N-1$. Hence, the ACF can be computed by (fast) convolution of the random signal with a time reversed version of itself.

Note in numerical implementations (e.g. MATLAB, Python), the computed ACF is stored in a vector of length $2 N - 1$. The positive indexes $0, 1, \dots, 2 N - 1$ of this vector cannot be directly interpreted as $\kappa$. The indexes of the vector have to be shifted by $N-1$ for a proper interpretation.

### Properties

The following properties of the ACF can be deduced from its definition

1. The ACF $\varphi_{xx}[\kappa]$ has a maximum for $\kappa = 0$. It is given as

    $$ \varphi_{xx}[0] = E\{x^2[k]\} = \sigma_x^2 + \mu_x^2 $$

    This is due to the fact that the signal is equal to itself for $\kappa = 0$. Please note that for periodic random signals more than one maximum will be present.

2. The ACF is a function with even symmetry

    $$ \varphi_{xx}[\kappa] = \varphi_{xx}[-\kappa] $$

3. For typical random signals, the ACF approaches the limiting value

    $$ \lim_{|\kappa| \to \infty} \varphi_{xx}[\kappa] = \mu_x^2 $$

    The similarity of a typical random signal is often low for large lags $\kappa$.

4. The ACF of a periodic signal $x[k] = x[k + P]$ is also periodic

    $$ \varphi_{xx}[\kappa] = \varphi_{xx}[\kappa + P] $$

    with the period $P \in \mathbb{N} \setminus 0$

A random signal $x[k]$ is said to be *uncorrelated* if

\begin{equation}
\varphi_{xx}[\kappa] = \mu_x^2 + \sigma_x^2 \cdot \delta[\kappa]
\end{equation}

and as correlated if this condition is not met. The samples of a signal which is uncorrelated show no dependencies between each other in a statistical sense.

### Example - Auto-Correlation Function of a Periodic Signal

A periodic signal $x[k]$ can be expressed as

\begin{equation}
x[k] = x_0[k] * \sum_{n = - \infty}^{\infty} \delta[k - nP]
\end{equation}

where $x_0[k]$ denotes the signal of one period. The DTFT $X(e^{j \Omega})$ of a periodic signal consists of a series of equidistant Dirac impulses (line spectrum). Expressing the ACF as convolution, transforming this into the frequency domain by the DTFT and having the multiplication property of the Dirac delta in mind it can be concluded that the ACF is periodic with period $P$. The ACF is then given by convolution over one period only and periodic continuation

\begin{equation}
\varphi_{xx}[\kappa] = \frac{1}{N} \left( x_0[\kappa] * x_0[-\kappa] \right) * \sum_{n = - \infty}^{\infty} \delta[k - nP]
\end{equation}

The ACF of a periodic signal is computed and plotted in the following.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

N = 8192  # total length of signal
P = 128  # period
K = 275  # upper/lower limit for lag in ACF

# generate periodic random signal
np.random.seed(1)
x0 = np.random.normal(size=P)
x = np.tile(x0, N//P)

# compute and truncate ACF
acf = 1/len(x) * np.correlate(x, x, mode='full')
acf = acf[(len(x)-1)-(K-1):(len(x)-1)+K]
kappa = np.arange(-(K-1), K)

# plot signal and its ACF
plt.figure(figsize = (10, 4))
plt.stem(x[:512])
plt.xlabel(r'$k$')
plt.ylabel(r'$x[k]$')
plt.grid()

plt.figure(figsize = (10, 4))
plt.stem(kappa, acf)
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$\hat{\varphi}_{xx}[\kappa]$')
plt.axis([-K, K, 1.1*min(acf), 1.1*max(acf)]);
plt.grid()
```

**Exercise**

* What period $P$ does the periodic random signal have?

Solution: The period of the random signal can be estimated from the ACF, as both have the same period. This is most easy by inspection of the regular peaks of the ACF.

### Example - Auto-Correlation Function of a Speech Signal

The following example estimates and plots the ACF of a short recorded speech signal.

```python
from scipy.io import wavfile

K = 30  # upper/lower limit for lag in ACF

# read audio file
fs, x = wavfile.read('../data/speech_8k.wav')
x = np.asarray(x, dtype=float)/2**15

# compute and truncate ACF
acf = 1/len(x) * np.correlate(x, x, mode='full')
acf = acf[(len(x)-1)-(K-1):(len(x)-1)+K]
kappa = np.arange(-(K-1), K)

# plot ACF
plt.figure(figsize = (10, 8))
plt.stem(kappa, acf)
plt.xlabel(r'$\kappa$')
plt.ylabel(r'$\hat{\varphi}_{xx}[\kappa]$')
plt.axis([-K, K, 1.1*min(acf), 1.1*max(acf)]);
plt.grid()
```

**Exercise**

* Does the ACF fulfill the properties stated above?
* The plot shows only a small part of the ACF. Increase the range `K` for $-K \leq \kappa < K$ and check if the last property is fulfilled for the speech signal.
* Is the speech signal uncorrelated or correlated?

Solution: It can be observed that the plotted ACF has its maximum value at $\kappa = 0$ and that it shows an even symmetry. The third property can be confirmed by increasing `K` in above example. The speech signal is correlated since $\varphi_{xx}[\kappa] \neq 0$ for $\kappa \neq 0$.

## Auto-Covariance Function

The [auto-covariance function](https://en.wikipedia.org/wiki/Covariance) is defined as the ACF of a zero-mean random signal. For a random signal $x[k]$ with linear mean $\mu_x \neq 0$ it is given as

\begin{equation}
\psi_{xx}[\kappa] = \varphi_{xx}[\kappa] - \mu_x^2
\end{equation}

## Cross-Correlation Function

The cross-correlation function (CCF) is a measure of similarity that two random signals $x[k]$ and $y[k]$ show in a statistical sense.

### Definition

For a continuous-amplitude real-valued random signal $x[k]$ the CCF is defined by the [second-order ensemble avarage](ensemble_averages.ipynb#Second-Order-Ensemble-Averages) of the two signals $x[k]$ and $y[k]$ at two different time-instants $k_x$ and $k_y$

\begin{equation}
\varphi_{xy}[k_x, k_y] = E\{ x[k_x] \cdot y[k_y] \}
\end{equation}

Under the assumption of [wide-sense stationarity](stationary_ergodic.ipynb#Wide-Sense-Stationary-Random-Processes) (WSS) the CCF does only depend on the difference $\kappa = k_x - k_y$ between the considered sample indexes

\begin{equation}
\varphi_{xy}[\kappa] = E\{x[k] \cdot y[k - \kappa] \} = E\{x[k + \kappa] \cdot y[k] \}
\end{equation}

The cross-correlation function (CCF) is a measure of similarity that two random signals $x[k]$ and $y[k - \kappa]$ have with respect to the shift $\kappa \in \mathbb{Z}$. If $x[k]$ and $y[k]$ are wide-sense ergodic processes, the CCF can be computed by averaging along one sample function

\begin{equation}
\varphi_{xy}[\kappa] = \lim_{K \to \infty} \frac{1}{2K + 1} \sum_{k=-K}^{K} x[k] \cdot y[k-\kappa]
\end{equation}

For random signals $x_N[k] = \text{rect}_N[k] \cdot x[k]$ and $y_M[k] = \text{rect}_M[k] \cdot y[k]$ of finite lengths $N$ and $M$ one yields

\begin{equation}
\varphi_{xy}[\kappa] = \frac{1}{N} \sum_{k=0}^{N-1} x_N[k] \cdot y_M[k-\kappa] = \frac{1}{N} \; x_N[k] * y_M[-k]
\end{equation}

where the CCF $\varphi_{xy}[\kappa] = 0$ for $\kappa < -(M-1)$ and $\kappa > N-1$. The CCF can be computed by (fast) convolution of one random signal with a time reversed version of the other random signal. Note in numerical implementations (e.g. MATLAB, Python), the computed CCF is stored in a vector of length $N + M - 1$. The positive indexes $0, 1, \dots, N + M - 1$ of this vector cannot be directly interpreted as $\kappa$. The indexes of the vector have to be shifted by $M-1$ for a proper interpretation.

Above results hold also for the CCF $\varphi_{yx}[\kappa]$ when exchanging $x[k]$ with $y[k]$ and $N$ with $M$.

### Properties

1. When exchanging the two random signals, the CCF exhibits the following symmetry

    $$ \varphi_{xy}[\kappa] = \varphi_{yx}[-\kappa] $$

2. For typical random processes, the CCF approaches the limiting value

    $$ \lim_{|\kappa| \to \infty} \varphi_{xy}[\kappa] = \mu_x \cdot \mu_y$$

Two random signals are said to be [*uncorrelated*](https://en.wikipedia.org/wiki/Uncorrelated_random_variables) if

\begin{equation}
\varphi_{xy}[\kappa] = \mu_x \cdot \mu_y
\end{equation}

and as correlated if this condition is not met. The samples of two signals which are uncorrelated to each other show no dependencies between each other in a statistical sense. $\varphi_{xy}[\kappa] = 0$ for uncorrelated signals if one of the two random processes is zero-mean.

### Example - Time-of-Arrival Estimation

The CCF can be used to estimate the [time-of-arrival (TOA)](https://en.wikipedia.org/wiki/Time_of_arrival) of a signal at transmitted from a sender to a receiver. The TOA is used for instance for [radiolocation](https://en.wikipedia.org/wiki/Radiolocation) or sound source localization. For ease of illustration it is assumed that a random signal $x[k]$ is only delayed by transmission. The received signal $y[k]$ is then just a delayed version of transmitted signal

\begin{equation}
y[k] = x[k - k_0] = x[k] * \delta[k - k_0]
\end{equation}

The aim is to estimate the delay $k_0$ by observation of the received signal under knowledge of the transmitted signal. Under the assumption of finite length signals, the CCF between the transmitted and the received signal is given as

\begin{equation}
\begin{split}
\varphi_{xy}[\kappa] &= \frac{1}{N} x_N[k] * y_M[-k] \\
&= \frac{1}{N} \left( x_N[k] * x_N[-k] \right) * \delta[-k+k_0] \\
&= \frac{1}{N} \varphi_{xx}[\kappa] * \delta[k - k_0]
\end{split}
\end{equation}

where the associativity of the convolution and the even symmetry of the Dirac pulse has been used. The delay between the two signals results in a shift of the ACF of the transmitted signal. The delay can be estimated by finding the maximum value of the CCF $\varphi_{xy}[\kappa]$, if the ACF $\varphi_{xx}[\kappa]$ of the transmitted signal $x[k]$ exhibits a pronounced maximum at $\kappa = 0$

\begin{equation}
\hat{k}_0 = \underset{\kappa}{\mathrm{argmax}} \{ \varphi_{xy}[\kappa] \}
\end{equation}

One can differentiate between to types of transmitted signals: (i) ones which have been designed explicitly for TOA estimation and (ii) application specific signals. The latter are for instance wireless or speech signals which are used in the applications where specific measurement signals may not be appropriate. For the former, there exist various choices for finite length random sequences with good auto-correlation properties.

The Gold code is a pseudo random noise (PRN) sequence used in the [Global Positioning System (GPS)](https://en.wikipedia.org/wiki/Global_Positioning_System) to estimate the TOA between satellites and the receiver. It constitutes a series of binary sequences with good auto- and cross-correlation properties. The latter is required to lower the interference between the sequences of different GPS satellites. The properties of these specific random signals are investigated and the application of these sequences to TOA estimation is illustrated. First the pre-computed sequences are loaded and a function for computing and plotting the CCF (and ACF) is defined for convenience

```python
# load set of PRN sequences (Gold codes)
prn = np.load('../data/gold_sequences.npz')['prn']

def compute_plot_CCF(x, y, ylabel, K=30):
    # compute and truncate CCF
    ccf = 1/len(x) * np.correlate(x, y, mode='full')
    ccf = ccf[(len(y)-1)-K:len(y)+K]
    kappa = np.arange(-K, K+1)

    # plot CCF
    plt.stem(kappa, ccf)
    plt.xlabel(r'$\kappa$')
    plt.ylabel(ylabel)
    plt.axis([-K, K, 1.1*min(ccf), 1.1*max(ccf)]);
    plt.grid()

    return ccf
```

The ACF for one particular sequence and the CCF between two different sequences is computed and plotted

```python
plt.figure(figsize = (10, 4))

plt.subplot(121)
compute_plot_CCF(prn[10, :], prn[10, :], r'$\hat{\varphi}_{x_n x_n}[\kappa]$')
plt.title('ACF')

plt.subplot(122)
compute_plot_CCF(prn[10, :], prn[11, :], r'$\hat{\varphi}_{x_n x_m}[\kappa]$')
plt.title('CCF between two PRN sequences')
plt.ylim([-1, 1])
plt.tight_layout()
```

The ACF features a pronounced peak for $\kappa = 0$ and the two PRN sequences are reasonably uncorrelated. Now one particular sequence is delayed in order to model the received signal. The CCF is plotted and the TOA is estimated by finding the maximum value of the CCF between the transmitted and received sequence

```python
k0 = 10  # true TOA

x = prn[10, :]  # pick one PRN sequence
y = x[k0:] # delay transmitted signal by k0 samples

# compute and plot CCF
plt.figure(figsize = (10, 4))
ccf = compute_plot_CCF(x, y, r'$\hat{\varphi}_{xx}[\kappa]$')
plt.title('CCF between transmitted and received signal')

# estimate the TOA
print('Estimated TOA is {:2.0f} samples'.format(np.argmax(ccf) - K))
```

**Exercise**

* Add a random signal with different levels to the received signal. How does this influence the TOA estimation?

Solution: Additive noise can be added for instance by extending above example with `y = y + 0.1 * np.random.normal(len(y))`. Additive noise corrupts the CCF. For high levels, the maximum value of the CCF might not correspond to the delay $k_0$ anymore.

### Example - Cross-Correlation Function of Two Random Signals

The following example estimates and plots the CCF of two random signals $x[k]$ and $y[k]$ of finite lengths $N$ and $M = 2 N$.

```python
N = 1024  # length of random signals

# generate two uncorrelated random signals
np.random.seed(2)
x = 2 + np.random.normal(size=N)
y = 1 + np.random.normal(size=2*N)

# compute CCF
ccf = 1/len(x) * np.correlate(x, y, mode='full')
kappa = np.arange(-(N-1), 2*N)

# print mean values of signals
print('Mean of signal x[k]: %f' %np.mean(x))
print('Mean of signal y[k]: %f' %np.mean(y))

# plot CCF
plt.figure(figsize = (10, 8))
plt.stem(kappa, ccf)
plt.title('Estimated cross-correlation function')
plt.ylabel(r'$\hat{\varphi}_{xy}[\kappa]$')
plt.xlabel(r'$\kappa$')
plt.axis([kappa[0], kappa[-1], 0, 1.1*max(ccf)]);
plt.grid()
```

**Exercise**

* Can the signals be assumed to be correlated or uncorrelated when considering the CCF in the range $0 < \kappa < N$ only?
* Why does the CCF of the two finite length signals have this overall trapezoid like shape?
* What would be its theoretic value for signals of infinte length?

Solution: The CCF is approximately constant for $0 < \kappa < N$. Consequently, the random signals $x[k]$ and $y[k]$ can be assumed to be uncorrelated. The trapezoidal shape of the CCF results from the truncation of the random signals to a finite number of samples. This truncation can be modeled as a multiplication of the infinite length signals by a rectangular signal $\text{rect}_N[k]$ and $\text{rect}_M[k]$, respectively. Interpreting the CCF as convolution and having in mind that the convolution of two rectangular signals results in a signal of trapezoidal shape explains the shape of the CCF shown above. Its theoretic value is constant with a value given by the multiplication of the two linear means of the random signals $\varphi_{xy}[\kappa] = \mu_x \cdot \mu_y$.

The output of the CCF were both correlated signals overlap completely, here $0 \leq \kappa < N$, is also termed as *valid* part of the CCF, the remaining contributions as boundary effects. The computation of the CCF can be limted to the valid part only with the parameter `mode=valid`, for instance `np.correlate(x, y, mode='valid')`.

## Cross-Covariance Function

The [cross-covariance function](https://en.wikipedia.org/wiki/Covariance) is defined as the CCF of two zero-mean random signals. For two random signals $x[k]$ and $y[k]$ with linear means $\mu_x \neq 0$ and $\mu_y \neq 0$ it is given as

\begin{equation}
\psi_{xy}[\kappa] = \varphi_{xy}[\kappa] - \mu_x \mu_y
\end{equation}

It follows from the properties of the CCF that the cross-covariance function $\psi_{xy}[\kappa]$ of two uncorrelated signals $x[k]$ and $y[k]$ is zero, $\psi_{xy}[\kappa] = 0$.

**Exercise**

* What would the plot for $\psi_{xy}[\kappa]$ look like for the [example above](#Example---Cross-Correlation-Function-of-Two-Random-Signals) under the assumption of signals of infinite length?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


## The SIR model

The SIR model of epidemiology partitions the population into three compartments: susceptibles, S, who can catch the disease; infectives, I, who have already caught the disease and infect susceptibles; and removed individuals, R. Since the disease is assumed not to be fatal, the sum $N=S+I+R$ remains constant. The rate at which the susceptibles get infected is
$$
\lambda(t) = \frac{\beta I}{N}
$$
where the parameter $\beta$ is the probability of infection on contact. The infected individuals remove from the disease at a rate $\gamma$. Then, the ordinary differential equations of the SIR model are

\begin{align}
\dot S &= -\lambda(t)S \\
\dot I &= \lambda(t)I - \gamma I \\
\dot R &= \gamma I
\end{align}

This example integrates the above equations to obtain what is called the **epidemic curve**: a plot of the number of susceptibles and infectives as a function of time.

Below we use the class `Model` to simulate SIR model with three age-groups

```python
# M=3, SIR with three age-groups

import numpy as np
import pyross
import copy

import matplotlib.pyplot as plt

model_spec = { "classes" : ["S", "I"],

             "S" : {"infection" : [ ["I","S", "-beta"] ]},  ## the I class passes infection to S class
             "I" : { "linear"    : [ ["I", "-gamma"] ],     ## this is recovery process for I class
                    "infection" : [ ["I", "S", "beta"]]}

              ## the recovered class R is internally determined by number conservation
             }

parameters = {'beta'  : 0.1,
              'gamma' : 0.1,
              }

M=3;  Ni=1000*np.ones(M);  N=np.sum(Ni)

# Initial conditions as an array
x0 = np.array([
    980, 980, 980,    # S
    20,   20,  20,    # I
])

# Or initial conditions as a dictionary
x0 = {'S': [n-20 for n in Ni], 'I':  [20, 20, 20]  }

CM = np.array( [[1,   0.5, 0.1],
               [0.5, 1,   0.5],
               [0.1, 0.5, 1  ]], dtype=float)

def contactMatrix(t):
    return CM

# duration of simulation and data file
Tf = 160;  Nf=Tf+1;

model = pyross.deterministic.Model(model_spec, parameters, M, Ni)

# simulate model
data = model.simulate(x0, contactMatrix, Tf, Nf)

# plot the data and obtain the epidemic curve
S = np.sum(model.model_class_data('S', data), axis=1)
I = np.sum(model.model_class_data('I', data), axis=1)
R = np.sum(model.model_class_data('R', data), axis=1)
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.fill_between(t, 0, S/N, color="#348ABD", alpha=0.3)
plt.plot(t, S, '-', color="#348ABD", label='$S$', lw=4)

plt.fill_between(t, 0, I/N, color='#A60628', alpha=0.3)
plt.plot(t, I, '-', color='#A60628', label='$I$', lw=4)

plt.fill_between(t, 0, R/N, color="dimgrey", alpha=0.3)
plt.plot(t, R, '-', color="dimgrey", label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Compartment value')
plt.xlabel('Days');
```

## A toy model for vaccinations, derived from SEIR

```python
model_spec = {
    "classes" : ["S", "E", "I", "R"],

    "S" : {
        "infection" : [ ["I", "S", "-beta"] ]
    },

    "E" : {
        "linear"    : [ ["E", "-gammaE"] ],
        "infection" : [ ["I", "S", "beta"] ]
    },

    "I" : {
        "linear"    : [ ["E", "gammaE"],
                        ["I", "-gammaI"]]
    },

    "R" : {
        "linear"    : [ ["I", "gammaI"] ],
    }
}

```

```python
parameters = {
    'beta' : 0.05,
    'gammaE' : 0.5,
    'gammaI' : 0.1
}
```

```python
M = 3
Ni = 50000*np.ones(M)
N = np.sum(Ni)

# initial conditions as a dictionary
E0 = [30, 30, 30]
S0 = [n-30 for n in Ni]
I0 = [0, 0, 0]
R0 = [0, 0, 0]

x0 = {
    'S' : S0,
    'E' : E0,
    'I' : I0,
    'R' : R0
}

CM = np.array([
    [3,   0.8, 0.1],  # youngest cohort spreading most
    [0.8, 2,   0.5],
    [0.1, 0.5, 1  ]
], dtype=float)

# the contact matrix is time-dependent
def contactMatrix(t):
    if t<40:
        xx = CM
    elif 40<=t<200:
        xx = 0.5*CM
    elif 200<=t<280:
        xx = 0.9*CM
    elif 280<=t<370:
        xx = 0.5*CM
    else:
        xx = CM
    return xx

# duration of simulation and data file
Tf = 1000;  Nf=Tf+1;

model = pyross.deterministic.Model(model_spec, parameters, M, Ni)
# simulate model
data = model.simulate(x0, contactMatrix, Tf, Nf)
```

```python
# plot the data and obtain the epidemic curve
S = np.sum(model.model_class_data('S', data), axis=1)
E = np.sum(model.model_class_data('E', data), axis=1)
I = np.sum(model.model_class_data('I', data), axis=1)
R = np.sum(model.model_class_data('R', data), axis=1)
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.semilogy(t, S, '-',  label='$S$', lw=4)
plt.semilogy(t, E, '-',  label='$E$', lw=4)
plt.semilogy(t, I, '-', label='$I$', lw=4)
plt.semilogy(t, R, '-',  label='$R$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Compartment value')
plt.xlabel('Days');
```

```python
# age structure
Ii = model.model_class_data('I', data)

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})
plt.stackplot(t,np.transpose(Ii),labels=('cohort 1','cohort 2','cohort 3'))
plt.legend(fontsize=26); plt.grid()
plt.ylabel('Infections')
plt.xlabel('Days');
```

Without vaccinations, herd immunity is reached after the 3rd wave (no further lockdown). Total number of people that had been infected, by age group:

```python
np.round(np.sum(Ii,axis=0)*parameters['gammaI'])
```

    array([34921., 28938., 12600.])

### The SEIR model with vaccinations

This code takes the `Model` specification of SEIR, extends the number of classes by the vaccinated version of classes and sets up all the transitions. There are additional infection terms, the linear terms are the same in the vaccinated compartments (though they might be changed later, to account for a reduction in severity), and finite resource terms move individuals from the unvaccinated to the vaccinated classes.

This avoids typing the whole model specification by hand, and should also work for our more detailed, calibrated models.

The vaccinated version of class X is labeled XV, The extension to different types/doses of vaccinations should be straightforward (XV1, XV2, et.c).

```python
model_spec_vaccinations = copy.deepcopy(model_spec)

for cl in model_spec["classes"]:
    clV = cl+'V'
    model_spec_vaccinations["classes"].append(clV)

    model_spec_vaccinations[clV] = {}
    if "infection" in model_spec[cl]:
        model_spec_vaccinations[clV]["infection"] = []
        for term in model_spec[cl]["infection"]:

            termI = term.copy() # vaccinated infecting unvaccinated
            termI[0] += 'V'
            termI[2] += "*thetaI"
            model_spec_vaccinations[cl]["infection"].append(termI)

            termS = term.copy() # unvaccinated infecting vaccinated
            termS[1] += 'V'
            termS[2] += "*thetaS"
            model_spec_vaccinations[clV]["infection"].append(termS)

            termSI = term.copy() # vaccinated infecting vaccinated
            termSI[0] += 'V'
            termSI[1] += 'V'
            termSI[2] += "*thetaS*thetaI"
            model_spec_vaccinations[clV]["infection"].append(termSI)

    if "linear" in model_spec[cl]:
        model_spec_vaccinations[clV]["linear"] = []
        for term in model_spec[cl]["linear"]:
            termV = term.copy()
            termV[0] += 'V'
            model_spec_vaccinations[clV]["linear"].append(termV)

    if "finite-resource" in model_spec[cl]:
        model_spec_vaccinations[clV]["finite-resource"] = []
        for term in model_spec[cl]["finite-resource"]:
            termV = term.copy()
            termV[0] += 'V'
            model_spec_vaccinations[clV]["finite-resource"].append(termV)
    else:
        model_spec_vaccinations[cl]["finite-resource"] = []
        model_spec_vaccinations[clV]["finite-resource"] = []
    term = [cl, "vaccination_rate", "vaccination_priority", "one"]   # one: probability that the vaccination is successful. Set to 1, because we already
                                                                     # account for the imperfectness of vaccinations through the theta parameters
    model_spec_vaccinations[clV]["finite-resource"].append(term.copy())
    term[1] = "-"+term[1]
    model_spec_vaccinations[cl]["finite-resource"].append(term)

model_spec_vaccinations

```

    {'classes': ['S', 'E', 'I', 'R', 'SV', 'EV', 'IV', 'RV'],
     'S': {'infection': [['I', 'S', '-beta'], ['IV', 'S', '-beta*thetaI']],
      'finite-resource': [['S',
        '-vaccination_rate',
        'vaccination_priority',
        'one']]},
     'E': {'linear': [['E', '-gammaE']],
      'infection': [['I', 'S', 'beta'], ['IV', 'S', 'beta*thetaI']],
      'finite-resource': [['E',
        '-vaccination_rate',
        'vaccination_priority',
        'one']]},
     'I': {'linear': [['E', 'gammaE'], ['I', '-gammaI']],
      'finite-resource': [['I',
        '-vaccination_rate',
        'vaccination_priority',
        'one']]},
     'R': {'linear': [['I', 'gammaI']],
      'finite-resource': [['R',
        '-vaccination_rate',
        'vaccination_priority',
        'one']]},
     'SV': {'infection': [['I', 'SV', '-beta*thetaS'],
       ['IV', 'SV', '-beta*thetaS*thetaI']],
      'finite-resource': [['S',
        'vaccination_rate',
        'vaccination_priority',
        'one']]},
     'EV': {'infection': [['I', 'SV', 'beta*thetaS'],
       ['IV', 'SV', 'beta*thetaS*thetaI']],
      'linear': [['EV', '-gammaE']],
      'finite-resource': [['E',
        'vaccination_rate',
        'vaccination_priority',
        'one']]},
     'IV': {'linear': [['EV', 'gammaE'], ['IV', '-gammaI']],
      'finite-resource': [['I',
        'vaccination_rate',
        'vaccination_priority',
        'one']]},
     'RV': {'linear': [['IV', 'gammaI']],
      'finite-resource': [['R',
        'vaccination_rate',
        'vaccination_priority',
        'one']]}}

```python
parameters = {
    'beta' : 0.05,
    'gammaE' : 0.5,
    'gammaI' : 0.1,
    'thetaS' : 0.6,
    'thetaI' : 0.6,
}
```

The age-dependence of vaccinations is introduced explicitly through an age-dependent vaccination rate.
Alternatively, one could have specified a scalar vaccination rate, and implicilty realised the age-dependence though age-dependent priorities.

```python

def vaccination_rate(t):
    rate=500*np.array([0.5*np.tanh((t-420)/10)-0.5*np.tanh((t-480)/10),
                       0.5*np.tanh((t-360)/10)-0.5*np.tanh((t-420)/10),
                       0.5*np.tanh((t-300)/20)-0.5*np.tanh((t-360)/10)  ])
    rate*=(rate>1)
    return rate

def parameter_mapping(input_parameters, t):
    output_parameters = {
        'beta' : input_parameters['beta'],
        'gammaE' : input_parameters['gammaE'],
        'gammaI' : input_parameters['gammaI'],
        'beta*thetaS' : input_parameters['beta']*input_parameters['thetaS'],
        'beta*thetaI' : input_parameters['beta']*input_parameters['thetaI'],
        'beta*thetaS*thetaI' : input_parameters['beta']*input_parameters['thetaS']*input_parameters['thetaI'],
        'one' : 1,
        'vaccination_priority' : 1,
        'vaccination_rate' : vaccination_rate(t)
    }

    return output_parameters
```

```python
t=np.linspace(200,600,1024)
v=[vaccination_rate(tt) for tt in t]
plt.plot(t,v)
plt.xlabel('days')
plt.ylabel('vaccination rate (by age)')
plt.show()
```

```python
# initial conditions as a dictionary
E0 = [30, 30, 30]
S0 = [n-30 for n in Ni]
I0 = [0, 0, 0]
R0 = [0, 0, 0]

x0 = {
    'S' : S0,
    'E' : E0,
    'I' : I0,
    'R' : R0
}

for cl in model_spec["classes"]:
    x0[cl+'V'] = np.zeros(M)

# duration of simulation and data file
Tf = 1000;  Nf=Tf+1;

model = pyross.deterministic.Model(model_spec_vaccinations, parameters, M, Ni, time_dep_param_mapping=parameter_mapping)
# simulate model
data = model.simulate(x0, contactMatrix, Tf, Nf)

```

```python
# plot the data and obtain the epidemic curve
S = np.sum(model.model_class_data('S', data), axis=1)
E = np.sum(model.model_class_data('E', data), axis=1)
I = np.sum(model.model_class_data('I', data), axis=1)
R = np.sum(model.model_class_data('R', data), axis=1)
SV = np.sum(model.model_class_data('SV', data), axis=1)
EV = np.sum(model.model_class_data('EV', data), axis=1)
IV = np.sum(model.model_class_data('IV', data), axis=1)
RV = np.sum(model.model_class_data('RV', data), axis=1)
V=SV+EV+IV+RV
t = data['t']

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.semilogy(t, S+SV, '-',  label='$S$', lw=4)
plt.semilogy(t, E+EV, '-',  label='$E$', lw=4)
plt.semilogy(t, I+IV, '-', label='$I$', lw=4)
plt.semilogy(t, R+RV, '-',  label='$R$', lw=4)
plt.semilogy(t, V, '-',  label='$V$', lw=4)

plt.legend(fontsize=26); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Compartment value')
plt.xlabel('Days');
```

```python
# age structure
Ii = model.model_class_data('I', data)
IVi = model.model_class_data('IV', data)

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})
plt.stackplot(t,np.transpose(Ii+IVi),labels=('cohort 1','cohort 2','cohort 3'))
plt.legend(fontsize=26); plt.grid()
plt.ylabel('Infections')
plt.xlabel('Days');
```

With vaccinations, the 3rd wave is reduced, despite the lifting of lockdown. Total number of people that had been infected, by age group:

```python
# unvaccinated
np.round(np.sum(Ii,axis=0)*parameters['gammaI'])
```

    array([13855.,  7407.,  2040.])

```python
# vaccinated
np.round(np.sum(IVi,axis=0)*parameters['gammaI'])
```

    array([2850., 2539.,  852.])


# Introduction to nonparametric regression

------------------------------------------------------
*Introduction to Data Science & Machine Learning*

*Pablo M. Olmos olmos@tsc.uc3m.es*

------------------------------------------------------

Today, we will study a different approach to perform regression. In a nutshell, nonparametric models refer to such methods for which the complexity of the underlying model to perform the task (e.g. regression for us) **grows with the number of datapoints we use to train**.

Note that this is in contrast to the linear regression models we have seen before, for which we first had to decide the interpolation function (e.g. a degree-5 polynomial) and then train the model. However, the complexity of the model itself does not vary as we increase the number of points in the dataset. It is always a degree-5 polynomial. This **does not happen with nonparametric models**.

Lets start with a simple method to perform nonparametric regression: [**K-nearest neighbors (K-NN)for regression**](http://scikit-learn.org/stable/auto_examples/neighbors/plot_regression.html).

----

## K-NN regression

Consider we have a **training** database of $N$ entries of the form $(\mathbf{x}^{(i)},y^{(i)})$, where $\mathbf{x}^{(i)}\in\mathbb{R}^D$ and $y^{(i)}\in\mathbb{R}$.

In K-NN regression, given a new point $\mathbf{x}^*$ the estimated target $y^*$ is computed as follows:

\begin{align}
y^* \approx \frac{1}{K}\sum_{k\in \mathcal{S}_{K}(\mathbf{x}^*)} y^{(k)},
\end{align}
where $\mathcal{S}_{K}$ is the set of the **$K$ training points that are closest to $\mathbf{x}^*$ according to a given distance metric $d(\mathbf{x}^{(i)},\mathbf{x}^*)$**. For instance, in real spaces the most common choice is the **ecludean distance**:

\begin{align}
d(\mathbf{x}^{(i)},\mathbf{x}^*) = \left|\left|\mathbf{x}^{(i)}-\mathbf{x}^*\right|\right|^2
\end{align}

There are many variants of the above expression. For instance, we can **weight differently the regression values according to the distance**. In this way, the closest points has a bigger influence in the regression value:

\begin{align}
y^* \approx \frac{1}{K}\sum_{k\in \mathcal{S}_{K}(\mathbf{x}^*)} \frac{d(\mathbf{x}^{(k)},\mathbf{x}^*)}{\sum_{q\in \mathcal{S}_{K}(\mathbf{x}^*)} d(\mathbf{x}^{(q)},\mathbf{x}^*)} y^{(k)},
\end{align}

### K-NN complexity

Note that complexity is determined by the [**search of the set of closest neighbors**](https://en.wikipedia.org/wiki/Nearest_neighbor_search). This complexity grows as $\mathcal{O}(DN)$, where $D$ is the dimension and $N$ is the number of points.

Researches in computer science have developed many approximate lower-complexity algorithms to perform this task in high dimension spaces. The most common ones are [**k-d trees**](https://en.wikipedia.org/wiki/K-d_tree) and [**Local Sensitive Hashing**](https://en.wikipedia.org/wiki/Locality-sensitive_hashing).

### Example of K-NN regression with sklearn

Lets analyze the behavior of K-NN with a simple 1-D regression example.

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn import neighbors
from sklearn.kernel_ridge import KernelRidge

%matplotlib inline

# use seaborn plotting defaults
import seaborn as sns; sns.set()
plt.rcParams["figure.figsize"] = [8,8]
```

```python
np.random.seed(0)

N_train = 50
R = 20
Xtrain = np.sort(np.random.uniform(-R,R,N_train), axis=0)       #Training points

Xtest = np.linspace(-R,R,500)      #Test points

def noise_sinc(X):
    Y = np.sin(X)/np.pi/(X+1e-6) + np.sqrt(1e-04) * np.random.randn(X.shape[0])
    return Y

Ytrain = noise_sinc(Xtrain)
Ytest = noise_sinc(Xtest)
```

```python
#K-NN training
n_neighbors = 1

knn = neighbors.KNeighborsRegressor(n_neighbors, weights='uniform')
Ypred = knn.fit(Xtrain.reshape([-1,1]), Ytrain.reshape([-1,1])).predict(Xtest.reshape([-1,1]))   #Function .fit does not work with a-dimensional vectors

```

```python
plt.plot(Xtrain,Ytrain,'go',ms=15,label='train')
plt.plot(Xtest,Ypred,'r*',ms=5,label='K-NN')
plt.plot(Xtest,Ytest,'b-',label='test (true)')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
```

### Exercise

Investigate how does the k-NN regressor behave as you vary either the number of training points and the number of neighbors (K). How do you identify that the model is overfitting? or if the model is too biased? how do you regularize?

## Kernel Linear Regression

Note that the regression function that we learn using a $K$-NN regression is in general non-continuous and non-differentiable at many points. In other words, it is a [**non-smooth function**](http://mathworld.wolfram.com/SmoothFunction.html). Interpolating with smooth functions it is always desirable, as the model will certainly generalize better.

There are many ways to construct smooth version of a $K$-NN regressor. Now, we will use the most important example: a linear regression based on **kernels**. **Nonparametric models based on kernels constitute a keystone in machine learning**. The following [video](https://www.youtube.com/watch?v=ldzmOuESx5w) is a very clear introduction to kernel machines.

### Ridge linear regression model

Consider we have a **training** database of $N$ entries of the form $(\mathbf{x}^{(i)},y^{(i)})$, where $\mathbf{x}^{(i)}\in\mathbb{R}^L$ and $y^{(i)}\in\mathbb{R}$.

Imagine that each of the input vectors is transformed to include **additional features** (such as polynomial expansions or any other transformation). We denote the **feature vector** as

\begin{align}
\phi(\mathbf{x}): \mathbb{R}^L \rightarrow \mathbb{R}^D,
\end{align}

where $D\geq L$. Assume that the first element  of $\phi(\mathbf{x})$ is $1$, i.e. $\phi(\mathbf{x})_0=1$, accomodating for the intercept. We will use this training set to fit a model of the form

$$y \approx \theta_0 + \theta_1 \phi(\mathbf{x})_1 + \theta_2 \phi(\mathbf{x})_2 + \ldots + \theta_D \phi(\mathbf{x})_D $$

If we use the **ridge loss function**, i.e.
\begin{align}
L(\boldsymbol{\theta},\lambda) = \frac{1}{N} \left[\sum_{i=1}^{N} (y^{(i)}-\boldsymbol{\theta}^T\phi(\mathbf{x}^{(i)}))^2 + \lambda \sum_{j=1}^{D+1} \theta_j^2\right]
\end{align}
then we know that the minimum is attained at
$$\boldsymbol{\theta}_\lambda = (\mathbf{\Phi}^T\mathbf{\Phi} + \mathbf{D}_\lambda)^{-1}\mathbf{\Phi}^T\mathbf{y},$$
where $\mathbf{\Phi}$ is the $N\times (D+1)$ feature matrix and
$$ \mathbf{D} = \left[ \begin{array}{cc} 0 & 0 \\ 0 & \lambda \mathbf{I}_{D\times D}\end{array}\right] $$

---

### The kernel-trick and the dual problem representation

Our whole derivation is based upon the following point. If we compute the gradient of $L(\boldsymbol{\theta},\lambda)$ w.r.t. $\boldsymbol{\theta}$ and set it to zero we get

\begin{align}
\boldsymbol{\theta}_\lambda = -\frac{1}{\lambda}\sum_{i=1}^N \left(y^{(i)}-\boldsymbol{\theta}_\lambda^T\phi(\mathbf{x}^{(i)})\right)\phi(\mathbf{x}^{(i)}) = \sum_{i=1}^{N} a^i \phi(\mathbf{x}^{(i)}) = \mathbf{\Phi}^T\mathbf{a},
\end{align}

where $a^i = -\frac{1}{\lambda} \left(y^{(i)}-\boldsymbol{\theta}_\lambda^T\phi(\mathbf{x}^{(i)})\right)$. Thus, the optimal solution belowns to the vector subpspace ** spanned by the feature training vectors**. Note that, given $\mathbf{a}$, the regression estimate for a new vector $\mathbf{x}^{*}$ is:

\begin{align}
y^* \approx \boldsymbol{\theta}_\lambda^T \phi(\mathbf{x}^*) &= \left(\mathbf{\Phi}^T\mathbf{a}\right)^T \phi(\mathbf{x}^*) = \left(\sum_{i=1}^{N} a^i \phi(\mathbf{x}^{(i)})\right)^T \phi(\mathbf{x}^*) = \sum_{i=1}^{N} a^i \phi(\mathbf{x}^{(i)})^T \phi(\mathbf{x}^*) \\ &= \sum_{i=1}^{N} a^i_\lambda k(\mathbf{x}^{(i)},\mathbf{x}^*) = \mathbf{a}^T \mathbf{k}(\mathbf{x}^*),
\end{align}
where

\begin{align}
\mathbf{k}(\mathbf{x}^*) = \left[\begin{array}{c} k(\mathbf{x}^{(1)},\mathbf{x}^*) \\ k(\mathbf{x}^{(2)},\mathbf{x}^*) \\ \vdots \\ k(\mathbf{x}^{(N)},\mathbf{x}^*)\end{array}\right] = \left[\begin{array}{c} \phi(\mathbf{x}^{(1)})^T \phi(\mathbf{x}^*) \\ \phi(\mathbf{x}^{(2)})^T \phi(\mathbf{x}^*) \\ \vdots \\ \phi(\mathbf{x}^{(N)})^T \phi(\mathbf{x}^*)\end{array}\right]
\end{align}

and $k(\mathbf{x}^{(i)},\mathbf{x}^*)$ is called the **kernel function** between points $\mathbf{x}^{(i)}$ and $\mathbf{x}^{*}$. In **kernel linear regression**, we will choose a kernel function as **similarity metric** between each pair points and **our goal is to find the vector $\mathbf{a}$ that minimizes the ridge loss function**:

\begin{align}
y^* \approx \sum_{i=1}^{N} a^i k(\mathbf{x}^{(i)},\mathbf{x}^*)
\end{align}

---

### Examples of kernel functions

In order to explot the kernel trick, we need to be able to construct valid kernel functions. One approach is to choose directly a featre space mapping $\phi(\mathbf{x})$. For instance, if $x\in\mathbb{R}$ we can choose
\begin{align}
\phi(x) =[1, x, ~ x^2] \Rightarrow  k(x^{(1)},x^{(2)}) = 1+ (x^{(1)}x^{(2)}) + (x^{(1)}x^{(2)})^2
\end{align}

The most common approach is to **choose the kernel directly**. In this case we must ensure that the function we choose ** corresponds to a scalar product in some (perhaps infinite dimensional) feature space** (technicaly, that [Mercer's theorem](http://people.cs.uchicago.edu/~niyogi/papersps/MinNiyYao06.pdf) is satisfied).

Check this [**link**](http://scikit-learn.org/stable/modules/metrics.html) to see possibles examples of kernels that can be used within sklearn. A very common one that actually maps over an **infinite dimensional feature space** is the radial basis function (RBF) or Gaussian kernel:

\begin{align}
k(\mathbf{x},\mathbf{x}^*) = \text{e}^ -\frac{||\mathbf{x}-\mathbf{x}^*||^2}{2\sigma^2},
\end{align}

where $sigma$ is called the bandwith and it is a parameter that typically has to be adjusted by **cross validation**.

**You can understand the kernel as a similarity metric so that the vectors closest to $\mathbf{x}^*$ in the training set are those that weight more in the prediction! (similarly as in K-NN regression but in a smoother fashion).**

---

### Finding the vector of coefficients $\mathbf{a}$

If we substitute $\mathbf{\theta} = \mathbf{\Phi}^T\mathbf{a}$ in the Ridge Loss function we get the dual optimization problem. After some manipulation, the loss function is expressed in the following way:

\begin{align}
L(\boldsymbol{a},\lambda) = \frac{1}{N}\left[\mathbf{a}^T \mathbf{K}^T\mathbf{K}\mathbf{a}-\mathbf{a}^T\mathbf{K}\mathbf{y}+\lambda\mathbf{a}^T\mathbf{K}\mathbf{a}\right],
\end{align}
where $\mathbf{K}$ is the $(N\times N)$ symmetric **kernel matrix** such that

\begin{align}
K_{nm} = k(\mathbf{x}^{(m)},\mathbf{x}^{(n)}).
\end{align}

$L(\boldsymbol{a},\lambda)$ is a convex function whose minimum is attained at

\begin{align}
\boldsymbol{a} = \left(\boldsymbol{K}+\lambda \boldsymbol{I}\right)^{-1} \boldsymbol{y}.
\end{align}

As you probably guess already, this solution is rather approximated by **gradient descent** due to better complexity and stability properties. Also, note that the complexity now increases as $\mathcal{O}(N^3)$ instead of the $\mathcal{O}(D^3)$ complexity that we had in the feature primal space.

### Example of using kernel ridge linear regression (KRR) for the example above

We will use [**sklearn implementation of KRR**](http://scikit-learn.org/stable/modules/generated/sklearn.kernel_ridge.KernelRidge.html#sklearn.kernel_ridge.KernelRidge)

```python
#K-NN training
clf = KernelRidge(gamma=1,kernel='rbf')   #gamma is the inverse of the bandwith

Ypred = clf.fit(Xtrain.reshape([-1,1]), Ytrain.reshape([-1,1])).predict(Xtest.reshape([-1,1]))   #Function .fit does not work with a-dimensional vectors

```

```python
plt.plot(Xtrain,Ytrain,'go',ms=15,label='train')
plt.plot(Xtest,Ypred,'r*',ms=5,label='Kernel LR')
plt.plot(Xtest,Ytest,'b-',label='test (true)')
plt.xlabel('x')
plt.ylabel('y')
plt.legend()
```

### Exercise

Investigate how does the kernel regressor behave as you vary either the number of training points and the bandwith in the RBF kernel. How do you identify that the model is overfitting? or if the model is too biased? how do you regularize?

Investigate also the effect of varying the kernel.

```python

```

Notebooks/Session 6 Nonparametric Regression/Nonparametric Regression.ipynb


# Визуализация, понижение размерности
### Пример 1

У выборки (яркие точки) три размерности (три признака).

Если убрать один признак (ось Z), то получится двумерная выборка (бледные точки), но классы всё еще хорошо будут разделяться между собой.

### Пример 2

Оба признака значимы, но все данные можно спроецировать на зелёную прямую, практически не потеряв никакой информации (максимум дисперсии, минимум потерь).

Если спроецировать данные на перпендикулярную зёленой прямой ось, то произойдет потеря наибольшего количества дисперсии и информации (многие точки совпадут, станут слабо отличимы друг от друга).

Признаки линейно зависимые (есть избыточность в данных).

В данном случае нельзя просто выбросить один из признаков (ось x или y), нужно сформировать новый признак на основе двух исходных.

### Пример 3

Можно спроецировать выборку на кривую сложной формы и сделать данные одномерными.

**Задача понижения размерности - это формирование новых признаков на основе исходных.**

**Количество признаков должно уменьшиться, но они должны сохранять в себе как можно больше информации, присутствующей в исходных данных.**

## 1. PCA (Principal Component Analysis), Метод главных компонент

Один из основных способов уменьшить размерность данных, потеряв наименьшее количество информации

#### Задача:
Пусть имеется набор $m$ точек ${x^{(1)} , …, x^{(m)}}$ в пространстве $ℝ^n$ , и мы хотим подвергнуть их сжатию с потерей информации, т. е. сохранить точки в меньшем объеме памяти, возможно, ценой некоторой потери точности. Но хотелось бы свести эти потери к минимуму.

У PCA есть три постановки задачи:
1. через матричное разложение исходной матрицы объекты-признаки
2. через поиск оптимальной проекционной гиперплоскости (минимизация ошибки проецирования)
3. через поиск проекционной гиперплоскости, при которой получается максимальная дисперсия после проецирования

Все эти постановки задач эквивалентны, приводят к одному и тому же решению.

### Максимизация дисперсии

Есть некоторое облако точек, где каждая точка — это объект выборки, и нужно найти такие оси, при проецировании на которые
мы сохраним как можно больше дисперсии исходной выборки.

Чем больше дисперсии мы сохраним,тем больше информации останется после понижения размерности.

Хотим выбрать прямую (один признак вместо двух), на которую будет оптимально спроецировать выборку.

Синяя прямая лучше, поскольку при проецировании на неё сохранится больше **информации** о выборке.

Чем больше **дисперсия** выборки после проецирования на прямую, тем лучше.

Дисперсия после проецирования на синюю прямую будет выше, чем после проецирования на красную

### Максимизация дисперсии
$$\begin{cases}
\sum \limits_{j=1}^d w_j^T X^T X w_j -> max_w \\
W^T W = I
\end{cases}
$$
- Верхняя строка - дисперсия выборки **после проецирования**
- Нижняя строка - ограничение, чтобы матрица весов $W$ была ортогональной. Это нужно для того, чтобы решение было единственным.
- (!) верхнее выражение будет означать именно дисперсию выборки только в том случае, если матрица объекты-признаки **центрирована**, то есть если среднее каждого признака равно нулю. Поэтому сначала выборку нужно центрировать (вычесть среднее из каждого столбца в матрице объекты-признаки).

#### Частный случай (первая компонента)
$$j=1$$
$w_1$ - вектор весов, который характеризует направление
$$\begin{cases}
 w_1^T X^T X w_1 -> max_{w_{1}} \\
w_1^T w_1 = I
\end{cases}
$$
- $w_1$ - вектор весов, который характеризует направление, имеет размер D
- ограничение вырождается в то, что $L_2$ норма вектора весов $w_1$ должна равняться 1

Это задача условной оптимизации. Чтобы её решить, нужно выписать её лагранжиан:
$$L(w_1, \lambda)= w_1^T X^T X w_1 - \lambda(w_1^T w_1 - 1)$$

(критерий, который мы максимизируем минус λ умножить на ограничение)

Далее этот лагранжиан нужно продифференцировать по тому, что мы хотим найти, то есть по $w_1$.

Воспользуемся формулами матричного дифференцирования, и получим:

$$\frac{\partial L}{\partial w_1} = 2X^T Xw_1 - 2\lambda w_1 = 0$$

Преобразуем формулу (перенесём одно слагаемое в правую часть и поделим на 2):
$$X^T Xw_1 = \lambda w_1$$

Это уравнение говорит о том, что вектор $w1$ является собственным вектором матрицы $X^T X$, поскольку при умножении матрицы на этот вектор, мы получаем этот же вектор, умноженный на некоторое число $λ$, и число $λ$ является собственным значением, соответствующим этому собственному вектору.

Если подставить полученное нами условие в функционал задачи, то видно,
что дисперсия выборки после проецирования будет равна $λ$, то есть будет равна собственному значению, соответствующему собственному вектору.
$$w_1^T X^T X w_1=\lambda$$

Таким образом, поскольку мы хотим максимизировать дисперсию,
нам нужно выбирать максимальное собственное значение и собственный вектор, который соответствует этому максимальному собственному значению.
- первая компонента в методе главных компонент — это собственный вектор матрицы $X^T X$, который соответствует максимальному собственному значению этой матрицы.
- $X^T X$ — это матрица ковариации, то есть именно та матрица, которая характеризует дисперсию выборки

#### Общий случай
Если аналогичным образом найти второе, третье и т.д. направления, то получится, что $w_2$ - второе направление в методе главных компонент - соответствует второму по величине собственному вектору матрицы $X^T X$, соответствующему второму по величине собственному значению и т.д.

Оптимальные векторы $w_1, w_2, ..., w_d$ - собственные векторы матрицы $X^T X$, соответствующие наибольшим собственным значениям $\lambda_1, \lambda_2, ..., \lambda_d$

Через собственные значения можно выразить долю дисперсии, которую мы сохранили при понижении размерности:

$$\frac{\sum_{i=1}^{d} \lambda_i} {\sum_{i=1}^{D} \lambda_i}$$

**Дисперсия всей исходной выборки** — это сумма всех собственных значений матрицы $X^T X$, то есть сумма от 1 до D

**Дисперсия выборки после проецирования на d главных компонент** — это сумма максимальных d собственных значений

При решении задачи метода главных компонент очень пригождается сингулярное разложение (SVD). Можно его использовать.
**Cингулярное разложение** матрицы $X$ представляет эту матрицу в виде произведения трех других матриц:
$$X=UDV^T$$

При этом $U$ и $V$ — это ортогональные матрицы, а $D$ — диагональная матрица.

- cтолбцы матрицы $U$ — это собственные векторы матрицы $XX^T$
- столбцы матрицы $V$ — это собственные векторы матрицы $X^{T}X$
- диагональ матрицы $D$ - собственные значения $U$ и $V$, которые совпадают и называются **сингулярными числами**

В столбцах матрицы $V$ лежат собственные векторы матрицы $X^{T}X$, отсортированные по убыванию собственных значений. Это и есть **главные компоненты**.
**Проекция на главные компоненты** $XV=UD$
**Вернуться в исходное пространство (линейная комбинация столбцов V)** $UDV^T=X$

**Диагональная матрица** - матрица, у которой вне главной диагонали - только нули
**Единичная матрица** $I$ - на главной диагонали - единицы, остальное - нули. Частный случай диагональной матрицы.
**Ортогональная матрица** - это матрица, для которой её транспонированная версия является обратной $$A^T A = AA^T = I$$

**Собственным вектором**(правым) квадратной матрицы $A$ называется ненулевой вектор $x$ - такой, что умножение $A$ на $x$ изменяет лишь масштаб $x$:
$$Ax=\lambda x$$

Скаляр $\lambda$ называется **собственным значением**, соответствующим этому собственному вектору.

### Сингулярное разложение (SVD, Singular Value Decomposition)

Вычисление главных компонент может быть сведено к вычислению сингулярного разложения матрицы данных или к вычислению собственных векторов и собственных значений ковариационной матрицы исходных данных 🤯

### Алгоритм PCA:
1. Найти сингулярное разложение матрицы $X$
2. Сформировать матрицу весов $W$ , из столбцов матрицы $V$(из собственных векторов), соответствующих максимальным сингулярным числам
3. Преобразовать признаки $$Z=XW$$

Чтобы спроецировать объекты, записанные в матрицу $X$ на главные компоненты, нужно умножить матрицу $X$ на $W$. Получится матрица $Z$, которая является матрицей объекты-признаки для нового сокращенного признакового описания.

## Резюме
- нужно найти собственные векторы матрицы ковариации $X^{T}X$ и выбрать те из них, которые соответствуют максимальным собственным значениям.
- проецирование на эти векторы будет позволять сохранить как можно больше дисперсии.
- через собственные значения этой матрицы можно выяснить, какую именно долю дисперсии мы сохранили при таком понижении размерности

- Оси для проекции в методе главных компонент - собственные векторы ковариационной матрицы
- Доля сохраненной дисперсии выражается через сингулярные числа

## Пример 1
В наборе данных 569 объектов, 30 признаков и 2 класса.

Построим гистограммы распределения значений признаков для двух классов, доброкачественных и злокачественных опухолей

```python
from sklearn.datasets import load_breast_cancer
import numpy as np
cancer = load_breast_cancer()
```

```python
print(cancer.DESCR)
```

    .. _breast_cancer_dataset:

    Breast cancer wisconsin (diagnostic) dataset
    --------------------------------------------

    **Data Set Characteristics:**

        :Number of Instances: 569

        :Number of Attributes: 30 numeric, predictive attributes and the class

        :Attribute Information:
            - radius (mean of distances from center to points on the perimeter)
            - texture (standard deviation of gray-scale values)
            - perimeter
            - area
            - smoothness (local variation in radius lengths)
            - compactness (perimeter^2 / area - 1.0)
            - concavity (severity of concave portions of the contour)
            - concave points (number of concave portions of the contour)
            - symmetry
            - fractal dimension ("coastline approximation" - 1)

            The mean, standard error, and "worst" or largest (mean of the three
            largest values) of these features were computed for each image,
            resulting in 30 features.  For instance, field 3 is Mean Radius, field
            13 is Radius SE, field 23 is Worst Radius.

            - class:
                    - WDBC-Malignant
                    - WDBC-Benign

        :Summary Statistics:

        ===================================== ====== ======
                                               Min    Max
        ===================================== ====== ======
        radius (mean):                        6.981  28.11
        texture (mean):                       9.71   39.28
        perimeter (mean):                     43.79  188.5
        area (mean):                          143.5  2501.0
        smoothness (mean):                    0.053  0.163
        compactness (mean):                   0.019  0.345
        concavity (mean):                     0.0    0.427
        concave points (mean):                0.0    0.201
        symmetry (mean):                      0.106  0.304
        fractal dimension (mean):             0.05   0.097
        radius (standard error):              0.112  2.873
        texture (standard error):             0.36   4.885
        perimeter (standard error):           0.757  21.98
        area (standard error):                6.802  542.2
        smoothness (standard error):          0.002  0.031
        compactness (standard error):         0.002  0.135
        concavity (standard error):           0.0    0.396
        concave points (standard error):      0.0    0.053
        symmetry (standard error):            0.008  0.079
        fractal dimension (standard error):   0.001  0.03
        radius (worst):                       7.93   36.04
        texture (worst):                      12.02  49.54
        perimeter (worst):                    50.41  251.2
        area (worst):                         185.2  4254.0
        smoothness (worst):                   0.071  0.223
        compactness (worst):                  0.027  1.058
        concavity (worst):                    0.0    1.252
        concave points (worst):               0.0    0.291
        symmetry (worst):                     0.156  0.664
        fractal dimension (worst):            0.055  0.208
        ===================================== ====== ======

        :Missing Attribute Values: None

        :Class Distribution: 212 - Malignant, 357 - Benign

        :Creator:  Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian

        :Donor: Nick Street

        :Date: November, 1995

    This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets.
    https://goo.gl/U2Uwz2

    Features are computed from a digitized image of a fine needle
    aspirate (FNA) of a breast mass.  They describe
    characteristics of the cell nuclei present in the image.

    Separating plane described above was obtained using
    Multisurface Method-Tree (MSM-T) [K. P. Bennett, "Decision Tree
    Construction Via Linear Programming." Proceedings of the 4th
    Midwest Artificial Intelligence and Cognitive Science Society,
    pp. 97-101, 1992], a classification method which uses linear
    programming to construct a decision tree.  Relevant features
    were selected using an exhaustive search in the space of 1-4
    features and 1-3 separating planes.

    The actual linear program used to obtain the separating plane
    in the 3-dimensional space is that described in:
    [K. P. Bennett and O. L. Mangasarian: "Robust Linear
    Programming Discrimination of Two Linearly Inseparable Sets",
    Optimization Methods and Software 1, 1992, 23-34].

    This database is also available through the UW CS ftp server:

    ftp ftp.cs.wisc.edu
    cd math-prog/cpo-dataset/machine-learn/WDBC/

    .. topic:: References

       - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction
         for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on
         Electronic Imaging: Science and Technology, volume 1905, pages 861-870,
         San Jose, CA, 1993.
       - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and
         prognosis via linear programming. Operations Research, 43(4), pages 570-577,
         July-August 1995.
       - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques
         to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994)
         163-171.

Для каждого признака строим гистограмму, подсчитывая частоту встречаемости точек данных в пределах границ интервалов (bins).

Каждый график содержит две наложенные друг на друга гистограммы, первая – для всех точек, относящихся к классу «доброкачественная опухоль» (синий цвет), а вторая – для всех точек, относящихся к классу «злокачественная опухоль» (оранжевый цвет).

Это дает представление о распределении каждого признака по двум классам и позволяет строить предположения о том, какие признаки лучше всего
дискриминируют злокачественные и доброкачественные опухоли.

- Признак «smoothness error» малоинформативен, потому что две гистограммы, построенные для данного признака, большей частью накладываются друг на друга

- Признак «worst concave points» информативен, поскольку гистрограммы, построенные для этого признака, практически не накрывают друг друга.

```python
fig, axes = plt.subplots(15, 2, figsize=(10, 20))
malignant = cancer.data[cancer.target == 0]
benign = cancer.data[cancer.target == 1]
ax = axes.ravel()
for i in range(30):
    _, bins = np.histogram(cancer.data[:, i], bins=50)
    ax[i].hist(malignant[:, i], bins=bins,  alpha=.5)
    ax[i].hist(benign[:, i], bins=bins, alpha=.5)
    ax[i].set_title(cancer.feature_names[i])
    ax[i].set_yticks(())
ax[0].set_xlabel("Значение признака")
ax[0].set_ylabel("Частота")
ax[0].legend(["доброкачественная", "злокачественная"], loc="best")
fig.tight_layout()
```

Отмасштабируем наши данные таким образом, чтобы каждый признак имел единичную дисперсию, воспользовавшись StandardScaler:

```python
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
```

```python
scaler = StandardScaler()
scaler.fit(cancer.data)
X_scaled = scaler.transform(cancer.data)
```

```python
pca = PCA(n_components=2)
pca.fit(X_scaled)
X_pca = pca.transform(X_scaled)
```

```python
print("Форма исходного массива: {}".format(str(X_scaled.shape)))
print("Форма массива после сокращения размерности: {}".format(str(X_pca.shape)))
```

    Форма исходного массива: (569, 30)
    Форма массива после сокращения размерности: (569, 2)

```python
plt.figure(figsize=(8, 8))
scatter = plt.scatter(X_pca[:, 0], X_pca[:, 1], c=cancer.target)

plt.gca().set_aspect("equal")
plt.xlabel("Первая главная компонента")
plt.ylabel("Вторая главная компонента");
```

- Точки можно неплохо разделить даже простой прямой
- Желтые benign (доброкачественные)
- Фиолетовые malignant (злокачественные)
- Главные компоненты представляют собой комбинации исходных признаков

```python
print("форма главных компонент: {}".format(pca.components_.shape))
```

    форма главных компонент: (2, 30)

Каждая строка в атрибуте components_ соответствует одной главной компоненте и они отсортированы по важности (первой приводится первая главная компонента и т.д.).

Столбцы соответствуют атрибуту исходных признаков для объекта PCA в этом примере, «mean radius», «mean texture» и т.д.

```python
print("Первая компонента PCA:\n{}".format(pca.components_[0]))
print("Вторая компонента PCA:\n{}".format(pca.components_[1]))
```

    Первая компонента PCA:
    [0.21890244 0.10372458 0.22753729 0.22099499 0.14258969 0.23928535
     0.25840048 0.26085376 0.13816696 0.06436335 0.20597878 0.01742803
     0.21132592 0.20286964 0.01453145 0.17039345 0.15358979 0.1834174
     0.04249842 0.10256832 0.22799663 0.10446933 0.23663968 0.22487053
     0.12795256 0.21009588 0.22876753 0.25088597 0.12290456 0.13178394]
    Вторая компонента PCA:
    [-0.23385713 -0.05970609 -0.21518136 -0.23107671  0.18611302  0.15189161
      0.06016536 -0.0347675   0.19034877  0.36657547 -0.10555215  0.08997968
     -0.08945723 -0.15229263  0.20443045  0.2327159   0.19720728  0.13032156
      0.183848    0.28009203 -0.21986638 -0.0454673  -0.19987843 -0.21935186
      0.17230435  0.14359317  0.09796411 -0.00825724  0.14188335  0.27533947]

```python
plt.matshow(pca.components_, cmap='viridis')
plt.yticks([0, 1], ["Первая компонента", "Вторая компонента"])
plt.colorbar()
plt.xticks(range(len(cancer.feature_names)),
           cancer.feature_names, rotation=60, ha='left')
plt.xlabel("Характеристика")
plt.ylabel("Главные компоненты");
```

## 2. t-SNE (t-distributed Stochastic Neighbor Embedding, Стохастическое вложение соседей с t-распределением)

Один из алгоритмов множественного обучения (manifold learning algorithms).

Идея, лежащая в основе алгоритма t-SNE, заключается в том, чтобы найти двумерное представление данных, **сохраняющее расстояния между точками** наилучшим образом. t-SNE начинает свою работу со случайного двумерного представления каждой точки данных, а затем пытается сблизить точки, которые в пространстве исходных признаков находятся близко друг к другу, и отдаляет друг от
друга точки, которые находятся далеко друг от друга. При этом t-SNE уделяет большее внимание сохранению расстояний между точками, близко расположенными друг к другу. Иными словами, он пытается сохранить информацию, указывающую на то, какие точки являются соседями друг другу.

```python
from sklearn.datasets import load_digits
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
```

```python
digits = load_digits()
```

```python
fig, axes = plt.subplots(2, 5, figsize=(10, 5),
                         subplot_kw={'xticks':(), 'yticks': ()})
for ax, img in zip(axes.ravel(), digits.images):
    ax.imshow(img, cmap='gray')
```

Применим PCA

```python
pca = PCA(n_components=2)
pca.fit(digits.data)
```

    PCA(copy=True, iterated_power='auto', n_components=2, random_state=None,
        svd_solver='auto', tol=0.0, whiten=False)

```python
# преобразуем данные рукописных цифр к первым двум компонентам
digits_pca = pca.transform(digits.data)
colors = ["#476A2A", "#7851B8", "#BD3430", "#4A2D4E", "#875525",
"#A83683", "#4E655E", "#853541", "#3A3120", "#535D8E"]
```

```python
plt.figure(figsize=(10, 10))
plt.xlim(digits_pca[:, 0].min(), digits_pca[:, 0].max())
plt.ylim(digits_pca[:, 1].min(), digits_pca[:, 1].max())
for i in range(len(digits.data)):
    # строим график, где цифры представлены символами вместо точек
    plt.text(digits_pca[i, 0], digits_pca[i, 1], str(digits.target[i]),
             color = colors[digits.target[i]],
             fontdict={'weight': 'bold', 'size': 9})
plt.xlabel("Первая главная компонента")
plt.ylabel("Вторая главная компонента");
```

Цифры неплохо разделены с помощью только первых двух компонент, но классы сильно перекрывают друг друга

Применим t-SNE

```python
from sklearn.manifold import TSNE
```

```python
tsne = TSNE(random_state=42, n_components=2)
digits_tsne = tsne.fit_transform(digits.data)
```

```python
plt.figure(figsize=(10, 10))
plt.xlim(digits_tsne[:, 0].min(), digits_tsne[:, 0].max() + 1)
plt.ylim(digits_tsne[:, 1].min(), digits_tsne[:, 1].max() + 1)
for i in range(len(digits.data)):
    # строим график, где цифры представлены символами вместо точек
    plt.text(digits_tsne[i, 0], digits_tsne[i, 1], str(digits.target[i]),
             color = colors[digits.target[i]],
             fontdict={'weight': 'bold', 'size': 9})
plt.xlabel("t-SNE признак 0")
plt.xlabel("t-SNE признак 1");
```

Разделение более точное, но некоторые кластеры распадаются (9 и 1)

```python
tsne = TSNE(random_state=42, n_components=3)
digits_3d = tsne.fit_transform(digits['data'])
from mpl_toolkits.mplot3d import Axes3D
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
plt.scatter(digits_3d[:, 0], digits_3d[:,1], digits_3d[:, 2],
            c=digits['target']);
```

# Домашнее задание 3
### Алгоритм PCA

Дана матрица данных $D$ и коэффициент $\alpha$:
1. вычисляем среднее значение $\mu = \frac{1}{n} \sum \limits_{i=1}^{n} x_i$;
2. центруем матрицу данных $Z = D - 1 \cdot \mu^T$;
3. вычисляем матрицу ковариаций $K = \frac{1}{n} (Z^T Z)$;
4. вычисляем собственные значения $(\lambda_1, \lambda_2, \ldots, \lambda_d)$;
5. вычисляем собственные векторы $U = (u_1 u_2 \ldots u_d)$;
6. вычисляем долю общей дисперсии $f(r) = \frac{\sum_{i=1}^{r} \lambda_i} {\sum_{i=1}^{n} \lambda_i}$;
7. выбираем наименьший $r$ так, чтобы $f(r) \geq \alpha$;
8. выбираем $r$ первых векторов $U = (u_1 u_2 \ldots u_r)$;
9. уменьшаем признаковое пространство: $A = \{ a_i \: | \: a_i = U_r^T x_i, \: для \: i = 1, \ldots, n \} $

Реализуйте алгоритм для первых двух главных компонент.

Найдите $\lambda_1$, $\lambda_2$ и $u_1$, $u_2$.

Спроецируйте данные на плоскость образуемую этими векторами, визуализируйте полученные результаты c метками классов.

Сравните реузльтаты с методом PCA из библиотеки sklearn.decomposition.

В качестве данных возьмите изображения с рукописными цифрами.

На основе полученных собственных значений ковариационной матриы $K$ постройте график зависимости покрываемой дисперсии трансформированных данных от количества главных компонент.

# Интерпретация компонент
## Корреляция с исходными признаками
Главные компоненты являются линейными кобинациями исходных признаков. В этой связи необходимо оценить связь каждой компоненты с исходными признаками.

Рассчет взаимосвязи будем проводить с помощью корреляции Пирсона:

\begin{equation}
r_{jk} = \frac{\sum_{i=1}^n (x_{ij} - \bar{x}_j) (x_{ik}' - \bar{x}_k')}{\sqrt{\sum_{i=1}^n (x_{ij} - \bar{x}_j)^2 \sum_{i=1}^n (x_{ik}' - \bar{x}_k')^2}}
\end{equation}

где
$\bar{x}_j$ - среднее значение $j$-го признака,
$\bar{x}_k'$ - среднее значение проекции на $k$-ю главную компоненту,
$n$ - количество объектов.

Отметим, что корреляция Пирсона изменяется от $-1$ до $+1$. Она равна $0$ в случае, когда величины независимы, и $\pm 1$, если они линейно зависимы.

Определите по коэффициентам корреляции, какие признаки в какую компоненту вошли.

```python

```


Universidade Federal do Rio Grande do Sul (UFRGS)
Programa de Pós-Graduação em Engenharia Civil (PPGEC)

# Analysis of 2D and 3D frame structure under seismic load.

### Daniel Barbosa Mapurunga Matos (PPGEC/UFRGS)

```python
import numpy as np
import Vibmec as Vib
import matplotlib.pyplot as plt
import scipy
from scipy import signal

```

## 1. Introduction

This notebook studies the behavior of a ten floor building under seismic load. The building is fixed in a rock foundation and is made from 25 Mpa reinforced concrete. The 2D frame is made from 70 bars and 44 nodes, each one with 3 degree of freedom. The seismic load is generated from the Kanai-Tajimi spectrum.

## 2. The process

First of all, the stifiness matrix and the massa matrix are build from the database spreadsheet

```python
K,M = Vib.Matriz('Resources/X.xlsx',44)  # Importing 2D frame from resources
```

Then, the nodes that are restricted are set on a list, making possible to build te restricted matrix $K_R$ and $M_R$.

```python
Ld = (0,1,2,3,4,5,6,7,8,9,10,11)

Kr,Mr = Vib.Restr(K,M,Ld)
```

Now, with the restricted matrix, its possible to calculate the eigenvalues and eigenvectors, without any numerical problem.

```python
fk,wk = Vib.Eig(Kr,Mr,3,3,10)
```

The eigenvalues corresponds to the natural vibration frequencys of the 2D building, and, with these values, its possible to build, now, the Damper matrix. In this study, its used the Rayleigh Matrix, with $ \zeta_1 = \zeta_2 = 0.01$

```python
Cr = Vib.Rayleigh(Kr,Mr,wk,0.01,0.01)
```

To visualise the consistence of stifness, mass and damping matrices, it's possible to plot then and verify if they have some problem

```python
plt.figure(1,figsize=(12,12))
plt.subplot(1,3,1)
plt.imshow(Kr,'Blues')
plt.subplot(1,3,2)
plt.imshow(Mr)
plt.subplot(1,3,3)
plt.imshow(Cr)
```

Now that the system's characteristics are ready, its necessary to build the seismic load. To make it possible, its used the Kanai-Tajimi spectrum and the Shinozuka and Jan equation to bring it to time domain

```python
t, age,test = Vib.Kanai_Tajimi(0.348,'rocha',50,0.01)

F = Vib.Sismo(Mr,age,t)
```

Note that a envoltory function its used to make the process non-estacionary and note that the load is proportional to the soil acceleration.

Now, its possible to solve the Dynamic equilibrium equation:
\begin{align}
m\ddot{u} + c\dot{u} + ku = F
\end{align}
This equation is solved using the Nermark's method.

```python
d,v,a = Vib.Newmark(Kr,Mr,Cr,F,t)

plt.figure(10,figsize=(8,4))

plt.plot(t,d[117,:]*100,'b')
plt.xlabel('Tempo (s)'); plt.ylabel('Denslocamento (cm)');
plt.xlim(0,50); plt.ylim(-20,20); plt.title('Deslocamento no 10º pavimento')
plt.grid(True)

plt.figure(11,figsize=(8,4))
plt.plot(t,a[117,:] ,'m')
plt.xlabel('Tempo (s)'); plt.ylabel('Aceleração (m/s²)');
plt.xlim(0,50); plt.ylim(-20,20); plt.title('Aceleração no 10º pavimento')
plt.grid(True)
```

Now, to visualize the natural frequencys of the system, it will be used ``Scipy`` to plot the periodogram

```python
plt.figure(16,figsize=(8,4))
f,Saz = scipy.signal.periodogram(a[117,:],(len(a[117,:])/np.max(t)))
plt.plot(f,Saz,'b')
plt.xlabel('frequência(Hz)'); plt.ylabel('Densidade espectral(m²/s³)');
plt.xlim(0,21); plt.title(' Espectro de aceleração')
plt.grid(True)
```

```python
f_max1 =f[Saz.argmax()]
f_max2 =f[137:][Saz.argmax()]

print(' The natural frequencys are:','\n',
     '  F 1:               {0:5.2f}hz'.format(f_max1),'\n',
     '  F 2:               {0:5.2f}hz'.format(f_max2),'\n'
    )

```

     The natural frequencys are:
       F 1:                1.32hz
       F 2:                4.06hz

The 3D frame follows the same procedure. The only differences are in the assembly of the stiffness and mass matrices.

```python
K,M = Vib.Matriz3D('X3D.xlsx',132)

Ld = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]

for i in range (24):
    l1 = Ld[i]+265
    Ld.append(l1)

for i in range (24):
    l1 = Ld[i]+529
    Ld.append(l1)

Kr,Mr = Vib.Restr(K,M,Ld)

fk,wk = Vib.Eig3D(Kr,Mr,2,3,10)

Cr = Vib.Rayleigh(Kr,Mr,wk,0.01,0.01)

```

```python

t, age = Vib.Kanai_Tajimi(0.348,'rocha',50,0.1)

F = Vib.Sismo3D(Mr,age,t)

d,v,a = Vib.Newmark(Kr,Mr,Cr,F,t)

plt.figure(10,figsize=(8,4))

plt.plot(t,d[714,:]*100,'b')
plt.xlabel('Tempo (s)'); plt.ylabel('Denslocamento (cm)');
plt.xlim(0,50); plt.ylim(-15,15); plt.title('Deslocamento no 10º pavimento')
plt.grid(True)

plt.figure(11,figsize=(8,4))
plt.plot(t,a[714,:] ,'m')
plt.xlabel('Tempo (s)'); plt.ylabel('Aceleração (m/s²)');
plt.xlim(0,50); plt.ylim(-20,20); plt.title('Aceleração no 10º pavimento')
plt.grid(True)

```

```python
plt.figure(16,figsize=(8,4))
f,Saz = scipy.signal.periodogram(a[117,:],(len(a[117,:])/np.max(t)))
plt.plot(f,Saz,'b')
plt.xlabel('frequência(Hz)'); plt.ylabel('Densidade espectral(m²/s³)');
plt.xlim(0,21); plt.title(' Espectro de aceleração')
plt.grid(True)
```

```python

```

Analysis of 2D and 3D frame structure under seismic load..ipynb


# Musterlösung zu Übungsblatt 1

 * [Aufgabe 1](#Aufgabe-1)
 * [Aufgabe 2](#Aufgabe-2)
 * [Aufgabe 3](#Aufgabe-3)

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats
plt.style.use('ggplot')
```

---
## Aufgabe 1
Gegeben sei eine parametrische Funktion $y = f(x)$, $y = 1 + a_1x + a_2x^2$ mit Parametern $a_1 = 2.0 ± 0.2$, $a_2 = 1.0 ± 0.1$ und Korrelationskoeffizient $ρ = −0.8$.

---

```python
a1, a1_err = 2.0, 0.2
a2, a2_err = 1.0, 0.1
rho = -0.8
```

---
### 1.1
Geben Sie die Kovarianzmatrix von $a_1$ und $a_2$ an.

---

Die Kovarianzmatrix von $a_1$ und $a_2$ lässt sich wie folgt berechnen:

$$\mathrm{Cov}(a) = \pmatrix{\sigma^2_{a_1} & \rho\sigma_{a_1}\sigma_{a_2} \\
                                               \rho\sigma_{a_1}\sigma_{a_2} & \sigma^2_{a_2}}$$

```python
c12 = rho * a1_err * a2_err
covariance = np.matrix([[a1_err ** 2, c12], [c12, a2_err ** 2]])
covariance
```

    matrix([[ 0.04 , -0.016],
            [-0.016,  0.01 ]])

---
### 1.2
Bestimmen Sie analytisch die Unsicherheit von $y$ als Funktion von $x$:

---

Dazu bestimmen wir zunaechst die Ableitungen von $y$ nach $a_1$ und $a_2$

$$\frac{\partial{}y}{\partial{}a_1} = x \,,\quad \frac{\partial{}y}{\partial{}a_2} = x^2 \,.$$

Daraus können wir die Kovarianz von $y$ nach

$$\sigma^2_y = \mathrm{Cov}(y) = \sum_{ij}\frac{\partial{}y}{\partial{}a_i}\frac{\partial{}y}{\partial{}a_j}\mathrm{Cov}(a)_{ij}$$

bestimmen. Einsetzen der eingangs bestimmten Ableitungen liefert

$$\sigma^2_y = c_{11}x^2 + 2c_{12}x^3 + c_{22}x^4$$

wobei die $c_{ij}$ die Einträge von $\mathrm{Cov}(a)$ sind. Der Ausdruck lässt sich weiter vereinfachen zu

$$\sigma^2_y = x^2\left(\sigma^2_{a_1} + \sigma^2_{a_2}x^2 + 2\rho\sigma_{a_1}\sigma_{a_2}x\right) \,.$$

Daraus ergibt sich für die Unsicherheit

$$ \sigma_y = \lvert{}x\rvert\sqrt{\sigma^2_{a_1} + \sigma^2_{a_2}x^2 + 2\rho\sigma_{a_1}\sigma_{a_2}x} \,.$$

---
#### 1.2.1
unter Vernachlässigung der Korrelation

---

(also für $\rho = 0$) vereinfacht sich der obige Ausdruck zu

$$\sigma_y = \lvert{}x\rvert\sqrt{\sigma^2_{a_1} + \sigma^2_{a_2}x^2} \,.$$

Mit den eingangs berechneten Werten aus `covariance` ergibt sich also

$$\sigma_y = \lvert{}x\rvert\sqrt{0.04 + 0.01x^2} = 0.2\lvert{}x\rvert{}\sqrt{1 + 0.25x^2} \,.$$

```python
def err_ana_wo(x):
    return 0.2 * np.abs(x) * np.sqrt(1 + 0.25 * x ** 2)

xs = np.linspace(-3, 3, 10000)
ys = 1 + a1 * xs + a2 * xs ** 2
errs = err_ana_wo(xs)

plt.plot(xs, ys)
plt.fill_between(xs, ys - errs, ys + errs, alpha=0.5)
plt.show()
```

---
#### 1.2.2
mit Berücksichtigung der Korrelation

---

$$\sigma_y = \lvert{}x\rvert\sqrt{0.04 + 0.01x^2 - 0.016x} = 0.2\lvert{}x\rvert\sqrt{1 - 0.8x + 0.25x^2} \,.$$

```python
def err_ana(x):
    return 0.2 * np.abs(x) * np.sqrt(1 + 0.25 * x ** 2 - 0.4 * x)

errs = err_ana(xs)

plt.plot(xs, ys)
plt.fill_between(xs, ys - errs, ys + errs, alpha=0.5)
plt.show()
```

---
### 1.3
Bestimmen Sie per Monte Carlo die Unsicherheit von $y$ als Funktion von $x$:

#### 1.3.1
Generieren Sie Wertepaare $(a_1, a_2)$ gemäß ihrer Kovarianzmatrix und visualisieren Sie diese, z.B. mit einem Scatter-Plot.

_Hinweis_: Wenn $x_1$ und $x_2$ zwei gaussverteilte Zufallszahlen mit Mittelwert null und Varianz eins sind, erhält man ein Paar korrelierter gaussverteilter Zufallszahlen $(y_1, y_2)$ mit Mittelwert null und Varianz eins durch $(y_1 = x_1; y_2 = x_1ρ + x_2\sqrt{1 − \rho^2})$.

---

```python
x1s, x2s = np.random.normal(size=(2, 10000))
plt.hist2d(x1s, x2s, bins=40)
plt.title('2-dim Normalverteilung')
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.show()

y1s = x1s
y2s = x1s * rho + x2s * np.sqrt(1 - rho ** 2)
plt.hist2d(y1s, y2s, bins=40)
plt.title('2-dim Normalverteilung mit Korrelation')
plt.xlabel('$y_1$')
plt.ylabel('$y_2$')
plt.show()

a1s = a1 + y1s * a1_err
a2s = a2 + y2s * a2_err
plt.hist2d(a1s, a2s, bins=40)
plt.title('Verteilung der $a_1$ und $a_2$')
plt.xlabel('$a_1$')
plt.ylabel('$a_2$')
plt.show()
```

---
#### 1.3.2
Bestimmen Sie die Verteilung von $y$ für $x = \{−1, 0, +1\}$ und vergleichen Sie Mittelwert und Varianz (Standardabweichung) mit den Resultaten der analytischen Rechnung.

---

```python
def y(x, a1, a2):
    return 1 + a1 * x + a2 * x ** 2

def var_analytical(x):
    xx = x ** 2
    #return xx * (covariance[0, 0] + covariance[1, 1] * xx + 2 * covariance[0, 1] * x)
    return 0.04 * xx * (1 + 0.25 * xx - 0.8 * x)

for x in (-1, 0, 1):
    ys = y(x, a1s, a2s)
    mean = np.mean(ys)
    var = np.var(ys)
    print('〈y({})〉= {:.3f}'.format(x, mean))
    print(' σ² = {:.3f}'.format(var))
    print(' analytical = {:.3f}'.format(var_analytical(x)))
    plt.hist(ys, bins=100)
    plt.xlabel('y({})'.format(x))
    plt.ylabel('Absolute Häufigkeit')
    plt.show()
```

Der Fall $x = 0$ ist hier besonders. Da alle Koeffizienten vor Potenzen von $x$ stehen, ergibt sich für den Fall immer $y=0$ unabhängig von den $a_i$. Wir können also keine Aussage über die Varianz treffen.

---
## Aufgabe 2
Betrachten Sie folgende Reparametrisierung von $y = f(x)$:

$$y = 1 + \frac{x(1+x)}{b_1} + \frac{x(1-x)}{b_2}$$

### 2.1
Bestimmen Sie analytisch die transformierten Parameter $b_1$ und $b_2$ und deren Kovarianzmatrix

---

Wir lösen die Reparametrisierung nach Koeffizienten von Potenzen von $x$ auf. Dabei können wir den Term $1$ vernachlässigen, weil er in beiden Definitionen gleichermaßen auftritt.

\begin{align}
 a_1 x + a_2 x^2 &= \frac{x(1 + x)}{b_1} + \frac{x(1 - x)}{b_2} \\
                 &= \frac{x}{b_1} + \frac{x^2}{b_1} + \frac{x}{b_2} - \frac{x^2}{b_2} \\
                 &= x\left(\frac{1}{b_1} + \frac{1}{b_2}\right) + x^2\left(\frac{1}{b_1} - \frac{1}{b_2}\right)
\end{align}

Damit ist

$$a_1 = \left(\frac{1}{b_1} + \frac{1}{b_2}\right) \quad\text{und}\quad a_2 = \left(\frac{1}{b_1} - \frac{1}{b_2}\right)$$

also

$$b_1 = \frac{2}{a_1 + a_2} \quad\text{und}\quad b_2 = \frac{2}{a_1 - a_2} \,.$$

Für die Jacobimatrix der Transformation ergibt sich

\begin{equation}
    M = \pmatrix{
        \frac{-2}{(a_1 + a_2)^2} & \frac{-2}{(a_1 + a_2)^2} \\
        \frac{-2}{(a_1 - a_2)^2} & \frac{+2}{(a_1 - a_2)^2}
    } \quad\text{wobei}\quad m_{ij} = \frac{\partial b_i}{\partial a_j}
\end{equation}

\begin{equation}
    M^T = \pmatrix{
        \frac{-2}{(a_1 + a_2)^2} & \frac{-2}{(a_1 - a_2)^2} \\
        \frac{-2}{(a_1 + a_2)^2} & \frac{+2}{(a_1 - a_2)^2}
    }
\end{equation}

Die transformierte Kovarianzmatrix ist dann $\mathrm{Cov}(b) = M\mathrm{Cov}(a)M^T$.

```python
b1 = 2 / (a1 + a2)
b2 = 2 / (a1 - a2)

denom1 = (a1 + a2) ** 2
denom2 = (a1 - a2) ** 2

M = np.matrix([[-2 / denom1, -2 / denom1],
               [-2 / denom2, 2 / denom2]])
cov_b = M * covariance * M.T
cov_b
```

    matrix([[ 0.00088889,  0.01333333],
            [ 0.01333333,  0.328     ]])

---
### 2.2
Bestimmen Sie die Kovarianzmatrix der transformierten Parameter per Monte Carlo

---

```python
b1s = 2 / (a1s + a2s)
b2s = 2 / (a1s - a2s)

print('b1 = {}'.format(np.mean(b1s)))
print('var = {}'.format(np.var(b1s)))
print('b2 = {}'.format(np.mean(b2s)))
print('var = {}'.format(np.var(b2s)))

plt.hist(b1s, bins=100)
plt.show()
plt.hist(b2s, bins=100)
plt.show()
```

Dabei tritt das Problem auf, dass für einige Kombinationen von Werten für $a_1$ und $a_2$ der Nenner sehr nah an `0` kommt. Dadurch ergeben sich sehr große (unrealistische) Werte für $b_2$. Wir können dem entgegenwirken, indem wir einen Bereich festlegen in dem wir die Werte für $b_2$ erwarten.

```python
cut = np.logical_and(b2s < 5, b2s > 0)

b1s_ = b1s[cut]
b2s_ = b2s[cut]

print('b2 gefiltert = {}'.format(np.mean(b2s_)))
print('var = {}'.format(np.var(b2s_)))

plt.hist(b1s_, bins=100)
plt.show()

plt.hist2d(b1s_, b2s_, bins=100)
plt.show()
```

Wenn wir uns das Leben etwas erleichtern wollen, koennen wir auch einfach die Funktion `cov` aus Numpy verwenden, die uns fuer zwei Arrays direkt die Kovarianzmatrix ausrechnet

```python
ncov_b = np.cov(b1s_, b2s_)
ncov_b
```

    array([[ 0.00083908,  0.01394795],
           [ 0.01394795,  0.44603336]])

---
### 2.3
Bestimmen Sie analytisch die Unsicherheit von $y$ als Funktion von $x$:

#### 2.3.1
unter Verwendung der analytisch bestimmten Kovarianzmatrix von $(b_1, b_2)$

---

Zunächst berechnen wir die partiellen Ableitungen von $y$ nach den Koeffizienten $b_1$ und $b_2$ und damit die Jacobimatrix $M$.

\begin{equation}
    M = \pmatrix{\frac{\partial y}{\partial b_1} \\
                 \frac{\partial y}{\partial b_2}}
      = \pmatrix{\frac{-x(1+x)}{b_1^2}\\
                  \frac{-x(1-x)}{b_2^2}}
\end{equation}

Damit ergibt sich für die Varianz

\begin{align}
    \sigma_y^2 &= M^T \mathrm{Cov}(b) M \\
               &= x^2\left[\frac{c_{11}}{b_1^4}(1+x)^2 + \frac{2c_{12}}{b_1^2b_2^2}(1 - x^2) + \frac{c_{22}}{b_2^4}(1-x)^2\right] \\
               &= x^2\left(\alpha + \beta x + \gamma x^2 \right)
\end{align}

mit den Koeffizienten

\begin{equation}
    \alpha = \left(\frac{c_{11}}{b_1^4} + \frac{2c_{12}}{b_1^2b_2^2} + \frac{c_{22}}{b_2^4}\right) \quad,\quad
    \beta = 2\left(\frac{c_{11}}{b_1^4} - \frac{c_{22}}{b_2^4}\right) \quad\text{und}\quad
    \gamma = \left(\frac{c_{11}}{b_1^4} - \frac{2c_{12}}{b_1^2b_2^2} + \frac{c_{22}}{b_2^4}\right) \,.
\end{equation}

In Zahlen ausgedrückt sind die Koeffizienten

```python
c11 = cov_b[0, 0]
c12 = cov_b[0, 1]
c22 = cov_b[1, 1]
print(cov_b)

a = c11 / b1 ** 4
b = 2 * c12 / b1 ** 2 / b2 ** 2
c = c22 / b2 ** 4

alpha = a + b + c
beta = 2 * (a - c)
gamma = a - b + c

np.sqrt(alpha), alpha/alpha, beta/alpha, gamma/alpha
```

    [[ 0.00088889  0.01333333]
     [ 0.01333333  0.328     ]]

    (0.20000000000000001, 1.0, -0.80000000000000004, 0.25)

Es ist also

\begin{equation}
    \sigma_y = 0.2\lvert{}x\rvert\sqrt{1 - 0.8x - 0.25x^2}
\end{equation}

was exakt das gleiche Ergbnis ist wie für die ursprüngliche Parametrisierung.

```python
def err_ana(x):
    return np.abs(x) * np.sqrt(alpha + beta * x + gamma * x ** 2)

xs = np.linspace(-10, 10, 10000)
ys = 1 + xs * (xs + 1) / b1 + xs * (xs - 1) / b2
errs = err_ana(xs)

plt.plot(xs, ys)
plt.fill_between(xs, ys - errs, ys + errs, alpha=0.5)
plt.show()
```

---
#### 2.3.2
unter Verwendung der numerisch bestimmten Kovarianzmatrix von $(b_1, b_2)$

---

Die gleiche Rechnung mit den per Monte Carlo bestimmten Werten

```python
c11_n = ncov_b[0, 0]
c12_n = ncov_b[0, 1]
c22_n = ncov_b[1, 1]

a_n = c11_n / b1 ** 4
b_n = 2 * c12_n / b1 ** 2 / b2 ** 2
c_n = c22_n / b2 ** 4

alpha_n = a_n + b_n + c_n
beta_n = 2 * (a_n - c_n)
gamma_n = a_n - b_n + c_n

np.sqrt(alpha_n), alpha_n/alpha_n, beta_n/alpha_n, gamma_n/alpha_n
```

    (0.21866956831133924, 1.0, -0.98833238462782003, 0.3436789259019577)

```python
def err_num(x):
    return np.abs(x) * np.sqrt(alpha_n + beta_n * x + gamma_n * x ** 2)

errs = err_num(xs)

plt.plot(xs, ys)
plt.fill_between(xs, ys - errs, ys + errs, alpha=0.5)
plt.show()
```

---
## Aufgabe 3
Lösen Sie die obigen Teilaufgaben für $y = f(x)$ mit

$$y = \ln\left(1 + a_1x + a_2x^2\right) \quad \text{bzw.} \quad y = \ln\left(1 + \frac{x(1+x)}{b_1} + \frac{x(x-1)}{b_2}\right)$$

Im folgenden nennen wir das neue $y$ der Einfachheit halber $z$.

\begin{equation}
    z = \ln(1 + a_1x + a_2x^2) = \ln(y)
\end{equation}

Für die Unsicherheit von $z$ ergibt sich

\begin{align}
    \sigma_z &= \sqrt{\left(\frac{\partial z}{\partial y}\right)^2\sigma_y^2} \\
             &= \sqrt{\left(\frac{1}{y}\right)^2\sigma_y^2} \\
             &= \left\lvert\frac{\sigma_y}{y}\right\rvert \\
             &= \frac{0.2\lvert x\rvert\sqrt{1 + 0.25x^2 - 0.4x}}{\lvert 1 + a_1 x + a_2 x^2 \rvert} \,.
\end{align}

Völlig analog ist die Rechnung für die Reparametrisierung. Hier ergibt sich

\begin{align}
    \sigma_z &= \left\lvert\frac{\sigma_y}{y}\right\rvert \\
             &= \frac{0.2\lvert x\rvert\sqrt{1 + 0.25x^2 - 0.4x}}{\left\lvert 1 + \frac{x(1+x)}{b_1} + \frac{x(x-1)}{b_2}\right\rvert} \,.
\end{align}

```python
def err1(x):
    return err_ana(x) / np.abs(1 + a1 * x + a2 * x ** 2)

def err2(x):
    return err_ana(x) / np.abs(1 + xs * (xs + 1) / b1 + xs * (xs - 1) / b2)

xs = np.linspace(-3, 3, 100)
ys1 = 1 + a1 * xs + a2 * xs ** 2
ys2 = np.log(1 + xs * (xs + 1) / b1 + xs * (xs - 1) / b2)
errs1 = err1(xs)
errs2 = err2(xs)

plt.plot(xs, ys1)
plt.fill_between(xs, ys1 - errs1, ys1 + errs1, alpha=0.5)
plt.show()

plt.plot(xs, ys2)
plt.fill_between(xs, ys2 - errs2, ys2 + errs2, alpha=0.5)
plt.show()
```


# Tutorial rápido de Python para Matemáticos

© Ricardo Miranda Martins, 2022 - http://www.ime.unicamp.br/~rmiranda/

## Índice

1. [Introdução](1-intro.html)
2. [Python é uma boa calculadora!](2-calculadora.html) [(código fonte)](2-calculadora.ipynb)
3. [Resolvendo equações](3-resolvendo-eqs.html)  [(código fonte)](3-resolvendo-eqs.ipynb)
4. [Gráficos](4-graficos.html)  [(código fonte)](4-graficos.ipynb)
5. [Sistemas lineares e matrizes](5-lineares-e-matrizes.html)  [(código fonte)](5-lineares-e-matrizes.ipynb)
6. **[Limites, derivadas e integrais](6-limites-derivadas-integrais.html)** [(código fonte)](6-limites-derivadas-integrais.ipynb)
7. [Equações diferenciais](7-equacoes-diferenciais.html) [(código fonte)](7-equacoes-diferenciais.ipynb)

# Limites

Conhece aquela famosa piadinha? "Tudo tem limites, menos $1/x$ com $x\rightarrow 0$." Matemáticos tem um senso de humor bem peculiar.

O Python sabe muito bem trabalhar com limites, graças ao pacote SymPy. Não é nada recomendável calcular limites fazendo "tabelinhas" ou "aproximações", então o pacote de cálculo simbólico é o ideal.

Para calcular $$\lim_{x\rightarrow a} f(x)$$ o comando é ```sp.limit(f,x,a)```. A variável $x$ precisa anter ser definida.

```python
import sympy as sp
x = sp.symbols('x')
sp.limit(x**2,x,2)
```

Bom, não vamos ficar usando o Python para calcular limites que sabemos calcular de cabeça, só substituindo os valores né? Vamos a alguns mais complicados. Por exemplo, que tal calcular $$\lim_{x\rightarrow 0} \dfrac{\sin(x)}{x}?$$

```python
sp.limit(sp.sin(x)/x,x,0)
```

Isso significa que as funções $f(x)=\sin(x)$ e $g(x)=x$ são muito parecidas, para valores pequenos de $x$. Vamos fazer um gráfico para conferir isso:

```python
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(-0.5, 0.5, 100)
y = np.sin(x)
plt.plot(x, y)
plt.plot(x, x)
```

E os limites que não existem? Também podem ser calculados!

```python
import sympy as sp
x = sp.symbols('x')
sp.limit(1/(x-2),x,2)
```

Podemos calcular também limites com $x\rightarrow\infty$. No SymPy, o símbolo para infinito é ```oo```, lembrando do prefixo ```sp```. Sugestivo, não?

```python
sp.limit(1/x,x,sp.oo)
```

Podemos ainda calcular limites laterais no Python, usando os símboos ```+``` ou ```-``` no comando.

```python
sp.limit(1/x, x, 0, '+')
```

```python
sp.limit(1/x, x, 0, '-')
```

Já vamos começar a falar sobre derivadas, mas podemos calculá-las pela definição, usando limites. Abaixo faremos um exemplo disso.

```python
import sympy as sp
x, h = sp.symbols('x h', real=True)
f=x**2
fh=f.subs(x,x+h)
sp.limit( (fh-f)/h , h, 0)
```

O Python trabalha muito bem com funções definidas por partes. Abaixo fazemos um exemplo disso.

```python
import sympy as sp
x = sp.symbols('x', real=True)
g=sp.Piecewise((x**2+1, x<1),(3*x, x>1))
```

```python
sp.limit(g,x,1,'-')
```

```python
sp.limit(g,x,1,'+')
```

Aviso importante: o Python não é muito bom com funções definidas por partes, tome cuidado com os resultados.

# Derivadas

Se você já fez um curso de cálculo, deve ter percebido que derivação é um processo extremamente mecânico. Não é difícil implementar um "derivador formal". O Python calcula derivadas de funções de uma ou mais variáveis de forma muito eficiente. Novamente vamos usar o SymPy.

Sem mais delongas vamos calcular a derivada de $f(x)=x^2+x$ com respeito à função $x$.

```python
import sympy as sp
x = sp.symbols('x')
f=x**2+x
sp.diff(f,x)
```

Foi rápido, né? A função pode ser muito mais complicada, como por exemplo $g(x)=e^{x^3+x}+1/x$.

```python
g=sp.exp(x**3+x)+1/x
sp.diff(g,x)
```

As funções podem ser de mais que uma variável:

```python
y = sp.symbols('y')
h=sp.cos(x*y)+(3*x+y)/(y**2+1)
```

```python
# derivada em x
sp.diff(h,x)
```

```python
# derivada em y
sp.diff(h,x)
```

```python
# derivada mista, em x depois em y
sp.diff(h,x,y)
```

Claro que as derivadas de ordem superior também podem ser pedidas de forma iterada:

```python
sp.diff(sp.diff(h,x),y)
```

Para derivadas de ordem superior, também é fácil. Abaixo calculamos a derivada de terceira ordem de $h(x,y)$ com respeito a $x$

```python
sp.diff(h,x,x,x)
```

Se você quer a derivada em um ponto, pode usar o ```subs``` para avaliar a derivada nesse ponto:

```python
sp.diff(h,x).subs(x,0).subs(y,0)
```

Claro que a função precisa estar definida no ponto, ou coisas podem dar errado..

```python
sp.diff(g,x).subs(x,0).subs(y,0)
```

## Expansão em séries

Uma importante aplicação das derivadas é o cálculo da expansão em série de Taylor. O SymPy tem dois comandos para isso, o ```fps``` e o ```series```. Vamos testá-los com a função $f(x)=x\cos(x)$.

```python
import sympy as sp
x = sp.symbols('x')
f=x*sp.cos(x)
```

```python
# calculando a expansao formal de f em serie de potencias
# com respeito aa variavel x, no ponto x=0, e truncando
# em ordem 10 usando o fps
fn=sp.fps(f,x,0).truncate(10)
display(fn)
```

```python
# o comando series tambem funciona: a sintaxe é
# series(f(x),x,x0,k), onde x0
fm=sp.series(f,x,0,20).as_expr()
display(fm)
```

Se você não está ligando o nome à pessoa, uma aplicação interessante das séries de potências de uma função é poder calcular aproximações de valores de uma função $f(x)$ que não permite cálculos "diretos".

Por exemplo, não sabemos calcular diretamente $e^2$, mas usando a série da exponencial, podemos aproximar esse valor tão bem quanto quisermos. Vamos usar o sufixo ```removeO``` para remover da série de Taylor a parte que tem os termos em $O(n)$.

```python
import sympy as sp
x = sp.symbols('x')
g = sp.exp(x)
gk = sp.series(g,x,0,10).removeO()
gk.subs(x,2)
```

Portanto, $20947/2835$ é uma boa aproximação racional para $e^2$. Bom, o que estamos fazendo é uma coisa totalmente sem propósito, já que estamos fazendo isso no computador e um simples comando poderia calcular. Mas é aquele famoso "toy problem". Só por curiosidade, note que:

```python
20947//2835
```

Portanto, $e^2$ está perto de 7. Como bônus, se você decorar a série abaixo, poderá tirar aquela onda no churrasco do fim de ano e calcular aproximações para $e^x$ para valores de $x$ próximos de $0$. Certamente, vai ser um sucesso.

```python
sp.series(g,x,0,20).as_expr().removeO()
```

# Integrais

E chegamos nas integrais. O integrador do SymPy é muito rápido e eficiente, e é o companheiro que gostaríamos de ter na hora da prova de cálculo. Resolver uma integral no Python é muito fácil: o comando é ```integrate(f,x)``` ou ```integrate(f,(x,a,b))``` se for uma integral definida, com $x\in[a,b]$.

```python
import sympy as sp
x = sp.symbols('x')
f=x**2
sp.integrate(f,x)
```

```python
import sympy as sp
x = sp.symbols('x')
g=x*sp.exp(x)
sp.integrate(g,x)
```

```python
import sympy as sp
x = sp.symbols('x')
h=sp.exp(x)*sp.cos(x)
sp.integrate(h,x)
```

Bom, como você percebeu, o SymPy teria perdido $0.1$ em cada integral anterior - esqueceu a constante de integração.. coisa feia, Python. Alguns exemplos de integrais definidas:

```python
p=x**3+x
sp.integrate(p,(x,0,1))
```

```python
# vamos tentar enganar o python?
q=1/(x-2)
sp.integrate(q,(x,0,3))
```

Garoto esperto..

O Python também consegue calcular integrais impróprias, lembrando que o símbolo ```oo``` é usado para "infinito".

Atenção: se você tem menos que 18 anos, não execute o próximo comando!!

```python
sp.integrate(sp.exp(-x**2), (x, 0, sp.oo))
```

Mais um exemplo de integral imprópria:

```python
sp.integrate(1/(x**(6)), (x, 1,sp.oo))
```

A integração de funções de várias variávies, principalmente se o domínio for um retângulo, também pode ser feita sem problemas no Python:

```python
x, y, a, b, c, d = sp.symbols("x y a b c d")
f = x*y
sp.integrate(f, (y, a, b), (x, c, d))
```

Também podemos integrar sobre regiões um pouco mais gerais, as chamadas regiões de tipo I/tipo II, ou regiões $R_x$ ou $R_y$:

```python
x, y, a, b, c, d = sp.symbols("x y a b c d")
f = x**2+y
sp.integrate(f, (y, 0, x+1), (x, 0, 1))
```

## Somas de Riemann

Na primeira aula de integrais, começamos com integrais definidas, fazendo umas figuras sobre somas de Riemann. Acredite em mim: é complicado fazer isso manualmente, pois os cálculos são complicados e os desenhos são chatos de fazer. Que tal usar o Python pra facilitar nossa vida? Facilitará tanto o trabalho do professor quanto do aluno, que entenderá melhor.

A implementação abaixo foi adaptada do Mathematical Python [(nesse site)](https://personal.math.ubc.ca/~pwalls/math-python/integration/riemann-sums/).

```python
import numpy as np
import matplotlib.pyplot as plt

# originamente, a função foi definida com a definicao lambda,
# que de fato é mais simples nesse caso - caso queira saber
# mais sobre ela, leia aqui:
# https://stackabuse.com/lambda-functions-in-python/
# f = lambda x : x**2

# definindo a função
def f(x):
    return x**2

# intervalo
a = 0; b = 10;

# numero de retangulos
N = 10

n = 10 # Use n*N+1 points to plot the function smoothly

# discretizando variáveis
x = np.linspace(a,b,N+1)
y = f(x)
X = np.linspace(a,b,n*N+1)
Y = f(X)

# iniciando plot
plt.figure(figsize=(15,5))
plt.subplot(1,3,1)
plt.plot(X,Y,'b')
x_left = x[:-1]
y_left = y[:-1]
plt.plot(x_left,y_left,'b.',markersize=10)
plt.bar(x_left,y_left,width=(b-a)/N,alpha=0.2,align='edge',edgecolor='b')
plt.title('Soma de Riemann com base na esquerda'.format(N))

plt.show()
```

```python

```