# Similarity Matching

Similarity matching attempts to identify similar individuals based on data known about them.

- Given a time series $Q$ in time series database (TSDB), and a **similarity/dissimilarity measure** $d(T,S)$ defined for each pair $T$, $S$ in TSDB, the former query deals with the search of the set of first $k$ time series in TSDB more similar to $Q$. The latter query finds the set $R$ of time series that are within distance $r$ from $Q$.

- There are two ways to post a similarity query in time series databases:
  - whole matching : given a TSDB of time series, each of length n, whole matching relates to computation of similarity matching among time series along their whole length.
  - subsequence matching : given a TSDB of $m$ time series $S_1$, $S_2$, ..., $S_m$, each of length $n_i$, and a short query time series $Q$ of length $n_q < n_i$, with $0 < i < m$, subsequence matching relates to finding matches of $Q$ into subsequences of every $S_i$, starting at every position.

- Distance measures play an important role for similarity problem, in data mining tasks. Concerning a distance measure, it is important to understand if it can be considered metric.

- A **metric function** on a TSDB is a function $f$ : TSDB $\times$ TSDB → $\mathbb{R}$. A metric function obeys to four fundamental properties:
  For all $X$, $Y$, $Z$ in TSDB,
  - **non-negativity** : $f(X,Y)\geq 0$
  - **identity** : $f(X,Y)=0$ if and only if $X=Y$
  - **symmetry** : $f(X,Y) = f(Y,X)$
  - **triangle inequality** : $f(X,Z) \leq f(X,Y) + f(Y,Z)$

## **Euclidean Distance**

**Euclidean Distance** is the most used distance function in many   applications. It is defined as
$$\text{dist}(X,Y) = ||X-Y|| = \sqrt{\sum_{i=1}^{n} (x_i-y_i)^2} \text{ , }$$
where $X=[x_1, x_2, \dots, x_n ]$ and $Y=[y_1, y_2, \dots, y_n]$ in a $n$-dimensional space.

- Euclidean Distance obeys to the metric properties on TSDB: non-negativity, identity, symmetry and triangle inequality.

- In every way, Euclidean distance makes inappropriate their use in certain applications:
    - It compares only time series of the same length.
    - It doesn’t handle outliers or noise.
    - It is very sensitive(敏感的) respect to six signal transformations: shifting, uniform amplitude scaling, uniform time scaling, uniform bi-scaling, time warping and non-uniform amplitude scaling.

## **Dynamic Time Warping**

**Dynamic Time Warping (DTW)** is an algorithm for measuring similarity between two time series which may vary (i.e. warp) in timing. DTW gives more robustness to the similarity computation, because it replaces the one-to-one point comparison with a many-to-one (and viceversa) comparison. The main feature of this distance measure is that it allows to recognize similar shapes, even if they present signal transformations, such as shifting and/or scaling.

- Given two time series $T=\{t_1,t_2,\cdots,t_n\}$ and $S=\{s_1,s_2,\cdots,s_m\}$ of length $n$ and $m$, respectively. The goal of DTW is to find a warping path $\{(p_1,q_1),(p_2,q_2),\cdots,(p_k,q_k)\}$, where $\max(n,m)<k<m+n-1$, such that the distance on this warping path
$$\sum_{i=1}^{k} |t_{p_i}−s_{q_i}|$$ is minimized with $(p_1,q_1)=(1,1)$ and $(p_k,q_k)=(n,m)$. This is a typical example of "anchored beginning" and "anchored end".

- The warping path is subject to several constraints :
  - **Boundary Conditions** : $(p_1,q_1)=(1,1)$ and $(p_k,q_k)=(n,m)$；
  - **Continuity** and **Monotonically non-Decreasing** : for any given node $(i,j)$ in the warping path, the possible fan-in nodes are restricted to $(i−1,j)$, $(i,j−1)$, $(i−1,j−1)$.
  Moreover, for any given element in $T$, we should be able to find at least one corresponding element in $S$, and vice versa.

### **Optimum Warping Path**

The optimum warping path can be efficiently computed using Dynamic Programming (DP, 動態規劃).

- Given two time series $T=\{t_1,t_2,\cdots,t_n\}$ and $S=\{s_1,s_2,\cdots,s_m\}$ of length $n$ and $m$, respectively, a $n \times m$ distance matrix is given by

\begin{equation*}
\text{distMatrix} =
\begin{pmatrix}
d(t_1,s_1) & d(t_1,s_2)  & \cdots & d(t_1,s_m)  \\
d(t_2,s_1)  & d(t_2,s_2)  & \cdots & d(t_2,s_m)  \\
\vdots  & \vdots  & \ddots & \vdots  \\
d(t_n,s_1)  & d(t_n,s_2)  & \cdots & d(t_n,s_m)
\end{pmatrix} \\
\end{equation*}

where $d(t_i, s_j)$, with $1\leq i\leq n$ and $1\leq j\leq m$, is the distance of $i$th point of $T$ and $j$th point of $S$.

- A **cumulative distance matrix $\gamma$** of the same dimension as the distMatrix, is created to store in the cell $(i, j)$ the following value

$$\gamma(i,j)=d(t_i,s_j)+min{\big\{\gamma(i−1,j−1), \gamma(i−1,j), \gamma(i,j−1)\big\}} \text{.}$$

- In many cases, this method can bring to undesired effects. The most criticized disadvantage of DTW is the complexity of space and time, which are both $O(nm)$. A common way to overcome this problem is to  **restrict the warping path in such a way it has to follow a direction along the diagonal**. To do this, we can restrict the path enforcing the recursion to stop at a certain depth, represented by a threshold $\delta$. Then, the cumulative distance matrix $\gamma$ will be calculated as follows:

\begin{equation}
    \gamma(i, j) =
    \begin{cases}
      d(t_i,s_j)+min{\big\{\gamma(i−1,j−1), \gamma(i−1,j), \gamma(i,j−1)\big\}}, \quad |i-j|<\delta \text{,}\\
      \infty, \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad otherwise.
    \end{cases}\
\end{equation}

This constraint, besides limiting extreme or degenerate mappings, allows to speed-up DTW distance calculation, because we need to store only distances which are at most $\delta$ positions away (in horizontal and vertical direction) from the distMatrix diagonal. This reduces the computational complexity to $O((n+m)\delta)$. The above proposed constraint is known also as Sakoe-Chiba Band, and it is classified as global constraint.

Another most common global constraint is the Itakura parallelogram.

# Demo Example Using **dtw** Package

References
  [1] [dtw-python: Dynamic Time Warping in Python](https://dynamictimewarping.github.io/python/)
  [2] [Source code](https://github.com/DynamicTimeWarping/dtw-python) of dtw-python

## Import Libaries

```python
!pip install dtw-python
```

    Collecting dtw-python
      Downloading dtw_python-1.1.10-cp37-cp37m-manylinux2010_x86_64.whl (580 kB)
         |████████████████████████████████| 580 kB 22.7 MB/s
    Requirement already satisfied: scipy>=1.1 in /usr/local/lib/python3.7/dist-packages (from dtw-python) (1.4.1)
    Requirement already satisfied: numpy>=1.19 in /usr/local/lib/python3.7/dist-packages (from dtw-python) (1.19.5)
    Installing collected packages: dtw-python
    Successfully installed dtw-python-1.1.10

```python
import random
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('default')
from dtw import *
```

    Importing the dtw module. When using in academic works please cite:
      T. Giorgino. Computing and Visualizing Dynamic Time Warping Alignments in R: The dtw Package.
      J. Stat. Soft., doi:10.18637/jss.v031.i07.

## Create Two Time Sequences

```python
# Two time series data
x1 = np.arange(0, 20, .5)
x2 = x1 + np.pi/2

s1 = np.sin(x1)
s2 = np.sin(x2)
```

```python
plt.subplots( figsize=(6,6) )

plt.subplot(211)
plt.plot(s1, color='black', label='s1')
plt.legend( loc=1 )
plt.xlim( [-1,41] )

plt.subplot(212)
plt.plot(s2, color='dodgerblue', label='s2')
plt.legend( loc=1 )
plt.xlim( [-1,41] )
plt.xlabel( 'Time Step' )

plt.show()
```

## Sakoe-Chiba Band

```python
alignment = dtw( s1, s2, keep_internals=True,
                 window_type='sakoechiba', window_args={'window_size':10}  )
```

```python
alignment.plot( type='threeway', xlab='s1', ylab='s2' )
plt.show()
```

```python
alignment.plot( type='twoway', offset=-2 )
plt.show()
```

# Demo Example Using **dtaidistance** Package

References
  [1] [DTAIDistance’s documentation](https://dtaidistance.readthedocs.io/en/latest/index.html)
  [2] [Source code](https://github.com/wannesm/dtaidistance)  of dtaidistance

## Import Libaries

```python
!pip install dtaidistance
```

    Collecting dtaidistance
      Downloading dtaidistance-2.3.2.tar.gz (1.1 MB)
         |████████████████████████████████| 1.1 MB 27.3 MB/s
      Installing build dependencies ... done
      Getting requirements to build wheel ... done
        Preparing wheel metadata ... done
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from dtaidistance) (1.19.5)
    Requirement already satisfied: cython>=0.29.6 in /usr/local/lib/python3.7/dist-packages (from dtaidistance) (0.29.24)
    Building wheels for collected packages: dtaidistance
      Building wheel for dtaidistance (PEP 517) ... done
      Created wheel for dtaidistance: filename=dtaidistance-2.3.2-cp37-cp37m-linux_x86_64.whl size=1586354 sha256=133e026c7935db284abda6550fa4fd522d375b6eae45fd25d334aa64f1dc97c0
      Stored in directory: /root/.cache/pip/wheels/5e/4a/9a/30f9efb3926e3c2f28231680685b36d0041466a33e4b8f8367
    Successfully built dtaidistance
    Installing collected packages: dtaidistance
    Successfully installed dtaidistance-2.3.2

```python
import random
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('default')
from dtaidistance import dtw
from dtaidistance import dtw_visualisation as dtwvis
```

## Create Two Time Sequences

```python
# Two time series data
x1 = np.arange(0, 20, .5)
x2 = np.arange(0, 20, .4)
s1 = np.sin(x1)
s2 = np.cos(x2)
```

```python
plt.subplots( figsize=(6,6) )

plt.subplot(211)
plt.plot(s1, color='black', label='s1')
plt.legend( loc=1 )
plt.xlim( [-1,51] )

plt.subplot(212)
plt.plot(s2, color='dodgerblue', label='s2')
plt.legend( loc=1 )
plt.xlim( [-1,51] )
plt.xlabel( 'Time Step' )

plt.show()
```

## DTW Distance Measure

```python
path = dtw.warping_path( s1, s2 )

dtwvis.plot_warping( s1, s2, path )
plt.show()
```

```python
d, paths = dtw.warping_paths( s1, s2, window=10 )
best_path = dtw.best_path( paths )
dtwvis.plot_warpingpaths( s1, s2, paths, best_path )
plt.show()
```

# References

[1] [Similarity Measures and Dimensionality Reduction Techniques for Time Series Data Mining](https://www.intechopen.com/chapters/39030), Carmelo Cassisi, Placido Montalto, Marco Aliotta, Andrea Cannata and Alfredo Pulvirenti, September 2012.
[2] [Time Series Similarity Using Dynamic Time Warping -Explained](https://medium.com/walmartglobaltech/time-series-similarity-using-dynamic-time-warping-explained-9d09119e48ec), Abhishek Mishra, December 2020.


```python
import sys, os
sys.path.insert(0, os.path.join(os.pardir, 'src'))
from fe_approx1D_numint import approximate, u_glob
from sympy import tanh, Symbol, lambdify
x = Symbol('x')

steepness = 20
arg = steepness*(x-0.5)

approximate(tanh(arg), symbolic=False, numint='GaussLegendre4',
            d=3, N_e=1, filename='fe_p3_tanh_1e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre4',
            d=3, N_e=2, filename='fe_p3_tanh_2e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre4',
            d=3, N_e=4, filename='fe_p3_tanh_4e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre5',
            d=4, N_e=1, filename='fe_p4_tanh_1e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre5',
            d=4, N_e=2, filename='fe_p4_tanh_2e')
approximate(tanh(arg), symbolic=False, numint='GaussLegendre5',
            d=4, N_e=4, filename='fe_p4_tanh_4e')

# Interpolation method
import numpy as np
import matplotlib.pyplot as plt
f = lambdify([x], tanh(arg), modules='numpy')

# Compute exact f on a fine mesh
x_fine = np.linspace(0, 1, 101)
f_fine = f(x_fine)

for d in 3, 4:
    for N_e in 1, 2, 4:
        h = 1.0/N_e  # element length
        vertices = [i*h for i in range(N_e+1)]
        cells = [[e, e+1] for e in range(N_e)]
        dof_map = [[d*e + i for i in range(d+1)] for e in range(N_e)]
        N_n = d*N_e + 1  # Number of nodes
        x_nodes = np.linspace(0, 1, N_n)  # Node coordinates
        U = f(x_nodes)  # Interpolation method samples node values
        x, u, _ = u_glob(U, vertices, cells, dof_map,
                         resolution_per_element=51)
        plt.figure()
        plt.plot(x, u, '-', x_fine, f_fine, '--',
                 x_nodes, U, 'bo')
        plt.legend(['%d P%d elements' % (N_e, d),
                    'exact', 'interpolation points'],
                   loc='upper left')
        plt.savefig('tmp_%d_P%d.pdf' % (N_e, d))
        plt.savefig('tmp_%d_P%d.png' % (N_e, d))
plt.show()

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/14_FE_TANH_P3P4.ipynb


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of a Passive Electrical Network

[Electrical networks](https://en.wikipedia.org/wiki/Electrical_network) composed of linear passive elements, like resistors, capacitors and inductors can be described by linear ordinary differential equations (ODEs) with constant coefficients. Hence, in view of the theory of signals and systems they can be interpreted as a linear time-invariant (LTI) system. The different ways to characterize the properties of an LTI system introduced before are illustrated at the example of a second-order analog [low-pass filter](https://en.wikipedia.org/wiki/Low-pass_filter).

It is assumed that no energy is stored in the capacitor and inductor for $t<0$. It is furthermore assumed that $x(t) = 0$ for $t<0$. Hence $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$. For illustration, the normalized values $L = .5$, $R = 1$, $C = .4$ are used for the elements of the electrical network.

### Differential Equation

The differential equation describing the input/output relation of the electrical network is derived by applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) to the network. This results in the following ODE

\begin{equation}
C L \frac{d^2 y(t)}{dt^2} + C R \frac{d y(t)}{dt} + y(t) = x(t)
\end{equation}

This ODE is defined in `SymPy`

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, L, R, C = sym.symbols('t L R C', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(L*C*y.diff(t, 2) + R*C*y.diff(t) + y, x)
ode
```

The normalized values of the network elements are stored in a dictionary for substitution

```python
RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}
RLC
```

### Impulse Response

The passive electrical network and the ODE describing its input/output relation can be interpreted as an LTI system. Hence, the system can be characterized by its [impulse response](impulse_response.ipynb) $h(t)$ which is defined as the output of the system for a Dirac impulse $x(t) = \delta(t)$ at the input. For the given system, the impulse response is calculated by explicit solution of the ODE

```python
solution_h = sym.dsolve(ode.subs(x, sym.DiracDelta(t)).subs(y, sym.Function('h')(t)))
solution_h
```

The integration constants $C_1$ and $C_2$ have to be determined from the initial conditions $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

```python
integration_constants = sym.solve( (solution_h.rhs.limit(t, 0, '-'), solution_h.rhs.diff(t).limit(t, 0, '-')), ['C1', 'C2'] )
integration_constants
```

Substitution of the values for the integration constants $C_1$ and $C_2$ into the result from above yields the impulse response of the low-pass

```python
h = solution_h.subs(integration_constants).rhs
h
```

The impulse response is plotted for the values of $R$, $L$ and $C$ given above

```python
sym.plot(h.subs(RLC), (t,0,10), ylabel=r'h(t)');
```

### Step Response

The [step response](step_response.ipynb) is derived by integrating over the impulse response $h(t)$. For ease of illustration this is performed for the specific values of the elements given above

```python
tau = sym.symbols('tau')

he = sym.integrate(h.subs(RLC).subs(t,tau).subs(sym.Heaviside(tau),1), (tau,0,t))
he
```

Let's plot the step response

```python
sym.plot(he, (t,0,10), ylabel=r'$h_\epsilon(t)$');
```

### Transfer Function

For an exponential input signal $x(t) = e^{s t}$, the [transfer function](eigenfunctions.ipynb#Transfer-Function) $H(s)$ represents the weight of the exponential output signal $y(t) = H(s) \cdot e^{s t}$. The transfer function is derived by introducing $x(t)$ and $y(t)$ into the ODE and solving for $H(s)$

```python
s = sym.symbols('s')
H = sym.Function('H')(s)

H, = sym.solve(ode.subs(x, sym.exp(s*t)).subs(y, H*sym.exp(s*t)).doit(), H)
H
```

The transfer characteristics of an LTI system for harmonic exponential signals $e^{j \omega} = \cos(\omega t) + j \sin(\omega t)$ are of special interest in the analysis of electrical circuits. It can be derived from $H(s)$ by substituting the complex frequency $s$ with $s = j \omega$. The resulting transfer function $H(j \omega)$ provides the attenuation and phase the system adds to a harmonic input signal.

```python
w = sym.symbols('omega')

Hjw = H.subs(s, sym.I * w)
Hjw
```

The magnitude of the transfer function $|H(j \omega)|$ is plotted for illustration for the specific values of the elements given above

```python
sym.plot(abs(Hjw.subs(RLC)), (w, -10, 10), ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

It is obvious, that the low frequencies pass through the system and that high frequencies are attenuated. This motivates the term 'low-pass' for such systems.

As alternative to the solution of the ODE, the transfer function $H(s)$ is [computed from the impulse response](eigenfunctions.ipynb#Link-between-Transfer-Function-and-Impulse-Response) and plotted for the specific values of the elements given above

```python
H2 = sym.integrate(h.subs(RLC)*sym.exp(-s*t), (t,0,sym.oo), conds='none')
H2 = H2.simplify()
H2
```

```python
sym.plot(abs(H2.subs(s, sym.I*w)), (w, -10, 10), ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# "Wild Magic Surges"
- categories: [dnd]
- image: images/dnd/rolls.png

### A Comparison of Two Homebrew Methods

In order to make Wild Magic Surges a more frequent occurence, we can tweak the rules to trigger them. Two such tweaks are:

1. The "Increasing Count" method. Start as usual with a Wild Magic Surge triggering when the player rolls a `1` on their Surge roll. Every time a Surge *does not occur*, increase the D.C. for avoiding the Surge by one: `1` $\rightarrow$ `2` $\rightarrow$ `3`, etc. When a Surge *does* occur, reset the D.C. to 1.

2. The "Decreasing Die" method. Start as usual with a Wild Magic Surge triggering when the player rolls a `1` on their `d20` Surge roll. Every time a surge *does not occur*, decrease the size of the die by one: `d20` $\rightarrow$ `d12` $\rightarrow$ `d10`, etc. When a Surge *does* occur, reset the die to a `d20`. If a player avoids triggering a Surge all the way down through a `d2` (a coin flip), their next Surge is automatic. Or, we can think of this as rolling a `1` on a "`d1`".

Below we calculate the probabilities of triggering a Wild Magic Surge under both of the above systems.

```python
import numpy as np
import pandas as pd
import altair as alt
dice = [20, 12, 10, 8, 6, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
count = [*range(1,21)]

def pDiceSurge(d):
    return 1.0/d
def pDiceNoSurge(d):
    return 1.0-(1.0/d)
def pCountSurge(dc):
    return dc/20.0
def pCountNoSurge(dc):
    return 1-(dc/20.0)
```

First, we calculate the $PDF$ for the Decreasing Dice method — that is, the probability of rolling a Wild Magic surge on the $i^{th}$ roll exactly, no earlier and no later, for each $i$.

This is equal to the probability that we *don't* rolls a surge for the first $i-1$ rolls, times the probability that we *do* roll a surge on the $i^{th}$ roll:

```python
marginalP = list(map(pDiceNoSurge, dice))
dicePDF = []
for i in range(0,20):
    if (i == 0):
        P = pDiceSurge(dice[0])
    else:
        P = np.prod(marginalP[:i]) * pDiceSurge(dice[i])
    dicePDF.append(P)
```

We also calculate the $PDF$ for the Increasing Count method:

```python
marginalP = list(map(pCountNoSurge, count))
countPDF = []
for i in range(0,20):
    if (i == 0):
        P = pCountSurge(count[0])
    else:
        P = np.prod(marginalP[:i]) * pCountSurge(count[i])
    countPDF.append(P)
```

Then we calculate the $CDF$ — that is, the probability of encountering a Wild Magic Surge in $k$ rolls or fewer, for each $k$.
This is just the sum from $i=1$ to $i=k$ of the probabilities of getting a surge in exactly $i$ rolls — a partial sum of the $PDF$ we calculated above:

```python
diceCDF = []
countCDF = []
for i in range(1,21):
    diceCDF.append(np.sum(dicePDF[:i]))
    countCDF.append(np.sum(countPDF[:i]))
```

And now, the fun part, we plot the results!

```python
#collapse
cData = []
pData = []
for i in range(0, len(diceCDF)):
    cData.append([i+1, diceCDF[i], 'Decreasing Die'])
    cData.append([i+1, countCDF[i], 'Increasing Count'])
    pData.append([i+1, dicePDF[i], 'Decreasing Die'])
    pData.append([i+1, countPDF[i], 'Increasing Count'])

df = pd.DataFrame(cData, columns=['Number of Rolls', 'Probability', 'Method'])
df.reset_index()

# Create a selection that chooses the nearest point & selects based on x-value
nearest = alt.selection(type='single', nearest=True, on='mouseover',
                        fields=['Number of Rolls'], empty='none')

points = alt.Chart(df).mark_circle().encode(
    x='Number of Rolls:O',
    y=alt.Y('Probability', title='Probability of a Surge'),
    color='Method',
    opacity=alt.condition(nearest, alt.value(1), alt.value(.6))
)

# Transparent selectors across the chart. This is what tells us
# the x-value of the cursor
selectors = alt.Chart(df).mark_point().encode(
    x='Number of Rolls:O',
    opacity=alt.value(0),
).add_selection(
    nearest
)

# Draw text labels near the points, and highlight based on selection
text = points.mark_text(align='left', dx=5, dy=-5).encode(
    text=alt.condition(nearest, 'Probability:Q',alt.value(' '))
)

# Draw a rule at the location of the selection
rules = alt.Chart(df).mark_rule(color='gray').encode(
    x='Number of Rolls:O',
).transform_filter(
    nearest
)

# Put the five layers into a chart and bind the data
alt.layer(
    selectors, points, rules, text
).properties(
    width=600, height=300
)
```

As you can see, the probability of a Wild Magic Surge is generally higher with the Increasing Count method. This is somewhat expected, as the probability is the same initially for both methods, then at the second roll we have:

$$\begin{align}
P_{\text{Increasing Count}}(S) = \frac{2}{20} &= \frac{1}{10}\\
P_{\text{Decreasing Die}}(S) &= \frac{1}{12}
\end{align}$$

Similarly for the third roll, where the Increasing Count gives a $3/20$ probability of a surge vs. a $1/10$ for the Decreasing Die. So the Increasing Count takes an early lead which it maintains until the seventh roll, where the probability is about even between the methods. After this roll, the Decreasing Die method takes the lead because it gives an automatic Surge from here on out.

My personal preference would lean towards the Increasing Count method, since you could conceivably get a string of rolls that build to a fairly high D.C., which feels a little more dramatic. On the other hand, the Decreasing Die method gives you a guaranteed Surge a fair bit sooner, which is part of the point of these tweaks in the first place.

We can also compute the expected number of rolls to get a Wild Magic Surge for both methods:

```python
def E(pdf):
    ex = 0
    for i in range(0, len(pdf)):
        ex += (i+1)*pdf[i]
    return ex

print("Expectation for Dice  Method:", E(dicePDF))
print("Expectation for Count Method:", E(countPDF))
```

    Expectation for Dice  Method: 5.504768880208333
    Expectation for Count Method: 5.293584586000901

Which is to say, on average the Increasing Count method will give us a Wild Magic Surge slightly sooner. Perhaps another reason to favor it.

For completeness, here is the same data as in the graph, but in a table view:

```python
df.pivot(index='Number of Rolls', columns='Method')
```

      Probability

      Method
      Decreasing Die
      Increasing Count

      Number of Rolls

      1
      0.050000
      0.050000

      2
      0.129167
      0.145000

      3
      0.216250
      0.273250

      4
      0.314219
      0.418600

      5
      0.428516
      0.563950

      6
      0.571387
      0.694765

      7
      0.785693
      0.801597

      8
      1.000000
      0.880958

      9
      1.000000
      0.934527

      10
      1.000000
      0.967264

      11
      1.000000
      0.985269

      12
      1.000000
      0.994107

      13
      1.000000
      0.997938

      14
      1.000000
      0.999381

      15
      1.000000
      0.999845

      16
      1.000000
      0.999969

      17
      1.000000
      0.999995

      18
      1.000000
      1.000000

      19
      1.000000
      1.000000

      20
      1.000000
      1.000000


# Matplotlib

[Matplotlib](http://matplotlib.org) is a Python package for 2D plotting and the `matplotlib.pyplot` sub-module contains many plotting functions to create various kinds of plots. Let's get started by importing `matplotlib.pyplot` and using `%matplotlib` [Jupyter magic](https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-matplotlib) to display plots in the notebook.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## Basic Plotting

### Procedure

The general procedure to create a 2D line plot is:

1. Create a sequence of $x$ values.
2. Create a sequence of $y$ values.
3. Enter `plt.plot(x,y,[fmt],**kwargs)` where `[fmt]` is a (optional) format string and `**kwargs` are (optional) keyword arguments specifying line properties of the plot.
4. Use `pyplot` functions to add features to the figure such as a title, legend, grid lines, etc.
5. Enter `plt.show()` to display the resulting figure.

Let's begin with a basic example with a few random points:

```python
x = [-5,-2,0,1,3]
y = [2,-1,1,-4,3]
plt.plot(x,y)
plt.show()
```

The main things to notice are:

1. The sequences `x` and `y` define the coordinates of the points in the plot.
2. The line in the plot is constructed by connecting the points by straight lines.

The second observation implies that if we want to plot a smooth curve then we need to plot lots of points otherwise the plot will not be smooth. For example, we could try plotting the parabola $y = x^2$ for $x \in [-2,2]$ using only 5 points:

```python
x = [-2,-1,0,1,2]
y = [4,1,0,1,4]
plt.plot(x,y)
plt.show()
```

This is too few points to plot a smooth curve such as $y = x^2$ and so we need more points! Let's try again using the NumPy function `np.linspace` to create 100 points!

```python
x = np.linspace(-2,2,100)
y = x**2
plt.plot(x,y)
plt.show()
```

That's a better representation of the parabola $y = x^2$. Note that the number of points we use in a line plot (100 in this case) is completely arbitrary but the goal is to show a smooth graph for a smooth curve and so we just need to pick a big enough number depending on the function. But be careful not to generate too many points since a *very* large number of points will take a *long* time to plot!

Now that we have the general idea, let's look at adding style and features to our plots!

### Line Properties

A line appearing in a plot has several properties including color, transparency, style, width and markers. We can set these properties when we call `plt.plot` using the following keyword arguments:

| Property | Description |
| :---: | :--- |
| `alpha` | transparency (0.0 transparent through 1.0 opaque) |
| `color` (or `c`) | any matplotlib color |
| `label` | text appearing in legend |
| `linestyle` (or `ls`) | `solid`, `dashed`, `dashdot`, `dotted` |
| `linewidth` (or `lw`) | set width of the line |
| `marker` | set marker style |
| `markeredgecolor` (or `mec`) | any matplotlib color |
| `markerfacecolor` (or `mfc`) | any matplotlib color |
| `markersize` (or `ms`) | size of the marker |

Note that we can specify a [matplotlib color](https://matplotlib.org/api/colors_api.html) in several different ways including by name such as `blue` or `red`, or by a [RGB](https://www.w3schools.com/colors/colors_rgb.asp) tuple such as `(1,0,1)` for purple. For example, let's plot the function

$$
y = e^{-x^2}\cos(2 \pi x) \ \ , \ \ x \in [-2,2]
$$

```python
x = np.linspace(-2,2,41)
y = np.exp(-x**2) * np.cos(2*np.pi*x)
plt.plot(x,y,alpha=0.4,label='Decaying Cosine',
         color='red',linestyle='dashed',linewidth=2,
         marker='o',markersize=5,markerfacecolor='blue',
         markeredgecolor='blue')
plt.ylim([-2,2])
plt.legend()
plt.show()
```

Notice that we used the pyplot function `plt.legend` to display the figure with a legend (showing the line label) and  and `plt.ylim` to set the limits on the vertical axis to `[-2,2]`.

### Format Strings

A [format string](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html#matplotlib-pyplot-plot) gives us a shortcut to add color, markers and line style to a line plot. For example, if we want to plot the function

$$
y = \frac{1}{1 + x^2} \ , \ x \in [-5,5]
$$

with a dashed black line and square markers, we could use keyword arguments:

```python
x = np.linspace(-5,5,41)
y = 1/(1 + x**2)
plt.plot(x,y,color='black',linestyle='dashed',marker='s')
plt.show()
```

Or we could use the corresponding format string `'ks--'` where `k` denotes a black line, `s` a square marker and `--` a dashed line:

```python
x = np.linspace(-5,5,41)
y = 1/(1 + x**2)
plt.plot(x,y,'ks--')
plt.show()
```

Much easier! See below for a list of colors, markers and linestyles.

#### Colors

| Character | Color |
| :---: | :---: |
| `b` | blue |
| `g` | green |
| `r` | red |
| `c` | cyan |
| `m` | magenta |
| `y` | yellow |
| `k` | black |
| `w` | white |

#### Markers

| Character | Marker |
| :---: | :---: |
| `.` | point |
| `o` | circle |
| `v` | triangle down |
| `^` | triangle up |
| `s` | square |
| `p` | pentagon |
| `*` |	star |
| `+` | plus |
| `x` |	x |
| `D` | diamond |

#### Line Styles

| Character | Line Style |
| :---: | :---: |
| `-` | solid line style |
| `--` | dashed line style |
| `-.` | dash-dot line style |
| `:` | dotted line style |

See the [matplotlib.pyplot.plot documentation](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.plot.html) for more options.

### Pyplot Functions

There are many `pyplot` functions available for us to customize our figures. For example:

| Fucntion | Description |
| ---: | :--- |
| `plt.xlim` | set $x$ limits |
| `plt.ylim` | set $y$ limits |
| `plt.grid` | add grid lines |
| `plt.title` | add a title |
| `plt.xlabel` | add label to the horizontal axis |
| `plt.ylabel` | add label to the vertical axis |
| `plt.axis` | set axis properties (`equal`, `off`, `scaled`, etc.) |
| `plt.xticks` | set tick locations on the horizontal axis |
| `plt.yticks` | set tick locations on the vertical axis |
| `plt.legend` | display legend for several lines in the same figure |
| `plt.savefig` | save figure (as .png, .pdf, etc.) to working directory |
| `plt.figure` | create a new figure and set its properties |

See the [pyplot documentation](https://matplotlib.org/api/pyplot_summary.html) for a full list of functions.

## Examples

### Taylor Polynomials

Plot the function $y = \cos(x)$ along with its [Taylor polynomials](https://en.wikipedia.org/wiki/Taylor_series) of degrees 2 and 4.

```python
x = np.linspace(-6,6,50)

# Plot y = cos(x)
y = np.cos(x)
plt.plot(x,y,'b',label='cos(x)')

# Plot degree 2 Taylor polynomial
y2 = 1 - x**2/2
plt.plot(x,y2,'r-.',label='Degree 2')

# Plot degree 4 Taylor polynomial
y4 = 1 - x**2/2 + x**4/24
plt.plot(x,y4,'g:',label='Degree 4')

# Add features to our figure
plt.legend()
plt.grid(True,linestyle=':')
plt.xlim([-6,6])
plt.ylim([-4,4])
plt.title('Taylor Polynomials of cos(x) at x=0')
plt.xlabel('x')
plt.ylabel('y')

plt.show()
```

### Heart Curve

Plot the heart curve:

\begin{align}
x &= 16 \sin^3(t) \\\
y &= 13 \cos(t) - 5 \cos(2t) - 2 \cos(3t) - \cos(4t)
\end{align}

for $t \in [0,2\pi]$.

```python
t = np.linspace(0,2*np.pi,100)

x = 16*np.sin(t)**3
y = 13*np.cos(t) - 5*np.cos(2*t) - 2*np.cos(3*t) - np.cos(4*t)

# Plot line with RGB tuple (red=1, green=0.2, blue=0.5)
# and 20pt line width
plt.plot(x,y,c=(1,0.2,0.5),lw=20)

# Add features to our figure
plt.title('Heart!')
plt.axis('equal')
plt.axis('off')

plt.show()
```

## Subplots

The `plt.subplot` function takes at least 3 inputs $n$, $m$ and $i$ and creates a figure with a $n$ by $m$ grid of subplots and then sets the $i$th subplot (counting across the rows) as the current plot (ie. current axes object).

For example, consider the [sawtooth wave](https://en.wikipedia.org/wiki/Sawtooth_wave)

$$
f(t) = \frac{1}{2} - \frac{1}{\pi} \sum_{k=1}^{\infty} (-1)^k \frac{\sin(2 \pi k t)}{k}
$$

and let $f_N(t)$ denote the $N$th partial sum of the sawtooth wave:

$$
f_N(t) = \frac{1}{2} - \frac{1}{\pi} \sum_{k=1}^{N} (-1)^k \frac{\sin(2 \pi k t)}{k}
$$

Create a 2 by 2 grid of subplots to plot the first 4 partial sums:

\begin{align}
f_1(t) &= \frac{1}{2} + \frac{\sin(2 \pi t)}{\pi} \\\
f_2(t) &= \frac{1}{2} + \frac{\sin(2 \pi t)}{\pi} - \frac{\sin(4 \pi t)}{2\pi} \\\
f_3(t) &= \frac{1}{2} + \frac{\sin(2 \pi t)}{\pi} - \frac{\sin(4 \pi t)}{2\pi} + \frac{\sin(6 \pi t)}{3\pi} \\\
f_4(t) &= \frac{1}{2} + \frac{\sin(2 \pi t)}{\pi} - \frac{\sin(4 \pi t)}{2\pi} + \frac{\sin(6 \pi t)}{3\pi} - \frac{\sin(8 \pi t)}{4\pi}
\end{align}

```python
t = np.linspace(0,4,200)

fN = 1/2
for N in [1,2,3,4]:
    fN = fN - (-1)**N * np.sin(2*N*np.pi*t)/(N*np.pi)
    plt.subplot(2,2,N)
    plt.plot(t,fN)
    plt.title('N = {}'.format(N))

plt.tight_layout()
plt.show()
```

See the [documentation](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.subplot.html) for more about subplots.

## Beyond Line Plots

### Scatter plots

A [scatter plot](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.scatter.html) has 4 dimensions: $x$ coordinate, $y$ coordinate, size and color. Let's make a random scatter plot:

```python
# Set the number of dots in the plot
N = 2000

# Create random x and y coordinates sampled uniformly from [0,1]
x = np.random.rand(N)
y = np.random.rand(N)

# Create random array sampled uniformly from [20,120]
# `size` array is used below to set the size of each dot
size = 100*np.random.rand(N) + 20

# Create random 4-tuples sampled uniformly from [0,1]
# `colors` array is used below to set the color
# (red,green,blue,alpha) of each dot
colors = np.random.rand(N,4)

# Create a figure of size 12 by 5 and create scatter plot
plt.figure(figsize=(12,5))
plt.scatter(x,y,c=colors,s=size)
plt.axis('off')
plt.show()
```

### Histograms

Generate an array of 10000 random numbers sampled from the normal distribution and create a [histogram](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.hist.html). Let's also superimpose the normal distribution:

$$
y = \frac{1}{\sqrt{2\pi}} e^{-x^2/2}
$$

```python
samples = np.random.randn(10000)
plt.hist(samples,bins=20,density=True,alpha=0.5,color=(0.3,0.8,0.1))
plt.title('Random Samples - Normal Distribution')
plt.ylabel('Frequency')

x = np.linspace(-4,4,100)
y = 1/(2*np.pi)**0.5 * np.exp(-x**2/2)
plt.plot(x,y,'b',alpha=0.8)

plt.show()
```

### Bar plots

Plot the [total precipitation in Vancouver](https://vancouver.weatherstats.ca/charts/precipitation-monthly.html) by month as a [bar plot](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.bar.html):

```python
month = range(1,13)
precipitation = [98.8,128.8,206.0,138.5,102.2,46.4,1.8,5.0,29.4,114.8,197.0,170.6]

plt.bar(month,precipitation)

plt.xticks(month)
plt.yticks(range(0,300,50))
plt.grid(True,alpha=0.5,linestyle='--')
plt.title('Precipitation in Vancouver, 2017')
plt.ylabel('Total Precipitation (mm)')
plt.xlabel('Month')

plt.show()
```

Python/3. Computational Sciences and Mathematics/Plotting Functions/Intro to Matplotlib.ipynb


The probability distribution for a random variable describes how the probabilities are distributed over the values of the random variable.

For a discrete random variable, x, the probability distribution is defined by a probability mass function, denoted by f(x). This function provides the probability for each value of the random variable.

For a continuous random variable , since there is an infinite number of values in any interval, the probability that a continuous random variable will lie within a given interval is considered. So here, the probability distribution is defined by probability density function, also denoted by f(x).

The output of a probability mass function is a probability whereas the area under the curve produced by a probability density function represents a probability.

Both probability functions must satisfy two requirements::

f(x) must be non-negative for each value of the random variable, and

The sum of the probabilities for each value (or integral over all values) of the random variable must equal one.

Common Data Types
Before we jump on to the explanation of distributions, let’s see what kind of data can we encounter. The data can be discrete or continuous.

Discrete Data, as the name suggests, can take only specified values. For example, when you roll a die, the possible outcomes are 1, 2, 3, 4, 5 or 6 and not 1.5 or 2.45.

Continuous Data can take any value within a given range. The range may be finite or infinite. For example, A girl’s weight or height, the length of the road. The weight of a girl can be any value from 54 kgs, or 54.5 kgs, or 54.5436kgs.

Types of Distributions

1. Bernoulli Distribution

All you cricket junkies out there! At the beginning of any cricket match, how do you decide who is going to bat or ball? A toss! It all depends on whether you win or lose the toss, right? Let’s say if the toss results in a head, you win. Else, you lose. There’s no midway.

A Bernoulli distribution has only two possible outcomes, namely 1 (success) and 0 (failure), and a single trial. So the random variable X which has a Bernoulli distribution can take value 1 with the probability of success, say p, and the value 0 with the probability of failure, say q or 1-p.

Here, the occurrence of a head denotes success, and the occurrence of a tail denotes failure.
Probability of getting a head = 0.5 = Probability of getting a tail since there are only two possible outcomes.

The probability mass function is given by:

\begin{equation}
p^{x}(1-p)^{1-x} \quad where\;x\in (0, 1)
\end{equation}

The probabilities of success and failure need not be equally likely, like the result of a fight between me and Undertaker. He is pretty much certain to win. So in this case probability of my success is 0.15 while my failure is 0.85

Here, the probability of success(p) is not same as the probability of failure.

Here, the probability of success = 0.15 and probability of failure = 0.85. The expected value is exactly what it sounds. If I punch you, I may expect you to punch me back. Basically expected value of any distribution is the mean of the distribution. The expected value of a random variable X from a Bernoulli distribution is found as follows:

\[E(X) = 1*p + 0*(1-p) = p\]

The variance of a random variable from a bernoulli distribution is:

\[V(X) = E(X²) – [E(X)]² = p – p² = p(1-p)\]

There are many examples of Bernoulli distribution such as whether it’s going to rain tomorrow or not where rain denotes success and no rain denotes failure and Winning (success) or losing (failure) the game.

2. Uniform Distribution

When you roll a fair die, the outcomes are 1 to 6. The probabilities of getting these outcomes are equally likely and that is the basis of a uniform distribution. Unlike Bernoulli Distribution, all the n number of possible outcomes of a uniform distribution are equally likely.

A variable X is said to be uniformly distributed if the density function is:

\begin {equation}
f(x) = \frac{1}{b-a} \quad for -\infty<a\leq x\leq b<\infty
\end {equation}

You can see that the shape of the Uniform distribution curve is rectangular, the reason why Uniform distribution is called rectangular distribution.

For a Uniform Distribution, a and b are the parameters.

The number of bouquets sold daily at a flower shop is uniformly distributed with a maximum of 40 and a minimum of 10.

Let’s try calculating the probability that the daily sales will fall between 15 and 30.

The probability that daily sales will fall between 15 and 30 is (30-15)*(1/(40-10)) = 0.5

Similarly, the probability that daily sales are greater than 20 is  = 0.667

The mean and variance of X following a uniform distribution is:

Mean $E(X) = \frac{a+b}{2}$

Variance $V(X) = \frac {(b-a)^2}{12}$

The standard uniform density has parameters a = 0 and b = 1.

3. Binomial Distribution
Suppose that you won the toss today and this indicates a successful event. You toss again but you lost this time. If you win a toss today, this does not necessitate that you will win the toss tomorrow. Let’s assign a random variable, say X, to the number of times you won the toss. What can be the possible value of X? It can be any number depending on the number of times you tossed a coin.

There are only two possible outcomes. Head denoting success and tail denoting failure. Therefore, probability of getting a head = 0.5 and the probability of failure can be easily computed as: q = 1- p = 0.5.

A distribution where only two outcomes are possible, such as success or failure, gain or loss, win or lose and where the probability of success and failure is same for all the trials is called a Binomial Distribution.

The outcomes need not be equally likely. Remember the example of a fight between me and Undertaker? So, if the probability of success in an experiment is 0.2 then the probability of failure can be easily computed as q = 1 – 0.2 = 0.8.

Each trial is independent since the outcome of the previous toss doesn’t determine or affect the outcome of the current toss. An experiment with only two possible outcomes repeated n number of times is called binomial. The parameters of a binomial distribution are n and p where n is the total number of trials and p is the probability of success in each trial.

On the basis of the above explanation, the properties of a Binomial Distribution are

Each trial is independent.

There are only two possible outcomes in a trial- either a success or a failure.

A total number of n identical trials are conducted.

The probability of success and failure is same for all trials. (Trials are identical.)

The mathematical representation of binomial distribution is given by:
\[P(x) = \frac {n!}{(n-x)! x!} p^x q^(n-x) \]

A binomial distribution graph where the probability of success does not equal the probability of failure looks like

Now, when probability of success = probability of failure, in such a situation the graph of binomial distribution looks like

The mean and variance of a binomial distribution are given by:

Mean $ µ = n*p$

Variance $Var (X) = n*p*q$

4.Normal Distribution

Normal distribution represents the behavior of most of the situations in the universe. The large sum of (small) random variables often turns out to be normally distributed, contributing to its widespread application. Any distribution is known as Normal distribution if it has the following characteristics:

The mean, median and mode of the distribution coincide.

The curve of the distribution is bell-shaped and symmetrical about the line x=μ.

The total area under the curve is 1.

Exactly half of the values are to the left of the center and the other half to the right.

A normal distribution is highly different from Binomial Distribution. However, if the number of trials approaches infinity then the shapes will be quite similar.

The PDF of a random variable X following a normal distribution is given by:
\[ f(x)=\frac {1}{\sqrt{(2\pi)}\sigma}e^{-\frac {(x-\mu)^2}{2\sigma}}\]

The mean and variance of a random variable X which is said to be normally distributed is given by:

Mean $ E(X) = \mu$

Variance $ Var(X) = (\sigma)^2$

Here, µ (mean) and σ (standard deviation) are the parameters.

A standard normal distribution is defined as the distribution with mean 0 and standard deviation. For such a case, the PDF becomes:
\[ f(x) = \frac {1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}} \]

5. Poisson Distribution

Suppose you work at a call center, approximately how many calls do you get in a day? It can be any number. Now, the entire number of calls at a call center in a day is modeled by Poisson distribution. Some more examples are

The number of emergency calls recorded at a hospital in a day.
The number of thefts reported in an area on a day.
The number of customers arriving at a salon in an hour.
The number of suicides reported in a particular city.
The number of printing errors at each page of the book.
You can now think of many examples following the same course. Poisson Distribution is applicable in situations where events occur at random points of time and space wherein our interest lies only in the number of occurrences of the event.

A distribution is called Poisson distribution when the following assumptions are valid:

Any successful event should not influence the outcome of another successful event.

The probability of success over a short interval must equal the probability of success over a longer interval.

The probability of success in an interval approaches zero as the interval becomes smaller.

Now, if any distribution validates the above assumptions then it is a Poisson distribution. Some notations used in Poisson distribution are:

$\lambda$ is the rate at which an event occurs,
t is the length of a time interval,
And X is the number of events in that time interval.
Here, X is called a Poisson Random Variable and the probability distribution of X is called Poisson distribution.

Let $\mu$ denote the mean number of events in an interval of length t. Then, µ = λ*t.

The PMF of X following a Poisson distribution is given by:
\begin{equation}
P(X=x)= e^{-\mu{\frac{\mu^x}{x!}}}
\end{equation}

The mean µ is the parameter of this distribution. µ is also defined as the λ times length of that interval.
The mean and variance of X following a Poisson distribution:

Mean $E(X) = \mu$

Variance $Var(X) = \mu$

6. Exponential Distribution

Let’s consider the call center example one more time. What about the interval of time between the calls ? Here, exponential distribution comes to our rescue. Exponential distribution models the interval of time between the calls.

Other examples are:

Length of time beteeen metro arrivals.

Length of time between arrivals at a gas station.

The life of an Air Conditioner.

Exponential distribution is widely used for survival analysis. From the expected life of a machine to the expected life of a human, exponential distribution successfully delivers the result.

A random variable X is said to have an exponential distribution with PDF:
\begin{equation}
f(x) =  \lambda e - \lambda x,  x \geq 0
\end{equation}

and parameter λ>0 which is also called the rate.

For survival analysis, λ is called the failure rate of a device at any time t, given that it has survived up to t.

Mean and Variance of a random variable X following an exponential distribution:

Mean $E(X) = \frac {1}{\lambda}$

Variance $Var(X) = \frac {1}{\lambda^2}$

Also, the greater the rate, the faster the curve drops and the lower the rate, flatter the curve.
To ease the computation, there are some formulas given below.
$P(X\leq x) = 1 – e-\lambda x$, corresponds to the area under the density curve to the left of x.

$ P(X>x) = e^-\lambda x$, corresponds to the area under the density curve to the right of x.

$ P(x1<X\leq x2) = e^{-\lambda x1} – e^{-\lambda x2}$, corresponds to the area under the density curve between x1 and x2.

Relations between the Distributions.
Relation between Bernoulli and Binomial Distribution.

Bernoulli Distribution is a special case of Binomial Distribution with a single trial.

There are only two possible outcomes of a Bernoulli and Binomial distribution, namely success and failure.

Both Bernoulli and Binomial Distributions have independent trails.

Relation between Poisson and Binomial Distribution

Poisson Distribution is a limiting case of binomial distribution under the following conditions:

The number of trials is indefinitely large or n → ∞.

The probability of success for each trial is same and indefinitely small or p →0.

np = λ, is finite.

Relation between Normal and Binomial Distribution & Normal and Poisson Distribution:

Normal distribution is another limiting form of binomial distribution under the following conditions:

The number of trials is indefinitely large, n → ∞.

Both p and q are not indefinitely small.

The normal distribution is also a limiting case of Poisson distribution with the parameter λ →∞.

Relation between Exponential and Poisson Distribution:

If the times between random events follow exponential distribution with rate λ, then the total number of events in a time period of length t follows the Poisson distribution with parameter λt.

```python

```


The Equations of Ideal Magnetohydrodynamics (MHD)

The ideal MHD equations are in terms of the eight variables in addition to time t which are conserved in a volume:
density $\rho$, momentum, $\rho \mathbf u$, magnetic flux density $\mathbf B$ and energy density, $E$.
These equations expressed in conservation form are as follows:

Conservation of Mass:
\begin{equation}
\frac{\partial \rho}{\partial t}+\boldsymbol{\nabla}\cdot(\rho \mathbf u)=0
\end{equation}
Conservation of Momentum:
\begin{equation}
\frac{\partial}{\partial t}\left(
\rho \mathbf u
\right)
+\boldsymbol{\nabla}
\cdot
\left[
\rho \mathbf u \otimes \mathbf u
+\left(
p_{B}
\right)
\mathbf{{\overline {\overline I}}}
-\mathbf{B} \otimes \mathbf{B}
\right]
=0
\end{equation}
Conservation of Flux:
\begin{equation}
\frac{\partial \mathbf{B} }{\partial t}+
\boldsymbol{\nabla}
\cdot
(
\mathbf{u} \otimes \mathbf{B}-
\mathbf{B} \otimes \mathbf{u})
= 0
\end{equation}
Conservation of Energy:
\begin{equation}
\frac{\partial E}{\partial t}
+\boldsymbol{\nabla}
\cdot
\left[
\left( E + p_{B}\right) \mathbf{u}
-(\mathbf{u}\cdot\mathbf{B})\mathbf{B}
\right]  = 0
\end{equation}
where $E$ is the total energy, kinetic, internal and magnetic
\begin{equation}
E =
\frac{1}{2}\rho |\mathbf{u}|^2 +
\frac{p}{\gamma-1}+
\frac{1}{2} |\mathbf{B}|^2
\end{equation}
and $p_{B}$ is the sum of thermal and magnetic pressure:
\begin{equation}
p_{B} = p + \frac{1}{2}|\mathbf{B}|^2
\end{equation}
The units are chosen so that $\mathbf B$ absorbs a factor of $1/\sqrt {4 \pi}$.
The adiabatic index is $\gamma = 5/3$ for a monatomic gas throughout the simulations.
$\mathbf{I}$ is the identity tensor.
There are eight equations and nine variables. In order to close the system one more equation is required, the equation of state.
The equation of state is chosen to be the ideal gas equation.

```python

```


# State Space Class : Python

## Discrete Time

* Let's consider a time-invariant discret time state space representation as follows.

$$
\begin{align}
    \mathbf{x}_{k+1}&=\mathbf{Ax}_{k}+\mathbf{Bu}_{k} \\
    \mathbf{y}_{k}&=\mathbf{Cx}_{k}+\mathbf{Du}_{k}
\end{align}
$$

* If $\mathbf{D} = 0$ we can rewrite as follows.

$$
\begin{align}
    \mathbf{x}_{k+1}&=\mathbf{Ax}_{k}+\mathbf{Bu}_{k} \\
    \mathbf{y}_{k}&=\mathbf{Cx}_{k}
\end{align}
$$

* Here, we can think about following analogies.

|              |        State space        |      Class       |
|:------------:|:-------------------------:|:----------------:|
| $\mathbf{x}$ | vector of state variables | member variables |
| $\mathbf{B}$ |       input matrix        |  mutator method  |
| $\mathbf{u}$ | vector of input variables | mutator method argument |
| $\mathbf{C}$ |       output matrix       |  reader method  |
| $\mathbf{y}$ |    measurement vector     | return values of reader method |

* Let's see if the analogies above works.

```python
import numpy as np

```

```python
class LTI_DT(object):

    def __init__(self, A, B, C, D, x0=None,):

        # Set state space representation
        self.A = A
        self.B = B
        self.C = C
        self.D = D

        # is A matrix square?
        assert A.shape[0] == A.shape[1]

        # number of state variables
        self.n = A.shape[0]

        # check number of rows of B matrix
        assert B.shape[0] == self.n

        # expected size of input
        self.m = B.shape[1]

        # Initialize state column vector
        # https://stackoverflow.com/questions/12575421/convert-a-1d-array-to-a-2d-array-in-numpy
        self.x = np.array(x0).reshape((self.n, 1))

    def get_y(self, u_array=None):
        """
        y[k] = C x[k] + D u[k]
        """
        result = self.C @ self.x
        try:
            if u_array is not None:
                result += self.D @ u_array.reshape((self.m, 1))
        except ValueError as e:
            print(f'self.D = \n{self.D}')
            print(f'self.D.shape = \n{self.D.shape}')
            print(f'u_array = \n{u_array}')
            print(f'u_array.shape = \n{u_array.shape}')
            print(f'self.D @ u_array = \n{self.D @ u_array}')
            print(f'(self.D @ u_array).shape = \n{(self.D @ u_array).shape}')
            print(f'result = \n{result}')
            print(f'result.shape = \n{result.shape}')
            raise e

        return result

    def get_next_x(self, u_array=None):
        """
        x[k+1] = A x[k] + B u[k]
        """
        next_x = self.A @ self.x

        if u_array is not None:
            next_x += self.B @ u_array.reshape((self.m, 1))

        self.x = next_x

```

* Let's consider one example

```python
# Peng & Chiu, Design of Digital Control Systems, 2000.

A = np.array(
    [
        [0, 1],
        [-0.8, 1.8]
    ]
)

B = np.array([[0, 1]]).T

C = np.array(
    [
        [1, 0],
        [0, 1],
        [-0.5, 1],
    ]
)

D = np.zeros((C.shape[0], B.shape[1]))

u_array = np.array([0.15, 0.075, 0.0375, 0.01875])

```

```python
y_list = []
k_list = []
x0 = np.array([1, 0.5]).T
ss_dt = LTI_DT(A, B, C, D, x0)

for k, u in enumerate(u_array):
    k_list.append(k)
    y_now = ss_dt.get_y(u)
    y_list.append(y_now)
    ss_dt.get_next_x(u)

y_array = np.array(y_list)

y_array.squeeze().T

```

* Let's think about another example

```python
# https://ccrma.stanford.edu/~jos/fp/State_Space_Simulation_Matlab.html

A = np.array(
    [
        [0, 1],
        [-1, 0]
    ]
)

B = np.array([[0, 1]]).T

C = np.array(
    [
        [1, 0],
        [0, 1],
        [0, 1],
    ]
)

D = np.zeros((C.shape[0], B.shape[1]))

u_array = np.zeros((10)).T
u_array[0] = 1.0

```

```python
y_list = []
k_list = []
x0 = np.array([0, 0.0]).T
ss_dt = LTI_DT(A, B, C, D, x0)

for k, u in enumerate(u_array):
    k_list.append(k)
    y_now = ss_dt.get_y(u)
    y_list.append(y_now)
    ss_dt.get_next_x(u)

y_array = np.array(y_list)

y_array.squeeze().T

```

* For a possible C++ version please see if you can find in another file.

## Continuous Time

* A state space representation may calculate time derivatives of the state vector as follows.

$$
\begin{align}
    \frac{d}{dt}\mathbf{x}(t)&=\mathbf{Ax}(t)+\mathbf{Bu}(t) \\
    \mathbf{y}(t)&=\mathbf{Cx}(t)+\mathbf{Du}(t)
\end{align}
$$

* Assume that at time $t_i$ we know the state $\mathbf{x}(t_i)$ and want to find state $\mathbf{x}(t_{i+1})$ at time $t_{i+1} = t_i + \Delta t$

* For simplicity, we may apply Euler's method to find $\mathbf{x}(t_{i+1})$.

$$
    \mathbf{x}(t_{i+1}) =  \mathbf{x}(t_{i}) + \Delta t \frac{d}{dt}\mathbf{x}(t_i)
$$

* Let's implement by inheriting discrete time model.

```python
class LTI_CT(LTI_DT):
    def __init__(self, A, B, C, D, delta_t, x0=None, t0=0):
        super().__init__(A, B, C, D, x0)
        self.delta_t = delta_t
        self.t = t0

    def get_dx_dt(self, u_t=None):
        """
        dx_dt(t[i]) = A x(t[i]) + B u(t[i])
        """

        dx_dt_now = self.A @ self.x

        if u_t is not None:
            dx_dt_now += self.B @ u_t.reshape((self.m, 1))

        return dx_dt_now

    def get_next_x(self, u_t=None):
        """
        Euler method
        x(t[i+1]) = x(t[i]) + delta_t * dx_dt(t[i])
        """

        next_x = self.x + self.delta_t * self.get_dx_dt(u_t)

        self.x = next_x
        self.t += self.delta_t

```

* Let's think about following vibration system:

$$
    \begin{align}
        m \frac{d^2y(t)}{{dt}^2} &+ c \frac{dy(t)}{dt} + k y(t) = u(t) \\
        m \frac{d^2y(t)}{{dt}^2} &= - k y(t) - c \frac{dy(t)}{dt} + u(t) \\
        \mathbf{x}(t) &= \left(y(t), \frac{dy(t)}{dt} \right)^T \\
        \frac{d}{dt}\mathbf{x}(t) &= \left(\frac{dy(t)}{dt} , \frac{d^2y(t)}{{dt}^2} \right)^T \\
        &= \begin{bmatrix}
                0 & 1 \\
                -\frac{k}{m} & -\frac{c}{m} \\
            \end{bmatrix}  \mathbf{x} (t)
          + \begin{bmatrix}
                0\\
                \frac{1}{m}\\
            \end{bmatrix}  u (t)
    \end{align}
$$

```python
m = 1 # kg
c = 100 # N / m / s
k = 10000 # N / m

A = np.array(
    [
        [0, 1],
        [-k/m, -c/m]
    ]
)

B = np.array([[0, 1/m]]).T

C = np.array(
    [
        [1, 0],
        [0, 1],
    ]
)

D = np.zeros((C.shape[0], B.shape[1]))

w_hz = 1.0

t_end_sec = 5 / w_hz
delta_t_sec = t_end_sec * 1e-3

t_array = np.arange(0, t_end_sec+delta_t_sec*0.5, delta_t_sec)
u_array = np.sin((2 * np.pi * w_hz) * t_array)

```

```python
import matplotlib.pyplot as plt

y_list = []
t_list = []

x0 = np.array([0, 0.0]).T
delta_t_sec = 1e-3

ss_ct = LTI_CT(A, B, C, D, delta_t_sec, x0)

for u in u_array:
    t_list.append(ss_ct.t)
    y_now = ss_ct.get_y(u)
    y_list.append(y_now)
    ss_ct.get_next_x(u)

y_array = np.array(y_list).squeeze()

plt.subplot(2, 1, 1)
plt.plot(t_list, y_array[:, 0])
plt.grid(True)
plt.xlabel('t(sec)')
plt.ylabel('y(m)')

plt.subplot(2, 1, 2)
plt.plot(t_list, y_array[:, 1])
plt.grid(True)
plt.xlabel('t(sec)')
plt.ylabel('dy/dt(m/s)')

plt.savefig('lti_ct_ss.svg')

# If plot does not show up, please uncomment following lines
# import IPython.display as disp
# disp.display(disp.SVG('lti_ct_ss.svg'))
# https://stackoverflow.com/questions/30334385/display-svg-in-ipython-notebook-from-a-function

```

## Exercise

### 00 Continuous time version in C++

* Try to implement Continuous Time State Space model in C++
* Group assignment possible

### 01 State Space Simulation

* Implement a state space simulation and visualize using `matplotlib`.

* Following table suggests some of the possibilities.

|     model     | language |      visualization       |
|:-------------:|:--------:|:------------------------:|
| bicycle model |  python  |      `matplotlib`        |
| bicycle model |  cython  |      `matplotlib`        |
| bicycle model |   c++    | file i/o + `matplotlib`  |
| bicycle model |   c++    | `cython` + `matplotlib`  |
| path tracker  |  python  |      `matplotlib`        |
| path tracker  |  cython  |      `matplotlib`        |
| path tracker  |   c++    | file i/o + `matplotlib`  |
| path tracker  |   c++    | `cython` + `matplotlib`  |

```python

```


# Comparing Data
You'll often want to compare data in your dataset, to see if you can discern trends or relationships.

## Univariate Data
*Univariate* data is data that consist of only one variable or feature. While it may initially seem as though there's not much we can do to analyze univariate data, we've already seen that we can explore its distribution in terms of measures of central tendency and measures of variance. We've also seen how we can visualize this distribution using histograms and box plots.

Here's a reminder of how you can visualize the distribution of univariate data, using our student grade data with a few additional observations in the sample:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

plt.figure()
df['Grade'].plot( kind='box', title='Grade Distribution')
plt.figure()
df['Grade'].hist(bins=9)
plt.show()
print(df.describe())
print('median: ' + str(df['Grade'].median()))
```

## Bivariate and Multivariate Data
It can often be useful to compare *bivariate* data; in other words, compare two variables, or even more (in which case we call it *multivariate* data).

For example, our student data includes three numeric variables for each student: their salary, the number of hours they work per week, and their final school grade. Run the following code to see an enlarged sample of this data as a table:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df[['Name', 'Salary', 'Hours', 'Grade']]
```

Let's suppose you want to compare the distributions of these variables. You might simply create a boxplot for each variable, like this:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df.plot(kind='box', title='Distribution', figsize = (10,8))
plt.show()
```

Hmm, that's not particularly useful is it?

The problem is that the data are all measured in different scales. Salaries are typically in tens of thousands, while hours and grades are in single or double digits.

### Normalizing Data
When you need to compare data in different units of measurement, you can *normalize* or *scale* the data so that the values are measured in the same proportional scale. For example, in Python you can use a MinMax scaler to normalize multiple numeric variables to a proportional value between 0 and 1 based on their minimum and maximum values. Run the following cell to do this:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.preprocessing import MinMaxScaler

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Normalize the data
scaler = MinMaxScaler()
df[['Salary', 'Hours', 'Grade']] = scaler.fit_transform(df[['Salary', 'Hours', 'Grade']])

# Plot the normalized data
df.plot(kind='box', title='Distribution', figsize = (10,8))
plt.show()
```

Now the numbers on the y axis aren't particularly meaningful, but they're on a similar scale.

### Comparing Bivariate Data with a Scatter Plot
When you need to compare two numeric values, a scatter plot can be a great way to see if there is any apparent relationship between them so that changes in the value of one variable affect the value of the other.

Let's look at a scatter plot of *Salary* and *Grade*:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Create a scatter plot of Salary vs Grade
df.plot(kind='scatter', title='Grade vs Hours', x='Grade', y='Salary')
plt.show()

```

Look closely at the scatter plot. Can you see a diagonal trend in the plotted points, rising up to the right? It looks as though the higher the student's grade is, the higher their salary is.

You can see the trend more clearly by adding a *line of best fit* (sometimes called a *trendline*) to the plot:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Create a scatter plot of Salary vs Grade
df.plot(kind='scatter', title='Grade vs Salary', x='Grade', y='Salary')

# Add a line of best fit
plt.plot(np.unique(df['Grade']), np.poly1d(np.polyfit(df['Grade'], df['Salary'], 1))(np.unique(df['Grade'])))

plt.show()
```

The line of best fit makes it clearer that there is some apparent *colinearity* between these variables (the relationship is *colinear* if one variable's value increases or decreases in line with the other).

### Correlation
The apparently colinear relationship you saw in the scatter plot can be verified by calculating a statistic that quantifies the relationship between the two variables. The statistic usually used to do this is *correlation*, though there is also a statistic named *covariance* that is sometimes used. Correlation is generally preferred because the value it produces is more easily interpreted.

A correlation value is always a number between ***-1*** and ***1***.
- A positive value indicates a positive correlation (as the value of variable *x* increases, so does the value of variable *y*).
- A negative value indicates a negative correlation (as the value of variable *x* increases, the value of variable *y* decreases).
- The closer to zero the correlation value is, the weaker the correlation between *x* and *y*.
- A correlation of exactly zero means there is no apparent relationship between the variables.

The formula to calculate correlation is:

\begin{equation}r_{x,y} = \frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})(y_{i} -\bar{y})}{\sqrt{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}(y_{i} -\bar{y})^{2}}}\end{equation}

**rx, y** is the notation for the *correlation between x and y*.

The formula is pretty complex, but fortunately Python makes it very easy to calculate the correlation by using the ***corr*** function:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Calculate the correlation between *Salary* and *Grade*
print(df['Grade'].corr(df['Salary']))
```

In this case, the correlation is just over 0.8; making it a reasonably high positive correlation that indicates salary increases in line with grade.

Let's see if we can find a correlation between *Grade* and *Hours*:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

r = df['Grade'].corr(df['Hours'])
print('Correlation: ' + str(r))

# Create a scatter plot of Salary vs Grade
df.plot(kind='scatter', title='Grade vs Hours', x='Grade', y='Hours')

# Add a line of best fit-
plt.plot(np.unique(df['Grade']), np.poly1d(np.polyfit(df['Grade'], df['Hours'], 1))(np.unique(df['Grade'])))
plt.show()

```

In this case, the correlation value is just under -0.8; meaning a fairly strong negative correlation in which the number of hours worked decreases as the grade increases. The line of best fit on the scatter plot corroborates this statistic.

It's important to remember that *correlation* **is not** *causation*. In other words, even though there's an apparent relationship, you can't say for sure that one variable is the cause of the other. In this example, we can say that students who achieved higher grades tend to work shorter hours; but we ***can't*** say that those who work shorter hours do so *because* they achieved a high grade!

## Least Squares Regression
In the previous examples, we drew a line on a scatter plot to show the *best fit* of the data. In many cases, your initial attempts to identify any colinearity might involve adding this kind of line by hand (or just mentally visualizing it); but as you may suspect from the use of the *numpy.**polyfit*** function in the code above, there are ways to calculate the coordinates for this line mathematically. One of the most commonly used techniques is *least squares regression*, and that's what we'll look at now.

Cast your mind back to when you were learning how to solve linear equations, and recall that the *slope-intercept* form of a linear equation lookes like this:

\begin{equation}y = mx + b\end{equation}

In this equation, *y* and *x* are the coordinate variables, *m* is the slope of the line, and *b* is the y-intercept of the line.

In the case of our scatter plot for our former-student's working hours, we already have our values for *x* (*Grade*) and *y* (*Hours*), so we just need to calculate the intercept and slope of the straight line that lies closest to those points. Then we can form a linear equation that calculates the a new *y* value on that line for each of our *x* (*Grade*) values - to avoid confusion, we'll call this new *y* value *f(x)* (because it's the output from a linear equation function based on *x*). The difference between the original *y* (*Hours*) value and the *f(x)* value is the *error* between our regression line of best fit and the actual *Hours* worked by the former student. Our goal is to calculate the slope and intercept for a line with the lowest overall error.

Specifically, we define the overall error by taking the error for each point, squaring it, and adding all the squared errors together. The line of best fit is the line that gives us the lowest value for the sum of the squared errors - hence the name *least squares regression*.

So how do we accomplish this? First we need to calculate the slope (*m*), which we do using this formula (in which *n* is the number of observations in our data sample):

\begin{equation}m = \frac{n(\sum{xy}) - (\sum{x})(\sum{y})}{n(\sum{x^{2}})-(\sum{x})^{2}}\end{equation}

After we've calculated the slope (*m*), we can use is to calculate the intercept (*b*) like this:

\begin{equation}b = \frac{\sum{y} - m(\sum{x})}{n}\end{equation}

Let's look at a simple example that compares the number of hours of nightly study each student undertook with the final grade the student achieved:

| Name     | Study | Grade |
|----------|-------|-------|
| Dan      | 1     | 50    |
| Joann    | 0.75  | 50    |
| Pedro    | 0.6   | 46    |
| Rosie    | 2     | 95    |
| Ethan    | 1     | 50    |
| Vicky    | 0.2   | 5     |
| Frederic | 1.2   | 57    |

First, let's take each *x* (Study) and *y* (Grade) pair and calculate *x2* and *xy*, because we're going to need these to work out the slope:

| Name     | Study | Grade | x2 | xy   |
|----------|-------|-------|------|------|
| Dan      | 1     | 50    | 1    | 50   |
| Joann    | 0.75  | 50    | 0.55 | 37.5 |
| Pedro    | 0.6   | 46    | 0.36 | 27.6 |
| Rosie    | 2     | 95    | 4    | 190  |
| Ethan    | 1     | 50    | 1    | 50   |
| Vicky    | 0.2   | 5     | 0.04 | 1    |
| Frederic | 1.2   | 57    | 1.44 | 68.4 |

Now we'll sum *x*, *y*, *x2*, and *xy*:

| Name     | Study | Grade | x2 | xy   |
|----------|-------|-------|------|------|
| Dan      | 1     | 50    | 1    | 50   |
| Joann    | 0.75  | 50    | 0.55 | 37.5 |
| Pedro    | 0.6   | 46    | 0.36 | 27.6 |
| Rosie    | 2     | 95    | 4    | 190  |
| Ethan    | 1     | 50    | 1    | 50   |
| Vicky    | 0.2   | 5     | 0.04 | 1    |
| Frederic | 1.2   | 57    | 1.44 | 68.4 |
| **Σ**      | **6.75**  | **353**   | **8.4025**| **424.5**  |

OK, now we're ready to calculate the slope for our *7* observations:

\begin{equation}m = \frac{(7\times 424.5) - (6.75\times353)}{(7\times8.4025)-6.75^{2}}\end{equation}

Which is:

\begin{equation}m = \frac{2971.5 - 2382.75}{58.8175-45.5625}\end{equation}

So:

\begin{equation}m = \frac{588.75}{13.255} \approx 44.4172\end{equation}

Now we can calculate *b*:

\begin{equation}b = \frac{353 - (44.4172\times6.75)}{7}\end{equation}

Which simplifies to:

\begin{equation}b = \frac{53.18389}{7} = 7.597699\end{equation}

Now we have our linear function:

\begin{equation}f(x) = 44.4172x + 7.597699\end{equation}

We can use this for each *x* (Study) value to calculate the *y* values for the regression line (*f(x)*), and we can subtract the original *y* (Grade) from these to calculate the error for each point:

| Name     | Study | Grade | *f(x)* | Error |
|----------|-------|-------|------|------ |
| Dan      | 1     | 50    |52.0149 |2.0149 |
| Joann    | 0.75  | 50    |40.9106 |-9.0894|
| Pedro    | 0.6   | 46    |34.2480 |-11.752|
| Rosie    | 2     | 95    |96.4321 |1.4321 |
| Ethan    | 1     | 50    |52.0149 |2.0149 |
| Vicky    | 0.2   | 5     |16.4811 |11.4811|
| Frederic | 1.2   | 57    |60.8983 |3.8983 |

As you can see, the *f(x)* values are mostly quite close to the actual *Grade* values, and the errors (which when we're comparing estimated values from a function with actual known values we we often call *residuals*) are generally pretty small.

Let's plot the least squares regression line with the actual values:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Study':[1,0.75,0.6,2,1,0.2,1.2],
                   'Grade':[50,50,46,95,50,5,57],
                   'fx':[52.0159,40.9106,34.2480,96.4321,52.0149,16.4811,60.8983]})

# Create a scatter plot of Study vs Grade
df.plot(kind='scatter', title='Study Time vs Grade Regression', x='Study', y='Grade', color='red')

# Plot the regression line
plt.plot(df['Study'],df['fx'])

plt.show()
```

In this case, the line fits the middle values fairly well, but is less accurate for the outlier at the low end. This is often the case, which is why statisticians and data scientists often *treat* outliers by removing them or applying a threshold value; though in this example there are too few data points to conclude that the data points are really outliers.

Let's look at a slightly larger dataset and apply the same approach to compare *Grade* and *Salary*:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Calculate least squares regression line
df['x2'] = df['Grade']**2
df['xy'] = df['Grade'] * df['Salary']
x = df['Grade'].sum()
y = df['Salary'].sum()
x2 = df['x2'].sum()
xy = df['xy'].sum()
n = df['Grade'].count()
m = ((n*xy) - (x*y))/((n*x2)-(x**2))
b = (y - (m*x))/n
df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

# Create a scatter plot of Grade vs Salary
df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

# Plot the regression line
plt.plot(df['Grade'],df['fx'])

plt.show()

# Show the original x,y values, the f(x) value, and the error
df[['Grade', 'Salary', 'fx', 'error']]
```

In this case, we used Python expressions to calculate the *slope* and *y-intercept* using the same approach and formula as before. In practice, Python provides great support for statistical operations like this; and you can use the ***linregress*** function in the *scipy.stats* package to retrieve the *slope* and *y-intercept* (as well as the *correlation*, *p-value*, and *standard error*) for a matched array of *x* and *y* values (we'll discuss *p-values* later!).

Here's the Python code to calculate the regression line variables using the ***linregress*** function:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Get the regression line slope and intercept
m, b, r, p, se = stats.linregress(df['Grade'], df['Salary'])

df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

# Create a scatter plot of Grade vs Salary
df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

# Plot the regression line
plt.plot(df['Grade'],df['fx'])

plt.show()

# Show the original x,y values, the f(x) value, and the error
df[['Grade', 'Salary', 'fx', 'error']]

```

Note that the *slope* and *y-intercept* values are the same as when we worked them out using the formula.

Similarly to the simple study hours example, the regression line doesn't fit the outliers very well. In this case, the extremes include a student who scored only 5, and a student who scored 95. Let's see what happens if we remove these students from our sample:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000,42000,47000,78000,119000,95000,49000,29000,130000],
                   'Hours':[41,40,36,17,35,39,40,45,41,35,30,33,38,47,24],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

df = df[(df['Grade'] > 5) & (df['Grade'] < 95)]

# Get the regression line slope and intercept
m, b, r, p, se = stats.linregress(df['Grade'], df['Salary'])

df['fx'] = (m*df['Grade']) + b
df['error'] = df['fx'] - df['Salary']

print('slope: ' + str(m))
print('y-intercept: ' + str(b))

# Create a scatter plot of Grade vs Salary
df.plot(kind='scatter', title='Grade vs Salary Regression', x='Grade', y='Salary', color='red')

# Plot the regression line
plt.plot(df['Grade'],df['fx'])

plt.show()

# Show the original x,y values, the f(x) value, and the error
df[['Grade', 'Salary', 'fx', 'error']]

```

With the outliers removed, the line is a slightly better overall fit to the data.

One of the neat things about regression is that it gives us a formula and some constant values that we can use to estimate a *y* value for any *x* value. We just need to apply the linear function using the *slope* and *y-intercept* values we've calculated from our sample data. For example, suppose a student named Fabio graduates from our school with a final grade of **62**. We can use our linear function with the *slope* and *y-intercept* values we calculated with Python to estimate what salary he can expect to earn:

\begin{equation}f(x) = (1424.50\times62) - 7822.24 \approx 80,497 \end{equation}

Statistics and Probability by Hiren/04-03-Comparing Data.ipynb


# Fitting transfer function with a ZPK model

In the last tutorial, we tried using generic transfer function model to fit a transfer function

\begin{equation}
    H(s) = \frac{s^2 + 3s + 2}{s^3 + 12s^2 + 47s + 60}\,.
\end{equation}

Although we ended up with a transfer function that fits the data, the parameters were not correct.

If we factorize the the numerator and denominator, in fact, we get

\begin{equation}
    H(s) = \frac{(s+1)(s+2)}{(s+3)(s+4)(s+5)}\,
\end{equation}

which is a more desirable form (The first form has 7 parameters, and the latter one has 6! That means the solution is not unique if we use the first form!).

Let's try to fit the data with ``kontrol.curvefit.model.SimpleZPK`` instead.

``kontrol.curvefit.model.SimpleZPK`` is defined as

\begin{equation}
    G(s; z_i, p_j, k) = k\frac{\prod_i s/z_i + 1}{\prod_j s/p_j + 1}\,.
\end{equation}

So the parameters we are looking for are [1, 2, 3, 4, 5, 1]. But we typically work in unit of Hz instead of rad/s.

So, the true parameters are **[0.15915494, 0.31830989, 0.47746483, 0.63661977, 0.79577472, 0.03333333]**.
Note that for the first two parameters (zeros), the order doesn't matter, and same goes for

```python
# Same as the previous tutorial
import control
import numpy as np
import matplotlib.pyplot as plt

f = np.linspace(0.001, 1000, 100000)
s = 1j*2*np.pi*f
tf = (s**2 + 3*s + 2) / (s**3 + 12*s**2 + 47*s + 60)

plt.figure(figsize=(15, 5))
plt.subplot(121)
plt.loglog(f, abs(tf))
plt.grid(which="both")
plt.ylabel("Amplitude")
plt.xlabel("Frequency (Hz)")
plt.subplot(122)
plt.semilogx(f, np.angle(tf))
plt.grid(which="both")
plt.ylabel("Phase (rad)")
plt.xlabel("Frequency (Hz)")
plt.show()
```

```python
import kontrol.curvefit
import scipy.optimize

a = kontrol.curvefit.CurveFit()
a.xdata = f
a.ydata = tf
a.model = kontrol.curvefit.model.SimpleZPK(nzero=2, npole=3, log_args=False)
error_func = kontrol.curvefit.error_func.tf_error
weight = 1/f
error_func_kwargs = {"weight": weight}
a.cost = kontrol.curvefit.Cost(error_func=error_func, error_func_kwargs=error_func_kwargs)
a.optimizer = scipy.optimize.minimize
np.random.seed(123)
true_args = np.array([0.15915494, 0.31830989, 0.47746483, 0.63661977, 0.79577472, 0.03333333])  # These are the true parameters
noise = np.random.normal(loc=0, scale=true_args/10, size=6)
x0 = true_args + noise  # Now the initial guess is assumed to be some deviation from the true values.

a.optimizer_kwargs = {"x0": x0,}

res = a.fit()
```

```python
plt.figure(figsize=(15, 5))
plt.subplot(121)
plt.loglog(f, abs(tf), label="Data")
plt.loglog(f, abs(a.model(f, x0)), label="Initial guess")
plt.loglog(f, abs(a.yfit), ls="--", label="Fit")
plt.legend(loc=0)
plt.grid(which="both")
plt.ylabel("Amplitude")
plt.xlabel("Frequency (Hz)")
plt.subplot(122)
plt.semilogx(f, np.angle(tf), label="Data")
plt.semilogx(f, np.angle(a.model(f, x0)), label="Initial guess")
plt.semilogx(f, np.angle(a.yfit), ls="--", label="Fit")
plt.legend(loc=0)
plt.grid(which="both")
plt.ylabel("Phase (rad)")
plt.xlabel("Frequency (Hz)")
plt.show()
```

```python
print("True parameters :", true_args)
print("Parameters from fit: ", a.optimized_args)
```

    True parameters : [0.15915494 0.31830989 0.47746483 0.63661977 0.79577472 0.03333333]
    Parameters from fit:  [0.15889944 0.32540938 0.54057908 0.55856226 0.81765228 0.03333333]

Let's convert it into transfer function and obtain the transfer function representation.

```python
print("True transfer function")
s = control.tf("s")
tf_ = (s**2 + 3*s + 2) / (s**3 + 12*s**2 + 47*s + 60)
tf_
```

    True transfer function

$$\frac{s^2 + 3 s + 2}{s^3 + 12 s^2 + 47 s + 60}$$

```python
print("Fitted transfer function")
a.model.args = a.optimized_args
a.model.tf.minreal()
```

    Fitted transfer function

$$\frac{s^2 + 3.043 s + 2.041}{s^3 + 12.04 s^2 + 47.4 s + 61.24}$$

Much better!

Now, with the ZPK model, we don't really have to use local minimization. We know the the location of poles and zeros are within the measured frequency band so we can put a bound on the model parameters. This means that we don't even need an initial guess.

```python
bounds = [(0.1, 1)] * 5  # Say we know the features are between 0.01 and 10 Hz.
bounds += [(abs(tf[0])/10, (abs(tf[0])*10))]  # around the lower frequency value
a.optimizer = scipy.optimize.differential_evolution
a.optimizer_kwargs = {"bounds": bounds, "workers": -1, "updating": "deferred"}
np.random.seed(123)  # Fix random seed for reproducibility
a.fit()

```

         fun: -0.11893392985453001
     message: 'Optimization terminated successfully.'
        nfev: 14480
         nit: 156
     success: True
           x: array([0.24229786, 0.17012243, 0.7466101 , 0.74350342, 0.35457821,
           0.03333341])

```python
plt.figure(figsize=(15, 5))
plt.subplot(121)
plt.loglog(f, abs(tf), label="Data")
# plt.loglog(f, abs(a.model(f, x0)), label="Initial guess")
plt.loglog(f, abs(a.yfit), ls="--", label="Fit")
plt.legend(loc=0)
plt.grid(which="both")
plt.ylabel("Amplitude")
plt.xlabel("Frequency (Hz)")
plt.subplot(122)
plt.semilogx(f, np.angle(tf), label="Data")
# plt.semilogx(f, np.angle(a.model(f, x0)), label="Initial guess")
plt.semilogx(f, np.angle(a.yfit), ls="--", label="Fit")
plt.legend(loc=0)
plt.grid(which="both")
plt.ylabel("Phase (rad)")
plt.xlabel("Frequency (Hz)")
plt.show()
```

```python
print("Fitted transfer function")
a.model.args = a.optimized_args
a.model.tf.minreal()
```

    Fitted transfer function

$$\frac{s^2 + 2.592 s + 1.627}{s^3 + 11.59 s^2 + 42.77 s + 48.82}$$