```python
import os.path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression
from traitlets import traitlets
from IPython.display import display
from ipywidgets import HBox, VBox, BoundedFloatText, BoundedIntText, Text, Layout, Button
```

# Processing demographic data

We have retrieved demographic data from the data portal of the **National Centre for Statistics & Information** of the *Sultanate of Oman* (https://data.gov.om/)

Data are formatted in a two-columns file, where

*  The __first__ column contains _years_.

*  The __second__ column, which must be named *Population*, contains integer numbers and reports the total population in the Muscat region per year [2010-2019]

```python
if os.path.exists("./playgrounds/data/JNNP_2020/Epidemiology_Oman.txt"):
    epi_file = "./playgrounds/data/JNNP_2020/Epidemiology_Oman.txt"
elif not os.path.exists("./data/JNNP_2020/Epidemiology_Oman.txt"):
    !git clone https://github.com/mazzalab/playgrounds.git
    epi_file = "./playgrounds/data/JNNP_2020/Epidemiology_Oman.txt"
else:
    epi_file = "./data/JNNP_2020/Epidemiology_Oman.txt"

df = pd.read_csv(epi_file, sep='\t', index_col=0)
print(df)
```

          Population
    Year
    2010      407006
    2011      418652
    2012      435149
    2013      451652
    2014      470085
    2015      487592
    2016      512039
    2017      528327
    2018      543930

## Predicting Muscat population growth

The observed period of time starts in **2014** and ends in the year specified below.

```python
style = {'description_width': 'initial'}

simulation_end_text = BoundedIntText(
    min=2015,
    max=2100,
    step=1,
    value=2050,
    description='Simulate from 2014 until:', style=style)

class GenerateTimeButton(Button):
    def __init__(self, value=None, *args, **kwargs):
        super(GenerateTimeButton, self).__init__(*args, **kwargs)
        # Create the value attribute.
        self.add_traits(value=traitlets.Any(value))

# Generate time period (list of years) to be simulated
def on_generate_time_button_clicked(button):
    button.value = np.arange(2014, simulation_end_text.value+1, 1).reshape((-1, 1))
    print("Simulation time points generated")

generate_time_button = GenerateTimeButton(
    description="Generate",
    button_style='info',
    tooltip='Generate simulation time points'
)
generate_time_button.value=np.array([])
generate_time_button.on_click(on_generate_time_button_clicked)

hbox_time = HBox([simulation_end_text, generate_time_button])
display(hbox_time)
```

    HBox(children=(BoundedIntText(value=2050, description='Simulate from 2014 until:', max=2100, min=2015, style=D…

    Simulation time points generated

Linear regression analysis is conducted on demographic data using the *LinearRegression* module from the Python **sklearn** package. The typical linear regression equation: \begin{align}y & = mx + b\end{align} is fitted and *coefficient of determination ($r^2$)*, *intercept* ($b$) and *slope* ($m$) are inferred.

```python
if not generate_time_button.value.any():
    simulation_end_text.value=2050
    generate_time_button.click()

x_new = generate_time_button.value

#######################################

%matplotlib inline
x = df.index.values.reshape((-1, 1))
y = df.Population
model = LinearRegression().fit(x, y)

r_sq = model.score(x, y)
print(" ")
print('coefficient of determination:', np.round(r_sq, 3))
print('intercept (b):', np.round(model.intercept_, 3))
print('slope (m):', np.round(model.coef_, 3))
print('')

# Predict response of known data
y_pred = model.predict(x)
print('time:', np.transpose(x)[0], sep='\t\t\t\t')
print('predicted response:', np.around(y_pred), sep='\t\t')

# Plot outputs
axis_scale = 1e5
plt.scatter(x, y/axis_scale,  color='black', label="act. population")
plt.plot(x, y_pred/axis_scale, color='blue',
         linewidth=3, label="Interpolation")

plt.title('Actual vs Predicted')
plt.xlabel('years')
plt.ylabel('population growth (1e5)')

plt.legend(loc='upper left', frameon=False)
plt.savefig('linear_regression.svg', format='svg', dpi=600)

# Predict response of future data [2018-2020]
y_pred = np.round(model.predict(x_new))
print('predicted response [2018]', y_pred[4], sep='\t')
print('actual data [Dec. 2018]:\t{}'.format(df.iloc[8]['Population']))

print('predicted response [2019]', y_pred[5], sep='\t')
print('actual data [Dec. 2019]:\t{}'.format(567851))

print('predicted response [2020]', y_pred[6], sep='\t')
print('actual data [Feb. 2020]:\t{}'.format(570196))
```

### Theoretical response with plateau and inflection point at 2030, 2040 and 2050

Population growth was simulated considering a plateau in 2030, 2040 and in 2050 through the exponential function: $y = Y_M-(Y_M-Y_0)\cdot(e^{-ax})$, where $Y_M$ is the maximum at which the plateau ends up, $Y_0$ is the starting population and $a$ is a rate constant that governs how fast it gets there. Here, $Y_0 = 5.44 \cdot 1e5$ (actual population in 2018), while $Y_M$ is equals to $7.57$, $9.35$ and $11.13$ per $100000$ individuals, as inferred by linear regression respectively for 2030, 2040 and 2050.

```python
import math

axis_scale = 1e5
y_pred = model.predict(x_new)/axis_scale

fig = plt.figure(figsize=(3,3))

# Plot outputs
plt.scatter(x, y/axis_scale,  color='black', label="actual", s=10)
plt.plot(x_new, y_pred, color='#984ea3', linewidth=3, label="linear")

def plateau(x, M, M0, a):
    return M - (M-M0)*math.exp(-a*x)
y_pl = np.vectorize(plateau)

M_2018 = y_pred[np.where(x_new==2018)[0][0]]
M_2030 = y_pred[np.where(x_new==2030)[0][0]]
M_2040 = y_pred[np.where(x_new==2040)[0][0]]
M_2050 = y_pred[np.where(x_new==2050)[0][0]]

y_2030= y_pl(x=np.arange(0,33), M=M_2030, M0=M_2018, a=.08)
y_2040= y_pl(x=np.arange(0,33), M=M_2040, M0=M_2018, a=.045)
y_2050= y_pl(x=np.arange(0,33), M=M_2050, M0=M_2018, a=.035)

plt.plot(x_new[4:], y_2030, color='#377eb8',
         linewidth=3, label="2030")
plt.plot(x_new[4:], y_2040, color='#ff7f00',
         linewidth=3, label="2040")
plt.plot(x_new[4:], y_2050, color='#4daf4a',
         linewidth=3, label="2050")

plt.axvline(x=2030, linewidth=.5, linestyle='dashdot', color="#377eb8", ymax=.40)
plt.axvline(x=2040, linewidth=.5, linestyle='dashdot', color="#ff7f00", ymax=.55)
plt.axvline(x=2050, linewidth=.5, linestyle='dashdot', color="#4daf4a", ymax=.72)

plt.xlabel('years')
plt.ylabel('population growth (1e5)')
plt.legend(loc='upper left', frameon=False)
plt.rcParams.update({'font.size': 11})

plt.savefig('fitted_vs_predicted.svg', format='svg', dpi=1200)
```

# Markov Chain design

A **discrete-time Markov chain (DTMC)** is designed to model the *death* event of HD patients and *onset* of the disease among inhabitants of Muscat. To do that, *birth/death* events were inferred from observational data collected from 2013 to 2019 in the Muscat population.

## Set the transition rates

The modeled process is stochastic and *memoryless*, in that it allows to make predictions based solely on its present state. The process passes through three states: **Healthy**, **HD Alive** and **HD Dead** as a result of *birth* and *death* events, driven in turn by: I = incidence rate of the disease and D = death rate because of the disease.

where:

* $I_{avg}$:  the average HD incidence rate in the world population (min=**0.38** per 100,000 per year, [https://doi.org/10.1002/mds.25075]; max=**0.9** per 100,000 per year, [https://doi.org/10.1186/s12883-019-1556-3])
* $I_{Muscat}$: the actual HD incidence rate in Muscat (**0.56** per 100,000 per year [2013-2019])
* $D_{avg}$:  the HD death rate (min=**1.55** per million population registered in England-Wales in 1960-1973, [https://pubmed.ncbi.nlm.nih.gov/6233902/]; max=**2.27** per million population registered in USA in 1971-1978, [https://doi.org/10.1212/wnl.38.5.769])
* $D_{Muscat}$: the actual HD death rate in Muscat (**1.82** per million population [2013-2019])

```python
# estimated incidence rate world-wide
## min = 0.38/year per 100,000 [https://doi.org/10.1002/mds.25075]
## max = 0.9/year per 100,000  [https://doi.org/10.1186/s12883-019-1556-3]
est_inc_rates = [0.38, 0.56, 0.7, 0.9]

# actual incidence rate (0.56 per 100,000 per year) ~ 3 new HD patients in Muscat in 2018 [https://data.gov.om/OMPOP2016/population?indicator=1000140&region=1000020-muscat&nationality=1000010-omani]
act_inc_rate = 0.56

# estimated death rate world-wide
## min = 1.55 per million (England-Wales in 1960-1973, https://www.ncbi.nlm.nih.gov/pubmed/6233902)
## max = 2.27 per million (United States in 1971-1978, https://doi.org/10.1212/wnl.38.5.769)
est_death_rates = [1.55, 1.819, 2.27]

# actual death rate ~ 1 patient in 2018 per 100.000 per year in Muscat
act_death_rate = 1.819

# starting HD individuals in Muscat in 2018 (32)
hd_2018 = 32
```

### Set the number of simulation traces

In order to approximate the posterior distribution of the HD prevalence, one performs a *Monte Carlo* simulation with $1000$ independent simulation runs from where calculating the *average number of incident cases* per year.

```python
simulation_traces = BoundedIntText(
    min=1,
    max=5000,
    step=1,
    value=1000,
    description='Simulation traces:', style=style)
display(simulation_traces)
```

    BoundedIntText(value=1000, description='Simulation traces:', max=5000, min=1, style=DescriptionStyle(descripti…

### Initialize state vectors

For each simulation step, $6$ vectors are updated:

* ***est_inc*** stores the variation of incidence over time according to $I_{avg}$
* ***act_inc*** stores the variation of incidence over time according to $I_{Muscat}$
* ***est_death*** stores the numbers of deaths over time according to $D_{avg}$
* ***act_death*** stores the numbers of deaths over time according to $D_{Muscat}$
* ***est_alive_HD*** stores the number of alive HD patients over time, based on ***est_inc*** and ***est_death***
* ***act_alive_HD*** astores the number of alive HD patients over time, based on ***act_inc*** and ***act_death***

```python
# record the number of simulation steps
sim_time = len(x_new[4:])

# get the number of simulation traces from the textfield above
num_traces = simulation_traces.value

# get number of possible Iavg and Davg
num_iavg = len(est_inc_rates)
num_davg = len(est_death_rates)

est_inc = np.zeros((num_iavg, num_davg, num_traces, sim_time), dtype=int)
act_inc = np.zeros((num_traces, sim_time), dtype=int)

est_death = np.zeros((num_iavg, num_davg, num_traces, sim_time), dtype=int)
act_death = np.zeros((num_traces, sim_time), dtype=int)

est_alive_HD = np.zeros((num_iavg, num_davg, num_traces, sim_time), dtype=int)
est_alive_HD[:, :, :, 0] = hd_2018
act_alive_HD = np.zeros((num_traces, sim_time), dtype=int)
act_alive_HD[:, 0] = hd_2018
```

### Trigger the Monte Carlo simulation

All possible estimates of $I_{avg}$ and $D_{avg}$ are shuffled here and, for each combination, 1000 independent simulations will be launched. For each simulation and for each time step, the number of alive HD patients will be calculated as the sum of the number of *currently alive* HD patients and that of *new* HD patients, from which the number of currently *deceased* patients is subtracted. This computation is performed for:

#### Linear estimate (regression)

```python
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        print("Evaluating Iavg={} and Davg={}".format(iavg_value, davg_value))
        for rep in range(num_traces):
            for t in range(sim_time-1):
                curr_pop = y_pred[t]*100000  # <-------

                if(act_inc[rep, t] == 0):
                    act_inc[rep, t] = np.random.poisson((act_inc_rate * curr_pop)/100000)
                    act_death[rep, t] = np.random.poisson((act_death_rate * curr_pop)/1000000)
                    act_alive_HD[rep, t+1] = act_alive_HD[rep, t] - act_death[rep, t] + act_inc[rep, t]

                est_inc[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (iavg_value * curr_pop)/100000)

                est_death[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (davg_value * curr_pop)/1000000)

                est_alive_HD[iavg_idx, davg_idx, rep, t+1] = est_alive_HD[iavg_idx, davg_idx, rep, t] - \
                    est_death[iavg_idx, davg_idx, rep, t] + est_inc[iavg_idx, davg_idx, rep, t]

```

    Evaluating Iavg=0.38 and Davg=1.55
    Evaluating Iavg=0.38 and Davg=1.819
    Evaluating Iavg=0.38 and Davg=2.27
    Evaluating Iavg=0.56 and Davg=1.55
    Evaluating Iavg=0.56 and Davg=1.819
    Evaluating Iavg=0.56 and Davg=2.27
    Evaluating Iavg=0.7 and Davg=1.55
    Evaluating Iavg=0.7 and Davg=1.819
    Evaluating Iavg=0.7 and Davg=2.27
    Evaluating Iavg=0.9 and Davg=1.55
    Evaluating Iavg=0.9 and Davg=1.819
    Evaluating Iavg=0.9 and Davg=2.27

##### Plot HD population predicted to be alive

N.b., __est. HD population__ accounts for the population that is predicted to be alive every year according to *incidence* and *death* rates that are the world average; __act. HD population__ refers instead to the actual rates calculated stright on the actual Muscat population records, as previously described.

A ranom plot (i.e., a random trace among 1000) will be generaed for each combination of $I_{avg}$ and $D_{avg}$.

```python
# Select a random trace
rtrace = np.random.randint(0,num_traces)

# Plot a random trace of est/act. alive HD cases in the Muscat region
# for each combination of Iavg and Davg
fig, ax = plt.subplots(len(est_inc_rates), len(est_death_rates), figsize=(12, 9))
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        ax[iavg_idx, davg_idx].plot(x_new[4:], est_alive_HD[iavg_idx, davg_idx, rtrace, :],
                                    lw=2, label='est. {}, {}'.format(iavg_value, davg_value))
        ax[iavg_idx, davg_idx].plot(x_new[4:], act_alive_HD[rtrace, :],
                                    lw=2, label='act. {}, {}'.format(act_inc_rate, str(round(act_death_rate, 2))))
        ax[iavg_idx, davg_idx].legend(loc='upper left', frameon=False)

plt.savefig('alive_population.svg', format='svg', dpi=1200)
```

#### Inflation at 2030

```python
# record the number of simulation steps
sim_time_2030 = len(y_2030)

# get the number of simulation traces from the textfield above
num_traces_2030 = simulation_traces.value

# get number of possible Iavg and Davg
num_iavg_2030 = len(est_inc_rates)
num_davg_2030 = len(est_death_rates)

est_inc_2030 = np.zeros((num_iavg_2030, num_davg_2030, num_traces_2030, sim_time_2030), dtype=int)
act_inc_2030 = np.zeros((num_traces_2030, sim_time_2030), dtype=int)

est_death_2030 = np.zeros((num_iavg_2030, num_davg_2030, num_traces_2030, sim_time_2030), dtype=int)
act_death_2030 = np.zeros((num_traces_2030, sim_time_2030), dtype=int)

est_alive_HD_2030 = np.zeros((num_iavg_2030, num_davg_2030, num_traces_2030, sim_time_2030), dtype=int)
est_alive_HD_2030[:, :, :, 0] = hd_2018
act_alive_HD_2030 = np.zeros((num_traces_2030, sim_time_2030), dtype=int)
act_alive_HD_2030[:, 0] = hd_2018
```

```python
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        print("Evaluating Iavg={} and Davg={}".format(iavg_value, davg_value))
        for rep in range(num_traces_2030):
            for t in range(sim_time_2030-1):
                curr_pop = y_2030[t]*100000  # <-------

                if(act_inc_2030[rep, t] == 0):
                    act_inc_2030[rep, t] = np.random.poisson((act_inc_rate * curr_pop)/100000)
                    act_death_2030[rep, t] = np.random.poisson((act_death_rate * curr_pop)/1000000)
                    act_alive_HD_2030[rep, t+1] = act_alive_HD_2030[rep, t] - act_death_2030[rep, t] + act_inc_2030[rep, t]

                est_inc_2030[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (iavg_value * curr_pop)/100000)
                est_death_2030[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (davg_value * curr_pop)/1000000)
                est_alive_HD_2030[iavg_idx, davg_idx, rep, t+1] = est_alive_HD_2030[iavg_idx, davg_idx, rep, t] - \
                    est_death_2030[iavg_idx, davg_idx, rep, t] + est_inc_2030[iavg_idx, davg_idx, rep, t]
```

    Evaluating Iavg=0.38 and Davg=1.55
    Evaluating Iavg=0.38 and Davg=1.819
    Evaluating Iavg=0.38 and Davg=2.27
    Evaluating Iavg=0.56 and Davg=1.55
    Evaluating Iavg=0.56 and Davg=1.819
    Evaluating Iavg=0.56 and Davg=2.27
    Evaluating Iavg=0.7 and Davg=1.55
    Evaluating Iavg=0.7 and Davg=1.819
    Evaluating Iavg=0.7 and Davg=2.27
    Evaluating Iavg=0.9 and Davg=1.55
    Evaluating Iavg=0.9 and Davg=1.819
    Evaluating Iavg=0.9 and Davg=2.27

##### Plot HD population predicted to be alive

```python
# Select a random trace
rtrace = np.random.randint(0,num_traces_2030)

# Plot a random trace of est/act. alive HD cases in the Muscat region
# for each combination of Iavg and Davg
fig, ax = plt.subplots(len(est_inc_rates), len(est_death_rates), figsize=(12, 9))
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        ax[iavg_idx, davg_idx].plot(x_new[4:], est_alive_HD_2030[iavg_idx, davg_idx, rtrace, :],
                                    lw=2, label='est. {}, {}'.format(iavg_value, davg_value))
        ax[iavg_idx, davg_idx].plot(x_new[4:], act_alive_HD_2030[rtrace, :],
                                    lw=2, label='act. {}, {}'.format(act_inc_rate, str(round(act_death_rate, 2))))
        ax[iavg_idx, davg_idx].legend(loc='upper left', frameon=False)

plt.savefig('alive_population_y2030.svg', format='svg', dpi=1200)
```

#### Inflation at 2040

```python
# record the number of simulation steps
sim_time_2040 = len(y_2040)

# get the number of simulation traces from the textfield above
num_traces_2040 = simulation_traces.value

# get number of possible Iavg and Davg
num_iavg_2040 = len(est_inc_rates)
num_davg_2040 = len(est_death_rates)

est_inc_2040 = np.zeros((num_iavg_2040, num_davg_2040, num_traces_2040, sim_time_2040), dtype=int)
act_inc_2040 = np.zeros((num_traces_2040, sim_time_2040), dtype=int)

est_death_2040 = np.zeros((num_iavg_2040, num_davg_2040, num_traces_2040, sim_time_2040), dtype=int)
act_death_2040 = np.zeros((num_traces_2040, sim_time_2040), dtype=int)

est_alive_HD_2040 = np.zeros((num_iavg_2040, num_davg_2040, num_traces_2040, sim_time_2040), dtype=int)
est_alive_HD_2040[:, :, :, 0] = hd_2018
act_alive_HD_2040 = np.zeros((num_traces_2040, sim_time_2040), dtype=int)
act_alive_HD_2040[:, 0] = hd_2018
```

```python
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        print("Evaluating Iavg={} and Davg={}".format(iavg_value, davg_value))
        for rep in range(num_traces_2040):
            for t in range(sim_time_2040-1):
                curr_pop = y_2040[t]*100000  # <-------

                if(act_inc_2040[rep, t] == 0):
                    act_inc_2040[rep, t] = np.random.poisson((act_inc_rate * curr_pop)/100000)
                    act_death_2040[rep, t] = np.random.poisson((act_death_rate * curr_pop)/1000000)
                    act_alive_HD_2040[rep, t+1] = act_alive_HD_2040[rep, t] - act_death_2040[rep, t] + act_inc_2040[rep, t]

                est_inc_2040[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (iavg_value * curr_pop)/100000)
                est_death_2040[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (davg_value * curr_pop)/1000000)
                est_alive_HD_2040[iavg_idx, davg_idx, rep, t+1] = est_alive_HD_2040[iavg_idx, davg_idx, rep, t] - \
                    est_death_2040[iavg_idx, davg_idx, rep, t] + est_inc_2040[iavg_idx, davg_idx, rep, t]
```

    Evaluating Iavg=0.38 and Davg=1.55
    Evaluating Iavg=0.38 and Davg=1.819
    Evaluating Iavg=0.38 and Davg=2.27
    Evaluating Iavg=0.56 and Davg=1.55
    Evaluating Iavg=0.56 and Davg=1.819
    Evaluating Iavg=0.56 and Davg=2.27
    Evaluating Iavg=0.7 and Davg=1.55
    Evaluating Iavg=0.7 and Davg=1.819
    Evaluating Iavg=0.7 and Davg=2.27
    Evaluating Iavg=0.9 and Davg=1.55
    Evaluating Iavg=0.9 and Davg=1.819
    Evaluating Iavg=0.9 and Davg=2.27

##### Plot HD population predicted to be alive

```python
# Select a random trace
rtrace = np.random.randint(0,num_traces_2040)

# Plot a random trace of est/act. alive HD cases in the Muscat region
# for each combination of Iavg and Davg
fig, ax = plt.subplots(len(est_inc_rates), len(est_death_rates), figsize=(12, 9))
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        ax[iavg_idx, davg_idx].plot(x_new[4:], est_alive_HD_2040[iavg_idx, davg_idx, rtrace, :],
                                    lw=2, label='est. {}, {}'.format(iavg_value, davg_value))
        ax[iavg_idx, davg_idx].plot(x_new[4:], act_alive_HD_2040[rtrace, :],
                                    lw=2, label='act. {}, {}'.format(act_inc_rate, str(round(act_death_rate, 2))))
        ax[iavg_idx, davg_idx].legend(loc='upper left', frameon=False)

plt.savefig('alive_population_y2040.svg', format='svg', dpi=1200)
```

#### Inflation at 2050

```python
# record the number of simulation steps
sim_time_2050 = len(y_2050)

# get the number of simulation traces from the textfield above
num_traces_2050 = simulation_traces.value

# get number of possible Iavg and Davg
num_iavg_2050 = len(est_inc_rates)
num_davg_2050 = len(est_death_rates)

est_inc_2050 = np.zeros((num_iavg_2050, num_davg_2050, num_traces_2050, sim_time_2050), dtype=int)
act_inc_2050 = np.zeros((num_traces_2050, sim_time_2050), dtype=int)

est_death_2050 = np.zeros((num_iavg_2050, num_davg_2050, num_traces_2050, sim_time_2050), dtype=int)
act_death_2050 = np.zeros((num_traces_2050, sim_time_2050), dtype=int)

est_alive_HD_2050 = np.zeros((num_iavg_2050, num_davg_2050, num_traces_2050, sim_time_2050), dtype=int)
est_alive_HD_2050[:, :, :, 0] = hd_2018
act_alive_HD_2050 = np.zeros((num_traces_2050, sim_time_2050), dtype=int)
act_alive_HD_2050[:, 0] = hd_2018
```

```python
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        print("Evaluating Iavg={} and Davg={}".format(iavg_value, davg_value))
        for rep in range(num_traces_2050):
            for t in range(sim_time_2050-1):
                curr_pop = y_2050[t]*100000  # <-------

                if(act_inc_2050[rep, t] == 0):
                    act_inc_2050[rep, t] = np.random.poisson((act_inc_rate * curr_pop)/100000)
                    act_death_2050[rep, t] = np.random.poisson((act_death_rate * curr_pop)/1000000)
                    act_alive_HD_2050[rep, t+1] = act_alive_HD_2050[rep, t] - act_death_2050[rep, t] + act_inc_2050[rep, t]

                est_inc_2050[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (iavg_value * curr_pop)/100000)
                est_death_2050[iavg_idx, davg_idx, rep, t] = np.random.poisson(
                    (davg_value * curr_pop)/1000000)
                est_alive_HD_2050[iavg_idx, davg_idx, rep, t+1] = est_alive_HD_2050[iavg_idx, davg_idx, rep, t] - \
                    est_death_2050[iavg_idx, davg_idx, rep, t] + est_inc_2050[iavg_idx, davg_idx, rep, t]
```

    Evaluating Iavg=0.38 and Davg=1.55
    Evaluating Iavg=0.38 and Davg=1.819
    Evaluating Iavg=0.38 and Davg=2.27
    Evaluating Iavg=0.56 and Davg=1.55
    Evaluating Iavg=0.56 and Davg=1.819
    Evaluating Iavg=0.56 and Davg=2.27
    Evaluating Iavg=0.7 and Davg=1.55
    Evaluating Iavg=0.7 and Davg=1.819
    Evaluating Iavg=0.7 and Davg=2.27
    Evaluating Iavg=0.9 and Davg=1.55
    Evaluating Iavg=0.9 and Davg=1.819
    Evaluating Iavg=0.9 and Davg=2.27

##### Plot HD population predicted to be alive

```python
# Select a random trace
rtrace = np.random.randint(0,num_traces_2050)

# Plot a random trace of est/act. alive HD cases in the Muscat region
# for each combination of Iavg and Davg
fig, ax = plt.subplots(len(est_inc_rates), len(est_death_rates), figsize=(12, 9))
for iavg_idx, iavg_value in enumerate(est_inc_rates):
    for davg_idx, davg_value in enumerate(est_death_rates):
        ax[iavg_idx, davg_idx].plot(x_new[4:], est_alive_HD_2050[iavg_idx, davg_idx, rtrace, :],
                                    lw=2, label='est. {}, {}'.format(iavg_value, davg_value))
        ax[iavg_idx, davg_idx].plot(x_new[4:], act_alive_HD_2050[rtrace, :],
                                    lw=2, label='act. {}, {}'.format(act_inc_rate, str(round(act_death_rate, 2))))
        ax[iavg_idx, davg_idx].legend(loc='upper left', frameon=False)

plt.savefig('alive_population_y2050.svg', format='svg', dpi=1200)
```

# Calculate average estimates of prevalence over traces

The **prevalence** of HD is calculated year by year until the end of simulation and for each simulation trace.

```python
# act_alive_HD_mu = np.around(np.mean(act_alive_HD, axis=0) / y_pred, 2)
# act_alive_HD_sigma = np.around(np.std(act_alive_HD, axis=0) / y_pred, 2)
# act_alive_HD_2030_mu = np.around(np.mean(act_alive_HD_2030 / y_2030, axis=0), 2)
# act_alive_HD_2030_sigma = np.around(np.std(act_alive_HD_2030 / y_2030, axis=0), 2)
# act_alive_HD_2040_mu = np.around(np.mean(act_alive_HD_2040 / y_2040, axis=0), 2)
# act_alive_HD_2040_sigma = np.around(np.std(act_alive_HD_2040 / y_2040, axis=0), 2)
# act_alive_HD_2050_mu = np.around(np.mean(act_alive_HD_2050 / y_2050, axis=0), 2)
# act_alive_HD_2050_sigma = np.around(np.std(act_alive_HD_2050 / y_2050, axis=0), 2)

# Iavg = [0.38, 0.56, 0.7, 0.9]
# Davg = [1.55, 1.819, 2.27]
# Select simulations with Iavg=0.56 (index=1) and Davg=2.27 (index=2) estimates
est_alive_HD_mu = np.around(np.mean(est_alive_HD[1,2,:,:], axis=0) / y_pred[4:], 2)
est_alive_HD_sigma = np.around(np.std(est_alive_HD[1,2,:,:], axis=0) / y_pred[4:], 2)
est_alive_HD_2030_mu = np.around(np.mean(est_alive_HD_2030[1,2,:,:]/ y_2030, axis=0), 2)
est_alive_HD_2030_sigma = np.around(np.std(est_alive_HD_2030[1,2,:,:]/ y_2030, axis=0), 2)
est_alive_HD_2040_mu = np.around(np.mean(est_alive_HD_2040[1,2,:,:]/ y_2040, axis=0), 2)
est_alive_HD_2040_sigma = np.around(np.std(est_alive_HD_2040[1,2,:,:]/ y_2040, axis=0), 2)
est_alive_HD_2050_mu = np.around(np.mean(est_alive_HD_2050[1,2,:,:]/ y_2050, axis=0), 2)
est_alive_HD_2050_sigma = np.around(np.std(est_alive_HD_2050[1,2,:,:]/ y_2050, axis=0), 2)

print("Linear regression (est.): {}".format(est_alive_HD_mu))
# print("Linear regression (act.): {}\n".format(act_alive_HD_mu))
print("flection at 2030 (est.): {}".format(est_alive_HD_2030_mu))
# print("flection at 2030 (act.): {}\n".format(act_alive_HD_2030_mu))
print("flection at 2040 (est.): {}".format(est_alive_HD_2040_mu))
# print("flection at 2040 (act.): {}\n".format(act_alive_HD_2040_mu))
print("flection at 2050 (est.): {}".format(est_alive_HD_2050_mu))
# print("flection at 2050 (act.): {}".format(act_alive_HD_2050_mu))
```

    Linear regression (est.): [ 5.88  5.98  6.08  6.19  6.29  6.38  6.49  6.6   6.73  6.85  6.97  7.11
      7.22  7.34  7.49  7.61  7.75  7.89  8.02  8.16  8.32  8.46  8.61  8.76
      8.91  9.05  9.19  9.33  9.46  9.6   9.74  9.89 10.04]
    flection at 2030 (est.): [ 5.88  6.05  6.21  6.4   6.59  6.81  7.01  7.24  7.45  7.68  7.9   8.14
      8.41  8.66  8.91  9.16  9.44  9.72  9.99 10.27 10.54 10.83 11.11 11.39
     11.68 11.98 12.27 12.57 12.87 13.18 13.49 13.81 14.11]
    flection at 2040 (est.): [ 5.88  6.02  6.17  6.32  6.49  6.68  6.86  7.06  7.26  7.46  7.66  7.88
      8.1   8.31  8.52  8.75  8.99  9.22  9.46  9.69  9.92 10.16 10.41 10.64
     10.89 11.14 11.39 11.63 11.9  12.17 12.42 12.69 12.95]
    flection at 2050 (est.): [ 5.88  6.    6.13  6.28  6.44  6.6   6.77  6.92  7.1   7.27  7.46  7.65
      7.83  8.02  8.21  8.39  8.59  8.8   9.01  9.23  9.45  9.68  9.91 10.13
     10.36 10.58 10.81 11.05 11.29 11.52 11.76 12.   12.25]

### Plot prevalence estimates

Calculate the **average prevalence** values ($\mu$) for each year and over all simulation traces, together with the **standard deviation** ($\sigma$) values.

Make a line plot with bands, with *years* on the X-axis and the *prevalence* values on the Y-axis.

```python
def plot_double_prevalence(splot_idx: int, ax, x_vector: list, y1_vector_mu: list, y1_vector_sigma:
                           list, y2_vector_mu: list, y2_vector_sigma: list,
                           enable_y_axis_label: bool = False):

    markers_on = [len(y1_vector_mu)-1]

    ax[splot_idx].fill_between(x_vector.flatten(), y1_vector_mu+y1_vector_sigma, y1_vector_mu-y1_vector_sigma,
                               facecolor='#377eb8', alpha=0.1)

    ax[splot_idx].fill_between(x_vector.flatten(), y2_vector_mu+y2_vector_sigma, y2_vector_mu-y2_vector_sigma,
                               facecolor='#ff7f00', alpha=0.1)

    ax[splot_idx].plot(x_vector, y1_vector_mu, '-gD', lw=2,
                       label=r'$I_{avg}=0.56$, $D_{avg}=1.82$', markevery=markers_on, color='#377eb8')
    ax[splot_idx].set_xticks([2014, 2015, 2025, 2035, 2045], minor=True)

    ax[splot_idx].plot(x_vector, y2_vector_mu, '-gD', lw=2,
                       label=r'$I_{avg}=0.56$, $D_{avg}=2.27$', markevery=markers_on, color="#ff7f00")
    if enable_y_axis_label:
        ax[splot_idx].set_ylabel('Est. prevalence')
    ax[splot_idx].set_xticks([2014, 2015, 2025, 2035, 2045], minor=True)

    ax[splot_idx].spines['top'].set_visible(False)
    ax[splot_idx].spines['right'].set_visible(False)
    ax[splot_idx].spines['bottom'].set_visible(True)
    ax[splot_idx].spines['left'].set_visible(True)

    ax[splot_idx].annotate(y1_vector_mu[-1],
                           (2050, y1_vector_mu[-1]),
                           textcoords="offset points",
                           xytext=(-30, 10),
                           ha='left')

    ax[splot_idx].annotate(y2_vector_mu[-1],
                           (2050, y2_vector_mu[-1]),
                           textcoords="offset points",
                           xytext=(-30, -20),
                           ha='left')

def plot_prevalence(ax, x_vector: list, y1_vector_mu: list, y1_vector_sigma: list,
                    y2_vector_mu: list, y2_vector_sigma: list,
                    y3_vector_mu: list, y3_vector_sigma: list,
                    y4_vector_mu: list, y4_vector_sigma: list):
    markers_on = [len(y1_vector_mu)-1]

    ax.fill_between(x_vector.flatten(), y1_vector_mu+y1_vector_sigma,
                    y1_vector_mu-y1_vector_sigma, facecolor='#377eb8', alpha=0.1)
    ax.plot(x_vector, y1_vector_mu, '-gD', lw=2, label=r'2030',
            markevery=markers_on, color="#377eb8")

    ax.fill_between(x_vector.flatten(), y2_vector_mu+y2_vector_sigma,
                    y2_vector_mu-y2_vector_sigma, facecolor='#ff7f00', alpha=0.1)
    ax.plot(x_vector, y2_vector_mu, '-gD', lw=2, label=r'2040',
            markevery=markers_on, color="#ff7f00")

    ax.fill_between(x_vector.flatten(), y3_vector_mu+y3_vector_sigma,
                    y3_vector_mu-y3_vector_sigma, facecolor='#4daf4a', alpha=0.1)
    ax.plot(x_vector, y3_vector_mu, '-gD', lw=2, label=r'2050',
            markevery=markers_on, color="#4daf4a")

    ax.fill_between(x_vector.flatten(), y4_vector_mu+y4_vector_sigma,
                    y4_vector_mu-y4_vector_sigma, facecolor='#984ea3', alpha=0.1)
    ax.plot(x_vector, y4_vector_mu, '-gD', lw=2, label=r'linear',
            markevery=markers_on, color="#984ea3")

    ax.set_ylabel('Est. prevalence')
    ax.set_xticks([2014, 2015, 2025, 2035, 2045], minor=True)

    ax.spines['top'].set_visible(False)
    ax.spines['right'].set_visible(False)
    ax.spines['bottom'].set_visible(True)
    ax.spines['left'].set_visible(True)

    ax.annotate(y1_vector_mu[-1],
                (2050, y1_vector_mu[-1]),
                textcoords="offset points",
                xytext=(10, 0),
                ha='left')
    ax.annotate(y2_vector_mu[-1],
                (2050, y2_vector_mu[-1]),
                textcoords="offset points",
                xytext=(10, 0),
                ha='left')
    ax.annotate(y3_vector_mu[-1],
                (2050, y3_vector_mu[-1]),
                textcoords="offset points",
                xytext=(10, -3),
                ha='left')
    ax.annotate(y4_vector_mu[-1],
                (2050, y4_vector_mu[-1]),
                textcoords="offset points",
                xytext=(10, -5),
                ha='left')

fig, ax = plt.subplots(1, 1, figsize=(2.5, 3))
plot_prevalence(ax, x_new[4:], est_alive_HD_2030_mu, est_alive_HD_2030_sigma, est_alive_HD_2040_mu, est_alive_HD_2040_sigma,
                est_alive_HD_2050_mu, est_alive_HD_2050_sigma, est_alive_HD_mu, est_alive_HD_sigma)

lines, labels = fig.axes[-1].get_legend_handles_labels()
fig.legend(lines, labels, bbox_to_anchor=(
    0.2, 0.9), loc='upper left', frameon=False)

fig.savefig('prevalence_est_I056_D227.svg', format='svg', dpi=1200)
```

### Generate prevalence tables

```python
prev_df = pd.DataFrame({'Years':x_new[4:].flatten(),
                        'Linear (avg)':est_alive_HD_mu, 'Linear (std)':est_alive_HD_sigma,
                        '2030 (avg)':est_alive_HD_2030_mu, '2030 (std)':est_alive_HD_2030_sigma,
                        '2040 (avg)':est_alive_HD_2040_mu, '2040 (std)':est_alive_HD_2040_sigma,
                        '2050 (avg)':est_alive_HD_2050_mu, '2050 (std)':est_alive_HD_2050_sigma
                       })
prev_df.to_excel("prevalence_estimates.xlsx", index=False)
print(prev_df.head())

```

       Years  Linear (avg)  Linear (std)  2030 (avg)  2030 (std)  2040 (avg)  \
    0   2018          5.88          0.00        5.88        0.00        5.88
    1   2019          5.96          0.33        6.02        0.37        6.03
    2   2020          6.05          0.48        6.17        0.52        6.16
    3   2021          6.14          0.56        6.35        0.63        6.32
    4   2022          6.25          0.64        6.54        0.71        6.51

       2040 (std)  2050 (avg)  2050 (std)
    0        0.00        5.88        0.00
    1        0.37        5.99        0.36
    2        0.52        6.13        0.51
    3        0.62        6.26        0.61
    4        0.71        6.39        0.71

### Miscellaneous plots

Create two plots of **frequency** of *adults*, *youngs* HD and *at-risk* subjects.

```python
bars_file = "./data/JNNP_2020/Bar_plots.xlsx"
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(6, 2.5), gridspec_kw={'width_ratios': [2, 1]})

######## Panel A ########
dfA = pd.read_excel(bars_file, sheet_name="Figure A") #index_col=0,
print(dfA)
print("")

labels = dfA.Patients
x = np.arange(len(labels))

adult_onset = dfA.iloc[:,1]
juvenile_onset = dfA.iloc[:,2]

width = 0.35*2  # the width of the bars
rects1 = ax1.bar(x, adult_onset, width, label='Adult onset',
                 color='white', edgecolor='black')
rects2 = ax1.bar(x, juvenile_onset, width, label='Juvenile onset',
                 color='lightgray', edgecolor='black', hatch="//////")

ax1.set_ylabel('HD subjects')
ax1.set_xticks(x)
ax1.set_yticks([10, 30, 50], minor=True)
ax1.set_xticklabels(labels)
ax1.legend(loc='upper center', bbox_to_anchor=(0.5, 1.25),ncol=2, prop={"size":8}, frameon=False)

ax1.spines['top'].set_visible(False)
ax1.spines['right'].set_visible(False)
ax1.spines['bottom'].set_visible(True)
ax1.spines['left'].set_visible(True)

######## Panel B ########
dfB = pd.read_excel(bars_file, sheet_name="Figure B")
print(dfB)
print("")

labels = dfB.Patients
more50 = dfB.iloc[:,1]
less50 = dfB.iloc[:,2]

x = np.arange(len(labels))
width = 0.35*2

rects1 = ax2.bar(x, more50, width, label='>50% risk',
                 color='white', edgecolor='black')
rects2 = ax2.bar(x, less50, width, label='≤50% risk',
                 color='lightgray', edgecolor='black', hatch="xXX")

ax2.set_ylabel('At-risk subjects')
ax2.set_xticks(x)
ax2.set_xticklabels(labels)
ax2.legend(loc='upper center', bbox_to_anchor=(0.5, 1.25),ncol=2, prop={"size":8}, frameon=False)

ax2.spines['top'].set_visible(False)
ax2.spines['right'].set_visible(False)
ax2.spines['bottom'].set_visible(True)
ax2.spines['left'].set_visible(True)

##### Plotting #####
fig.tight_layout()

plt.rcParams.update({'font.size': 10})
plt.show()

def on_save_misc_button(but):
    fig.savefig('misc_plots.svg', format='svg', dpi=1200)
    print('Figure saved')

save_misc_button = Button(
    description="Save SVG",
    button_style='info',
    tooltip='Save to SVG file'
)
save_misc_button.on_click(on_save_misc_button)
display(save_misc_button)
```

# Print system and required packages information

```python
%load_ext watermark
%watermark -v -m -p numpy,pandas,matplotlib,sklearn,traitlets,IPython,ipywidgets

# date
print(" ")
%watermark -u -n -t -z
```

    CPython 3.7.6
    IPython 7.12.0

    numpy 1.18.1
    pandas 1.0.1
    matplotlib 3.1.3
    sklearn 0.22.1
    traitlets 4.3.3
    IPython 7.12.0
    ipywidgets 7.5.1

    compiler   : MSC v.1916 64 bit (AMD64)
    system     : Windows
    release    : 10
    machine    : AMD64
    processor  : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
    CPU cores  : 8
    interpreter: 64bit

    last updated: Sun May 24 2020 12:30:06 W. Europe Summer Time


# Series de tiempo

Las [series de tiempo](https://en.wikipedia.org/wiki/Time_series) son mediciones de variables relacionadas con un fenómeno en instantes de tiempo conocidos, de manera que corresponden también a secuencias de tiempo discreto de dichas variables. Se usan modelos para extraer información sobre las relaciones causales entre estas variables y del fenómeno relacionado.

Una variable medida $x_t$ puede entenderse como la combinación de distintas componentes:

\begin{equation}
x_t = T_t + E_t + C_t + R_t
\end{equation}

Donde:
- $T_t$ es la tendencia "ideal" de la variable.
- $E_t$ es una variación estacional de la variable. Hace referencia a variaciones de periodos "cortos" respecto al fenómeno.
- $C_t$ es una variación cíclica de la variable. Hace referencia a variaciones de periodos "largos" respecto al fenómeno.
- $R_t$ es una variación aleatoria o ruido de la variable.

**Nota** La combinación de estas componentes puede modelarse de otras formas.

La serie puede denotarse como

\begin{align}
X &= \left\{ X1, X2, ... \right \} \\
X &= \left\{ X_k \right \}_{k \geq 1} \\
X &= \left\{ X_k : t \in  T \right \}
\end{align}

## Modelos univariantes

- En los modelos univariantes se considera solamente una variable endógena.
- No se consideran variables exógenas.
- Se busca predecir valores de la serie a partir de información del pasado.

## Polinomio autorregresivo (AR)

\begin{equation}
z_t = c + \phi z_{t-1} + a_t
\end{equation}

\begin{equation}
z_t = c + \phi_1 z_{t-1} + \phi_2 z_{t-1} + a_t
\end{equation}

## Modelo autorregresivo de media móvil (ARMA)

\begin{equation}
z_t = \mu_z + \phi z_{t-1} + a_t
\end{equation}

## Modelo autorregresivo integrado de media móvil (ARIMA)

## Modelo autorregresivo de media móvil con entrada exógena (ARMAX).


# Welcome to interactive INTRO to Fourier Transform (FT) (TODO: code cleaning, comments)
  * I will not go deep into math in any way
  * But I will try to provide motivation, why FT is such a strong concept

1. Any decent function can be approximated by a fourier series, sum of sines and cosines.
2. It is especially useful for oscillatory signals.
3. Yes, the problem is the series is infinite.
3. However, if you get the fourier series, the terms are ordered by descending significance. This means you are always taking the most important terms first.
4. The rest of the terms are corrections to what you already have.
5. Crucial is that I have oscillatory function in time, and I am able to extract frequencies (lots of simple sine waves added to each other), which completely reconstruct my crazy looking oscillations in time.

```python
## Only necessary to run this for Google colabs
import subprocess
import sys

def install_module(module):
    subprocess.call([sys.executable, "-m", "pip", "install", module])

install_module('dash')
install_module('jupyter_dash')
```

```python
# scientific modules
import numpy as np
from scipy import signal, fft

# imports for interactive plots
from dash import Dash, html, dcc, Input, Output
from jupyter_dash import JupyterDash
import plotly.express as px
import plotly.graph_objects as go

# standard plotting
import matplotlib.pyplot as plt
from matplotlib import cm

plt.rcParams['figure.facecolor'] = 'white'
```

```python
# create a squared function
t = np.linspace(0, 2, 500, endpoint=False)
plt.plot(t, signal.square(2 * np.pi * t))
plt.plot(t, np.sin(2*np.pi*t))
plt.axhline(0, color='k', lw=0.4)
plt.ylim(-1.5, 1.5)
plt.show()
```

Square function is very far from sine right.

It oscillates, which is good, but how on Earth can you produce the sharp edge just using sines and cosines?

---

At University, this is fairly easy calculation on paper, however we take just the result from wikipedia (if you want to see more details check this [Wolfram explanation](https://mathworld.wolfram.com/FourierSeriesSquareWave.html)

\begin{align}
F(t)=\dfrac{4}{\pi}\sum_{k=1}^{\infty}{\dfrac{\sin{(2\pi(2k-1)ft)}}{2k-1}} = \dfrac{4}{\pi}\left(\sin{(\omega t)} + \dfrac{1}{3}\sin{(3\omega t)} + \dfrac{1}{5}\sin{(5\omega t)} + ...\right)
\end{align}
, for $\omega=2\pi ft$

```python
## fourier series
def ft_series_square(x_axis, freq, n_harmonics):
    # first create array of zeros, where I am going to add the sin terms
    f_values = np.zeros(len(x_axis))
    # k is the k from the formula above
    for k in range(1, n_harmonics):
        # for higher and higher frequencies, adding the sin term to my existing values
        f_values = f_values + np.sin(2*np.pi*(2*k-1)*freq*x_axis)/(2*k-1)
    return f_values*4/np.pi  # multiplying only once the whole thing, insted of n_harmonics-times (in each iteration)

## FFT stands for fast fourier transform, it is standard and very fast algorithm to calculate FT
def do_fft(x, f):
    t_step = abs(x[1] - x[0])
    f_fft = fft.fft(f)
    x_fft = fft.fftfreq(len(x), t_step)[:len(x)//2]
    return x_fft, 2.0/len(x) * np.abs(f_fft[0:len(x)//2])
```

## The following part is the interactive plotting definitions
  * It looks a bit complicated, but this sort of knowledge can get you actually hired one day. Because:
  * These plots run on the web, so it nicely integrates any kind of webpage.
  * You might notice that dash makes interface to html, so you create html page using python style of coding.
  * And you can share it easily with the world.
  * Read my comments if you are interested. If not, we are calculating the same as above. Generating the FT series for the left graph and doing FT of it in the second one.

```python
app = JupyterDash(__name__)
app.layout = html.Div([
    html.H1('How FFT works.'),

    html.Div([
        html.Div([
            dcc.Graph(id='time_domain_lines')
        ], style={'width':'49%', 'display': 'inline-block'}),
    html.Div([
        dcc.Graph(id='fft_lines')
    ], style={'width': '49%', 'display': 'inline-block'}),

    html.H3('Number of harmonics'),
    html.Div(dcc.Slider(
        1, 20, step=1,
        id='n_harmonics_slider',
        value=2
    ), style={'width': '100%', 'display': 'inline-block'}),
], style={'width': '100%', 'display': 'inline-block'}),

], style={'backgroundColor':'white'})

@app.callback(
    Output('time_domain_lines', 'figure'),
    Input('n_harmonics_slider', 'value'))

def update_graph(n_harmonics_value):
    t = np.linspace(0, 2, 500, endpoint=False)
    f = ft_series_square(t, 1, n_harmonics_value)
    fig = go.Figure()
    fig.add_trace(go.Line(x=t, y=f, name='FT sine series'))

    fig.add_trace(go.Line(x=t, y=signal.square(2 * np.pi * t), name='Square wave'))
    fig.update_layout(yaxis_range=[-1.5,1.5])
    fig.update_layout(legend=dict(
        yanchor="top",
        y=1.2,
        xanchor="left",
        x=0.01
        ))
    fig.update_layout(
        margin=dict(l=20, r=20, t=20, b=20),
        )
    return fig

@app.callback(
    Output('fft_lines', 'figure'),
    Input('n_harmonics_slider', 'value'))

def create_fft(n_harmonics_value):
    t = np.linspace(0, 2, 500, endpoint=False)
    x,y = do_fft(t, ft_series_square(t, 1, n_harmonics_value))
    fig = go.Figure()
    fig.add_trace(go.Line(x=x, y=y))
    fig.update_layout(xaxis_range=[0,50])
    fig.update_layout(
        margin=dict(l=20, r=20, t=20, b=20),
        )
    return fig
```

```python
del app.config._read_only["requests_pathname_prefix"]
```

```python
app.run_server(mode='inline')#, port = 8090)
```

```python

```


# Jones Vectors

This is a sample notebook to perform basic calculations on the "Jones Calculus". The basis is chosen so that a jones vector is given by a <b>normalized</b> vector whose components represent the magnitude and phase of the $x$ and $y$ components of the electric field of a beam propagatin along $\textbf{k} = k\hat{z}$:

\begin{equation}
\textbf{j} = \vert{\textbf{E}}\vert^{-1} \left( \begin{array}{c} E_x \\ E_y \end{array} \right)
\end{equation}

The Jones vector is a class and right now it has very limited functionality: you define the polarization manually:

- Horizontal     = Jones(1,0)
- Vertical       = Jones(0,1)
- $\pm45^\circ$  = Jones(1,$\pm$1)
- Right Circular = Jones(1,1j)
- Left Circular  = Jones(1,-1j)

Rotating the polarization by angle 'theta' is accomplished by

    rot_mat(theta)

which performs the rotation matrix:

$$
R(\phi) \equiv \left(\begin{matrix} \cos\phi & -\sin\phi \\ \sin\phi & \cos\phi \end{matrix}\right)
$$

half-wave plates with fast axis horizontal are defined by

$$
H_0 \equiv e^{i\pi/2}\left(\begin{matrix} 1 & 0 \\ 0 & -1 \end{matrix}\right)
$$

and quarter wave plates as:

$$
Q_0 \equiv e^{i\pi/4}\left(\begin{matrix} 1 & 0 \\ 0 & -i \end{matrix}\right)
$$

Wave plates at arbitrary angles are then applied by rotating polarization into the basis defined by the wave-plate, applying the in-axis wave plate transformation, and rotating back into the "lab frame".

\begin{eqnarray}
Q(\theta) & = & R(\theta)Q_0R(-\theta) \\
H(\theta) & = & R(\theta)H_0R(-\theta)
\end{eqnarray}

There is also a primitive method to plot the polarization ellipses of an array of jones vectors via:
    j1 = Jones(1,2)
    #j2 = ...
    # jvecs = [j1,j2,...]
    plot_pol(j1) # plot a single Jones Vector
    plot_pols(jvecs) # plot an array of Jones Vectors

This couled easily be expanded upon.

```python
%pylab inline
class Jones:
    def __init__(self, horiz=1, vert=0):
        mg =sqrt(abs(horiz)**2 + abs(vert)**2)
        self.h = horiz/mg
        self.v = vert/mg

    def rotmat(self,theta):
        return Jones(self.h*cos(theta) - self.v*sin(theta), self.v*cos(theta) + self.h*sin(theta))

    def hwp0(self):
        return Jones(self.h,-self.v)
    def qwp0(self):
        return Jones(self.h,1j*self.v)

    def hwp(self,theta):
        return self.rotmat(-theta).hwp0().rotmat(theta)
    def qwp(self,theta):
        return self.rotmat(-theta).qwp0().rotmat(theta)

def plot_pol(j):
    figure(figsize=[5,5])
    wt = linspace(0,2*pi,1e2)

    plot(real(j.h *exp(1j*wt)),real(j.v *exp(1j*wt)))
    xlim([-1,1])
    ylim([-1,1])
    grid(True)
    xlabel('$E_x$')
    ylabel('$E_y$')

def plot_pols(jvex):
    figure(figsize=[5,5])
    wt = linspace(0,2*pi,1e2)
    for j in jvex:
        plot(real(j.h *exp(1j*wt)),real(j.v *exp(1j*wt)))
    xlim([-1,1])
    ylim([-1,1])
    grid(True)
    xlabel('$E_x$')
    ylabel('$E_y$')
```

    Populating the interactive namespace from numpy and matplotlib

```python
# Example: action of half wave plate on linear polarization
j_h = Jones(1,0)
j_p = j_h.hwp(pi/8)

plot_pols([j_h,j_p])
title('HWP at $\\theta$ rotates Linear Polarization by $2\\theta$')
```

```python
# Example: a field with some arbitrary Jones vector
j = Jones(1,1+1j)
plot_pol(j)
title('Some Aribitrary Polarization')
```

```python
# Example: action of quarter wave plate on linear polarization
j_h = Jones(1,0)

plz = []
thz = linspace(0,pi/2,10)
for t in thz:
    plz = append(plz,j_h.qwp(t))
plot_pols(plz)
title('H Polarization Through QWP for Various $\\theta$')
```

```python
# Example: action of quarter wave plate on circular polarization
j_c = Jones(1,1j)
plz = []
thz = linspace(0,pi,10)
for t in thz:
    plz = append(plz,j_c.qwp(t))
plot_pols(plz)
title('R Polarization Through QWP for Various $\\theta$')
```


# Support Vector Machines using cvxopt
> Implementation of hard margin and soft margin classifiers using cvxopt.
- author: "Axel Mendoza"
- categories: [svm, kernel-trick, pytorch, eda, from-scratch]
- toc: false
- comments: true
- badges: true
- image: images/svm.jpg

This blog post is about implementing Support Vector Machines from scratch using CVXOPT.
We will go through the math behind the SVM method and test out the kernel RBF and linear kernel on generated data.

```python
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
from cvxopt import matrix as cvxmat
from cvxopt import solvers as solver
from sklearn.datasets import make_blobs
from sklearn.preprocessing import StandardScaler
```

```python
def generate_data(n_samples=200, n_features=2, centers=2, center_box=(-15., 15.), cluster_std=1.):
    """Generates the clusters for testing linear SVMs

    Args:
        n_samples: An integer for the number of data points to be generated.
        n_features: An integer for the number of the columns in the output data.
        centers: An integer for the number of cluster to generate.
        center_box: A tuple of float for the location of the clusters.
        cluster_std: A float that controls the standard deviation of the clusters.
    """
    X, y = make_blobs(n_samples=n_samples, n_features=n_features, centers=centers,
                      center_box=center_box, cluster_std=cluster_std)
    y[y == 0] = -1
    y = y.astype(float)

    st = StandardScaler()
    X_s = st.fit_transform(X)
    return X_s, y
```

```python
X, y = generate_data()
```

```python
plt.scatter(X[:, 0], X[:, 1], c=y)
```

Here are two linearly separable clusters that we will use to test the Hard Margin SVM method.

## The kernel Trick
First let's define the kernel trick to understand the magic behind SVMs.
A kernel function $\kappa(\boldsymbol{x}, \boldsymbol{x^{'}})$ is a measure of similarity between two objects $\boldsymbol{x}, \boldsymbol{x^{'}} \in \chi$ where $\chi$ is an abstract space of higher dimension. Let's assume that we have $\boldsymbol{X}$ a non-linearly separable data. We would like to transform the data into a higher dimensional space to make it linearly separable. Let us define $\phi$ a function that maps the data into a higher dimensional space. A classical approach would be to compute $\phi(\boldsymbol{X})$ but this transformation may be very computationally heavy. Instead we use the kernel trick.

The kernel trick method stands for calculating the relationship of data in higher dimension without actually transforming the whole data. The kernel trick reduces the amount of computation required for kernelized algorithms. The main idea is, instead of computing $\phi(\boldsymbol{X})$, we will compute the dot product $\phi(\boldsymbol{X}_i)^\top\phi(\boldsymbol{X}_j)$ for all the components of $\boldsymbol{X}$. In other words, we will know the relationship of the data in higher dimension, without ever computing the transformation of the whole data.

The kernel function is defined as:
$$
\kappa(\boldsymbol{x}, \boldsymbol{x^{'}}) = \phi(\boldsymbol{x})^\top \phi(\boldsymbol{x^{'}})
$$

The kernel function has the following properties:
$$
\begin{align}
                \kappa(\boldsymbol{x}, \boldsymbol{x^{'}}) &\in \mathbb{R} \\
                \kappa(\boldsymbol{x}, \boldsymbol{x}^{'}) &= \kappa(\boldsymbol{x}^{'}, \boldsymbol{x}) \\
                \kappa(\boldsymbol{x}, \boldsymbol{x^{'}}) &\geq 0 \\
            \end{align}
$$
There is a multitude of different kernels like:
$$
            \begin{align}
                \kappa(\boldsymbol{x}, \boldsymbol{x^{'}}) &= \boldsymbol{x}^\top \boldsymbol{x}^{'} && \text{linear kernel} \\
                \kappa(\boldsymbol{x}, \boldsymbol{x^{'}}) &= \left(\boldsymbol{x}^\top \boldsymbol{x^{'}} + r\right)^d && \text{polynomial kernel} \\
                \kappa(\boldsymbol{x}, \boldsymbol{x^{'}}) &= \exp\left(-\frac{|| \boldsymbol{x} -  \boldsymbol{x^{'}} ||^2}{2\sigma^{2}}\right) && \text{kernel RBF} \\
            \end{align}
$$
where $r$ is the coefficient of the polynomial and $d$ its degree, while $\sigma$ is known as the bandwith of the radial basis function (RBF).

## Hard Margin Classifier
Support vector machines were originally designed for binary classification but can be extended to regression and multi-class classification.
SVMs are not probabilistic models, their output is a raw prediction. We will only discuss how SVMs work for classification. Let's get the basic intuition with the help of the following figure.

SVMs separate the data in two classes using a hyperplane. The distance between the dotted hyperplanes and the hyperplane at the center is known as margin.
A margin is passing by one or multiple points, these points are known as support vectors.
Any point above the red hyperplane will be classified as $1$ respectively $-1$ for the points below the hyperplane.
In order to predict on an input sample $\boldsymbol{x}$, we compute the linear equation:
$$
\begin{align}
    f(\boldsymbol{x}) &= \boldsymbol{w}^\top \boldsymbol{x} + b \\
    \hat{y} &=
    \begin{cases}
        \;\;\;1 & \text{if} \quad f(\boldsymbol{x}) \geq 0 \\
        -1 & \text{otherwise}
    \end{cases}
\end{align}
$$

SVMs try to maximize the margin when separating the data. This fact plays a role in the generalization of SVM models. We can find the value of the margin by solving the equation:
$$
    \begin{align}
        \boldsymbol{w}^\top (\boldsymbol{x}_2 - \boldsymbol{x}_1) &= 2 \\
        \frac{\boldsymbol{w}^\top}{||\boldsymbol{w}||} (\boldsymbol{x}_2 - \boldsymbol{x}_1) &= \frac{2}{||\boldsymbol{w}||}
    \end{align}
$$

In other words, we need to solve the optimization problem known as primal:
$$
    \begin{equation}
        \min_{\boldsymbol{w}, b}\frac{1}{2}||\boldsymbol{w}||^2 \quad \text{s.t} \quad y_i(\boldsymbol{w}^\top \boldsymbol{x}_i + b) \geq 1, \quad \forall i \in \{1, \dots, N\}
    \end{equation}
$$

The reason we minimize $||\boldsymbol{w}||^2$ instead of $||\boldsymbol{w}||$ is that minimizing the second option is non-convex.
The condition ensures that all samples are on the right side of the hyperplane. Note that this classifier does not allow any misclassification and is known as hard margin classifier.

## Primal form to Dual form
The primal form is can be transformed to the dual for which is more suited for implementation.
We can use the generalized Lagrangian to get the dual form that has more feasible constrains:
$$
    \begin{equation}
        \min_{\boldsymbol{x}, b}L(\boldsymbol{x}, b, \boldsymbol{\sigma}) = \frac{1}{2}||\boldsymbol{w}||^2 - \sum_{i=1}^{N} \sigma_i y_i(\boldsymbol{w}^\top \phi(\boldsymbol{x}_i) + b - 1)
        \label{eq:lagrange}
    \end{equation}
$$

We also need to satisfy the KKT conditions, such as:
$$
    \begin{align}
        \frac{\partial L}{\partial \boldsymbol{w}} = \boldsymbol{w} - \sum_{i=1}^{N} \sigma_i y_i\phi(\boldsymbol{x}_i) &= 0 \Rightarrow \boldsymbol{w} = \sum_{i=1}^{N} \sigma_i y_i\phi(\boldsymbol{x}_i) \\
        \frac{\partial L}{\partial \boldsymbol{b}} = -\sum_{i=1}^{N}\sigma_i y_i &= 0 \Rightarrow \sum_{i=1}^{N}\sigma_i y_i = 0
    \end{align}
$$

Because the dual problem is a lower bound of the primal problem, we need to maximize the equation above:
$$
    \begin{equation}
        \max_{\boldsymbol{\sigma}}\min_{\boldsymbol{x}, b}L(\boldsymbol{x}, b, \boldsymbol{\sigma}) \sim \min_{\boldsymbol{w}}f(\boldsymbol{x})
    \end{equation}
$$

By substituting the formulas that we got while solving1 the KKT conditions in $L$ we get the final dual form:
$$
    \begin{align}
        &\max_{\boldsymbol{\sigma}} \: \sum_{i=1}^{N}\sigma_i - \frac{1}{2} \sum_{i=1}^{N}\sum_{j=1}^{N} \sigma_i \sigma_j y_i y_j \phi(\boldsymbol{x}_i)^\top \phi(\boldsymbol{x}_j) \\
        &\text{s.t.} \quad \sigma_i \geq 0 && \forall i \in \{1, \dots, N\}\\
        &\text{s.t.} \quad \sum_{i=1}^{N}\sigma_iy_i = 0 && \forall i \in \{1, \dots, N\}\\
    \end{align}
$$

Then, we transform the optimization problem into a minimization and introducting the Gram matrix $\boldsymbol{K}_{ij} = \phi(\boldsymbol{x}_i)^\top \phi(\boldsymbol{x}_j)$:
$$
    \begin{align}
    &\min_{\boldsymbol{\sigma}; \boldsymbol{\sigma} \geq 0}  \frac{1}{2} \sum_{i=1}^{N}\sum_{j=1}^{N} \sigma_i \sigma_j y_i y_j \boldsymbol{K}_{ij} -  \sum_{i=1}^{N}\sigma_i \\
        &\text{s.t.} \quad \sigma_i \geq 0 && \forall i \in \{1, \dots, N\}\\
        &\text{s.t.} \quad \sum_{i=1}^{N}\sigma_iy_i = 0 && \forall i \in \{1, \dots, N\}\\
    \end{align}
$$

If the data is not linearly separable, even after using the kernel trick, we can use a soft margin classifier.

## Soft Margin Classifier
Soft-margin classifiers introduce slack variables $\zeta_i \geq 0$ as $\zeta_i = |y_i - f_i|$ to allow the classifier to mis-classify some samples.

In fact, we allow the model to make mistakes to be able to separate non linearly separable data where:
- $\zeta_i = 0$ means that the point lies in the correct margin boundary.
- $0 < \zeta_i \leq 1$ the point lies inside the margin and on the correct side.
- $\zeta_i > 1$ the point lies in the incorrect boundary.

The primal optimization is then:
$$
    \begin{align}
        &\min_{\boldsymbol{w}, b}\frac{1}{2}||\boldsymbol{w}||^2 - C\sum_{i=1}^{N}\zeta_i\\
        &\text{s.t} \quad y_i(\boldsymbol{w}^\top \boldsymbol{x}_i + b) \geq 1 - \zeta_i, && \forall i \in \{1, \dots, N\} \\
        &\text{s.t} \quad \zeta_i \geq 0 && \forall i \in \{1, \dots, N\}
    \end{align}
$$
where $C$ is the regularization parameter that controls the number of errors we are willing to tolerate on the training set.

Then the dual problem for the soft margin model is:
$$
    \begin{align}
        &\max_{\boldsymbol{\sigma}}\sum_{i=1}^{N}\sigma_i - \frac{1}{2} \sum_{i=1}^{N}\sum_{j=1}^{N} \sigma_i \sigma_j y_i y_j \phi(\boldsymbol{x}_i)^\top \phi(\boldsymbol{x}_j) \\
        &\text{s.t.} \quad 0 \leq \sigma_i \leq C && \forall i \in \{1, \dots, N\}\\
        &\text{s.t.} \quad \sum_{i=1}^{N}\sigma_iy_i = 0 && \forall i \in \{1, \dots, N\}\\
    \end{align}
$$

## Implementation
We will compute the convex optimizations using the CVXOPT package.
The solver performs the following optimization:
$$
\begin{align}
    \min &\frac{1}{2} \boldsymbol{\sigma}^\top \boldsymbol{P} \boldsymbol{\sigma} - 1^\top\boldsymbol{\sigma} \\
    &\text{s.t.} \\
    &- \sigma_i \leq 0 \\
    &\boldsymbol{y}^\top \boldsymbol{\sigma} = 0
\end{align}
$$

As you have probably guess, we will have to convert the dual problem in matrix form, let's define:
$$
\begin{align}
    &P_{ij} = y_i y_j \boldsymbol{K}_{ij} \\
    &\boldsymbol{w} = \sum_{i=1}^{N} y_i \sigma_i\phi(\boldsymbol{x}_i) \\
    &b = y_s - \sum_{m \in S} \sigma_m y_m \phi(\boldsymbol{x}_m)^\top \phi(\boldsymbol{x}_s) \\
\end{align}
$$
where $S$ is the set of support vectors such as $\sigma_i > 0$.

We can solve the hard margin optimization problem by solving:
$$
\begin{align}
    \min &\frac{1}{2} \boldsymbol{x}^\top\boldsymbol{P}\boldsymbol{x} + \boldsymbol{q}^\top\boldsymbol{x}\\
    &\text{s.t.} \\
    &\boldsymbol{G}\boldsymbol{x} \leq h \\
    &\boldsymbol{A}\boldsymbol{x} = 0
\end{align}
$$
where $h$ is a column of ones, $\boldsymbol{A}$ contains the labels and $\boldsymbol{G}$ is a diagonal matrix full of $-1$.

Here is the pseudo-algorithm for training an SVM:
- Compute $\boldsymbol{P}$.
- Compute $\boldsymbol{w}$.
- Find the support vectors $S$ by finding any data point where $\sigma_i > 0$.
- Compute $b$.
- To predict, compute:
$$
    y_{test} = \text{sign}\left(\boldsymbol{w}^\top \phi({\boldsymbol{x}_{test}}) + b \right)
$$

In matrix form the dual optimization is:
$$
\begin{align}
    \min &\frac{1}{2} \boldsymbol{\sigma}^\top \boldsymbol{P} \boldsymbol{\sigma} - 1^\top\boldsymbol{\sigma} \\
    &\text{s.t.} \\
    &- \sigma_i \leq 0 \\
    &\boldsymbol{y}^\top \boldsymbol{\sigma} = 0
\end{align}
$$

For training a soft margin classifier, we add a constraint on $C$:
$$
\begin{align}
    \min &\frac{1}{2} \boldsymbol{\sigma}^\top \boldsymbol{P} \boldsymbol{\sigma} - 1^\top\boldsymbol{\sigma} \\
    &\text{s.t.} \\
    &- \sigma_i \leq 0 \\
    &\sigma_i \leq C \\
    &\boldsymbol{y}^\top \boldsymbol{\sigma} = 0
\end{align}
$$

```python
def linear_kernel(X_1, X_2):
    """Computes a dot product between two vectors"""
    return np.dot(X_1, X_2)
```

```python
class SupportVectorClassifier():
    """Class for both the hard margin and soft margin classifiers

    Attributes:
        kernel: A function for the kernel function to apply.
        C: If using soft margin, it is a float controling the tolerance to misclassification.
        sv_sigma: A torch tensor for the slack variables of the support vectors.
        sv: A torch tensor for the features of the support vectors.
        sv_y: A torch tensor for the labels of the support vectors.
        w: A torch tensor for the weights of the model, only used if the kernel is linear.
        b: A float for the bias.
    """
    def __init__(self, kernel, C=None):
        """Init function of the SVC"""
        self.kernel = kernel
        self.C = C

    def fit(self, X, y):
        """Trains the SVC model

        Args:
            X: A torch tensor for the data.
            y: A torch tensor for the labels.
        """
        # Building the cvxopt matrices
        if len(y.shape) == 1:
            y = np.expand_dims(y, axis=1)
        y = y.astype(float)
        n_samples, n_features = X.shape
        K = np.zeros((n_samples, n_samples))
        for i in range(n_samples):
            for j in range(n_samples):
                K[i, j] = self.kernel(X[i, :], X[j, :])
        q = cvxmat(np.ones((n_samples, 1)) * -1.)
        A = cvxmat(y.T)
        b = cvxmat(0.)
        P = cvxmat(np.outer(y, y) * K)

        if self.C is None:
            G = cvxmat(np.eye(n_samples) * -1.)
            h = cvxmat(np.zeros((n_samples, 1)))
        else:
            col_1 = np.eye(n_samples) * -1.
            col_2 = np.identity(n_samples)
            G = cvxmat(np.vstack((col_1, col_2)))

            col_1 = np.zeros(n_samples)
            col_2 = np.ones(n_samples) * self.C
            h = cvxmat(np.hstack((col_1, col_2)))

        # Compute the convex optimization
        sigma = np.array(solver.qp(P, q, G, h, A, b)['x'])

        # Gather support vectors
        sv_idx = sigma > 1e-5
        self.sv_sigma = np.expand_dims(sigma[sv_idx], axis=1)
        sv_idx = np.squeeze(sv_idx)
        self.sv = X[sv_idx, :]
        self.sv_y = y[sv_idx]

        # If linear kernel, compute w, otherwise calculate it in pairwise
        # fashion once we predict
        if self.kernel == linear_kernel:
            w = np.sum(self.sv_y * self.sv_sigma * self.sv, axis=0)
            self.w = np.expand_dims(w, axis=1)
        else:
            self.w = None

        # Gather all pairwise distance of the support vectors
        idx = sv_idx.astype(int).nonzero()[0]
        b = 0
        sv_sigma_flat = self.sv_sigma.flatten()
        sv_y_flat = self.sv_y.flatten()
        for i in range(len(idx)):
            b += self.sv_y[i]
            b -= np.sum(sv_sigma_flat * sv_y_flat * K[idx[i], sv_idx])
        b /= len(idx)
        self.b = b

    def project(self, X):
        """Projects the data into the SVC hyperspace"""
        if self.w is not None:
            return np.dot(X, self.w) + self.b
        else:
            pred = np.zeros(X.shape[0])
            for i in range(X.shape[0]):
                cur_pred = 0
                for j in range(self.sv_sigma.shape[0]):
                    cur_pred += self.sv_sigma[j] * self.sv_y[j] * self.kernel(X[i], self.sv[j])
                pred[i] = cur_pred
            return pred + self.b

    def predict(self, X):
        """Projects the data than return the sign of the prediction as label"""
        return np.sign(self.project(X))
```

## Hard Margin Classifier with Linear Kernel

```python
svc = SupportVectorClassifier(linear_kernel)
svc.fit(X, y)
```

         pcost       dcost       gap    pres   dres
     0: -1.6591e+02 -3.9116e+02  2e+02  4e-14  2e+00
     1: -3.5588e+02 -3.6335e+02  7e+00  5e-15  1e+00
     2: -1.7374e+04 -1.7380e+04  6e+00  7e-12  1e+00
     3: -5.0598e+07 -5.0598e+07  2e+02  1e-08  1e+00
     4: -5.4654e+10 -5.4655e+10  2e+05  2e-05  1e+00
    Terminated (singular KKT matrix).

```python
pred = svc.predict(X)
```

```python
def plot_svc(X, y, svc, pred, plot_sv=True):
    """Plots the SVC hyperplane"""
    plt.scatter(X[:, 0], X[:, 1], c=y)
    if plot_sv:
        plt.scatter(svc.sv[:, 0], svc.sv[:, 1])
    axis_min, axis_max = plt.gca().get_xlim()

    w = svc.w

    a = -w[0] / w[1]
    xx = np.linspace(axis_min, axis_max)
    yy = a * xx - svc.b[0] / w[1]

    # Plot the hyperplane
    b = svc.sv[np.where(svc.sv_y == 1.)[0]][0]

    yy_down = a * xx + (b[1] - a * b[0])
    b = svc.sv[np.where(svc.sv_y == -1.)[0]][0]
    yy_up = a * xx + (b[1] - a * b[0])

    plt.plot(xx, yy, 'k-')
    plt.plot(xx, yy_down, 'k--')
    plt.plot(xx, yy_up, 'k--')
```

```python
plot_svc(X, y, svc, pred)
```

The SVC separates very well the data, the support vectors appear in blue.

## Soft Margin Classifier with Linear Kernel

```python
X, y = generate_data(n_samples=200, n_features=2, centers=2, center_box=(-5, 5), cluster_std=2.5)
plt.scatter(X[:, 0], X[:, 1], c=y)
```

This time we generated non separable data.

```python
svc_soft = SupportVectorClassifier(linear_kernel, C=0.1)
```

         pcost       dcost       gap    pres   dres
     0: -4.2577e+01 -4.1240e+01  1e+03  3e+01  2e-15
     1: -7.5598e+00 -3.8621e+01  9e+01  1e+00  1e-15
     2: -5.1212e+00 -1.8202e+01  1e+01  3e-02  1e-15
     3: -6.0958e+00 -8.1423e+00  2e+00  4e-03  7e-16
     4: -6.7688e+00 -7.1911e+00  4e-01  5e-04  5e-16
     5: -6.8928e+00 -7.0247e+00  1e-01  1e-04  5e-16
     6: -6.9357e+00 -6.9653e+00  3e-02  2e-05  6e-16
     7: -6.9460e+00 -6.9522e+00  6e-03  3e-06  5e-16
     8: -6.9488e+00 -6.9489e+00  9e-05  4e-08  6e-16
     9: -6.9489e+00 -6.9489e+00  9e-07  4e-10  6e-16
    Optimal solution found.

```python
pred = svc_soft.predict(X)
```

```python
plot_svc(X, y, svc_soft, pred, plot_sv=False)
```

The soft margin classifier places the classification plane in the middle of the two cluster in the direction of highest variance. Well done !

## Hard Margin Classifier with Kernel RBF

```python
def generate_non_linear(mean_1, mean_2, mean_3, mean_4, cov):
    """Generates non linear data from gaussian distributions

    Mean 1 to 4 is are the means that generate 4 clusters and cov is their covariance matrix.
    """
    X_1 = np.random.multivariate_normal(mean_1, cov, 50)
    X_1 = np.vstack((X_1, np.random.multivariate_normal(mean_3, cov, 50)))
    y_1 = np.ones(len(X_1))
    X_2 = np.random.multivariate_normal(mean_2, cov, 50)
    X_2 = np.vstack((X_2, np.random.multivariate_normal(mean_4, cov, 50)))
    y_2 = np.ones(len(X_2)) * -1

    X = np.vstack([X_1, X_2])
    y = np.hstack([y_1, y_2])
    X = StandardScaler().fit_transform(X)
    return X, y
```

```python
X, y = generate_non_linear(mean_1=[-2, 2], mean_2=[1, -1],
                           mean_3=[4, -4], mean_4=[-4, 4],
                           cov=[[1.0,0.8], [0.8, 1.0]])
```

```python
plt.scatter(X[:, 0], X[:, 1], c=y)
```

Even the linear soft margin model cannot fit well this data. For solving this problem we will use the kernel radial basis function (RBF).

```python
def polynomial_kernel(X_1, X_2, p):
    """Computes the polynomial kernel where p is the degree of the polynomial"""
    return (1 + np.dot(X_1, X_2)) ** p

def kernel_rbf(X_1, X_2, sigma=1.0):
    """Computes the kernel RBF where sigma is the bandwith"""
    return np.exp(-np.linalg.norm(X_1 - X_2) ** 2 / (2 * sigma ** 2))
```

```python
svc_rbf = SupportVectorClassifier(kernel_rbf)
svc_rbf.fit(X, y)
```

         pcost       dcost       gap    pres   dres
     0: -1.5702e+01 -4.7310e+01  5e+02  2e+01  2e+00
     1: -7.4712e+00 -5.0258e+01  9e+01  3e+00  2e-01
     2: -1.0976e+01 -4.0857e+01  5e+01  1e+00  9e-02
     3: -1.9295e+01 -4.0147e+01  2e+01  1e-01  1e-02
     4: -2.7419e+01 -3.4022e+01  7e+00  1e-02  1e-03
     5: -3.0658e+01 -3.2568e+01  2e+00  2e-03  2e-04
     6: -3.1971e+01 -3.2192e+01  2e-01  2e-04  1e-05
     7: -3.2154e+01 -3.2158e+01  4e-03  2e-06  2e-07
     8: -3.2157e+01 -3.2157e+01  5e-05  2e-08  2e-09
     9: -3.2157e+01 -3.2157e+01  5e-07  2e-10  2e-11
    Optimal solution found.

```python
pred = svc_rbf.predict(X)
```

```python
def plot_svc_rbf(X, y, svc):
    """Plots the hyperplane and margins of a SVC with rbf kernel"""
    plt.scatter(X[:, 0], X[:, 1], c=y)
    sv = svc.sv
    plt.scatter(sv[:, 0], sv[:, 1], c='g')
    axis_min, axis_max = plt.gca().get_xlim()

    X1, X2 = np.meshgrid(np.linspace(axis_min, axis_max, 50), np.linspace(axis_min, axis_max,50))
    X = np.array([[x1, x2] for x1, x2 in zip(np.ravel(X1), np.ravel(X2))])
    Z = svc.project(X).reshape(X1.shape)
    plt.contour(X1, X2, Z, [0.0], colors='k', linewidths=1, origin='lower')
    plt.contour(X1, X2, Z + 1, [0.0], colors='grey', linewidths=1, origin='lower')
    plt.contour(X1, X2, Z - 1, [0.0], colors='grey', linewidths=1, origin='lower')

    plt.axis("tight")
    plt.show()

```

```python
plot_svc_rbf(X, y, svc_rbf)
```

## Conclusion
Support Vector Machines have a variety of kernels to fit the most complex data. SVMs work really well when there is a clear margin of separation between classes and are very effective on data with high dimensionality. Unfortunately, SVMs are not suited for large scale datasets due to their computational complexity. Furthermore, there is no probabilistic explanation of their prediction and they do not perform well on very noisy data.


# Continuous Signals

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Standard Signals

Certain [signals](https://en.wikipedia.org/wiki/Signal_%28electrical_engineering%29) play an important role in the theory and practical application of [signal processing](https://en.wikipedia.org/wiki/Signal_processing). They emerge from the theory of signals and systems, are used to characterize the properties of linear time-invariant (LTI) systems or frequently occur in practical applications. These standard signals are introduced and illustrated in the following. The treatise is limited to one-dimensional deterministic time- and amplitude-continuous signals.

### Complex Exponential Signal

The complex exponential signal over time $t$ is defined by the [complex exponential function](https://en.wikipedia.org/wiki/Exponential_function#Complex_plane)

\begin{equation}
x(t) = e^{s t}
\end{equation}

where $s = \sigma + j \omega$ denotes the complex frequency with $\sigma, \omega \in \mathbb{R}$ and $j$ the imaginary unit $(j^2=-1)$. The signal is often used as a generalized representation of harmonic signals. Using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula) above definition can be reformulated as

\begin{equation}
x(t) = e^{(\sigma + j \omega) t} = e^{\sigma t} \cos(\omega t) + j e^{\sigma t} \sin(\omega t)
\end{equation}

The real/imaginary part of the exponential signal is given by a weighted cosine/sine with angular frequency $\omega = 2 \pi f$. For $t>0$, the time-dependent weight $e^{\sigma t}$ is

* exponentially decaying over time for $\sigma < 0$,
* constantly one for $\sigma = 0$,
* exponentially growing over time for $\sigma > 0$,

and vice-versa for $t<0$. The complex exponential signal is used to model harmonic signals with constant or exponentially decreasing/increasing amplitude.

**Example**

The following example illustrates the complex exponential signal and its parameters. The Python module [SymPy](http://docs.sympy.org/latest/index.html) is used for this purpose. It provides functionality for symbolic variables and functions, as well as their calculus. The required symbolic variables need to be defined explicitly before usage. In the example $t$, $\omega$ and $\sigma$ are defined as real-valued symbolic variables, followed by the definition of the exponential signal.

```python
import sympy as sym
sym.init_printing()

t, sigma, omega = sym.symbols('t sigma omega', real=True)
s = sigma + 1j*omega
x = sym.exp(s*t)
x
```

Now specific values for the complex frequency $s = \sigma + j \omega$ are considered for illustration. For this purpose a new signal is defined by substituting both $\sigma$ and $\omega$ with specific values. The real and imaginary part of the signal is plotted for illustration.

```python
y = x.subs({omega: 10, sigma: -.1})

sym.plot(sym.re(y), (t, 0, 2*sym.pi), ylabel=r'Re{$e^{st}$}')
sym.plot(sym.im(y), (t, 0, 2*sym.pi), ylabel=r'Im{$e^{st}$}');
```

**Exercise**

* Try other values for `omega` and `sigma` to create signals with increasing/constant/decreasing amplitudes and different angular frequencies.

### Dirac Impulse

The Dirac impulse is one of the most important signals in the theory of signals and systems. It is used for the characterization of LTI systems and the modeling of impulse-like signals. The Dirac impulse is defined by way of the [Dirac delta function](https://en.wikipedia.org/wiki/Dirac_delta_function) which is not a function in the conventional sense. It is a generalized function or *distribution*. The Dirac impulse is denoted as $\delta(t)$. The Dirac delta function is defined by its effect on other functions. A rigorous treatment is beyond the scope of this course material. Please refer to the literature for a detailed discussion of the mathematical foundations of the Dirac delta distribution. Fortunately it is suitable to consider only certain properties for its application in signal processing. The most relevant ones are

1. **Sifting property**
    \begin{equation}
    \int_{-\infty}^{\infty} \delta(t) \cdot x(t) = x(0)
    \end{equation}
    where $x(t)$ needs to be differentiable at $t=0$. The sifting property implies $\int_{-\infty}^{\infty} \delta(t) = 1$.

2. **Multiplication**
    \begin{equation}
    x(t) \cdot \delta(t) = x(0) \cdot \delta(t)
    \end{equation}
    where $x(t)$ needs to be differentiable at $t=0$.

3. **Linearity**
    \begin{equation}
    a \cdot \delta(t) + b \cdot \delta(t) = (a+b) \cdot \delta(t)
    \end{equation}

4. **Scaling**
    \begin{equation}
    \delta(a t) = \frac{1}{|a|} \delta(t)
    \end{equation}
    where $a \in \mathbb{R} \setminus 0$. This implies that the Dirac impulse is a function with even symmetry.

5. **Derivation**
    \begin{equation}
    \int_{-\infty}^{\infty} \frac{d \delta(t)}{dt} \cdot x(t) \; dt = - \frac{d x(t)}{dt} \bigg\vert_{t = 0}
    \end{equation}

6. **Convolution**

    Generalization of the sifting property yields
    \begin{equation}
    \int_{-\infty}^{\infty} \delta(\tau) \cdot x(t - \tau)  \, d\tau = x(t)
    \end{equation}

    This operation is known as [convolution](https://en.wikipedia.org/wiki/Convolution) and will be introduced later in more detail. It may be concluded already here that the Dirac delta function constitutes the neutral element of the convolution.

It is important to note that the product $\delta(t) \cdot \delta(t)$ of two Dirac impulses is not defined.

**Example**

This example illustrates some of the basic properties of the Dirac impulse. Let's first define a Dirac impulse by way of the Dirac delta function

```python
delta = sym.DiracDelta(t)
delta
```

Now let's check the sifting property by defining an arbitrary signal (function) $f(t)$ and integrating over its product with the Delta impulse

```python
f = sym.Function('f')(t)
sym.integrate(delta*f, (t, -sym.oo, sym.oo))
```

**Exercise**

* Derive the sifting property for a shifted Dirac impulse $\delta(t-\tau)$ and check your results by modifying above example.

### Heaviside Signal

The Heaviside signal is defined by the [Heaviside step function](https://en.wikipedia.org/wiki/Heaviside_step_function)

\begin{equation}
\epsilon(t) = \begin{cases} 0 & t<0 \\  \frac{1}{2} & t=0 \\ 1 & t > 0 \end{cases}
\end{equation}

Note that alternative definitions exist, which differ with respect to the value of $\epsilon(t)$ at $t=0$. The Heaviside signal may be used to represent a signal that switches on at a specified time and stays switched on indefinitely. The Heaviside signal can be related to the Dirac impulse by

\begin{equation}
\epsilon(t) = \int_{-\infty}^{t} \delta(\tau) \; d\tau
\end{equation}

**Example**

In the following, a Heaviside signal $\epsilon(t)$ is defined and plotted. Note that `Sympy` denotes the Heaviside function by $\theta(t)$.

```python
step = sym.Heaviside(t)
step
```

```python
sym.plot(step, (t, -2, 2), ylim=[-0.2, 1.2], ylabel=r'$\epsilon(t)$');
```

Let's construct a harmonic signal $\cos(\omega t)$ with $\omega=2$ which is switched on at $t=0$. Considering the definition of the Heaviside function, the desired signal is given as

\begin{equation}
x(t) = \cos(\omega t) \cdot \epsilon(t)
\end{equation}

```python
x = sym.cos(omega*t) * sym.Heaviside(t)
sym.plot(x.subs(omega, 2), (t, -2, 10), ylim=[-1.2, 1.2], ylabel=r'$x(t)$');
```

### Rectangular Signal

The rectangular signal is defined by the [rectangular function](https://en.wikipedia.org/wiki/Rectangular_function)

\begin{equation}
\text{rect}(t) = \begin{cases} 1 & |t| < \frac{1}{2} \\ \frac{1}{2} & |t| = \frac{1}{2} \\ 0 & |t| > \frac{1}{2} \end{cases}
\end{equation}

Its time limits and amplitude are chosen such that the area under the function is $1$.

Note that alternative definitions exist, which differ with respect to the value of $\text{rect}(t)$ at $t = \pm \frac{1}{2}$. The rectangular signal is used to represent a signal which has finite duration, respectively is switched on for a limited period of time. The rectangular signal can be related to the Heaviside signal by

\begin{equation}
\text{rect}(t) = \epsilon \left(t + \frac{1}{2} \right) - \epsilon \left(t - \frac{1}{2} \right)
\end{equation}

**Example**

The Heaviside function is used to define a rectangular function in `Sympy`. This function is then used as rectangular signal.

```python
class rect(sym.Function):

    @classmethod
    def eval(cls, arg):
        return sym.Heaviside(arg + sym.S.Half) - sym.Heaviside(arg - sym.S.Half)
```

```python
sym.plot(rect(t), (t, -1, 1), ylim=[-0.2, 1.2], ylabel=r'rect$(t)$');
```

**Exercise**

* Use $\text{rect}(t)$ to construct a harmonic signal $\cos(\omega t)$ with $\omega=2$ which is switched on at $t=-\frac{1}{2}$ and switched off at $t=+\frac{1}{2}$.

### Sign Signal

The sign signal is defined by the [sign/signum function](https://en.wikipedia.org/wiki/Sign_function) which evaluates the sign of its argument

\begin{equation}
\text{sgn}(t) = \begin{cases} 1 & t>0 \\ 0 & t=0 \\ -1 & t < 0  \end{cases}
\end{equation}

The sign signal is useful to represent the absolute value of a real-valued signal $x(t) \in \mathbb{R}$ by a multiplication

\begin{equation}
|x(t)| = x(t) \cdot \text{sgn}(x(t))
\end{equation}

It is related to the Heaviside signal by

\begin{equation}
\text{sgn}(t) = 2 \cdot \epsilon(t) - 1
\end{equation}

when following above definition with $\epsilon(0)=\frac{1}{2}$.

**Example**

The following example illustrates the sign signal $\text{sgn}(t)$. Note that the sign function is represented as $\text{sign}(t)$ in `Sympy`.

```python
sgn = sym.sign(t)
sgn
```

```python
sym.plot(sgn, (t, -2, 2), ylim=[-1.2, 1.2], ylabel=r'sgn$(t)$');
```

**Exercise**

* Check the values of $\text{sgn}(t)$ for $t \to 0^-$, $t = 0$ and $t \to 0^+$ as implemented in `SymPy`. Do they conform to above definition?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Task 5: Time-dependent expectation values

## Time-dependent overlap

We compute the time-dependent overlap between two Slater determinants (with orthonormal atomic orbital basis functions) by
\begin{align}
    P(t, t_0)
    &= \bigl|\langle \Phi(t) | \Phi(t_0) \rangle \bigr|^2
    = \bigl|\det\bigl(C^{\dagger}_o(t) C_o(t_0)\bigr)\bigr|^2,
\end{align}
where $C_o(t)$ is the coefficient matrix of the time-evolved state with $o$ denoting the occupied columns.

## One-body expectation values

Given a one-body operator $\hat{A}$ we wish to compute the time-dependent expectation values.
```{note}
A one-body operator is an operator that acts on a single particle at a time.
For example, the position operator $\hat{x}$ is a one-body operator, whereas the Coulomb interaction $\hat{u}$ is a two-body operator as it applies to the interaction between two particles at a time.
```
Having already found the matrix elements of $\hat{A}$ in a static (time-independent) atomic basis $\{\psi_{\mu}\}_{\mu = 1}^{l}$, we can compute the expectation value of this operator by
\begin{align}
    \langle A(t) \rangle
    &= \langle \psi_{\mu} | \hat{A} | \psi_{\nu} \rangle D_{\nu \mu}(t),
\end{align}
where $D_{\nu \mu}(t) = \sum_{i = 1}^{n} C^{*}_{\mu i}(t) C_{\nu i}(t)$ is the time-evolved Hartree-Fock density matrix.
Alternatively, by transforming the one-body operator to the time-evolved molecular orbital basis $\{\psi_p(t)\}_{p = 1}^{k}$ we can compute the expectation value from
\begin{align}
    \langle A(t) \rangle
    &=
    \langle \phi_{p}(t) | \hat{A} | \phi_q(t) \rangle
    \rho_{q p},
\end{align}
where $\rho_{q p}$ is the one-body density matrix in the time-evolved Hartree-Fock basis.
This matrix is given by
\begin{align}
    \boldsymbol{\rho}
    &= \begin{pmatrix}
        \boldsymbol{I}_{n \times n} & \boldsymbol{0}_{n \times (k - n)} \\
        \boldsymbol{0}_{(k - n) \times n} & \boldsymbol{0}_{(k - n) \times (k - n)}
    \end{pmatrix},
\end{align}
that is, the matrix is the identity in the occupied block and zero elsehwere.
Both ways of computing the expectation value give the same value.


# ルンゲクッタ法による常微分方程式の数値解析

ここでは，オイラー法よりも計算精度が良く，広く用いられている4次のルンゲ・クッタ（Runge-Kutta）法を学ぶ．

オイラー法のときと同様に，2階常微分方程式を1階化した

$$
\begin{cases}
 \frac{dx(t)}{dt} = v(t), \\
 \frac{dv(t)}{dt} = f(x(t), v(t))
\end{cases}
\tag{2, 再掲}
$$

を考える．式（2）の数値解 $x_i$, $v_i$，$i=0, 1, 2, \ldots,$は，４次のルンゲ・クッタ法により次のように計算される．

$$
\begin{eqnarray}
x_{i+1} &= x_i + \frac{h}{6} \left(k_{x1} + 2k_{x2} + 2k_{x3} + k_{x4} \right),
\tag{9}
\\
%
v_{i+1} &= v_i + \frac{h}{6} \left(k_{v1} + 2k_{v2} + 2k_{v3} + k_{v4} \right)
\tag{10}
\end{eqnarray}
$$

ここで，

$$
\begin{eqnarray}
&k_{x1} = v_i, \quad
&k_{v1} = f(x_i, v_i),\\
%
&k_{x2} = v_i + \frac{h k_{v1}}{2}, \quad
&k_{v2} = f\left(x_i + \frac{h k_{x1}}{2}, v_i + \frac{h k_{v1}}{2} \right),\\
%
&k_{x3} = v_i + \frac{h k_{v2}}{2}, \quad
&k_{v3} = f\left(x_i + \frac{h k_{x2}}{2}, v_i + \frac{h k_{v2}}{2} \right),\\
%
&k_{x4} = v_i + h k_{v3}, \quad
&k_{v4} = f(x_i + h k_{x3}, v_i + h k_{v3}),
\end{eqnarray}
$$

であり，$h$は積分刻みである．

では，ルンゲクッタ法をPythonで実装してみよう．ここでは

$$
\begin{align}
  f = -x(t) \tag{5}\\
\end{align}
$$

とする．また，初期条件を$x(0)=1.0, y(0)=0$とする．

このとき，ルンゲクッタ法を実装すると下記のようになる．

（TODO） 下記のコードを理解し，セルを実行せよ

```
# 必要なライブラリのインポート
import numpy as np
from matplotlib import pyplot as plt

# 関数定義
def f(x, v):
  return -x # dv/dt = f(x, y) = -x の実装

# 計算条件を変数として定義
x = 1.0 # xの初期値
v = 0 # yの初期値
t_ini = 0 # 初期時刻
h = 0.2 # 積分刻み
t_max = 10 + h # 終了時刻

# 結果保存用の配列定義
ts = np.arange(start=t_ini, stop=t_max, step=h) # tの時系列
xs = np.zeros(len(ts)) # xの時系列
vs = np.zeros(len(ts)) # yの時系列

# 初期値の代入
xs[0] = x
vs[0] = v

# ルンゲクッタ法
for i in range(1, len(ts)): # i=1 から i=len(ts) までループ
  kx1 = v
  kv1 = f(x, v)
  kx2 = v + h * kv1 / 2.0
  kv2 = f(x + h * kx1 / 2.0, v + h * kv1 / 2.0)
  kx3 = v + h * kv2 / 2.0
  kv3 = f(x + h * kx2 / 2.0, v + h * kv2 / 2.0)
  kx4 = v + h * kv3
  kv4 = f(x + h * kx3, v + h * kv3)
  # v, xの更新
  v += (kv1 + 2.0 * kv2 + 2.0 * kv3 + kv4) * h / 6.0
  x += (kx1 + 2.0 * kx2 + 2.0 * kx3 + kx4) * h / 6.0
  # 結果の保存
  xs[i] = x
  vs[i] = v
```

ルンゲ・クッタ法で計算された数値解が配列`xs`, `ys`に格納される．また，数値計算の各時刻は`ts`に格納される．

（TODO） 下記のコードを実行し，`ts`を横軸，`xs`を縦軸にプロットせよ．

```
plt.plot(ts, xs, '.')
plt.xlabel('t')
plt.ylabel('x')
```

（TODO） 下記のコードを実行し，`ts`を横軸，`vs`を縦軸にプロットせよ．

```
plt.plot(ts, vs, '.')
plt.xlabel('t')
plt.ylabel('v')
```

オイラー法のときと同様に，得られた数値解と厳密解を比較してみよう．

（TODO） 下記のコードを実行し，得られた厳密解とオイラー法の数値解を同時にplotせよ．

```
xs_exact = np.cos(ts) # 厳密解

# 厳密解とオイラー法の数値解をplot
plt.plot(ts, xs_exact, '-r', label='Exact')
plt.plot(ts, xs, '.g', label='RK') # 既にルンゲ・クッタ法で計算済み
plt.xlabel('t')
plt.ylabel('x')
plt.legend() # labelを表示する
```

（TODO） 次のコードを実行し，ルンゲ・クッタ法で得た数値解の配列`xs`と，厳密解の配列`xs_exact`のMAEを計算せよ．

```
np.mean(np.abs(xs - xs_exact))
```

出力された結果から，今回のルンゲ・クッタ法では，$4.0\times10^{-5}$程度のMAEとなるはずだ．よって，ルンゲ・クッタ法はオイラー法よりも高精度な数値解が得られることが確かめられた．

# ここまでのまとめ

数値解析手法であるオイラー法とルンゲ・クッタ法をPythonを用いて実装し，その動作を確認した．両手法ともに，常微分方程式の1階化ができれば，適用できることがわかっただろう．

ここまでの内容を一通り理解したら，[実験のTopページ](https://github.com/yyamnk/numerical-methods-py3/blob/master/uu_experiment.md) に戻り，レポート課題に取り組んでほしい．


# Understanding the SVD

```python
import numpy as np
```

### Useful reference

- [A Singularly Valuable Decomposition](https://datajobs.com/data-science-repo/SVD-[Dan-Kalman].pdf)

## Sketch of lecture

### Singular value decomposition

Our goal is to understand the following forms of the SVD.

$$
A = U \Sigma V^T
$$

$$
A = \begin{bmatrix}
U_1 & U_2
\end{bmatrix}\begin{bmatrix}
\Sigma_1 & 0 \\
0 & 0
\end{bmatrix}\begin{bmatrix}
V_1^T \\
V_2^T
\end{bmatrix}
$$

$$
A = \sum_{i=1}^r \sigma u_i v_i^T
$$

### (1) The matrix A

#### What does a matrix do?

A linear function is one that satisfies the property that

$$
f(a_1x_1 + a_2x_2 + \cdots + a_nx_n) = a_1 f(x_1) + a_2 f(x_2) + \ldots + a_n f(x_n)
$$

Let $f(x) = Ax$, where $A$ is a matrix and $x$ is a vector. You can check that the matrix $A$ fulfills the property of being a linear function. If $A$ is $m \times n$, then it is a linear map from $\mathbb{R}^n \mapsto \mathbb{R}^m$.

Let's consider: what does a matrix *do* to a vector?  Matrix multiplication has a *geometric* interpretation.  When we multiply a vector, we either rotate, reflect, dilate or some combination of those three. So multiplying by a matrix *transforms* one vector into another vector.  This is known as a *linear transformation*.

Important Facts:

* Any matrix defines a linear transformation
* The matrix form of a linear transformation is NOT unique
* We need only define a transformation by saying what it does to a *basis*

Suppose we have a matrix $A$ that defines some transformation.  We can take any invertible matrix $B$ and

$$BAB^{-1}$$

defines the same transformation.  This operation is called a *change of basis*, because we are simply expressing the transformation with respect to a different basis.

**Example**

Let $f(x)$ be the linear transformation that takes $e_1=(1,0)$ to $f(e_1)=(2,3)$ and $e_2=(0,1)$ to $f(e_2) = (1,1)$.  A matrix representation of $f$ would be given by:

$$A = \left(\begin{matrix}2 & 1\\3&1\end{matrix}\right)$$

This is the matrix we use if we consider the vectors of $\mathbb{R}^2$ to be linear combinations of the form

$$c_1 e_1 + c_2 e_2$$

Now, consider a second pair of (linearly independent) vectors in $\mathbb{R}^2$, say $v_1=(1,3)$ and $v_2=(4,1)$. We first find the transformation that takes $e_1$ to $v_1$ and $e_2$ to $v_2$.  A matrix representation for this is:

$$B = \left(\begin{matrix}1 & 4\\3&1\end{matrix}\right)$$

Our original transformation $f$ can be expressed with respect to the basis $v_1, v_2$ via

$$B^{-1}AB$$

#### Fundamental subspaces of $A$

- Span and basis
- Inner and outer products of vectors
- Rank of outer product is 1
- $C(A)$, $N(A)$, $(C(A^T))$ and $N(A^T)$ mean
- Dimensions of each space and its rank
- How to find a basis for each subspace given a $m \times n$ matrix $A$
- Sketch the diagram relating the four fundamental subspaces

### (2) Orthogonal matrices $U$ and $V^T$

- Orthogonal (perpendicular) vectors
- Orthonormal vectors
- Orthogonal matrix
- $Q^TQ = QQ^T = I$
- Orthogonal matrices are rotations (and reflections)
- Orthogonal matrices preserve norms (lengths)
- 2D orthogonal matrix is a rotation matrix
$$ V =
\begin{bmatrix}
\cos\theta & -\sin \theta \\
\sin \theta & \cos \theta
\end{bmatrix}
$$
- $V^T$ rotates the perpendicular frame spanned by $V$ into the standard frame spanned by $e_i$
- $V$ rotates the standard frame into the frame spanned by $V$
-
$$\text{proj}_v x = \frac{\langle x, v \rangle}{\langle v, v \rangle} v
$$
- Matrix form
$$
P = \frac{vv^T}{v^Tv}
$$
- Gram-Schmidt for converting $A$ into an orthogonal matrix $Q$
- QR decomposition

### (3) Diagonal matrix $S$

- Recall that a matrix $A$ is a transform with respect to some basis
- It is desirable to find the simplest similar matrix $B$ in some other basis
- $A$ and $B$ represent the exact same linear transform, just in different coordinate systems
- $Av = \lambda v$ defines the eigenvectors and eigenvalues of $A$
- When a square matrix $A$ is real, symmetric and has all non-negative eigenvalues, it has an eigen-space decomposition (ESD)
$$
A = V \Lambda V^T
$$
where $V$ is orthogonal and $\Lambda$ is diagonal
- The columns of $V$ are formed from the eigenvectors of $A$
- The diagonals of $\Lambda$ are the eigenvalues of $A$ (arrange from large to small in absolute value)

## (4) SVD $U\Sigma V^T$

- The SVD is a generalization of ESD for general $m \times n$ matrices $A$
- If $A$ is $(m \times n)$, we cannot perform an ESD
- $A^TA$ is diagonalizable (note this is the dot product of all pairs of column vectors in $A$)
-
$$
A^TA = V \Lambda V^T
$$
- Let $\Lambda = \Sigma^2$
- Let $U = AV\Sigma^{-1}$
- The $A = U\Sigma V^T$
- Show $U$ is orthogonal
- Show $U$ is formed from eigenvectors of $AA^T$
- Geometric interpretation of SVD
    - rotate orthogonal frame $V$ onto standard frame
    - scale by $\Sigma$
    - rotate standard frame into orthogonal frame $U$

### Covariance, PCA and SVD

Remember the formula for covariance

$$
\text{Cov}(X, Y) = \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1}
$$

where $\text{Cov}(X, X)$ is the sample variance of $X$.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import scipy.linalg as la
```

```python
np.set_printoptions(precision=3)
```

```python
def cov(x, y):
    """Returns covariance of vectors x and y)."""
    xbar = x.mean()
    ybar = y.mean()
    return np.sum((x - xbar)*(y - ybar))/(len(x) - 1)
```

```python
X = np.random.random(10)
Y = np.random.random(10)
```

```python
np.array([[cov(X, X), cov(X, Y)], [cov(Y, X), cov(Y,Y)]])
```

    array([[0.077, 0.027],
           [0.027, 0.097]])

Using `numpy` function

```python
np.cov(X, Y)
```

    array([[0.077, 0.027],
           [0.027, 0.097]])

```python
Z = np.random.random(10)
np.cov([X, Y, Z])
```

    array([[0.077, 0.027, 0.01 ],
           [0.027, 0.097, 0.014],
           [0.01 , 0.014, 0.06 ]])

#### Eigendecomposition of the covariance matrix

```python
mu = [0,0]
sigma = [[0.6,0.2],[0.2,0.2]]
n = 1000
x = np.random.multivariate_normal(mu, sigma, n).T
```

```python
A = np.cov(x)
```

```python
m = np.array([[1,2,3],[6,5,4]])
ms = m - m.mean(1).reshape(2,1)
np.dot(ms, ms.T)/2
```

    array([[ 1., -1.],
           [-1.,  1.]])

```python
e, v = la.eigh(A)
```

```python
plt.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e, v.T):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3])
plt.title('Eigenvectors of covariance matrix scaled by eigenvalue.');
```

### PCA

Principal Components Analysis (PCA) basically means to find and rank all the eigenvalues and eigenvectors of a covariance matrix. This is useful because high-dimensional data (with $p$ features) may have nearly all their variation in a small number of dimensions $k$, i.e. in the subspace spanned by the eigenvectors of the covariance matrix that have the $k$ largest eigenvalues. If we project the original data into this subspace, we can have a dimension reduction (from $p$ to $k$) with hopefully little loss of information.

Numerically, PCA is typically done using SVD on the data matrix rather than eigendecomposition on the covariance matrix. The next section explains why this works. Numerically, the condition number for working with the covariance matrix directly is the square of the condition number using SVD, so SVD minimizes errors.

For zero-centered vectors,

\begin{align}
\text{Cov}(X, Y) &= \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1} \\
  &= \frac{\sum_{i=1}^nX_iY_i}{n-1} \\
  &= \frac{XY^T}{n-1}
\end{align}

and so the covariance matrix for a data set X that has zero mean in each feature vector is just $XX^T/(n-1)$.

In other words, we can also get the eigendecomposition of the covariance matrix from the positive semi-definite matrix $XX^T$.

Note: Here $x$ is a matrix of **row** vectors

```python
X = np.random.random((5,4))
X
```

    array([[0.027, 0.212, 0.602, 0.276],
           [0.118, 0.095, 0.058, 0.736],
           [0.595, 0.283, 0.537, 0.475],
           [0.39 , 0.952, 0.45 , 0.16 ],
           [0.493, 0.958, 0.811, 0.184]])

```python
Y = X - X.mean(1)[:, None]
```

```python
np.around(Y.mean(1), 5)
```

    array([-0.,  0., -0., -0., -0.])

```python
Y
```

    array([[-0.252, -0.067,  0.323, -0.003],
           [-0.134, -0.157, -0.194,  0.484],
           [ 0.123, -0.19 ,  0.064,  0.003],
           [-0.098,  0.464, -0.038, -0.328],
           [-0.119,  0.346,  0.2  , -0.427]])

```python
np.cov(X)
```

    array([[ 0.057, -0.007,  0.001, -0.006,  0.024],
           [-0.007,  0.105,  0.001, -0.07 , -0.095],
           [ 0.001,  0.001,  0.018, -0.034, -0.023],
           [-0.006, -0.07 , -0.034,  0.111,  0.102],
           [ 0.024, -0.095, -0.023,  0.102,  0.119]])

```python
np.cov(Y)
```

    array([[ 0.057, -0.007,  0.001, -0.006,  0.024],
           [-0.007,  0.105,  0.001, -0.07 , -0.095],
           [ 0.001,  0.001,  0.018, -0.034, -0.023],
           [-0.006, -0.07 , -0.034,  0.111,  0.102],
           [ 0.024, -0.095, -0.023,  0.102,  0.119]])

```python
e1, v1 = np.linalg.eig(np.dot(x, x.T)/(n-1))
```

#### Principal components

Principal components are simply the eigenvectors of the covariance matrix used as basis vectors. Each of the original data points is expressed as a linear combination of the principal components, giving rise to a new set of coordinates.

```python
plt.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e1, v1.T):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

#### Change of basis

Suppose we have a vector $u$ in the standard basis $B$ , and a matrix $A$ that maps $u$ to $v$, also in $B$. We can use the eigenvalues of $A$ to form a new basis $B'$. As explained above, to bring a vector $u$ from $B$-space to a vector $u'$ in $B'$-space, we multiply it by $Q^{-1}$, the inverse of the matrix having the eigenvctors as column vectors. Now, in the eigenvector basis, the equivalent operation to $A$ is the diagonal matrix $\Lambda$ - this takes $u'$ to $v'$. Finally, we convert $v'$ back to a vector $v$ in the standard basis by multiplying with $Q$.

#### We get the principal components by a change of basis

```python
plt.scatter(ys[0,:], ys[1,:], alpha=0.2)
for e_, v_ in zip(e1, np.eye(2)):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

For example, if we only use the first column of `ys`, we will have the projection of the data onto the first principal component, capturing the majority of the variance in the data with a single feature that is a linear combination of the original features.

#### Transform back to original coordinates

We may need to transform the (reduced) data set to the original feature coordinates for interpretation. This is simply another linear transform (matrix multiplication).

```python
zs = np.dot(v1, ys)
```

```python
plt.scatter(zs[0,:], zs[1,:], alpha=0.2)
for e_, v_ in zip(e1, v1.T):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

```python
u, s, v = np.linalg.svd(x)
u.dot(u.T)
```

    array([[1.00e+00, 1.11e-16],
           [1.11e-16, 1.00e+00]])

#### Dimension reduction via PCA

We have the spectral decomposition of the covariance matrix

$$
A = Q^{-1}\Lambda Q
$$

Suppose $\Lambda$ is a rank $p$ matrix. To reduce the dimensionality to $k \le p$, we simply set all but the first $k$ values of the diagonal of $\Lambda$ to zero. This is equivalent to ignoring all except the first $k$ principal components.

What does this achieve? Recall that $A$ is a covariance matrix, and the trace of the matrix is the overall variability, since it is the sum of the variances.

```python
A
```

    array([[0.589, 0.191],
           [0.191, 0.191]])

```python
A.trace()
```

    0.7798589520198774

```python
e, v = np.linalg.eigh(A)
D = np.diag(e)
D
```

    array([[0.114, 0.   ],
           [0.   , 0.665]])

```python
D.trace()
```

    0.7798589520198773

```python
D[0,0]/D.trace()
```

    0.14677219641978512

Since the trace is invariant under change of basis, the total variability is also unchanged by PCA. By keeping only the first $k$ principal components, we can still "explain" $\sum_{i=1}^k e[i]/\sum{e}$ of the total variability. Sometimes, the degree of dimension reduction is specified as keeping enough principal components so that (say) $90\%$ of the total variability is explained.

### Using SVD for PCA

SVD is a decomposition of the data matrix $X = U S V^T$ where $U$ and $V$ are orthogonal matrices and $S$ is a diagonal matrix.

Recall that the transpose of an orthogonal matrix is also its inverse, so if we multiply on the right by $X^T$, we get the following simplification

\begin{align}
X &= U S V^T \\
X X^T &= U S V^T (U S V^T)^T \\
 &= U S V^T V S U^T \\
 &= U S^2 U^T
\end{align}

Compare with the eigendecomposition of a matrix $A = W \Lambda W^{-1}$, we see that SVD gives us the eigendecomposition of the matrix $XX^T$, which as we have just seen, is basically a scaled version of the covariance for a data matrix with zero mean, with the eigenvectors given by $U$ and eigenvealuse by $S^2$ (scaled by $n-1$)..

```python
u, s, v = np.linalg.svd(x)
```

```python
e2 = s**2/(n-1)
v2 = u
plt.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e2, v2):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

```python
v1 # from eigenvectors of covariance matrix
```

    array([[ 0.928, -0.373],
           [ 0.373,  0.928]])

```python
v2 # from SVD
```

    array([[-0.928, -0.373],
           [-0.373,  0.928]])

```python
e1 # from eigenvalues of covariance matrix
```

    array([0.665, 0.115])

```python
e2 # from SVD
```

    array([0.665, 0.115])