# Ridge, Lasso and Elasticnet

Ora che abbiamo capito come effettuare la regressione lineare anche con più di una variabile indipendente(MLR), è ora necessario vedere come applicare altri metodi lineari per ovviare ai problemi che possono influire sulle previsioni o sul training, ed essi sono:

- collinearità: ovvero la possibilità di due o più feature di presentare una relazione di tipo lineare consistente nel dataset, questo in genere è un bene solo nel caso in cui sia perfetta, altrimenti può portare all'insordenza di problemi
- dimensionalità: in alcuni casi potremmo avere un numero di feature molto grandi o addirittura maggiori del numero di samples al suo interno
- overfitting  o underfitting: il modello non riesce ad estrapolare relazioni generali a causa della sua complessità maggiore o minore al problema.

Per questo motivo, basandoci sempre sulla tecnica **OLS** sono state creati altre tecniche più robuste per la regressione, in genere queste tecniche che andremo ad analizzare si basano sulla __[Regolarizzazione](https://it.wikipedia.org/wiki/Regolarizzazione_(matematica))__ ovvero sull'**imposizione sulla formula di minimizzazione di un termine che penalizzi una specifica condizione legata al problema al fine di ridurne l'errore di generalizzazione**.

La regoralizzazione viene imposta attraverso le __[condizioni di Karush-Kuhn-Tucker(KKT)](http://onmyphd.com/?p=kkt.karush.kuhn.tucker)__ dove quindi introduciamo dei termini di penalizzazione sugli unici termini che possiamo controllare: **i pesi($w$)**. In questa lezione vedremo che la condizione sui pesi come **la norma, la norma quadratica di $w$ o la loro condizione ibrida** porta a diversi tipi di regressori esprimibili matematicamente come:
- Ridge Regression : $\large\min_{w} \parallel wX - Y \parallel^2 + \alpha \parallel w \parallel^2$
- Lasso Regression : $\large\min_{w} \parallel wX - Y \parallel^2 + \beta \parallel w \parallel$
- Elasticnet : $\large\min_{w} \parallel wX - Y \parallel^2 + \alpha \parallel w \parallel^2 + \beta \parallel w \parallel$

I parametri $\alpha$ e $\beta$ sono detti anche **Iperparametri-Hyperparameters** e vengono determinati usando **criteri di statistica o la cross-validation scegliendo tra quelli che minimizzano l'errore di testing**.

## Ridge Regression

Iniziamo ad analizzare la __[Ridge regression](https://scikit-learn.org/stable/modules/linear_model.html#ridge-regression-and-classification)__ riportando la formula definita in precedenza, in cui il termine modulo quadro della norma viene anche detto ***termine $l_{2}$***

\begin{equation}
\Large\min_{w} \parallel wX - Y \parallel^2 + \alpha \parallel w \parallel^2
\end{equation}

la trattazione può essere vista anche in questo modo

\begin{equation}
\Large\min_{w} \parallel wX - Y \parallel^2 , \parallel w \parallel^2 \leq \alpha
\end{equation}

ovvero la condizione risulta che i pesi che devono soddisfare la **OLS** devono avere un modulo quadro **minore di $\alpha$** per poter essere ammesi dalla formula, ma qualè l'effetto prodotto dalla $\alpha$ sui fit?

```python
%matplotlib widget

import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split
from ipywidgets import widgets, interactive_output

#load the dataset
data = load_boston()
df = pd.DataFrame(data.data, columns = data.feature_names)
df['MEDV'] = data.target
#prepare the data
X_train, X_test, y_train, y_test = train_test_split(df.LSTAT.values, df.MEDV.values,
                                                    random_state=0, test_size = 0.2)
#define a first regressor
ridge = Ridge(alpha = 1)
ridge.fit(X_train.reshape(-1, 1), y_train)

#setting plot
fig, ax = plt.subplots(figsize=(8, 6))
ax.scatter(X_train, y_train, label = "train", color = "blue")
ax.scatter(X_test, y_test, label = "test", color = "green")
line, = ax.plot(X_train, ridge.predict(X_train.reshape(-1, 1)), label = "fit", color = "red")
plt.legend()

#define what happens when slider value change
def update(alpha):
    ridge = Ridge(alpha = alpha)
    ridge.fit(X_train.reshape(-1, 1), y_train)
    line.set_ydata(ridge.predict(X_train.reshape(-1, 1)))
    fig.canvas.draw()

#define slider
alpha = widgets.FloatSlider(
        value=1,
        min = 0, # min value
        max= 1e5, # max value
        step= 1, # step
        description = "\u03B1") # unicode character for alpha

# An HBox lays out its children horizontally
ui = widgets.HBox([alpha])
#define interactive output
plot = interactive_output(update, {'alpha':alpha})
#show slider and plot
display(plot,ui)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Output()

    HBox(children=(FloatSlider(value=1.0, description='α', max=100000.0, step=1.0),))

Quindi possiamo vedere che alzando il valore di $\alpha$ quello che succede è che i pesi tendono a zero e la linea diventa orizzontaele questo poichè il termine dominante della funzione di regressione diventa il termine di regolarizzazione, in tal caso l'unico modo per ridure complessivamente il valore è fare tendere a zero $\parallel w \parallel^2$ ovvero ponendo i termini $w$ nulli o molto vicini ad esso.

Questo aspetto può risultare molto efficacie nei problemi in cui il dataset presenta **multicollinearità, ovvero i dati presentano correlazioni tra di loro** e noi vogliamo invece tenere conto di feature quanto più possibili indipendenti e per la **riduzione delle dimensionalità**.

### Visione geometrica e riduzione della dimensionalità

La visione geometrica può essete interpretata partendo dalla formula che otteniamo per la pendenza dei coefficienti ottenibile come:

\begin{equation}
    \Large y = X \hat{\beta}^{ridge} = X (X^{T}X + \alpha I)^{-1}X^{T}y = \sum_{i=1}^{p} u_j \frac{d_{j}^{2}}{d_{j}^{2} + \alpha} u_{j}^Ty
\end{equation}

Ora focalizzandoci sulla formula possiamo notare che se noi consideriamo il caso in cui i dati sono centrati possiamo definire

\begin{equation}
    \large Var(\hat(B)_j) = \frac{u_{j}^{T}u_j}{d_{j}^2} \qquad \hat{\beta}^{ridge} = \sum_{i=1}^{p} \frac{d_{j}^{2}}{d_{j}^{2} + \alpha} u_{j}^T y
\end{equation}

pertanto il termine $\alpha$ riduce la proiezione delle direzioni $u_{j}$ che presentano una varianza più bassa, questo vuol dire che più è alto il valore di $\alpha$ e avremo componenti con varianza sempre più alta. questo principio a simile a quello che vedremo essere come PCA.

    La cosa forse più interessante dela Ridge Regression è la sua capacità di fittare su dati con molte dimensionalità avendo un piccolo numero di dati riuscendo a cogliere gli aspetti fondamentali della struttura dei dati! Per maggiori infor guardate il video qui sotto!

```python
from IPython.display import YouTubeVideo

#put link yotube video, only final part
YouTubeVideo('Q81RR3yKn30')
```

```python
%matplotlib widget
#prepare the data
X_train, X_test, y_train, y_test = train_test_split(df.drop('MEDV', axis = 1).values, df.MEDV.values,
                                                    random_state=0, test_size = 0.2)

#define a first regressor
ridge = Ridge(alpha = 1000)
ridge.fit(X_train, y_train)

#variables used for plots
names = df.columns.tolist()[:-1]#exclude last element
variance = df.drop('MEDV', axis = 1).var(axis = 0)
ypos = range(len(names))

#show data model as dataframe
#define dataframe
df_model = pd.DataFrame(columns=names)
df_model.loc['weights'] = ridge.coef_.tolist()
df_model.loc['variance'] = variance.tolist()
#display dataframe
display("Variance and weights with \u03B1 = 1000")
display(df_model)
#model with alpha =1
ridge = Ridge(alpha = 1)
ridge.fit(X_train, y_train)
df_model.loc['weights'] = ridge.coef_.tolist()
#display dataframe
display("Variance and weights with \u03B1 = 1")
display(df_model)

#plot setting
fig, (ax1, ax2) = plt.subplots(1, 2, figsize = (10,6))
#first subplot
bar = ax1.barh(ypos, ridge.coef_, align = 'center', color = 'red')
ax1.set_yticks(ypos)
ax1.set_yticklabels(names)
ax1.set_xlabel('Weight value')
ax1.set_title('Weights of every feature')
#second subplot
ax2.barh(ypos, variance, align = 'center', color = 'green')
ax2.set_yticks(ypos)
ax2.set_yticklabels(names)
ax2.set_xlabel('Variance value')
ax2.set_title('Variance of every feture')
plt.show()

#define what happens when slider value change
def update(alpha):
    ridge = Ridge(alpha = alpha)
    ridge.fit(X_train, y_train)
    for i in range(len(ridge.coef_)):
        bar[i].set_width(ridge.coef_[i])
    df_model.loc['weights'] = ridge.coef_.tolist()
    fig.canvas.draw()

#define slider
alpha = widgets.FloatSlider(
        value=1,
        min = 1, # min value
        max= 1000, # max value
        step= 1, # step
        description = "\u03B1") # unicode character fro alpha
# An HBox lays out its children horizontally
ui = widgets.HBox([alpha])
#define interactive output
plot = interactive_output(update, {'alpha':alpha})
#show slider and plot
display(plot,ui)
```

    'Variance and weights with α = 1000'

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      weights
      -0.087876
      0.059909
      -0.044061
      0.084989
      -0.008172
      0.454157
      0.028739
      -0.546187
      0.233914
      -0.015136
      -0.685913
      0.006599
      -0.718846

      variance
      73.986578
      543.936814
      47.064442
      0.064513
      0.013428
      0.493671
      792.358399
      4.434015
      75.816366
      28404.759488
      4.686989
      8334.752263
      50.994760

    'Variance and weights with α = 1'

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      weights
      -0.116808
      0.046003
      -0.023762
      2.278150
      -8.557796
      3.755135
      -0.010414
      -1.280095
      0.222038
      -0.011526
      -0.969288
      0.008535
      -0.498849

      variance
      73.986578
      543.936814
      47.064442
      0.064513
      0.013428
      0.493671
      792.358399
      4.434015
      75.816366
      28404.759488
      4.686989
      8334.752263
      50.994760

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Output()

    HBox(children=(FloatSlider(value=1.0, description='α', max=1000.0, min=1.0, step=1.0),))

## Lasso Regression

Ora che abbiamo capito come funziona la ridge regression, vediamo ora come funziona la **Lasso Regression**, partiamo dalla sua formula matematica che a differenza precedente predente un termine di norma detto anche ***termine $l_1$***

\begin{equation}
\Large\min_{w} \parallel wX - Y \parallel^2 + \beta \parallel w \parallel
\end{equation}

la trattazione può essere vista anche in questo modo

\begin{equation}
\Large\min_{w} \parallel wX - Y \parallel^2 , \parallel w \parallel \leq \beta
\end{equation}

Vediamo cosa succede qualora usiamo questo regressore variando il parametro $\beta$.

```python
%matplotlib widget
#import the model
from sklearn.linear_model import Lasso
#prepare the data
X_train, X_test, y_train, y_test = train_test_split(df.LSTAT.values, df.MEDV.values,
                                                    random_state=0, test_size = 0.2)
#define a first regressor
lasso = Lasso(alpha = 1)
lasso.fit(X_train.reshape(-1, 1), y_train)

#setting plot
fig, ax = plt.subplots(figsize=(8, 6))
ax.scatter(X_train, y_train, label = "train", color = "blue")
ax.scatter(X_test, y_test, label = "test", color = "green")
line, = ax.plot(X_train, lasso.predict(X_train.reshape(-1, 1)), label = "fit", color = "red")
plt.legend()

#define what happens when slider value change
def update(alpha):
    lasso = Lasso(alpha = alpha)
    lasso.fit(X_train.reshape(-1, 1), y_train)
    line.set_ydata(lasso.predict(X_train.reshape(-1, 1)))
    fig.canvas.draw()

#define slider
alpha = widgets.FloatSlider(
        value=1,
        min = 1, # min value
        max= 100, # max value
        step= 1, # step
        description = "\u03B2") # unicode character fro alpha
# An HBox lays out its children horizontally
ui = widgets.HBox([alpha])
#define interactive output
plot = interactive_output(update, {'alpha':alpha})
#show slider and plot
display(plot,ui)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Output()

    HBox(children=(FloatSlider(value=1.0, description='β', min=1.0, step=1.0),))

Come possiamo vedere in questo caso il valore di $\beta$(nel codice è sempre alpha) porta a **delle soluzioni dei coefficienti che tendono a zero per valori $\beta$ molto più piccoli rispetto alla ridge**, il motivo è da ricerca il fatto che ***la penalizzazione ora è l1***.

### Visione geometrica- Lasso Regression
La lasso regression non può essere riespressa come prima a causa del fatto che ora **non si ha una soluzione analitica a causa della non linearità in Y nella definizione**, questo può portare a dei problemi, fortunamente è ancora possibile visualizzare geometricamente il problema con il video qui sotto, rimane pur sempre la questione di capire cosa succede, la risposta risulta essere che il lasso tenda a fornire ***sparse solution, ovvero soluzione con un basso numero di soluzioni diverse da zero.***

Questo aspetto è fondamentale poiché grazie ad esso possiamo dire anche che ***il lasso tende a selezionare le feature più importanti e che riducono la varianza del modello sui dati al prezzo di un minimo di bais; per questo motivo la lasso viene anche usata come feature selction algorithm.***

L'algoritmo usato da scikit è quello del coordinate descent, non dello SGD e questo tipo di algoritmo richiede funzioni convesse.

```python
#put link yotube video, only final part
YouTubeVideo('14MKVkhvMus')
```

```python
#prepare the data
X_train, X_test, y_train, y_test = train_test_split(df.drop('MEDV', axis = 1).values, df.MEDV.values,
                                                    random_state=0, test_size = 0.2)

#define a first regressor
lasso = Lasso(alpha = 100)
lasso.fit(X_train, y_train)

#display dataframe
df_model.loc['weights'] = lasso.coef_.tolist()
display("Variance and weights with \u03B2 = 100")
display(df_model)

#model with alpha =1
lasso = Lasso(alpha = 1)
lasso.fit(X_train, y_train)
df_model.loc['weights'] = lasso.coef_.tolist()
#display dataframe
display("Variance and weights with \u03B2 = 1")
display(df_model)

#plot setting
fig, ax = plt.subplots(figsize = (10,6))
#first subplot
bar = ax.barh(ypos, lasso.coef_, align = 'center', color = 'red')
ax.set_yticks(ypos)
ax.set_yticklabels(names)
ax.set_xlabel('Weight value')
ax.set_title('Weights of every feature')

#define what happens when slider value change
def update(alpha):
    lasso = Lasso(alpha = alpha)
    lasso.fit(X_train, y_train)
    for i in range(len(lasso.coef_)):
        bar[i].set_width(lasso.coef_[i])
    df_model.loc['weights'] = lasso.coef_.tolist()
    fig.canvas.draw()

#define slider
alpha = widgets.FloatSlider(
        value=1,
        min = 1, # min value
        max= 100, # max value
        step= 1, # step
        description = "\u03B2") # unicode character fro alpha
# An HBox lays out its children horizontally
ui = widgets.HBox([alpha])
#define interactive output
plot = interactive_output(update, {'alpha':alpha})
#show slider and plot
display(plot,ui)
```

    'Variance and weights with β = 100'

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      weights
      -0.000000
      0.000000
      -0.000000
      0.000000
      -0.000000
      0.000000
      -0.000000
      0.000000
      -0.000000
      -0.023137
      -0.000000
      0.004687
      -0.00000

      variance
      73.986578
      543.936814
      47.064442
      0.064513
      0.013428
      0.493671
      792.358399
      4.434015
      75.816366
      28404.759488
      4.686989
      8334.752263
      50.99476

    'Variance and weights with β = 1'

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      weights
      -0.058890
      0.053177
      -0.000000
      0.000000
      -0.000000
      0.679550
      0.016841
      -0.648766
      0.198738
      -0.013994
      -0.864220
      0.006603
      -0.73121

      variance
      73.986578
      543.936814
      47.064442
      0.064513
      0.013428
      0.493671
      792.358399
      4.434015
      75.816366
      28404.759488
      4.686989
      8334.752263
      50.99476

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Output()

    HBox(children=(FloatSlider(value=1.0, description='β', min=1.0, step=1.0),))

## ElasticNet

Ora che abbiamo visto come funzionano la Lasso e la Ridge regression, vediamo la __[ElasticNet Regression](https://core.ac.uk/reader/4406480)__ che è un intermezzo tra le due regressioni precedenti e assume la formula:

\begin{equation}
\Large \min_{w} \parallel wX - Y \parallel^2 + \alpha \parallel w \parallel^2 + \beta \parallel w \parallel
\end{equation}

dove quindi abbiamo un **termine di regolarizzazione sia su il modulo quadro di $w^2$ che il modulo $w$**.

La formula di scikit per l'elasticnet è leggermente diversa ed è del tipo:

\begin{equation}
\large \min_{w} { \frac{1}{2n_{\text{samples}}} ||X w - y||_2 ^ 2 + \alpha \rho ||w||_1 +
\frac{\alpha(1-\rho)}{2} ||w||_2 ^ 2}
\end{equation}

in questa formula il termine $\rho$ deve essere un numero compreso tra 0 e 1 e ci dice in che misura contare i termini L1 e L2, metre $\alpha$ è il solito termoine di penalizzazione.

    Nel 2014 è stato scoperto che Elasticnet è come una Support Vector Machine lineare, vedremo in seguito cosa sono!

```python
from sklearn.linear_model import ElasticNet
#prepare the data
X_train, X_test, y_train, y_test = train_test_split(df.LSTAT.values, df.MEDV.values,
                                                    random_state=0, test_size = 0.2)
#define a first regressor
elastic = ElasticNet(alpha = 1, l1_ratio = 0.5)
elastic.fit(X_train.reshape(-1, 1), y_train)

#setting plot
fig, ax = plt.subplots(figsize=(8, 6))
ax.scatter(X_train, y_train, label = "train", color = "blue")
ax.scatter(X_test, y_test, label = "test", color = "green")
line, = ax.plot(X_train, elastic.predict(X_train.reshape(-1, 1)), label = "fit", color = "red")
plt.legend()

alpha =  widgets.FloatSlider(value=1, min = 1, max= 1e3, step= 1, description = "\u03B1")
rho =  widgets.FloatSlider(value=0.5, min = 0.01, max= 1, step= 0.01, description = "\u03C1")

#define what happens when slider value change
def update(alpha, rho):
    elastic = ElasticNet(alpha = alpha, l1_ratio = rho)
    elastic.fit(X_train.reshape(-1, 1), y_train)
    line.set_ydata(elastic.predict(X_train.reshape(-1, 1)))
    fig.canvas.draw()

# An HBox lays out its children horizontally
ui = widgets.HBox([alpha, rho])
#define interactive output
plot = interactive_output(update, {'alpha':alpha, 'rho':rho})
#show slider and plot
display(plot,ui)
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Output()

    HBox(children=(FloatSlider(value=1.0, description='α', max=1000.0, min=1.0, step=1.0), FloatSlider(value=0.5, …

### Interpretazione geometrica dell'elasticnet

L'elasticnet può essere visto come un'intermezzo tra la ridge e il lasso, dal punto di vista geometrico potete interpretarlo con questa immagine:

il parametro $\rho$ nella formula ci dice come vogliamo sia la formula di penalizzazione, con il parametro uguale a 1 abbiamo un lasso, mentre con zero abbiamo un comportamento ridge, con 0.5 abbiamo un comportamento intermedio tra i due.

```python
#prepare the data
X_train, X_test, y_train, y_test = train_test_split(df.drop('MEDV', axis = 1).values, df.MEDV.values,
                                                    random_state=0, test_size = 0.2)
#model with alpha =100
elastic = ElasticNet(alpha = 100, l1_ratio = 0.5)
elastic.fit(X_train, y_train)
df_model.loc['weights'] = elastic.coef_.tolist()

#display dataframe
display("Variance and weights with \u03B2 = 100, \u03C1 = 0.5")
display(df_model)

#define a first regressor
elastic = ElasticNet(alpha = 1, l1_ratio=0.5)
elastic.fit(X_train, y_train)

#display dataframe
df_model.loc['weights'] = elastic.coef_.tolist()
display("Variance and weights with \u03B1 = 1, \u03C1 = 0.5")
display(df_model)

#plot setting
fig, ax = plt.subplots(figsize = (10,6))
#first subplot
bar = ax.barh(ypos, elastic.coef_, align = 'center', color = 'red')
ax.set_yticks(ypos)
ax.set_yticklabels(names)
ax.set_xlabel('Weight value')
ax.set_title('Weights of every feature')

#define sliders
alpha =  widgets.FloatSlider(value=1, min = 1, max= 100, step= 1, description = "\u03B1")
rho =  widgets.FloatSlider(value=0.5, min = 0.01, max= 1, step= 0.01, description = "\u03C1")

#define what happens when slider value change
def update(alpha, rho):
    elastic = ElasticNet(alpha = alpha, l1_ratio = rho)
    elastic.fit(X_train, y_train)
    for i in range(len(elastic.coef_)):
        bar[i].set_width(elastic.coef_[i])
    df_model.loc['weights'] = elastic.coef_.tolist()
    fig.canvas.draw()

# An HBox lays out its children horizontally
ui = widgets.HBox([alpha, rho])
#define interactive output
plot = interactive_output(update, {'alpha':alpha, 'rho':rho})
#show slider and plot
display(plot,ui)
```

    'Variance and weights with β = 100, ρ = 0.5'

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      weights
      -0.000000
      0.007871
      -0.000000
      0.000000
      -0.000000
      0.000000
      -0.000000
      0.000000
      0.000000
      -0.023099
      -0.000000
      0.010668
      -0.00000

      variance
      73.986578
      543.936814
      47.064442
      0.064513
      0.013428
      0.493671
      792.358399
      4.434015
      75.816366
      28404.759488
      4.686989
      8334.752263
      50.99476

    'Variance and weights with α = 1, ρ = 0.5'

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      weights
      -0.079223
      0.056378
      -0.024847
      0.000000
      -0.000000
      0.842704
      0.015518
      -0.757884
      0.228696
      -0.014712
      -0.856892
      0.006905
      -0.714712

      variance
      73.986578
      543.936814
      47.064442
      0.064513
      0.013428
      0.493671
      792.358399
      4.434015
      75.816366
      28404.759488
      4.686989
      8334.752263
      50.994760

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Output()

    HBox(children=(FloatSlider(value=1.0, description='α', min=1.0, step=1.0), FloatSlider(value=0.5, description=…

# Come trovare i parametri migliori?

Per trovare i **parametri migliori dei modelli, detto anche hyperparameter tuning(poichè chiamati anche iperparametri)** consiste nel trovare dei metodi attraverso cui possiamo stabilire quali siano i valori di $\alpha$, $\beta$ o di qualsiasi altro parametro che il nostro modello usa per allenarsi sui dati, per questa trattazione vedremo tre modi : criterio di informazione di Akaike, criterio di informazione Baesiano e Cross validation.

## Akaike Information Criterion (AIC)

L'__[AIC](https://en.wikipedia.org/wiki/Akaike_information_criterion)__ è un criterio che si basa sulla seguente formula:

\begin{equation}
    \Large AIC = 2k - 2\ln{(\hat{L})}
\end{equation}

dove $k$ indica il numero di parametri del modello, $\hat{L} = p(x | \theta, M)$ è il massimo valore di verosomiglianza del modello con x i dati, $\theta$ i parametri del modello e $M$ il modello stesso, dove il modello migliore è quello che minimizza l'AIC.

## Bayesian information criterion(BIC)

Il __[BIC](https://en.wikipedia.org/wiki/Bayesian_information_criterion)__ invece ha la seguente formula:

\begin{equation}
    \Large BIC = k\ln(n) - 2\ln{(\hat{L})}
\end{equation}

dove il parametro $n$ indica il numero di dati presenti.

## Cross Validation

Nel caso della __[Cross Validation](https://en.wikipedia.org/wiki/Cross-validation_%28statistics%29)__ queello che usiamo è la __[coordinate descent](https://en.wikipedia.org/wiki/Coordinate_descent)__ dove la formula su cui ci basiamo nel caso la funzione sia differenziabile può essere rappresentata come:

\begin{equation}
    \Large x_{i+1} = x_{i} - \alpha \frac{\partial F}{\partial x_{i}}(\textbf{x})
\end{equation}

in cui $\alpha$ rappresenta il passo da effettuare nello spazio, la F è la nostra funzione da minimizzare e possiamo decidere di fissarci su una singola componente o su un intero blocco di esse. Usiamo in tal caso la Cross Validation per essere sicuri che i parametri trovati siano effettivamente generali e non specifici al nostro dataset di training.

## Least Angle Regression (LARS)

La libreria scikit(__[link alla libreria](https://scikit-learn.org/stable/modules/linear_model.html#least-angle-regression)__) permette di usare un particolare algoritmo molto utile chiamato __[LARS](https://en.wikipedia.org/wiki/Least-angle_regression)__, che permette di allenare modelli anche ad alta dimensionalità e che permette di estrapolare le varibili maggiormente correlate al target ed i loro coefficienti, vedremo che con questo algoritmo è possibile applicare i modelli appena visti.

```python
import time #libreria per tenere conto del tempo impiegato
from numpy import arange

#importiamo i metodi per trovare gli iperparametri
from sklearn.linear_model import RidgeCV, LassoCV, LassoLarsCV, LassoLarsIC, ElasticNetCV

#definiamo i dati da usare
X, Y = df.drop('MEDV', axis = 1).values, df.MEDV.values

print('Preparing the data for training and testing')
#prepare the data for the criterion optimization and testing
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=0, test_size = 0.2)
print('Data prepared, the 80% will be used for training, and the remaining 20% will be for testing')
#separator
print('-'*80)

#range of alphas to try on ridge
alpha = range(0,100,1)
betas = arange(0.01, 1, 0.01)

#definiamo una Ridge cross validation con score r2 e 10 fold con il nostro insieme di parametri alpha
print('Ridge Model optimized using 10 folds croos validation and searching all integers alphas between 1 and 100')
ridge = RidgeCV(alphas = alpha, scoring = 'r2', cv = 10)
ridge_start = time.time()
ridge.fit(X_train, Y_train)
ridge_end = time.time()
print(f'Ridge best alpha: {ridge.alpha_}, R^2 score testing: {ridge.score(X_test, Y_test)}\n'
      f' time taken : {ridge_end - ridge_start}s')
#separator
print('-'*80)

#Lasso model using cv
print('Lasso Model optimized using 10 folds cross validation and searching all integers alphas between 1 and 100')
#even in this case you need to pass the number of parameters
lassocv = LassoCV(alphas = betas, cv = 10)
lassocv_start = time.time()
lassocv.fit(X_train, Y_train)
lassocv_end = time.time()
print(f'Lasso best alpha: {lassocv.alpha_}, R^2 score testing: {lassocv.score(X_test, Y_test)}, '
      f'numero iterazioni: {lassocv.n_iter_}\n time taken : {lassocv_end - lassocv_start}s')
#separator
print('-'*80)

#lasso using cv and lars
print('Lasso Model using Lars algorithm optimized using 10 folds cross validation and searching all integers alphas between 1 and 100')
#here i pass the maximum number of of alphas used on the path to compute the residual
lassolarscv = LassoLarsCV(max_n_alphas=100, cv = 10)
lassolarscv_start = time.time()
lassolarscv.fit(X_train, Y_train)
lassolarscv_end = time.time()
print(f'Lasso Lars best alpha: {lassolarscv.alpha_}, R^2 score testing: {lassolarscv.score(X_test, Y_test)}, '
      f'numero iterazioni: {lassolarscv.n_iter_}\n time taken : {lassolarscv_end - lassolarscv_start}s')
#separator
print('-'*80)

#lasso model using aic
print('Lasso Model using AIC criterion optimization')
#here i pass the maximum number of of alphas used on the path to compute the residual
lassolarsaic = LassoLarsIC(criterion='aic')
lassolarsaic_start = time.time()
lassolarsaic.fit(X_train,Y_train)
lassolarsaic_end = time.time()
print(f'Lasso Lars using AIC best alpha: {lassolarsaic.alpha_}, R^2 score testing: {lassolarsaic.score(X_test, Y_test)}, '
      f'numero iterazioni : {lassolarsaic.n_iter_}\n time taken : {lassolarsaic_end - lassolarsaic_start}s')
#separator
print('-'*80)

#lasso model using bic
print('Lasso Model using BIC criterion optimization')
#here i pass the maximum number of of alphas used on the path to compute the residual
lassolarsbic = LassoLarsIC(criterion='bic')
lassolarsbic_start = time.time()
lassolarsbic.fit(X_train,Y_train)
lassolarsbic_end = time.time()
print(f'Lasso Lars using BIC best alpha: {lassolarsbic.alpha_}, R^2 score testing: {lassolarsbic.score(X_test, Y_test)}, '
      f'numero iterazioni : {lassolarsbic.n_iter_}\n time taken : {lassolarsbic_end - lassolarsbic_start}s')
#separator
print('-'*80)

#Lasso model using cv
print('Elasticnet Model optimized using 10 folds cross validation and searching l1_ratio between 0 and 1')
#even in this case you need to pass the number of parameters
elasticv = ElasticNetCV(l1_ratio=betas, n_alphas=100, cv = 10)
elasticv_start = time.time()
elasticv.fit(X_train, Y_train)
elasticv_end = time.time()
print(f'Elasticnet best alpha: {elasticv.alpha_}, best l1_ratio: {elasticv.l1_ratio_}, R^2 score testing: {elasticv.score(X_test, Y_test)}, '
      f'numero iterazioni: {elasticv.n_iter_}\n time taken : {elasticv_end-  elasticv_start}s')
#separator
print('-'*80)
```

    Preparing the data for training and testing
    Data prepared, the 80% will be used for training, and the remaining 20% will be for testing
    --------------------------------------------------------------------------------
    Ridge Model optimized using 10 folds croos validation and searching all integers alphas between 1 and 100
    Ridge best alpha: 0, R^2 score testing: 0.5892223849182503
     time taken : 2.002999782562256s
    --------------------------------------------------------------------------------
    Lasso Model optimized using 10 folds cross validation and searching all integers alphas between 1 and 100
    Lasso best alpha: 0.01, R^2 score testing: 0.58499838480617, numero iterazioni: 64
     time taken : 0.16098308563232422s
    --------------------------------------------------------------------------------
    Lasso Model using Lars algorithm optimized using 10 folds cross validation and searching all integers alphas between 1 and 100
    Lasso Lars best alpha: 0.0009600789694834831, R^2 score testing: 0.5858944312103755, numero iterazioni: 14
     time taken : 0.05199694633483887s
    --------------------------------------------------------------------------------
    Lasso Model using AIC criterion optimization
    Lasso Lars using AIC best alpha: 0.0001988059860358497, R^2 score testing: 0.5884444412846026, numero iterazioni : 15
     time taken : 0.007001399993896484s
    --------------------------------------------------------------------------------
    Lasso Model using BIC criterion optimization
    Lasso Lars using BIC best alpha: 0.06987936879637764, R^2 score testing: 0.5047826091816949, numero iterazioni : 15
     time taken : 0.006998538970947266s
    --------------------------------------------------------------------------------
    Elasticnet Model optimized using 10 folds cross validation and searching l1_ratio between 0 and 1
    Elasticnet best alpha: 0.812788907603632, best l1_ratio: 0.99, R^2 score testing: 0.508981141057887, numero iterazioni: 46
     time taken : 14.50524616241455s
    --------------------------------------------------------------------------------

Testando tutti i modelli abbiamo capito che il modello migliore tra quelli visti è **il Lasso, il migliore score nel miglior tempo lo otteniamo usando il criterio AIC**, in genere *se siete decisi tra questi due modelli iniziate con testare il modello Elasticnet, se il termine l1_ratio tende a uno dei due estremi allora crecate i migliori valori per tale modello controllando sempre che il modello abbia **uno score sul testing** più alto del modello iniziale*, __[per maggiori info](https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html?highlight=aic)__.
Qualora incontraste lunge attese nell'esecuzione in alcuni di questi algortimi è presente l'opzione ***n_jobs*** ponete questo valore uguale a -1 per fare in modo che l'algoritmo usi tutti i core del pc per eseguire le operazioni o un numero compreso tra 1 e il numero di core del vostro pc, ma state attenti che userete molto più risorse!

---

AVETE FINITO LA LEZIONE SULLA REGRESSIONE LINEARE E LA SUA REGORALIZZAZIONE, A PRESTO!


```python
## import required packages
import numpy as np
import sympy as sp
import plotly.graph_objects as go
```

```python
# Transformation matrix that defines pirtoation about an axis in 3-space
def pirotation (m):
    m = m/np.linalg.norm(m)
    m1, m2, m3 = m.item(0), m.item(1), m.item(2)
    P = np.matrix([[2*m1**2-1, 2*m1*m2, 2*m1*m3],[2*m2*m1, 2*m2**2-1, 2*m2*m3],[2*m3*m1, 2*m3*m2, 2*m3**2-1]])
    return P
    # P is the matrix that defines pirotation about m

def hklplane(K,Pm,s,n):
  # ax + by + cz + d = 0 is the equation of a plane
  # normal to this plane is (a,b,c)
  normal = K @ np.linalg.inv(Pm)
  a,b,c = normal

  if a != 0:
    # x, y, z
    point = Pm @ [1/a,0,0]
  elif b != 0:
    # x, y, z
    point = Pm @ [0,1/b,0]
  elif c != 0:
    # x, y, z
    point = Pm @ [0,0,1/c]
  # d = - xa - yb - zc
  #d = -np.sum(point*normal)

  xx,yy = np.meshgrid(np.arange(-n,n),np.arange(-n,n))
  # z = (-a*x - a * y - d)/c
  zz = (-normal[0]*xx - normal[1]*yy - s)/normal[2]
  return xx, yy, zz

```

```python
# define 3 symbolic variables
beta, gamma, Gamma = sp.symbols("beta gamma Gamma")
```

```python
## Required variables for all type II twinning mode in m-unit cell framework

# Matrix transformation from m-space to 3-space
Pm = sp.Matrix([[1.,beta*sp.cos(Gamma),0.],[0.,beta*sp.sin(Gamma),0.],[0.,0.,gamma]])
Pm = sp.lambdify((beta,gamma,Gamma),Pm,"numpy")

# Irrational component of twinnning mode for k1 = (011)
q1 = 2*beta*sp.cos(Gamma)/(gamma**2 - beta**2)
q1 = sp.lambdify((beta,gamma,Gamma),q1,"numpy")
r1 = 2*beta*gamma**2 * sp.cos(Gamma) / (gamma**2 - (beta*sp.sin(Gamma))**2)
r1 = sp.lambdify((beta,gamma,Gamma),r1,"numpy")

# Irrational component of twinnning mode for k1 = (101)
q2 = 2*beta*sp.cos(Gamma)/(gamma**2 - 1)
q2 = sp.lambdify((beta,gamma,Gamma),q2,"numpy")
r2 = 2*gamma**2*sp.cos(Gamma)/(beta*(gamma**2 - sp.sin(Gamma)**2))
r2 = sp.lambdify((beta,gamma,Gamma),r2,"numpy")
```

```python
## Lattice parameters of monoclinic unit cell in m-space framework
a, b, c, Beta = 1, 0.85 , 0.7, np.radians(96)

## Numerical output of formal solution
Pm = Pm(b/a,c/a,Beta)
q1, r1 = q1(b/a,c/a,Beta), r1(b/a,c/a,Beta)
q2, r2 = q2(b/a,c/a,Beta), r2(b/a,c/a,Beta)
```

# Type II Twinning Mode

$\mathbf{K}_1 = \mathbf{k}_2 = (q_1 \bar{1} 1)_m$

$\mathbf{K}_2 = \mathbf{k}_1 = (011)_m$

$\mathbf{\eta}_1 = \mathbf{\gamma}_2 = [0\bar{1} \bar{1}]_m$

$\mathbf{\eta}_2 = \mathbf{\gamma}_1 = [\overline{r_1} 1 \bar{1}]_m$

```python
# Define the rotation axis
k1, k2, eta1 = [0,1,1], [q1,-1,1], [0,-1,-1]
# rotation axis for type I
m_typeI = k1 @ np.linalg.inv(Pm)
# rotation axis for type II
m_typeII = Pm @ eta1

# no of unit cells
n = 5
```

```python
## Define parent lattice, twin lattice and 2-fold axis
R = pirotation(m_typeI)
X, Y, Z = np.meshgrid(np.arange(-n,n),np.arange(-n,n),np.arange(-n,n))
M = np.row_stack((X.flatten(),Y.flatten(),Z.flatten()))
parent_lattice = Pm @ M
twin_lattice = np.asarray(np.linalg.inv(R) @ Pm @ M)
```

```python
## Plot the figure
# call
fig = go.Figure()

# parent lattice
fig.add_trace(go.Scatter3d(
    x=parent_lattice[0,:],y=parent_lattice[1,:],z=parent_lattice[2,:],
    mode='markers',name='Matrix',
    marker=dict(size=11, opacity=0.85)
    ))

# twin lattice
fig.add_trace(go.Scatter3d(
    x=twin_lattice[0,:],y=twin_lattice[1,:],z=twin_lattice[2,:],
    mode='markers',name='Twin',
    marker=dict(size=10,opacity=0.5)
    ))

# sets of K1 planes
xx,yy,zz = hklplane(k1,Pm,-2,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))

xx,yy,zz = hklplane(k1,Pm,-1,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))

xx,yy,zz = hklplane(k1,Pm,0,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))

xx,yy,zz = hklplane(k1,Pm,1,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))

xx,yy,zz = hklplane(k1,Pm,2,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))

# figure aspect ratio, range, and no. of ticks
scenes = dict(
    aspectratio=dict(x=1, y=1, z=1),
    xaxis = dict(nticks=1, range=[-6,6],),
    yaxis = dict(nticks=1, range=[-6,6],),
    zaxis = dict(nticks=1, range=[-6,6],)
  )

# camera view
camera = dict(
    eye=dict(x=3,y=0,z=0),
    up = dict(x=0,y=0,z=1)
  )

# title
name = 'Type I Twinning'

# figure layout
fig.update_layout(height=800, width=800,
    scene = scenes,
    scene_camera = camera,
    title = name
  )

fig.show()
```

<meta charset="utf-8" />

```python
## Define parent lattice, twin lattice and 2-fold axis
R = pirotation(m_typeII)
X, Y, Z = np.meshgrid(np.arange(-n,n),np.arange(-n,n),np.arange(-n,n))
M = np.row_stack((X.flatten(),Y.flatten(),Z.flatten()))
parent_lattice = Pm @ M
twin_lattice = np.asarray(np.linalg.inv(R) @ Pm @ M)
```

```python
## Plot the figure
fig = go.Figure()
fig.add_trace(go.Scatter3d(
    x=parent_lattice[0,:],y=parent_lattice[1,:],z=parent_lattice[2,:],
    mode='markers',name='Matrix',
    marker=dict(size=11, opacity=0.9)
    ))
fig.add_trace(go.Scatter3d(
    x=twin_lattice[0,:],y=twin_lattice[1,:],z=twin_lattice[2,:],
    mode='markers',name='Twin',
    marker=dict(size=10,opacity=0.5)
    ))
xx,yy,zz = hklplane(k2,Pm,-2,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))
xx,yy,zz = hklplane(k2,Pm,-1,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))
xx,yy,zz = hklplane(k2,Pm,0,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))
xx,yy,zz = hklplane(k2,Pm,1,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))
xx,yy,zz = hklplane(k2,Pm,2,n)
fig.add_trace(go.Mesh3d(x=xx.flatten(),y=yy.flatten(),z=zz.flatten(),opacity=0.2,color='blue'))
fig.update_layout(scene=dict(
    aspectratio=dict(x=1, y=1, z=1),
    xaxis = dict(nticks=1, range=[-6,6],),
    yaxis = dict(nticks=1, range=[-6,6],),
    zaxis = dict(nticks=1, range=[-6,6],)
    ))
fig.show()
```

<meta charset="utf-8" />

```python

```


```python
%matplotlib inline
import pandas as pd
import numpy as np
```

# Definitions
## Option price
Assuming underlying forward rate follows Normal distribution:
$$
{\rm d} F_t = \mu dt + \sigma dW_t,
$$
The price for a call option can be described as below:
$$
C = \left( F - K \right) N \left( d \right) + \sigma \sqrt{\tau} n \left( d \right), \\
d = \frac{F - K}{\sigma \sqrt{\tau}},
$$
where $N$ is c.d.f of Normal, $n$ is p.d.f of Normal, $F$ is forward rate, $K$ is strike, $\sigma$ is normal volatility
and $\tau$ is time to maturity.

## Greeks
Now we are interested in how greeks behave as underlying forward rate moves. Focal greeks are:
$$
\begin{align}
{\rm Gamma} &:= \frac{{\partial^2 C}}{\partial F^2} = \frac{1}{\sigma \sqrt{\tau}} n\left( d \right)\\
{\rm Theta} &:= \frac{{\partial C}}{\partial \tau} = \frac{\sigma}{2 \sqrt{\tau}} n\left( d \right)\\
{\rm Vega} &:= \frac{{\partial C}}{{\partial \sigma}} = \sqrt{\tau} n\left( d \right)\\
{\rm Vanna} &:= \frac{{\partial^2 C}}{{\partial \sigma}{\partial F}} = -\frac{d}{\sigma} n\left( d \right)\\
{\rm Volga} &:= \frac{{\partial^2 C}}{{\partial \sigma^2}} = \frac{d^2 \sqrt{\tau}}{\sigma}n\left( d \right)\\
{\rm Veta \ (Vega \ decay)} &:= \frac{{\partial^2 C}}{{\partial \sigma}{\partial \tau}} = \frac{1}{2 \sqrt{\tau}}n\left( d \right) \left( 1-d^2 \right)\\
\end{align}
$$

# Greeks vs forward rate
## Parameters

```python
strike = 3
sigma = 0.5
tau = 2
```

## Greeks

```python
data = pd.DataFrame(np.linspace(1, 5, 101), columns=["Forward"]).set_index("Forward")
data.head()
```

      Forward

      1.00

      1.04

      1.08

      1.12

      1.16

```python
data = data.assign(
    d=lambda df: (df.index - strike) / (sigma * tau ** 0.5)
).assign(
    n_d=lambda df: (1/((2 * np.pi) ** 0.5)) * np.exp(-0.5 * df["d"] ** 2)
).assign(
    gamma=lambda df: (1/(sigma * tau ** 0.5)) * df["n_d"],
    theta=lambda df: (sigma/(2 * tau ** 0.5)) * df["n_d"],
    vega=lambda df: (tau ** 0.5) * df["n_d"],
    vanna=lambda df: -1 * df["d"] / sigma * df["n_d"],
    volga=lambda df: ((tau ** 0.5) * df["d"] ** 2)/sigma * df["n_d"],
    veta=lambda df: 1/(2 * tau ** 0.5) * df["n_d"] * (1 - df["d"] ** 2),
)

data.head()
```

      d
      n_d
      gamma
      theta
      vanna
      vega
      veta
      volga

      Forward

      1.00
      -2.828427
      0.007307
      0.010333
      0.001292
      0.041334
      0.010333
      -0.018084
      0.165336

      1.04
      -2.771859
      0.008561
      0.012107
      0.001513
      0.047460
      0.012107
      -0.020229
      0.186042

      1.08
      -2.715290
      0.009998
      0.014140
      0.001767
      0.054297
      0.014140
      -0.022527
      0.208499

      1.12
      -2.658721
      0.011640
      0.016461
      0.002058
      0.061893
      0.016461
      -0.024975
      0.232718

      1.16
      -2.602153
      0.013507
      0.019102
      0.002388
      0.070295
      0.019102
      -0.027560
      0.258686

## Gamma

```python
def plot_greeks(column, title):
    ax = data[column].plot(grid=True, title=title)
    ax.axhline(0, color="black")
```

```python
plot_greeks("gamma", "Gamma")
```

## Theta

```python
plot_greeks("theta", "Theta")
```

## Vega

```python
plot_greeks("vega", "Veta")
```

## Vanna

```python
plot_greeks("vanna", "Vanna")
```

## Volga

```python
plot_greeks("volga", "Volga")
```

## Veta

```python
plot_greeks("veta", "Veta")
```


# Logistic Regression

## Introduction
In linear regression our main interest was centered on learning the
coefficients of a functional fit (say a polynomial) in order to be
able to predict the response of a continuous variable on some unseen
data. The fit to the continuous variable $y_i$ is based on some
independent variables $\hat{x}_i$. Linear regression resulted in
analytical expressions for standard ordinary Least Squares or Ridge
regression (in terms of matrices to invert) for several quantities,
ranging from the variance and thereby the confidence intervals of the
parameters $\hat{\beta}$ to the mean squared error. If we can invert
the product of the design matrices, linear regression gives then a
simple recipe for fitting our data.

Classification problems, however, are concerned with outcomes taking
the form of discrete variables (i.e. categories). We may for example,
on the basis of DNA sequencing for a number of patients, like to find
out which mutations are important for a certain disease; or based on
scans of various patients' brains, figure out if there is a tumor or
not; or given a specific physical system, we'd like to identify its
state, say whether it is an ordered or disordered system (typical
situation in solid state physics); or classify the status of a
patient, whether she/he has a stroke or not and many other similar
situations.

The most common situation we encounter when we apply logistic
regression is that of two possible outcomes, normally denoted as a
binary outcome, true or false, positive or negative, success or
failure etc.

Logistic regression will also serve as our stepping stone towards
neural network algorithms and supervised deep learning. For logistic
learning, the minimization of the cost function leads to a non-linear
equation in the parameters $\hat{\beta}$. The optimization of the
problem calls therefore for minimization algorithms. This forms the
bottle neck of all machine learning algorithms, namely how to find
reliable minima of a multi-variable function. This leads us to the
family of gradient descent methods. The latter are the working horses
of basically all modern machine learning algorithms.

We note also that many of the topics discussed here on logistic
regression are also commonly used in modern supervised Deep Learning
models, as we will see later.

## Basics

We consider the case where the dependent variables, also called the
responses or the outcomes, $y_i$ are discrete and only take values
from $k=0,\dots,K-1$ (i.e. $K$ classes).

The goal is to predict the
output classes from the design matrix $\hat{X}\in\mathbb{R}^{n\times p}$
made of $n$ samples, each of which carries $p$ features or predictors. The
primary goal is to identify the classes to which new unseen samples
belong.

Let us specialize to the case of two classes only, with outputs
$y_i=0$ and $y_i=1$. Our outcomes could represent the status of a
credit card user that could default or not on her/his credit card
debt. That is

$$
y_i = \begin{bmatrix} 0 & \mathrm{no}\\  1 & \mathrm{yes} \end{bmatrix}.
$$

Before moving to the logistic model, let us try to use our linear
regression model to classify these two outcomes. We could for example
fit a linear model to the default case if $y_i > 0.5$ and the no
default case $y_i \leq 0.5$.

We would then have our
weighted linear combination, namely

$$
\begin{equation}
\hat{y} = \hat{X}^T\hat{\beta} +  \hat{\epsilon},
\label{_auto1} \tag{1}
\end{equation}
$$

where $\hat{y}$ is a vector representing the possible outcomes, $\hat{X}$ is our
$n\times p$ design matrix and $\hat{\beta}$ represents our estimators/predictors.

The main problem with our function is that it takes values on the
entire real axis. In the case of logistic regression, however, the
labels $y_i$ are discrete variables. A typical example is the credit
card data discussed below here, where we can set the state of
defaulting the debt to $y_i=1$ and not to $y_i=0$ for one the persons
in the data set (see the full example below).

One simple way to get a discrete output is to have sign
functions that map the output of a linear regressor to values $\{0,1\}$,
$f(s_i)=sign(s_i)=1$ if $s_i\ge 0$ and 0 if otherwise.
We will encounter this model in our first demonstration of neural networks. Historically it is called the "perceptron" model in the machine learning
literature. This model is extremely simple. However, in many cases it is more
favorable to use a ``soft" classifier that outputs
the probability of a given category. This leads us to the logistic function.

## The logistic function

The perceptron is an example of a ``hard classification" model. We
will encounter this model when we discuss neural networks as
well. Each datapoint is deterministically assigned to a category (i.e
$y_i=0$ or $y_i=1$). In many cases, it is favorable to have a "soft"
classifier that outputs the probability of a given category rather
than a single value. For example, given $x_i$, the classifier
outputs the probability of being in a category $k$.  Logistic regression
is the most common example of a so-called soft classifier. In logistic
regression, the probability that a data point $x_i$
belongs to a category $y_i=\{0,1\}$ is given by the so-called logit function (or Sigmoid) which is meant to represent the likelihood for a given event,

$$
p(t) = \frac{1}{1+\mathrm \exp{-t}}=\frac{\exp{t}}{1+\mathrm \exp{t}}.
$$

Note that $1-p(t)= p(-t)$.

The following code plots the logistic function, the step function and other functions we will encounter from here and on.

```python
%matplotlib inline

"""The sigmoid function (or the logistic curve) is a
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm."""

import numpy
import matplotlib.pyplot as plt
import math as mt

z = numpy.arange(-5, 5, .1)
sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
sigma = sigma_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, sigma)
ax.set_ylim([-0.1, 1.1])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('sigmoid function')

plt.show()

"""Step Function"""
z = numpy.arange(-5, 5, .02)
step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
step = step_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, step)
ax.set_ylim([-0.5, 1.5])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('step function')

plt.show()

"""tanh Function"""
z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
t = numpy.tanh(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, t)
ax.set_ylim([-1.0, 1.0])
ax.set_xlim([-2*mt.pi,2*mt.pi])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('tanh function')

plt.show()
```

## Two parameters

We assume now that we have two classes with $y_i$ either $0$ or $1$. Furthermore we assume also that we have only two parameters $\beta$ in our fitting of the Sigmoid function, that is we define probabilities

$$
\begin{align*}
p(y_i=1|x_i,\hat{\beta}) &= \frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}},\nonumber\\
p(y_i=0|x_i,\hat{\beta}) &= 1 - p(y_i=1|x_i,\hat{\beta}),
\end{align*}
$$

where $\hat{\beta}$ are the weights we wish to extract from data, in our case $\beta_0$ and $\beta_1$.

Note that we used

$$
p(y_i=0\vert x_i, \hat{\beta}) = 1-p(y_i=1\vert x_i, \hat{\beta}).
$$

## Maximum likelihood

In order to define the total likelihood for all possible outcomes from a
dataset $\mathcal{D}=\{(y_i,x_i)\}$, with the binary labels
$y_i\in\{0,1\}$ and where the data points are drawn independently, we use the so-called [Maximum Likelihood Estimation](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation) (MLE) principle.
We aim thus at maximizing
the probability of seeing the observed data. We can then approximate the
likelihood in terms of the product of the individual probabilities of a specific outcome $y_i$, that is

$$
\begin{align*}
P(\mathcal{D}|\hat{\beta})& = \prod_{i=1}^n \left[p(y_i=1|x_i,\hat{\beta})\right]^{y_i}\left[1-p(y_i=1|x_i,\hat{\beta}))\right]^{1-y_i}\nonumber \\
\end{align*}
$$

from which we obtain the log-likelihood and our **cost/loss** function

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n \left( y_i\log{p(y_i=1|x_i,\hat{\beta})} + (1-y_i)\log\left[1-p(y_i=1|x_i,\hat{\beta}))\right]\right).
$$

Reordering the logarithms, we can rewrite the **cost/loss** function as

$$
\mathcal{C}(\hat{\beta}) = \sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\beta$.
Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that

$$
\mathcal{C}(\hat{\beta})=-\sum_{i=1}^n  \left(y_i(\beta_0+\beta_1x_i) -\log{(1+\exp{(\beta_0+\beta_1x_i)})}\right).
$$

This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression,
in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression.

The cross entropy is a convex function of the weights $\hat{\beta}$ and,
therefore, any local minimizer is a global minimizer.

Minimizing this
cost function with respect to the two parameters $\beta_0$ and $\beta_1$ we obtain

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_0} = -\sum_{i=1}^n  \left(y_i -\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right),
$$

and

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \beta_1} = -\sum_{i=1}^n  \left(y_ix_i -x_i\frac{\exp{(\beta_0+\beta_1x_i)}}{1+\exp{(\beta_0+\beta_1x_i)}}\right).
$$

Let us now define a vector $\hat{y}$ with $n$ elements $y_i$, an
$n\times p$ matrix $\hat{X}$ which contains the $x_i$ values and a
vector $\hat{p}$ of fitted probabilities $p(y_i\vert x_i,\hat{\beta})$. We can rewrite in a more compact form the first
derivative of cost function as

$$
\frac{\partial \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}} = -\hat{X}^T\left(\hat{y}-\hat{p}\right).
$$

If we in addition define a diagonal matrix $\hat{W}$ with elements
$p(y_i\vert x_i,\hat{\beta})(1-p(y_i\vert x_i,\hat{\beta})$, we can obtain a compact expression of the second derivative as

$$
\frac{\partial^2 \mathcal{C}(\hat{\beta})}{\partial \hat{\beta}\partial \hat{\beta}^T} = \hat{X}^T\hat{W}\hat{X}.
$$

Within a binary classification problem, we can easily expand our model to include multiple predictors. Our ratio between likelihoods is then with $p$ predictors

$$
\log{ \frac{p(\hat{\beta}\hat{x})}{1-p(\hat{\beta}\hat{x})}} = \beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p.
$$

Here we defined $\hat{x}=[1,x_1,x_2,\dots,x_p]$ and $\hat{\beta}=[\beta_0, \beta_1, \dots, \beta_p]$ leading to

$$
p(\hat{\beta}\hat{x})=\frac{ \exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}{1+\exp{(\beta_0+\beta_1x_1+\beta_2x_2+\dots+\beta_px_p)}}.
$$

## Including more classes

Till now we have mainly focused on two classes, the so-called binary
system. Suppose we wish to extend to $K$ classes.  Let us for the sake
of simplicity assume we have only two predictors. We have then
following model

1
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\log{\frac{p(C=2\vert x)}{p(K\vert x)}} = \beta_{20}+\beta_{21}x_1,
$$

and so on till the class $C=K-1$ class

$$
\log{\frac{p(C=K-1\vert x)}{p(K\vert x)}} = \beta_{(K-1)0}+\beta_{(K-1)1}x_1,
$$

and the model is specified in term of $K-1$ so-called log-odds or
**logit** transformations.

In our discussion of neural networks we will encounter the above again
in terms of a slightly modified function, the so-called **Softmax** function.

The softmax function is used in various multiclass classification
methods, such as multinomial logistic regression (also known as
softmax regression), multiclass linear discriminant analysis, naive
Bayes classifiers, and artificial neural networks.  Specifically, in
multinomial logistic regression and linear discriminant analysis, the
input to the function is the result of $K$ distinct linear functions,
and the predicted probability for the $k$-th class given a sample
vector $\hat{x}$ and a weighting vector $\hat{\beta}$ is (with two
predictors):

$$
p(C=k\vert \mathbf {x} )=\frac{\exp{(\beta_{k0}+\beta_{k1}x_1)}}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}}.
$$

It is easy to extend to more predictors. The final class is

$$
p(C=K\vert \mathbf {x} )=\frac{1}{1+\sum_{l=1}^{K-1}\exp{(\beta_{l0}+\beta_{l1}x_1)}},
$$

and they sum to one. Our earlier discussions were all specialized to
the case with two classes only. It is easy to see from the above that
what we derived earlier is compatible with these equations.

To find the optimal parameters we would typically use a gradient
descent method.  Newton's method and gradient descent methods are
discussed in the material on [optimization
methods](https://compphysics.github.io/MachineLearning/doc/pub/Splines/html/Splines-bs.html).


```python
import sympy as sp
sp.init_printing(use_latex = True)
%matplotlib inline
```

```python
M_s, x, w, y, h, a, beta, t, nu = sp.symbols('M_s, x, w, y, h, a, beta, t, nu')
```

```python
H_x = M_s/(4*sp.pi) * (sp.log(((x+w)**2 + (y-h)**2)/((x+w)**2 + (y+h)**2)) - sp.log(((x-w)**2 + (y-h)**2)/((x-w)**2 + (y+h)**2)))
H_x
```

```python
H_y = M_s/(2*sp.pi) * (sp.atan((2*h*(x+w))/((x+w)**2 + y**2 - h**2)) - sp.atan((2*h*(x-w))/((x-w)**2 + y**2 - h**2)))
H_y
```

```python
H = sp.sqrt(H_x**2 + H_y**2)
H
```

```python
diffx = sp.diff(H, x)
diffx
```

```python
diffy = sp.diff(H, y)
diffy
```

```python
g = sp.simplify(((a*x)/(1-beta*t)) * sp.sqrt(a/(nu*(1-beta*t))) * sp.sqrt(a/(nu*(1-beta*t))))
g
```

```python
gg = ((a*x)/(1-beta*t)) * sp.sqrt(a/(nu*(1-beta*t)))
gg**2
```

```python
eta = y * sp.sqrt(a/(nu*(1 - beta*t)))
eta
```

```python
ddt = sp.diff(eta, t)
ddt
```

```python
sp.simplify((a/(nu*(1-beta*t))) * sp.sqrt(a/(nu*(1-beta*t))))
```

```python
bb = sp.sqrt((a*nu)/(1-beta*t)) * sp.sqrt(a/(nu*(1-beta*t)))
sp.simplify(bb)
```

```python
sp.simplify(diffx)
```

```python
print(sp.simplify(diffx))
```

    M_s**2*(-4*h*((4*h**2*(w - x)**2 + (-h**2 + y**2 + (w - x)**2)**2)*(-h**2 + y**2 - (w + x)**2) + (4*h**2*(w + x)**2 + (-h**2 + y**2 + (w + x)**2)**2)*(h**2 - y**2 + (w - x)**2))*((-h + y)**2 + (-w + x)**2)*((-h + y)**2 + (w + x)**2)*((h + y)**2 + (-w + x)**2)*((h + y)**2 + (w + x)**2)*(atan(2*h*(-w + x)/(-h**2 + y**2 + (-w + x)**2)) - atan(2*h*(w + x)/(-h**2 + y**2 + (w + x)**2))) + (4*h**2*(w - x)**2 + (-h**2 + y**2 + (w - x)**2)**2)*(4*h**2*(w + x)**2 + (-h**2 + y**2 + (w + x)**2)**2)*(-(w + x)*(-(-h + y)**2 + (h + y)**2)*((-h + y)**2 + (-w + x)**2)*((h + y)**2 + (-w + x)**2) + ((-w + x)*((h + y)**2 + (-w + x)**2) + (w - x)*((-h + y)**2 + (-w + x)**2))*((-h + y)**2 + (w + x)**2)*((h + y)**2 + (w + x)**2))*(log(((-h + y)**2 + (-w + x)**2)/((h + y)**2 + (-w + x)**2)) - log(((-h + y)**2 + (w + x)**2)/((h + y)**2 + (w + x)**2))))/(2*pi*sqrt(M_s**2*((log(((h - y)**2 + (w - x)**2)/((h + y)**2 + (w - x)**2)) - log(((h - y)**2 + (w + x)**2)/((h + y)**2 + (w + x)**2)))**2 + 4*(atan(2*h*(w - x)/(-h**2 + y**2 + (w - x)**2)) + atan(2*h*(w + x)/(-h**2 + y**2 + (w + x)**2)))**2))*(4*h**2*(w - x)**2 + (-h**2 + y**2 + (w - x)**2)**2)*(4*h**2*(w + x)**2 + (-h**2 + y**2 + (w + x)**2)**2)*((-h + y)**2 + (-w + x)**2)*((-h + y)**2 + (w + x)**2)*((h + y)**2 + (-w + x)**2)*((h + y)**2 + (w + x)**2))


# Neural Networks for Data Science Applications
## Mid-term Homework: Implementing a custom activation function

**Name**: *Michele Meo*

**Matricola**: *1599032*

Send the completed notebook before 26/11/2020 back to **simone.scardapane@uniroma1.it** with the object "[NNDS] Homework_1_\<id\>".

```python
import tensorflow as tf
print(tf.__version__)
```

    2.2.0

### Introduction

The **exponential linear unit** (ELU) is an activation function defined as [1]:

$$
\phi(x) =
\Biggl\{
\begin{align}
x & \;\; \text{ if } x \ge 0 \\
\alpha \left(\exp\left(x\right)- 1\right) & \;\; \text{ otherwise }
\end{align}
\Bigr.
\,,
$$

where $\alpha$ is a hyper-parameter. The function is implemented in `tf.keras.layers.ELU` (see the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/ELU)).

The **parametric ELU** (PELU) extends the ELU activation function as [2]:

$$
\phi(x) =
\Biggl\{
\begin{align}
\frac{\alpha}{\beta}x & \;\; \text{ if } x \ge 0 \\
\alpha \left(\exp\Bigl(\frac{x}{\beta}\Bigr)- 1\right) & \;\; \text{ otherwise }
\end{align}
\Bigr.
\,,
$$

where the major difference is that $\alpha,\beta > 0$ are *trainable* parameters, i.e., a pair of $(\alpha, \beta)$ values is trained for each unit in the network.

### Exercise 1: implement the PELU

In TensorFlow, it is possible to implement new layers by subclassing `tf.keras.layers.Layer`:

+ [Making new Layers and Models via subclassing](https://www.tensorflow.org/guide/keras/custom_layers_and_models)
+ [Custom layers](https://www.tensorflow.org/tutorials/customization/custom_layers)
+ [tf.keras.layers.Layer (documentation)](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer)

**Exercise 1**: *After carefully reading the guides*, complete the following implementation of the PELU activation function.

```python
class PELU(tf.keras.layers.Layer):
    def __init__(self, units=32):
        super(PELU, self).__init__()
        self.units = units

    def build(self, input_shape):

        self.alpha = self.add_weight( shape = (self.units,),
                                    initializer = tf.keras.initializers.RandomUniform(0.1, 1.),
                                    constraint = tf.keras.constraints.NonNeg(),  # avoid negative values during training phase
                                    trainable = True)

        self.beta = self.add_weight( shape = (self.units,),
                                    initializer = tf.keras.initializers.RandomUniform(0.1, 1.),
                                    constraint = tf.keras.constraints.NonNeg(),
                                    trainable = True)

    def call(self, inputs):
        # In order to avoid overflow errors I had to add a 0.1 constant term to each denominator
        # in which the beta parameter appears, I even tried with smaller values like 1e-6 but it
        # didn't work

        self.a = tf.cast((inputs >= 0), dtype=tf.float32) * ((self.alpha/(self.beta+0.1))*inputs) + \
                    tf.cast((inputs < 0), dtype=tf.float32) * (self.alpha*(tf.exp(inputs/(self.beta+0.1))-1))

        return self.a
```

**Hints for a correct implementation**:

1. The layer (probably) requires two sets of trainable variables, whose shape depends on the number of units.
2. From the definition of the PELU, $\alpha, \beta$ are required to be positive in order to ensure differentiability. The simplest way to handle this is to use a [constraint callable](https://www.tensorflow.org/api_docs/python/tf/keras/constraints) when creating the weight (see also the [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer) for `add_weight`).

### Exercise 2: some preliminary tests

To evaluate your implementation, let us start by creating a single PELU function:

```python
pelu = PELU(units=1)
```

**Exercise 2.1**: plot the function using the skeleton code below.

```python
import matplotlib.pyplot as plt
x_range = tf.linspace(-5., 5., 200) # An equispaced grid of 200 points in [-5, +5]
y_range = pelu(x_range)

plt.plot(x_range.numpy(), y_range.numpy())  # check the pelu shape
```

The derivative of a PELU function with respect to the $\alpha$ parameter is given by [2]:

$$
\frac{d\phi(x)}{d\alpha} =
\Biggl\{
\begin{align}
\frac{x}{\beta} & \;\; \text{ if } x \ge 0 \\
 \left(\exp\Bigl(\frac{x}{\beta}\Bigr)- 1\right) & \;\; \text{ otherwise }
\end{align}
\Bigr.
\,,
$$

**Exercise 2.2**: using a `tf.GradientTape` object, compute the derivative above using automatic differentiation, and check its correctness up to a certain numerical precision.

**Hints for a correct implementation**:

1. `tf.GradientTape` allows to compute the derivative *at a single point x*. If you prefer to avoid a loop over all possible points, consider using the `jacobian` function to obtain them in a single pass ([Advanced Automatic Differentiation](https://www.tensorflow.org/guide/advanced_autodiff)).
2. Given two tensors x and y, a simple way to compute elementwise similarity up to a certain precision (e.g., $10^{-4}$), is given by `tf.reduce_all(tf.abs(x - y) < 1e-4)`.

**Exercise 2.3 (optional)**: try the same for the $\beta$ parameter (you can check the analytical formula for the gradient in the original paper [2]). **Careful**: the equation in the original paper has a missing $h$ (thanks to Davide Aureli and Federico Siciliano for spotting this). See [the correct derivation](https://www.wolframalpha.com/input/?i=d%28a*%28exp%28h%2Fb%29-1%29%29%2Fdb) on Wolfram Alpha.

```python
# Ex. 2.2: Verifying the gradient w.r.t. alpha obtained from automatic differentiation

# Note: the 0.1 const. term addition in the pelu expression must be considered
#       also in the evaluation of derivatives not implemented with the automatic differentiation

with tf.GradientTape() as t:
    y_range = pelu(x_range)

y_automatic_grad_alpha = t.jacobian(y_range, pelu.alpha)

y_grad_alpha = tf.where(x_range >= 0,
                          x = x_range/(pelu.beta+0.1),
                          y = tf.exp(x_range/(pelu.beta+0.1))-1)
y_grad_alpha = tf.reshape(y_grad_alpha, (-1, 1))

tf.reduce_all(tf.abs(y_grad_alpha - y_automatic_grad_alpha) < 1e-4)  # check automatic gradient with the known expression
```

    <tf.Tensor: shape=(), dtype=bool, numpy=True>

```python
# Ex. 2.3: Verifying the gradient w.r.t. beta obtained from automatic differentiation

with tf.GradientTape() as t:
    y_range = pelu(x_range)

y_automatic_grad_beta = t.jacobian(y_range, pelu.beta)

y_grad_beta = tf.where(x_range >= 0,
                        x = -(pelu.alpha*x_range)/(0.1 + pelu.beta)**2,
                        y = -(pelu.alpha*x_range*tf.exp(x_range/(0.1 + pelu.beta)))/(0.1 + pelu.beta)**2)
y_grad_beta = tf.reshape(y_grad_beta, (-1, 1))

tf.reduce_all(tf.abs(y_grad_beta - y_automatic_grad_beta) < 1e-4)
```

    <tf.Tensor: shape=(), dtype=bool, numpy=True>

### Exercise 3: PELU in practice

Consider a simple model built with the PELU activation function, as below.

```python
model = tf.keras.Sequential(layers=[
      tf.keras.layers.Dense(50),
      PELU(50),
      tf.keras.layers.Dense(11, activation='softmax')
])
```

**Exercise 3**: load any classification dataset, and train the model above (using either a custom training loop or `model.fit(...)`). Additionally, compare with a standard ReLU activation.

```python
# I decided to use the dataset from second laboratory lecture to implement the training loop

import pandas as pd

sensorless = pd.read_csv('/home/michele/Projects/DataScience/NN/Lab3/Sensorless_drive_diagnosis.txt',
                         header=None, sep=' ')

X = sensorless.values[:, 0:-1].astype('float32')
y = sensorless.values[:, -1:].astype('int64') - 1
```

```python
from sklearn import model_selection
from sklearn.preprocessing import MinMaxScaler  #  I normalize each data feature in the [0,1] range

X_tr, X_tst, y_tr, y_tst = model_selection.train_test_split(X, y, stratify=y)

scaler = MinMaxScaler()
scaler.fit(X_tr)

X_tr = scaler.transform(X_tr)
X_tst = scaler.transform(X_tst)
```

```python
train_dataset = tf.data.Dataset.from_tensor_slices((X_tr, y_tr)).batch(32)
test_dataset = tf.data.Dataset.from_tensor_slices((X_tst, y_tst)).batch(32)
```

```python
cross_entropy = tf.keras.losses.SparseCategoricalCrossentropy()
adam = tf.keras.optimizers.Adam(learning_rate=1e-3)  # best performance obtained with Adam optimizer
acc = tf.keras.metrics.SparseCategoricalAccuracy()

model.compile(optimizer=adam, loss=cross_entropy, metrics=[acc])
```

```python
history = model.fit(train_dataset, epochs=15, validation_data=test_dataset)
```

    Epoch 1/15
    1372/1372 [==============================] - 4s 3ms/step - loss: 1.0972 - sparse_categorical_accuracy: 0.6368 - val_loss: 0.5818 - val_sparse_categorical_accuracy: 0.8078
    Epoch 2/15
    1372/1372 [==============================] - 3s 2ms/step - loss: 0.5224 - sparse_categorical_accuracy: 0.8173 - val_loss: 0.4535 - val_sparse_categorical_accuracy: 0.8372
    Epoch 3/15
    1372/1372 [==============================] - 3s 2ms/step - loss: 0.4370 - sparse_categorical_accuracy: 0.8372 - val_loss: 0.4022 - val_sparse_categorical_accuracy: 0.8428
    Epoch 4/15
    1372/1372 [==============================] - 3s 2ms/step - loss: 0.3894 - sparse_categorical_accuracy: 0.8490 - val_loss: 0.3664 - val_sparse_categorical_accuracy: 0.8532
    Epoch 5/15
    1372/1372 [==============================] - 5s 3ms/step - loss: 0.3567 - sparse_categorical_accuracy: 0.8608 - val_loss: 0.3362 - val_sparse_categorical_accuracy: 0.8648
    Epoch 6/15
    1372/1372 [==============================] - 4s 3ms/step - loss: 0.3346 - sparse_categorical_accuracy: 0.8686 - val_loss: 0.3164 - val_sparse_categorical_accuracy: 0.8711
    Epoch 7/15
    1372/1372 [==============================] - 3s 3ms/step - loss: 0.3185 - sparse_categorical_accuracy: 0.8751 - val_loss: 0.3018 - val_sparse_categorical_accuracy: 0.8759
    Epoch 8/15
    1372/1372 [==============================] - 4s 3ms/step - loss: 0.3051 - sparse_categorical_accuracy: 0.8810 - val_loss: 0.2899 - val_sparse_categorical_accuracy: 0.8792
    Epoch 9/15
    1372/1372 [==============================] - 5s 3ms/step - loss: 0.2937 - sparse_categorical_accuracy: 0.8847 - val_loss: 0.2792 - val_sparse_categorical_accuracy: 0.8833
    Epoch 10/15
    1372/1372 [==============================] - 4s 3ms/step - loss: 0.2835 - sparse_categorical_accuracy: 0.8884 - val_loss: 0.2713 - val_sparse_categorical_accuracy: 0.8863
    Epoch 11/15
    1372/1372 [==============================] - 4s 3ms/step - loss: 0.2738 - sparse_categorical_accuracy: 0.8921 - val_loss: 0.2608 - val_sparse_categorical_accuracy: 0.8908
    Epoch 12/15
    1372/1372 [==============================] - 4s 3ms/step - loss: 0.2646 - sparse_categorical_accuracy: 0.8962 - val_loss: 0.2517 - val_sparse_categorical_accuracy: 0.8948
    Epoch 13/15
    1372/1372 [==============================] - 3s 2ms/step - loss: 0.2559 - sparse_categorical_accuracy: 0.8998 - val_loss: 0.2425 - val_sparse_categorical_accuracy: 0.8993
    Epoch 14/15
    1372/1372 [==============================] - 3s 2ms/step - loss: 0.2467 - sparse_categorical_accuracy: 0.9043 - val_loss: 0.2327 - val_sparse_categorical_accuracy: 0.9056
    Epoch 15/15
    1372/1372 [==============================] - 3s 2ms/step - loss: 0.2362 - sparse_categorical_accuracy: 0.9091 - val_loss: 0.2242 - val_sparse_categorical_accuracy: 0.9115

```python
model = tf.keras.Sequential(layers=[
                           tf.keras.layers.Dense(50, activation='relu'),
                           tf.keras.layers.Dense(11, activation='softmax')])

model.compile(optimizer=adam, loss=cross_entropy, metrics=[acc])
```

```python
history_relu = model.fit(train_dataset, epochs=15, validation_data=test_dataset)
```

    Epoch 1/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 1.4010 - sparse_categorical_accuracy: 0.6436 - val_loss: 1.0221 - val_sparse_categorical_accuracy: 0.7244
    Epoch 2/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.8777 - sparse_categorical_accuracy: 0.7525 - val_loss: 0.7513 - val_sparse_categorical_accuracy: 0.8063
    Epoch 3/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.6881 - sparse_categorical_accuracy: 0.8027 - val_loss: 0.6202 - val_sparse_categorical_accuracy: 0.8207
    Epoch 4/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.5905 - sparse_categorical_accuracy: 0.8191 - val_loss: 0.5469 - val_sparse_categorical_accuracy: 0.8314
    Epoch 5/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.5329 - sparse_categorical_accuracy: 0.8289 - val_loss: 0.5014 - val_sparse_categorical_accuracy: 0.8377
    Epoch 6/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.4948 - sparse_categorical_accuracy: 0.8341 - val_loss: 0.4684 - val_sparse_categorical_accuracy: 0.8424
    Epoch 7/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.4658 - sparse_categorical_accuracy: 0.8404 - val_loss: 0.4431 - val_sparse_categorical_accuracy: 0.8452
    Epoch 8/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.4428 - sparse_categorical_accuracy: 0.8444 - val_loss: 0.4231 - val_sparse_categorical_accuracy: 0.8473
    Epoch 9/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.4243 - sparse_categorical_accuracy: 0.8477 - val_loss: 0.4063 - val_sparse_categorical_accuracy: 0.8508
    Epoch 10/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.4086 - sparse_categorical_accuracy: 0.8516 - val_loss: 0.3917 - val_sparse_categorical_accuracy: 0.8540
    Epoch 11/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.3948 - sparse_categorical_accuracy: 0.8555 - val_loss: 0.3787 - val_sparse_categorical_accuracy: 0.8591
    Epoch 12/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.3825 - sparse_categorical_accuracy: 0.8598 - val_loss: 0.3671 - val_sparse_categorical_accuracy: 0.8651
    Epoch 13/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.3714 - sparse_categorical_accuracy: 0.8642 - val_loss: 0.3569 - val_sparse_categorical_accuracy: 0.8690
    Epoch 14/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.3618 - sparse_categorical_accuracy: 0.8672 - val_loss: 0.3484 - val_sparse_categorical_accuracy: 0.8720
    Epoch 15/15
    1372/1372 [==============================] - 2s 1ms/step - loss: 0.3536 - sparse_categorical_accuracy: 0.8696 - val_loss: 0.3410 - val_sparse_categorical_accuracy: 0.8749

I show two plots in which the accuracy value for each epoch is represented: in the first plot a comparison between the accuracy obtained on the train set and on the validation set, in the second plot a comparison between the accuracy obtained on validation set with PELU and ReLU.

```python
plt.plot(range(1, 16), history.history['sparse_categorical_accuracy'], label='train acc')
plt.plot(range(1, 16), history.history['val_sparse_categorical_accuracy'], label='validation acc', color='orange')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.title('Accuracy on Train and Validation (PELU)')
plt.legend()
plt.show()
```

```python
plt.plot(range(1, 16), history.history['val_sparse_categorical_accuracy'], label='val acc PELU', color='orange')
plt.plot(range(1, 16), history_relu.history['val_sparse_categorical_accuracy'], label='val acc ReLU', color='green')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.title('PELU vs ReLU')
plt.legend()
plt.show()
```

### Optional: understanding saving/loading of models

TensorFlow has several options for saving or loading objects from the disk:

1. [Save and load Keras models](https://www.tensorflow.org/guide/keras/save_and_serialize/)

In many cases, custom classes require the implementation of a `get_config` / `from_config` functions to define the serialization behaviour.

**Exercise 4 (optional)**: implement the `get_config` method and test your implementation as below (taken from the guide on saving and loading models).

```python
model.save('pelu_model')
del PELU # This is needed to remove any reference to PELU from memory
reloaded_model = tf.keras.models.load_model('pelu_model')
```

```python
print("Original model:", model)
print("Loaded model:", reloaded_model) # Observe that the object has been dynamically recreated in absence of the configuration options
```

### References

[1] Clevert, D.A., Unterthiner, T. and Hochreiter, S., 2015. [Fast and accurate deep network learning by exponential linear units (ELUs)](https://arxiv.org/abs/1511.07289). arXiv preprint arXiv:1511.07289.

[2] Trottier, L., Gigu, P. and Chaib-draa, B., 2017. [Parametric exponential linear unit for deep convolutional neural networks](https://arxiv.org/abs/1605.09332). In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 207-214). IEEE.


#  Neutrino-Driven Wind Transsonic Velocity Solver

By Brian Nevins

Image from:  Newsweek

---
# Authors
Brian Nevins

Dr. Luke Roberts, Michigan State University

---
# Abstract

When a massive star, between 10 and ~29 times the mass of our sun, dies, most of its outer layers are blown out to space in a massive explosion called a supernova. The inner core, about 1.4 solar masses, forms a neutron star - essentially a giant, super-dense atomic nucleus with a radius of roughly 10km. In a short period after the supernova explosion, some of the outer material from the dead star falls back onto the surface of the neutron star. This material is then blown back into space by the huge numbers of neutrinos being released from the newly formed neutron star, as what is known as the neutrino-driven wind.

The neutrino-driven wind can be well approximated as a steady-state, time independent system, with a constant mass loss rate. The hydrodynamic equations governing the wind are:
\begin{equation}
\dot{M}=4\pi r^2 v \rho
\end{equation}

\begin{equation}
v \frac{\partial v}{\partial r}=-\frac{1}{\rho}\frac{\partial P}{\partial r}-\frac{G M}{r^2}
\end{equation}

\begin{equation}
v \frac{\partial s}{\partial r}=\frac{S_\epsilon}{n_B T}
\end{equation}

and the equation of state for the wind, which gives the specific pressure ($p$), speed of sound ($c_s$), and specific entropy ($s$). For this code we use a simple gamma law equation of state, where
\begin{equation}
p=\rho T
\end{equation}

\begin{equation}
c_s^2=\gamma T
\end{equation}

\begin{equation}
s=\frac{1}{\gamma-1}\ln(T \rho^{1-\gamma})
\end{equation}

The governing equations can then be rewritten in terms of a timelike variable $\psi$, with respect to which we can computationally integrate using a Runge-Kutta method. The governing equations, in terms of dimensionless variables $x=\ln(\frac{r}{r_0}), u=\ln(\frac{v}{c_s}), w=\ln(\frac{T}{T_0})$, become:
\begin{equation}
\frac{dx}{d\psi}=f_1
\end{equation}

\begin{equation}
\frac{du}{d\psi}=f_2
\end{equation}

\begin{equation}
\frac{dw}{d\psi}=-(\gamma-1)(2f_1+f_2)
\end{equation}
where $f_1=1-e^{2u-w},$ $f_2=a e^{-w-x}-2$, and $a=\frac{GM}{c_s^2 r_0}$. Numerically, $a\approx.1$ from an order-of-magnitude calculation. We integrate this with an adaptive RK4 method, specifying starting values of $r=r_0$ and $T=T_0$. Our independent variable is the starting value of $v=v_0$, which we vary in order to find a realistic solution curve when we plot $v$ vs $r$. We can then use $v_0$ to find the mass loss rate $\dot{M}$.

----
# Statement of Need

The neutrino-driven wind is a site for interesting nucleosynthesis reactions. The nuclei that can be formed depend on a number of parameters, especially the mass loss rate. This code will be useful in determining mass loss rates for different equations of state, and can be adapted to incorporate other parameters such as secondary heating to determine their effect on nucleosynthesis.

----
# Installation instructions

This code can be run as-is, if the git repository is properly cloned. The Adiabatic_wind_solver.py file, which contains the main code, is self-contained and can be moved as needed. Dependencies are listed below, and can be installed via pip:

    numpy
    matplotlib.pyplot
    os
    contextlib
    Ipython.display

----
# Tests

Several testing methods are included in the test_Adiabatic_wind_solver.py file. These tests check several of the methods in the main python file. The most important tests are the last two, which check the RK method and the bisection search for the critical velocity for simple cases and compare the results to known values. If these do not succeed, something has been changed that significantly alters the critical velocity for multiple gamma values. These tests can be run by entering the pytest command in the main folder of the project.

```python
!pytest
```

    ============================= test session starts =============================
    platform win32 -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
    rootdir: D:\Documents\Brian\Important\MSU\Neutrino-driven-winds-research\neutrino-winds
    plugins: arraydiff-0.3, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.2
    collected 4 items

    tests\test_Adiabatic_wind_solver.py ....                                 [100%]

    ========================== 4 passed in 66.44 seconds ==========================

---
# Example Usage
This code is designed to approximate the critical transsonic velocity of the neutrino-driven wind from the surface of a proto-neutron star, where the wind follows a gamma law equation of state. This is accomplished by evolving a set of ODEs until a sign change occurs in one of two characteristic functions of the system $f_1$ and $f_2$. The critical velocity is found by adjusting the starting velocity of the wind so that the two functions cross zero at the same time using a bisection method.

The difference in the zeros of these functions is easy to see in the isothermal case $(\gamma = 1)$. The curves that approach zero at large r represent "breeze" solutions, where material is lifted off the surface of the star but does not have sufficient energy to escape, and falls back to the surface. The curves that bend upward are nonphysical. The behavior of the curves is determined by which function crosses zero first.

```python
%matplotlib inline

import Adiabatic_wind_solver as aws
s=aws.solver(1,10)
s.makePlots(.001,.01,.0005,False,20,5);
```

There exists a solution between these two curve sets for which the two functions cross zero at the same time. We use a bisection method to determine the bounds on the critical velocity for this to take place, as shown below.

```python
v0avg=s.findV0(.001,.006,.0001)
s.makePlot(v0avg)
```

This solution does not return to zero, and represents a true wind solution where the material blown off continues out into space.

We can also determine the critical velocity for an isentropic wind that follows an ideal gas equation of state $(\gamma=5/3)$. It is harder to see the different solution sets, but the bisection method is just as effective for finding the critical velocity.

```python
s1=aws.solver(5/3,.1)
v0=s1.findV0(.0001,.009,.0001)
s1.makePlot(v0,xrange=50)
```

We can also see how the critical velocity depends on the value of $\gamma$, for a general gamma law equation of state. The gammaSearch function simply iterates through given range of $\gamma$ values, finds the critical velocity, and plots those velocities. The system seems to destabilize just below $\gamma=1.5$, as the critical velocity drops dramatically to zero.

```python
#This function takes a long time to evaluate
g=s1.gammaSearch(a=.1,g0=10,dg=-.01,glim=1,lower=.01,upper=.9,itermax=100)
```

---
# Methodology

My original proposal outlined several goals surrounding the integration and visualization of the ODEs, including a minimization algorithm suggested by my advisor. All of these goals were met, and the minimization algorithm (the search method for the critical velocity) was implemented. Useful visuals were generated, as I suggested would be beneficial. These visuals were not generated in real-time, as I initially thought, but the primary code is fast enough that this did not seem necessary. The overarching goal of the project was to integrate the ODEs and find the critical velocity, and that was accomplished. I did not deviate significantly from my project proposal. I did go a step further, by investigating how the critical velocity depended on $\gamma$ for a gamma law equation of state, which fits into the broader goal of seeing how solutions to the ODEs depend on the various physics involved.

---
# Concluding Remarks

I successfully found critical velocities for winds that follow a gamma law equation of state, of which the isothermal case is a member with $\gamma=1$. All of my stated project goals were accomplished. Future work will involve generalizing this code for any equation of state, to more accurately model real neutron stars. I would like to make inputting this equation of state as intuitive as possible, which may involve using symbolic python or something similar. It will also be helpful to numerically convert the critical velocity I found to the mass loss rate it corresponds to. Although finding the critical velocity is fairly fast in python, my investigation of the relationship between $\gamma$ and the critical velocity shows that increasing efficiency should also be a goal for future work. This will likely involve converting my code into C, and then writing a python wrapper for easier use.

----
# References

Lamers, Henny J. G. L. M., and Joseph P. Cassinelli. Introduction to Stellar Winds. Cambridge University Press, 1999.

Roberts, Luke. Notes on the Basic Physics of the Neutrino Driven Wind. 18 June 2017.

Arcones, A., and F. K. Thielemann. “Neutrino-Driven Wind Simulations and Nucleosynthesis of Heavy Elements.” Journal of Physics G: Nuclear and Particle Physics, vol. 40, no. 1, Jan. 2013, p. 013201. arXiv.org, doi:10.1088/0954-3899/40/1/013201.

Code is stored in a repository on Github.

```python

```