*Sebastian Raschka*
last modified: 03/31/2014

I am really looking forward to your comments and suggestions to improve and extend this tutorial! Just send me a quick note
via Twitter: [@rasbt](https://twitter.com/rasbt)
or Email: [bluewoodtree@gmail.com](mailto:bluewoodtree@gmail.com)

### Problem Category
- Statistical Pattern Recognition
- Supervised Learning
- Parametric Learning
- Bayes Decision Theory
- Univariate data
- 2-class problem
- different variances
- equal priors
- Cauchy model (2 parameters)
- With conditional Risk (1-0 loss functions)

# Sections

• Given information

• Deriving the decision boundary

• Plotting the posterior probabilities and decision boundary

• Classifying some random example data

• Calculating the empirical error rate

## Given information:

[back to top]

####model: continuous univariate Cauchy distribution for the class-conditional densities

$p(x | \omega_j) = \frac{1}{\pi b} \; \dot \; \frac{1}{1 \; + \; \bigg(\frac{x - a_i}{b} \bigg)^2} \quad i = 1, 2$

####Prior probabilities:

$P(\omega_1) = P(\omega_2) = 0.5$

####Loss functions:

where

$\lambda(\alpha_i|\omega_j) = \lambda_{ij}$,

the loss occured if $action_i$ is taken if the actual true class is $ \omega_j$ (assuming that $action_i$ classifies sample as $\omega_i$)

$\begin{equation}
\lambda = \begin{pmatrix}
\lambda_{11} \quad \lambda_{12} \\
\lambda_{21} \quad \lambda_{22}
\end{pmatrix}
= \begin{pmatrix}
0 \quad 1 \\
1 \quad 0 \\
\end{pmatrix}\\
\end{equation}$

#### Parameters

$a_1 = 0, \quad a_2 = 2 ,\quad b=0.5$

## Deriving the decision boundary

[back to top]

### Bayes' Rule:

$P(\omega_j|x) = \frac{p(x|\omega_j) * P(\omega_j)}{p(x)}$

### Risk Functions:

$R(\alpha_1|x) = \lambda_{11}P(\omega_1|x) + \lambda_{12}P(\omega_2|x)$

$R(\alpha_2|x) = \lambda_{21}P(\omega_1|x) + \lambda_{22}P(\omega_2|x)$

###Decision Rule:

Decide $\omega_1 $ if $ R(\alpha_2|x) > R(\alpha_1|x) $ else decide $ \omega_2$.

$\Rightarrow \lambda_{21}P(\omega_1|x) + \lambda_{22}P(\omega_2|x) > \lambda_{11}P(\omega_1|x) + \lambda_{12}P(\omega_2|x)$

$\Rightarrow (\lambda_{21} - \lambda_{11}) \; P(\omega_1|x) > (\lambda_{12} - \lambda_{22}) \; P(\omega_2|x)$

$\Rightarrow \frac{P(\omega_1|x)}{P(\omega_2|x)} > \frac{(\lambda_{12} - \lambda_{22})}{(\lambda_{21} - \lambda_{11})} $

$\Rightarrow \frac{p(x|\omega_1) \; P{(\omega_1)}}{p(x|\omega_2) \; P{(\omega_2)}} > \frac{(\lambda_{12} - \lambda_{22})}{(\lambda_{21} - \lambda_{11})}$

$\Rightarrow \frac{p(x|\omega_1)}{p(x|\omega_2)} > \frac{(\lambda_{12} - \lambda_{22}) \; P{(\omega_2)}}{(\lambda_{21} - \lambda_{11}) \; P{(\omega_1)}}$

$\Rightarrow \frac{p(x|\omega_1)}{P(x|\omega_2)} > \frac{(1 - 0) \; (0.5)}{(1 - 0) \; (0.5)}$

$\Rightarrow \frac{p(x|\omega_1)}{P(x|\omega_2)} > 1$

$\Rightarrow \frac{1}{\pi b} \; \dot \; \frac{1}{1 \; + \; \bigg(\frac{x - a_1}{b} \bigg)^2} > \frac{1}{\pi b} \; \dot \; \frac{1}{1 \; + \; \bigg(\frac{x - a_2}{b} \bigg)^2}$

$\Rightarrow  \frac{1}{1 \; + \; \bigg(\frac{x - a_1}{b} \bigg)^2} > \frac{1}{1 \; + \; \bigg(\frac{x - a_2}{b} \bigg)^2}$

$\Rightarrow  \bigg(\frac{x - a_1}{b} \bigg)^2 > \bigg(\frac{x - a_2}{b} \bigg)^2$

$\Rightarrow x-a_1 > x-a_2$

$\Rightarrow x < \frac{a_1+a_2}{2}$

Decide $\omega_1 $ if $ x < \frac{a_1+a_2}{2}$ else decide $\omega_2$.

## Plotting the class posterior probabilities and decision boundary

[back to top]

```
%pylab inline

import numpy as np
from matplotlib import pyplot as plt

def cauchy(x, a, b):
    """
    Calculates the Cauchy distribution's probability density
    function.

    """
    term1 = 1 / (np.pi * b)
    term2 = 1 / (1 + ((x - a)/b)**2)
    return term1 * term2

def decision_boundary(a1, a2):
    return (a1 + a2) / 2

def posterior(likelihood, prior):
    """
    Calculates the posterior probability (after Bayes Rule) without
    the scale factor p(x) (=evidence).

    """
    return likelihood * prior

# generating some sample data for plotting the posterior probabilities
x = np.arange(-10, 10, 0.05)

# probability density functions
posterior1 = posterior(cauchy(x, a=0, b=0.5), prior=0.5)
posterior2 = posterior(cauchy(x, a=2, b=0.5), prior=0.5)

# plotting probability density functions
plt.plot(x, posterior1)
plt.plot(x, posterior2)
plt.title('Posterior Probabilities w. Decision Boundary for a Cauchy Distribution')
plt.ylabel('P(w)')
plt.xlabel('random variable x')
plt.legend(['P(w_1|x)', 'P(w_2|x)'], loc='upper right')
plt.ylim([0,0.4])
plt.xlim([-4,6])

# plotting the decision boundary
bound = decision_boundary(a1=0, a2=2)
plt.vlines(bound, -10, 10, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(0.0, 0.35), xytext=(0.0, 0.35))
plt.annotate('R2', xy=(2.0, 0.35), xytext=(2.0, 0.35))
plt.show()
```

## Classifying some random example data

[back to top]

```
import math
import random as rnd

def cauchy(location, scale):
    """
    Generates random cauchy distributed data
    reference: http://www.johndcook.com/python_cauchy_rng.html

    """
    p = 0.0
    while p == 0.0:
        p = rnd.random()

    return location + scale*math.tan(math.pi*(p - 0.5))

# Parameters
a_1 = 0
a_2 = 2
b = 0.5

# Generating 20 Cauchy distributed random samples for class 1 & 2
x1_samples = [cauchy(a_1, b) for i in range(20)]
x2_samples = [cauchy(a_2, b) for i in range(20)]
y = [0.1 for i in range(20)]

plt.scatter(x1_samples, y, marker='o', color='green', s=40, alpha=0.5)
plt.scatter(x2_samples, y, marker='^', color='blue', s=40, alpha=0.5)
plt.ylim([0,0.4])
plt.xlim([-4,6])

# Plotting the decision boundary
bound = decision_boundary(a1=0, a2=2)
plt.vlines(bound, -10, 10, color='r', alpha=0.8, linestyle=':', linewidth=2)
plt.annotate('R1', xy=(0.0, 0.2), xytext=(0.0, 0.2))
plt.annotate('R2', xy=(2.0, 0.2), xytext=(2.0, 0.2))

# Plot annotation
plt.title('Classifying random example data from 2 classes')
plt.ylabel('')
plt.xlabel('random variable x')
plt.legend(['w_1', 'w_2'], loc='upper right')

plt.show()
```

## Calculating the empirical error rate

[back to top]

```
w1_as_w2, w2_as_w1 = 0, 0
for x1,x2 in zip(x1_samples, x2_samples):
    if x1 > decision_boundary(a1=0, a2=2):
        w1_as_w2 += 1
    if x2 <= decision_boundary(a1=0, a2=2):
        w2_as_w1 += 1

emp_err =  (w1_as_w2 + w2_as_w1) / float(len(x1_samples) + len(x2_samples))

print('Empirical Error: {}%'.format(emp_err * 100))
```

    Empirical Error: 10.0%

## Calculating the Bayes Error

$P(error) = \int_{x=-\infty}^{\infty} P(error/x) \; \cdot \; p(x) \; dx$

$P(error|x) = \left\{
\begin{array}{l l}
P(\omega_1 | x) & \quad \text{if $x > \frac{a_1 + a_2}{2}$}\\
P(\omega_2 | x) & \quad \text{if $x < \frac{a_1 + a_2}{2}$}\end{array} \right.$

$\Rightarrow \int_{x=-\infty}^{\frac{a_1+a_2}{2}} P(\omega_2|x) \; \cdot \; p(x) \; dx  + \int_{x=\frac{a_1+a_2}{2}}^{\infty} P(\omega_1|x) \; \cdot \; p(x) \; dx$

$\Rightarrow \int_{x=-\infty}^{\frac{a_1+a_2}{2}} p(x|\omega_2) \; \cdot \; P(\omega_2) \; dx  + \int_{x=\frac{a_1+a_2}{2}}^{\infty} p(x|\omega_1) \; \cdot \; P(\omega_1) \; dx$

$\Rightarrow \int_{x=-\infty}^{\frac{a_1+a_2}{2}} \frac{1}{\pi b} \; \dot \; \frac{1}{1 \; + \; \bigg(\frac{x - a_2}{b} \bigg)^2} \; \frac{1}{2} \; dx  + \int_{x=\frac{a_1+a_2}{2}}^{\infty} \frac{1}{\pi b} \; \dot \; \frac{1}{1 \; + \; \bigg(\frac{x - a_1}{b} \bigg)^2} \; \frac{1}{2} \; dx$

$\Rightarrow \frac{1}{2} - \frac{1}{\pi} \; tan^{-1} \bigg| \frac{a_2 - a_1}{2b} \bigg|$

```
def bayes_error(a1, a2, b):
    """ Calculates the Bayes Error for 2 Cauchy densities ."""
    term1 = 1/2.0 - 1/np.pi
    term2 = np.arctan(np.fabs((a2-a1)/(2*b)))
    return term1 * term2

print('Bayes Error: {:.2f}%'.format(100* bayes_error(0, 2, 0.5)))

```

    Bayes Error: 20.12%

```

```


# Learning Math with Jupyter Notebooks

---

### Jan Reimann, Department of Mathematics, Penn State University Park

### May 13, 2021

## What is a Jupyter notebook?

Jupyter notebooks are interactive notebooks that combine

- content input/output via **Markdown**

- interactive computations using a REPL kernel (IPython, Sage, Julia, R, ...)

**REPL:** **R**ead–**E**val–**P**rint **L**oop

```python
6*8
```

    48

- A notebook is a **dynamic document** consisting of **cells**.

- A cell can be either **Markdown** (for text, images, etc) or **code**.

- A user can **add/delete cells** at any time.

## What can I do with a Jupyter notebook?

- Easily input and display math/latex:

> Let $H_n = \sum\limits_{j=1}^n \frac{1}{j}$ be the $n$-th
harmonic number.
>
> __Hypothesis:__
For each $n \ge 1$, it holds that
$$\sum\limits_{d|n} d \le H_n + \exp (H_n) \log (H_n),$$
with equality only for $n = 1.$

- Include links:

> Some helpful resources for Jupyter:
>
>    - [The Jupyter Project](https://jupyter.org)
>
>    - [Teaching and Learning with Jupyter](https://jupyter4edu.github.io/jupyter-edu-book/)
>
>    - [Markdown Guide](https://www.markdownguide.org)
>
>    - [IPython documentation](https://ipython.readthedocs.io/en/stable/)

- Include graphics:

- Embed videos:

```python
from IPython.display import IFrame
IFrame("https://www.youtube.com/embed/fs4ImYH0GRg?rel=0&controls=0&showinfo=0", '80%', '600px')
```

- Run code (Python, Sage, Julia, R, many more...)

```python
from sympy import diff, sympify, Symbol
x = Symbol('x')

f = sympify('x^3+2/x')

diff(f)
```

$\displaystyle 3 x^{2} - \frac{2}{x^{2}}$

- **Prepare slides!**

    [Creating Presentations With Jupyter Notebook](https://rise.readthedocs.io/en/stable/index.html)

- Prepare handouts (pdf, ...)
- Write books
- ...

## Possibilities / Advantages

- Open source

- Authoring and delivery in the **same** platform

- Notebook format text-based, easily portable (`Pandoc`, `nbconvert`)

- Interactivity

- Engagement: students can add cells to
    - perform additional computations
    - add content or notes

    __It is "their" document.__

## Possibilities / Advantages

- Ubiquitous platform, becoming de-facto standard in the data science world.

- Plenty of tools to compute and graph functions

(`Sympy`, `Numpy`, `matplotlib`, ...)

- Better data-oriented activities

(`pandas`, `seaborn`, `scikit-learn`, ...)

[
Sample Notebook: Logistic functions (Calculus)
](https://mybinder.org/v2/gh/28left/psumathnotebooks/master?filepath=logistic.ipynb)

---

## Challenge

Combine these tremendous possibilities into one learning tool that is

    user friendly

    accessible

    does not require programming knowledge

    can be easily deployed

## Phase 1

[
Tutorial Notebook
](https://mybinder.org/v2/gh/28left/psumathnotebooks/master?filepath=110_0_jupyter_tutorial_pure.ipynb)

---

## Phase 2

## The Team
---
### TLT
Bradley Antill, Matt Hansen, Brad Kozlek, Jenay Roberts, JooYoung Seo, Philip Voorhees

### Math Department
Nawaf Alansari, Tigran Eritsyan, Raymond Friend

### PSU Cloud Services
Paul Crum

Questions? Interested?

- Contact me at <jan.reimann@psu.edu>

- I can give a more thorough walk-through and add you to our Binder server.

- Pilot in Math 110 WC, Fall 2021


# Fundamentals of Data Science
Winter Semester 2021

## Prof. Fabio Galasso, Guido D'Amely, Alessandro Flaborea, Luca Franco, Muhammad Rameez Ur Rahman and Alessio Sampieri
<galasso@di.uniroma1.it>, <damely@di.uniroma1.it>, <flaborea@di.uniroma1.it>, <franco@diag.uniroma1.it>, <rahman@di.uniroma1.it>, <alessiosampieri27@gmail.com>

## Exercise 2: Classification

In Exercise 2, you will re-derive and implement logistic regression and optimize the parameters with Gradient Descent and with the Newton's method. Also, in this exercise you will re-derive and implement Gassian Discriminant Analysis.
We will use datasets generated from the make_classification function from the SkLearn library. Its first output contains the feature values $x^{(i)}_1$ and $x^{(i)}_2$ for the $i$-th data sample $x^{(i)}$. The second contains the ground truth label $y^{(i)}$ for each corresponding data sample.

The completed exercise should be handed in as a single notebook file. Use Markdown to provide equations. Use the code sections to provide your scripts and the corresponding plots.
Submit it by sending an email to galasso@di.uniroma1.it, flaborea@di.uniroma1.it, franco@diag.uniroma1.it and alessiosampieri27@gmail.com  by Wednesday November 17th 2021, 23:59.

## Notation

- $x^i$ is the $i^{th}$ feature vector
- $y^i$ is the expected outcome for the $i^{th}$ training example
- $m$ is the number of training examples
- $n$ is the number of features

Let's start by setting up our Python environment and importing the required libraries:

```python
%matplotlib inline
import numpy as np # imports a fast numerical programming library
import scipy as sp # imports stats functions, amongst other things
import matplotlib as mpl # this actually imports matplotlib
import matplotlib.cm as cm # allows us easy access to colormaps
import matplotlib.pyplot as plt # sets up plotting under plt
import pandas as pd # lets us handle data as dataframes
from sklearn.datasets import make_classification
import seaborn as sns

# sets up pandas table display
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)

import seaborn as sns # sets up styles and gives us more plotting options
```

## Question 1: Logistic Regression with Gradient Ascent **(10 Points)**

### Code and Theory

#### Exercise 1.a **(3 Points)** Equations for the log likelihood, its gradient, and the gradient ascent update rule.

Write and simplify the likelihood $L(\theta)$ and log-likelihood $l(\theta)$ of the parameters $\theta$.

Recall the probabilistic interpretation of the hypothesis $h_\theta(x)= P(y=1|x;\theta)$ and that $h_\theta(x)=\frac{1}{1+\exp(-\theta^T x)}$.

Also derive the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ and write the gradient update equation.

Question: Are we looking for a local minimum or a local maximum using the gradient ascent rule?

################# Do not write above this line #################

- $P(Y=1|x; \theta) = h_{\theta}(x)$

- $P(Y=0|x; \theta) = 1 - h_{\theta}(x)$

Given these two points, we can say that in general we have:

$P(y|x;\theta)=h_{\theta}(x)^{y} (1+h_{\theta}(x))^{1-y}$

We must observe that for $y=1$, we get only $h_{\theta}(x)^{y}$, instead for $y=0$, we just get $(1+h_{\theta}(x))^{1-y}$.

The Likelihood $L(\theta)$ is given by:

$$L(\theta) = P(\vec{y}|x; \theta)  \\
 = \prod_{i=1}^{m} P(y^{(i)}|x^{(i)};\theta)  \\
 = \prod_{i=1}^{m} h_{\theta}(x^{(i)})^{y^{(i)}}(1-h_{\theta}(x^{(i)}))^{1-y^{(i)}}  \\
 = \prod_{i=1}^{m} \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)^{y^{(i)}}\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)^{1-y^{(i)}}
$$

Instead, in order to find the **Log-likelihood $l(\theta)$**, we must apply the natural logarithm function to the likelihood function previously found, and we get:
$$
l(\theta) = \log L(\theta)
 = \sum_{i=1}^{m} y^{(i)}\log h_{\theta}(x^{(i)})+(1-y^{(i)})\log(1-h_{\theta}(x^{(i)}))  \\
 = \sum_{i=1}^{m} y^{(i)}\log \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg) + (1-y^{(i)})\log\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)
$$

We proceed deriving the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ as follows:
$$
\frac{\delta l(\theta)}{\delta \theta_j} = \frac{\delta}{\delta \theta_j} \Bigg(\sum_{i=1}^{m} y^{(i)}\log \bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg) + (1-y^{(i)})\log\bigg(1-\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)\Bigg)\\
= \Bigg[y^{(i)}-\bigg(\frac{1}{1+\exp(-\theta^T x^{(i)})}\bigg)\Bigg]x^{(i)}
$$

The procedure of the _gradient ascent rule_ leads us to a local maximum of our (differentiable) function.

################# Do not write below this line #################

#### Exercise 1.b **(7 Points)** Implementation of logistic regression with Gradient Ascent

Code up the equations above to learn the logistic regression parameters. The dataset used here is created using the make_classification function present in the SkLearn library. $x^{(i)}_1$ and $x^{(i)}_2$ represent the two features for the $i$-th data sample $x^{(i)}$ and $y^{(i)}$ is its ground truth label.

```python
X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

Adding a column of 1's to $X$ to take into account the zero intercept

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
```

```python
[x[:5,:],x[-5:,:]] # Plot the first and last 5 lines of x, now containing features x0 (constant=1), x1 and x2
```

    [array([[ 1.        ,  2.25698215, -1.34710915],
            [ 1.        ,  1.43699308,  1.28420453],
            [ 1.        ,  0.57927295,  0.23690172],
            [ 1.        ,  0.42538132, -0.24611145],
            [ 1.        ,  1.13485101, -0.61162683]]),
     array([[ 1.        ,  1.56638944,  0.81749944],
            [ 1.        , -1.94913831, -1.90601147],
            [ 1.        ,  1.53440506, -0.11687238],
            [ 1.        , -0.39243599,  1.39209018],
            [ 1.        , -0.11881249,  0.96973739]])]

```python
[y[:5],y[-5:]] # Plot the first and last 5 lines of y
```

    [array([1, 1, 1, 0, 1]), array([1, 0, 0, 0, 1])]

Define the sigmoid function "sigmoid", the function to compute the gradient of the log likelihood  "grad_l" and the gradient ascent algorithm.

################# Do not write above this line #################

```python
def sigmoid(x):
    '''
    Function to compute the sigmoid of a given input x.

    Input:
    x: it's the input data matrix. The shape is (N, H)

    Output:
    g: The sigmoid of the input x
    '''

    #####################################################

    #####################################################

    return  1 / (1 + np.exp(-x))

def log_likelihood(theta,features,target):
    '''
    Function to compute the log likehood of theta according to data x and label y

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)
    target: the label array

    Output:
    log_g: the log likehood of theta according to data x and label y
    '''

    t = features.dot(theta)
    log_l= np.sum(target * np.log(sigmoid(t)) + (1 - target) * np.log(1 - sigmoid(t))) / features.shape[0]

    return log_l

def predictions(features, theta):
    '''
    Function to compute the predictions for the input features

    Input:
    theta: it's the model parameter matrix.
    features: it's the input data matrix. The shape is (N, H)

    Output:
    preds: the predictions of the input features
    '''

    preds = sigmoid(features.dot(theta))

    return preds

def update_theta(theta, target, preds, features, lr):
    '''
    Function to compute the gradient of the log likelihood
    and then return the updated weights

    Input:
    theta: the model parameter matrix.
    target: the label array
    preds: the predictions of the input features
    features: it's the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the updated model parameter matrix.
    '''
    prediction = predictions(features, theta)
    theta = theta + ((lr/features.shape[0]) * (features.T).dot(target-prediction))

    return theta

def gradient_ascent(theta, features, target, lr, num_steps):
    '''
    Function to execute the gradient ascent algorithm

    Input:
    theta: the model parameter matrix.
    target: the label array
    num_steps: the number of iterations
    features: the input data matrix. The shape is (N, H)
    lr: the learning rate

    Output:
    theta: the final model parameter matrix.
    log_likelihood_history: the values of the log likelihood during the process
    '''

    log_likelihood_history = np.zeros(num_steps)

    m = len(target)

    for it in range(num_steps):

        prediction = np.dot(features,theta)

        theta = update_theta(theta, target, prediction, features, lr)

        log_likelihood_history[it]  = log_likelihood(theta,features,target)

    return theta, log_likelihood_history

```

################# Do not write below this line #################

Check your grad_l implementation:
grad_l applied to the theta_test (defined below) should provide a value for log_l_test close to the target_value (defined below); in other words the error_test should be 0, up to machine error precision.

```python
target_value = -1.630501731599431

output_test  = log_likelihood(np.array([-7,4,1]),x,y)
error_test=np.abs(output_test-target_value)

print("{:f}".format(error_test))
```

    0.000000

Let's now apply the function gradient_ascent and print the final theta as well as theta_history

```python
# Initialize theta0
theta0 = np.zeros(x.shape[1])

# Run Gradient Ascent method
n_iter=1000
theta_final, log_l_history = gradient_ascent(theta0,x,y,lr=0.5,num_steps=n_iter)
print(theta_final)
```

    [-0.46097042  2.90036399  0.23146846]

Let's plot the log likelihood over iterations

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)),log_l_history,'b.')
```

 boundary:

```python
# Generate vector to plot decision boundary
x1_vec = np.linspace(X[:,0].min(),X[:,1].max(),2)

# Plot raw data
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X)

# Plot decision boundary
plt.plot(x1_vec,(-x1_vec*theta_final[1]-theta_final[0])/theta_final[2], color="red")
plt.ylim(X[:,1].min()-1,X[:,1].max()+1)
# Save the theta_final value for later comparisons
theta_GA = theta_final.copy()
```

################# Do not write above this line #################

Discuss these two points:
1. You have implemented the gradient ascent rule. Could we have also used gradient descent instead for the proposed problem? Why/Why not?

- Since we know that we can derive the cost function from the log-likelihood simply multiplying it by $-1$:
$J(\theta)=-\frac{1}{m}L(\theta)$ ,
we could have turned a maximization problem into a minimization one, then using the gradient descend.

2. Let's deeply analyze how the learning rate $\alpha$ and the number of iterations affect the final results. Run the algorithm you have written for different values of $\alpha$ and the number of iterations and look at the outputs you get. Is the decision boundary influenced by these parameters change? Why do you think these parameters are affecting/not affecting the results?

- We analyzed how the learning rate $\alpha$ and the number of iterations affect the final results:
1) if we don't change lr's value but we consider a lower number of iterations, theta_final doesn't change considerably;
2) the same thing happens if we take the same number of iterations but we increase the learning rate $\alpha$ or we increase both the learning rate and the number of iterations;
3) a little difference in our results appears if we decrease at the same time both the learning rate and the number of iterations, but still not so considerably;
4) we notice an abnormal behaviour if we take a large value of $\alpha$ (with the same number of iterations).

- We can justify our results because generally, a large value of the learning rate makes the learning faster, at the cost of a less optimal result, instead a smaller learning rate may allow the model to learn a more optimal or even globally optimal result but may take significantly longer time to execute.

```python
#EXAMPLE with a very big value of alpha
# Run Gradient Ascent method
n_iter=1500
theta_final, log_l_history = gradient_ascent(theta0,x,y,lr=300,num_steps=n_iter)
print(theta_final)
#stesso lr ma n_iter aumentato restituisce stessi risultati
```

    <ipython-input-8-78a00349af4b>:33: RuntimeWarning: divide by zero encountered in log
      log_l= np.sum(target * np.log(sigmoid(t)) + (1 - target) * np.log(1 - sigmoid(t))) / features.shape[0]
    <ipython-input-8-78a00349af4b>:33: RuntimeWarning: invalid value encountered in multiply
      log_l= np.sum(target * np.log(sigmoid(t)) + (1 - target) * np.log(1 - sigmoid(t))) / features.shape[0]
    <ipython-input-8-78a00349af4b>:17: RuntimeWarning: overflow encountered in exp
      return  1 / (1 + np.exp(-x))

    [-45.43579893 158.26593918  23.81026057]

################# Do not write below this line #################

## Question 2: Logistic Regression with non linear boundaries (7 points)

#### Exercise 2.a **(4 Points)** Polynomial features for logistic regression

Define new features, e.g. of 2nd and 3rd degrees, and learn a logistic regression classifier by using the new features, by using the gradient ascent optimization algorithm you defined in Question 1.

In particular, we would consider a polynomial boundary with equation:

$f(x_1, x_2) = c_0 + c_1 x_1 + c_2 x_2 + c_3 x_1^2 + c_4 x_2^2 + c_5 x_1 x_2 + c_6 x_1^3 + c_7 x_2^3 + c_8 x_1^2 x_2 + c_9 x_1 x_2^2$

We would therefore compute 7 new features: 3 new ones for the quadratic terms and 4 new ones for the cubic terms.

Create new arrays by stacking x and the new 7 features (in the order x1x1, x2x2, x1x2, x1x1x1, x2x2x2, x1x1x2, x1x2x2). In particular create x_new_quad by additionally stacking with x the quadratic features, and x_new_cubic by additionally stacking with x the quadratic and the cubic features.

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=500, n_features=2, n_informative=2, n_redundant=0, n_classes=2, random_state=5)
X.shape, y.shape
```

    ((500, 2), (500,))

```python
x = np.hstack([np.ones((X.shape[0], 1)), X])
x,X
```

    (array([[ 1.        ,  2.25698215, -1.34710915],
            [ 1.        ,  1.43699308,  1.28420453],
            [ 1.        ,  0.57927295,  0.23690172],
            ...,
            [ 1.        ,  1.53440506, -0.11687238],
            [ 1.        , -0.39243599,  1.39209018],
            [ 1.        , -0.11881249,  0.96973739]]),
     array([[ 2.25698215e+00, -1.34710915e+00],
            [ 1.43699308e+00,  1.28420453e+00],
            [ 5.79272954e-01,  2.36901723e-01],
            [ 4.25381322e-01, -2.46111450e-01],
            [ 1.13485101e+00, -6.11626834e-01],
            [-1.09782664e+00,  7.04230042e-01],
            [ 1.84526958e+00, -1.34246886e+00],
            [ 1.36824415e+00,  1.24281523e+00],
            [-1.23489899e+00,  3.15214148e-01],
            [ 9.54474387e-01,  7.10230238e-01],
            [-1.46938265e+00,  1.00057176e+00],
            [ 3.40800429e-01, -4.90453030e-01],
            [ 9.84443926e-01,  6.16869610e-01],
            [-6.09726633e-01,  5.24250777e-01],
            [ 1.39133026e+00, -1.92101724e+00],
            [ 1.08992742e+00, -1.95115805e+00],
            [-5.63519400e-01,  1.73139452e-01],
            [ 5.37671698e-01, -1.05068253e+00],
            [ 3.03659865e-01,  1.12107863e+00],
            [-2.10308813e-01,  5.67656233e-01],
            [ 1.29354367e+00, -8.16332244e-01],
            [ 6.05762154e-01, -1.51525184e+00],
            [-1.11623634e+00, -1.04828377e+00],
            [-5.75286728e-01, -7.54688035e-01],
            [ 9.95695462e-01,  6.42419350e-01],
            [ 1.41405784e+00, -1.49961632e+00],
            [ 1.70550275e-01, -1.10286016e+00],
            [ 1.86445118e+00, -2.04310500e+00],
            [-1.03856732e+00,  1.17052999e+00],
            [ 6.14031868e-01,  1.28423281e+00],
            [ 6.83489809e-01,  8.16482325e-01],
            [-8.41876834e-01,  8.77353710e-01],
            [ 1.09475375e+00,  9.48227868e-01],
            [-1.99683223e+00, -1.45114488e+00],
            [ 1.11171557e+00,  8.07482808e-02],
            [ 1.40365431e+00,  7.63661484e-01],
            [ 1.39661898e+00, -2.17886730e+00],
            [-1.34937829e+00,  8.64699375e-01],
            [ 1.18847714e+00, -5.22316372e-01],
            [ 9.36695968e-02,  8.62311690e-01],
            [ 1.04749771e+00,  6.78648212e-01],
            [ 1.01154939e+00,  9.36482001e-01],
            [-8.42373719e-01,  9.12937336e-01],
            [-8.74214777e-01, -4.76148013e-01],
            [-2.05347216e-01, -3.79020736e-01],
            [ 1.19734459e+00, -1.93244611e+00],
            [-2.39637077e+00, -1.30203467e+00],
            [ 8.62671904e-01,  1.76967232e-01],
            [ 1.52752663e+00, -3.48306621e+00],
            [-8.18724327e-01, -1.03430333e+00],
            [ 2.51982893e-01, -4.08938430e-01],
            [ 1.07465437e+00,  1.22534269e+00],
            [-1.25189413e+00,  7.50534087e-01],
            [-1.75773938e+00, -1.15003095e+00],
            [ 8.76823127e-01,  1.26942747e+00],
            [-2.17214723e+00, -1.46649214e+00],
            [ 1.55322570e+00, -2.37783611e+00],
            [-1.37811924e+00, -1.87130036e+00],
            [-1.65246802e+00, -7.04743781e-01],
            [ 1.46148614e+00,  7.85319401e-01],
            [-1.12995011e+00, -1.37891040e+00],
            [-2.06258777e+00, -1.15659192e+00],
            [ 9.47942535e-01,  8.21811422e-01],
            [ 9.33555294e-01,  7.90706921e-01],
            [ 5.06261018e-01, -6.59472190e-02],
            [-5.51058991e-01, -5.31980063e-01],
            [ 1.21820383e-01,  2.06344558e+00],
            [ 9.80884649e-01, -7.09318982e-01],
            [ 2.32708573e+00, -2.15388563e+00],
            [ 1.26091368e+00,  2.66643731e-02],
            [-1.69893114e+00, -1.37102636e+00],
            [-2.91235878e+00, -1.51067117e+00],
            [ 1.29643728e+00,  1.01721260e+00],
            [-1.78318905e-01,  1.50843058e+00],
            [ 1.08382132e+00,  9.10831858e-01],
            [ 2.88741335e-01,  7.12099901e-01],
            [-6.61271022e-01, -5.59858374e-01],
            [ 5.51795491e-02, -5.79408164e-01],
            [ 5.38425172e-03, -1.15849616e+00],
            [-1.51855231e+00, -1.81824339e+00],
            [ 9.20433214e-01,  6.95349561e-01],
            [ 1.36723486e+00,  9.93021263e-01],
            [ 6.27677006e-01, -4.92517711e-01],
            [-1.32977261e+00, -1.17567774e+00],
            [ 4.12178124e-01,  7.50921075e-01],
            [ 6.05136131e-01, -3.07482550e-02],
            [-2.41069195e+00, -1.65512413e+00],
            [ 1.90505900e+00, -9.69768438e-01],
            [ 8.63071205e-01,  8.53547389e-01],
            [ 3.89277554e-01,  9.88080327e-01],
            [ 1.85216876e+00,  1.27355649e+00],
            [ 1.14181726e+00, -6.21856050e-01],
            [-2.44526936e+00, -5.28050809e-01],
            [ 9.11348794e-01, -6.57022623e-01],
            [-2.44171334e+00, -5.04454010e-01],
            [ 1.52494564e+00,  6.73102993e-01],
            [-6.53726920e-01,  1.40311941e+00],
            [-3.03575720e-01,  1.66076550e+00],
            [-9.73795774e-01, -7.57334533e-01],
            [-6.87049772e-01, -3.40503846e-01],
            [-1.79376989e+00,  1.83249040e-01],
            [ 1.46081470e+00,  1.13888176e+00],
            [ 3.08729586e-01, -1.08722144e+00],
            [ 1.45361953e+00,  1.26820691e+00],
            [-6.65369415e-01,  1.63560886e+00],
            [-2.03608082e+00, -1.62701872e+00],
            [-2.46582691e-01,  1.55627908e+00],
            [ 1.10649419e+00,  1.40039468e+00],
            [ 7.85416877e-01, -8.19246217e-01],
            [-8.64964823e-01,  1.02574941e+00],
            [ 1.08818523e+00,  1.11525468e+00],
            [-1.43515936e+00,  1.38993633e+00],
            [-1.07776760e+00,  1.17085152e+00],
            [ 1.86458889e+00, -1.80157466e+00],
            [ 1.01044914e+00,  2.06492453e-01],
            [ 1.70004154e+00, -2.47850060e+00],
            [-1.42501296e+00, -1.98477928e+00],
            [-9.18372375e-01, -1.55178154e+00],
            [-4.59143035e-01, -1.25664044e+00],
            [ 1.56767506e+00, -9.95752079e-01],
            [ 7.62283849e-01,  9.59691571e-01],
            [-9.44690875e-01,  9.39255119e-01],
            [-4.19079588e-01, -6.55141724e-01],
            [ 6.88240400e-01,  5.56185531e-01],
            [ 1.58425289e+00, -2.16342153e+00],
            [ 1.02999549e+00, -1.45285233e+00],
            [ 1.09644346e+00, -1.57436777e+00],
            [ 1.42200867e+00,  1.15419503e+00],
            [-2.45803222e-01,  1.71211314e+00],
            [ 4.94889976e-01,  7.63839990e-01],
            [ 1.02134180e+00,  9.45960244e-01],
            [-1.67470722e+00,  5.08146054e-01],
            [ 1.31150495e+00, -1.09500977e+00],
            [-5.74292799e-01,  1.00181604e+00],
            [ 2.47038168e+00, -2.45631217e+00],
            [ 1.72951693e+00, -9.63105809e-01],
            [ 1.47998852e-01,  3.01710864e-01],
            [ 3.37083142e-01, -8.65016030e-01],
            [-1.80207349e+00,  3.37331961e-01],
            [-1.23927164e+00, -1.20011796e+00],
            [ 1.44152874e+00, -1.85846428e+00],
            [-5.20668439e-01, -2.40394792e-01],
            [ 7.71938518e-01, -8.74740658e-01],
            [ 1.78947253e+00, -1.01245757e+00],
            [ 3.09113475e-02,  2.41104654e+00],
            [ 4.59431851e-01,  2.36445144e+00],
            [ 1.01666812e+00, -1.56502399e+00],
            [-1.88645356e+00, -1.67943557e+00],
            [ 2.45648912e-01,  1.11218999e+00],
            [ 1.27077097e+00, -1.35827817e+00],
            [-2.87581849e-01,  1.46731772e+00],
            [ 8.25097241e-01, -2.62233083e-02],
            [-3.38919592e-01,  1.58977198e+00],
            [ 6.62954338e-01, -4.87995370e-01],
            [-1.93613629e+00,  7.35795411e-01],
            [ 3.41563043e-01,  9.90219366e-01],
            [-1.97693583e+00,  7.15732080e-01],
            [-3.23927163e+00, -1.13605309e+00],
            [ 2.78278955e-02, -9.19745096e-02],
            [-7.27701405e-01, -1.20616195e+00],
            [-7.28708839e-01,  1.72186934e+00],
            [-1.55234255e+00,  9.76583606e-01],
            [ 1.82882615e+00, -1.11569468e+00],
            [ 1.45730617e+00, -4.38473505e-01],
            [ 6.03113999e-01, -9.08056191e-01],
            [-2.81244610e-01, -5.84348532e-01],
            [ 1.23558642e+00, -1.79416588e+00],
            [-6.39466351e-01,  1.02526476e+00],
            [ 1.30555956e+00,  9.05457995e-01],
            [ 1.44296338e+00,  1.03176775e+00],
            [ 3.98092665e-01,  4.73502757e-01],
            [-1.13986104e+00,  5.70490953e-01],
            [ 7.20807541e-01,  1.41238389e-01],
            [-2.45625317e+00, -1.49734574e+00],
            [ 1.43092192e+00,  1.36186926e+00],
            [-1.41998835e+00,  5.18215231e-01],
            [ 1.11205856e+00, -1.82124432e+00],
            [ 1.85940785e+00, -1.72584343e+00],
            [-4.06404869e-01,  1.73902841e+00],
            [ 4.51142545e-01, -7.27901808e-02],
            [-6.21164903e-01, -3.81532899e-01],
            [ 4.70012478e-03,  5.76598817e-01],
            [-1.70544416e+00, -1.94646892e+00],
            [-7.33850688e-01, -1.33540603e+00],
            [ 1.46759956e+00,  8.69091793e-01],
            [-2.86465203e+00, -5.57864027e-01],
            [ 1.26920944e-01, -2.04818378e-01],
            [-9.69589889e-01,  8.20997459e-01],
            [-8.99102189e-01,  8.67680525e-01],
            [-2.83936041e-01,  1.65405743e+00],
            [ 1.23871259e+00, -1.79538307e+00],
            [ 5.97699538e-01,  1.93048554e-01],
            [ 1.28184220e+00, -1.04166614e+00],
            [ 3.50648934e-01,  1.14813286e+00],
            [ 1.16758824e+00, -7.01064115e-01],
            [-2.15539734e+00, -1.27091136e-01],
            [ 1.15196480e+00, -1.07420684e+00],
            [-1.52258812e+00,  4.30328639e-01],
            [ 1.80576473e+00,  1.10638905e+00],
            [ 4.07069853e-01, -8.91786889e-01],
            [-7.46639934e-01,  1.19685663e+00],
            [ 8.03787965e-01,  9.82127613e-02],
            [ 9.63138657e-01, -9.88789802e-01],
            [ 1.36353175e+00,  1.59257857e+00],
            [ 9.21953518e-01,  1.37616744e+00],
            [ 1.04148839e+00,  6.66985537e-01],
            [ 1.16303905e+00, -1.20872918e+00],
            [ 4.55947344e-01,  6.11750718e-01],
            [ 6.50504434e-01, -1.01997598e+00],
            [ 1.51219493e+00, -1.67725623e+00],
            [ 1.29627205e+00, -1.23504364e+00],
            [-1.01670517e+00,  1.33476233e+00],
            [-8.14054012e-01,  1.15566929e+00],
            [-1.48750587e+00,  3.80053034e-01],
            [-9.96788316e-01, -2.03616049e+00],
            [-1.36594229e+00,  1.09842908e+00],
            [-1.02465652e-01,  5.53956591e-01],
            [-2.17052027e+00, -1.38461987e+00],
            [-1.42169644e+00, -1.57498446e+00],
            [ 7.23070754e-01,  1.38839195e+00],
            [ 1.21871404e+00,  9.69850246e-01],
            [ 8.94458453e-01,  9.83666354e-01],
            [-6.75387920e-02, -2.30337105e-02],
            [ 1.04562215e+00, -9.71037149e-01],
            [-9.12096820e-01, -1.28417814e+00],
            [ 8.18584252e-01,  1.19769758e+00],
            [ 1.37260767e+00,  1.37404115e+00],
            [-1.04701697e+00,  6.16527216e-01],
            [-1.09318842e+00, -1.00062068e+00],
            [-2.51813750e+00, -1.16167981e+00],
            [ 1.81890518e+00, -2.17722894e+00],
            [ 2.08836195e-01, -2.56430339e-01],
            [ 1.11209587e+00,  8.05689793e-01],
            [ 8.17205373e-01,  8.35979534e-01],
            [ 1.95827213e-01,  2.98636212e-01],
            [ 4.28916544e-01, -2.75411267e-01],
            [ 1.12031409e+00,  1.02263204e+00],
            [ 1.26556164e+00, -8.70789391e-01],
            [-9.70489919e-01,  9.61056816e-01],
            [-5.80980454e-01, -6.99105141e-01],
            [ 1.90596608e+00, -1.08647739e+00],
            [ 6.87860140e-02, -9.82164724e-02],
            [-3.04030394e-01,  1.51417742e+00],
            [ 4.14881820e-01, -2.68812687e-01],
            [-1.66514696e+00, -1.02370552e+00],
            [ 1.74238008e+00, -1.30866640e+00],
            [-8.38621671e-01,  1.45227885e+00],
            [ 5.15075096e-01, -3.78777911e-01],
            [-5.11663756e-01, -1.63334211e+00],
            [ 1.49867796e+00,  4.60927728e-01],
            [ 4.13789590e-01,  1.00260120e-02],
            [-9.37267039e-01,  8.64482897e-01],
            [-1.24633643e+00, -1.67258773e+00],
            [-1.86211983e+00,  3.89944837e-01],
            [-1.51648769e+00,  5.18181882e-01],
            [-1.38464521e+00,  4.84336694e-01],
            [-2.05237035e+00,  2.02799919e-01],
            [-5.62225233e-01,  1.00454642e+00],
            [ 6.43327315e-01, -2.45733723e-01],
            [ 1.88022483e+00, -2.25160705e+00],
            [ 2.07926203e+00, -2.15835954e+00],
            [-8.26989678e-01,  7.38588099e-01],
            [ 1.23412521e+00,  1.17475680e+00],
            [-6.84901011e-01,  9.31611798e-01],
            [ 9.44432699e-01, -1.47958240e+00],
            [-1.63824370e+00, -9.55968138e-01],
            [ 3.43007702e-01, -3.31247014e-01],
            [ 4.73208554e-01,  1.26070897e+00],
            [ 1.42790173e+00, -3.95053081e-01],
            [-8.52766283e-01,  1.16950866e+00],
            [-2.19807522e+00, -1.13011775e+00],
            [ 1.35270338e-01, -9.90359853e-01],
            [ 5.05688086e-01, -7.06586051e-01],
            [ 2.15773288e+00, -2.60852823e+00],
            [ 9.85841386e-01,  6.55448584e-01],
            [-1.64324760e+00, -1.31084193e+00],
            [ 1.59791807e+00,  1.04230599e+00],
            [ 9.73974302e-01,  8.67307977e-02],
            [ 9.87577918e-01,  9.49110878e-01],
            [ 9.38665030e-01,  5.11062073e-01],
            [ 7.04563943e-01,  7.05779271e-01],
            [-7.89582205e-01,  1.53437979e+00],
            [ 5.23509525e-01,  2.95470234e-01],
            [ 8.51938096e-01,  1.22834153e+00],
            [ 5.56012195e-01,  1.26758641e+00],
            [-9.77574151e-01, -1.33756042e+00],
            [-5.52933016e-01,  1.10262513e+00],
            [ 1.28226893e+00,  8.55404115e-01],
            [ 9.13124609e-01, -1.72264444e-02],
            [ 5.89857039e-01, -1.25074350e+00],
            [ 5.74223542e-01, -2.46798649e-01],
            [-4.86510366e-01,  1.46016428e+00],
            [-9.88057749e-01,  8.30595481e-01],
            [ 9.07543785e-01, -1.11639163e+00],
            [-1.31058296e+00,  7.37289269e-01],
            [ 8.63508499e-01,  6.29903584e-01],
            [ 1.48841863e+00, -1.62284673e+00],
            [ 2.05184262e+00, -9.36240725e-01],
            [ 1.10803590e+00,  1.10489624e+00],
            [-1.03193364e+00,  7.08488331e-01],
            [ 1.26953908e+00,  1.10615212e+00],
            [-1.12484612e+00,  1.02647079e+00],
            [-1.89715176e+00, -4.78526109e-01],
            [ 1.20965589e+00,  7.28719527e-01],
            [ 2.45401828e+00, -2.47597680e+00],
            [-1.79718681e+00, -1.04367650e+00],
            [ 1.34182617e-03,  4.76781608e-01],
            [ 1.91732681e+00, -1.82940957e+00],
            [ 1.02558850e+00,  1.31343677e+00],
            [-1.95635584e+00, -1.19695707e+00],
            [-1.19188733e+00,  7.37035516e-01],
            [-7.78407817e-01,  1.53137460e+00],
            [-1.39473524e+00, -9.62794110e-01],
            [ 2.38987138e-01,  9.86079182e-01],
            [ 2.17455317e-01, -6.25083895e-01],
            [ 1.31038735e+00,  1.02461018e+00],
            [ 4.68164777e-01, -9.83594541e-01],
            [ 1.38594054e+00,  1.10779611e+00],
            [-1.26212956e+00, -7.17065244e-01],
            [-6.68015712e-01,  1.49493760e+00],
            [-2.35964542e+00, -2.53704881e+00],
            [ 7.38255616e-01,  2.34899416e-02],
            [ 9.44345246e-01,  8.31534285e-01],
            [ 1.57526683e-01, -7.51845330e-01],
            [ 2.18601774e-01,  1.40791696e-01],
            [-1.06574474e+00,  1.08427400e+00],
            [ 1.31122458e+00, -9.51143878e-01],
            [-2.78073406e+00, -1.43729161e+00],
            [-1.51272990e+00,  6.48855176e-01],
            [-2.07815666e+00, -1.78493569e+00],
            [-1.63901568e+00, -1.14957508e+00],
            [ 6.07389196e-02,  6.13783841e-01],
            [ 2.23376997e+00, -1.47514210e+00],
            [ 1.02441741e+00, -1.16709857e+00],
            [-9.27659357e-01,  8.06614288e-01],
            [ 1.22136810e+00, -2.26586483e-01],
            [ 1.45548437e+00,  8.91044158e-01],
            [-2.52328791e+00, -1.95271198e+00],
            [ 6.23093167e-01,  1.08491715e+00],
            [ 9.26320747e-01,  1.21944700e+00],
            [ 1.03545560e+00,  1.26666498e+00],
            [-3.13822450e+00, -9.68560552e-01],
            [ 6.70625167e-01, -2.89735747e-01],
            [ 1.10573824e+00, -5.51445127e-01],
            [ 2.70876085e+00,  7.78570201e-01],
            [ 2.51678240e-01, -7.62615644e-01],
            [-1.73085150e+00, -3.52419630e-02],
            [-1.28460692e+00,  8.33113539e-01],
            [ 1.95789390e-01, -5.33733046e-01],
            [-8.35631901e-01, -8.97651230e-01],
            [-1.36419226e+00, -1.23540466e+00],
            [ 1.23545150e-01,  1.58169997e+00],
            [ 1.90017257e-01,  6.69784472e-01],
            [ 7.69223884e-01,  1.33403864e+00],
            [-4.75616453e-01,  1.73688190e+00],
            [ 1.21434847e+00, -7.14758388e-01],
            [-1.51655423e+00,  4.78214539e-01],
            [ 2.07698737e+00,  1.43864367e+00],
            [-2.93432373e+00, -1.27879247e+00],
            [-8.37349919e-01, -3.30536628e-01],
            [-9.85276515e-01, -5.62866138e-01],
            [-1.09675919e+00,  1.09201910e+00],
            [ 8.89226048e-01,  8.77748387e-01],
            [-3.22428308e-01, -8.51721846e-01],
            [ 2.70368080e-01, -5.25612838e-01],
            [ 7.24426707e-01,  1.42219097e+00],
            [-2.06065486e-01,  1.60670342e+00],
            [-1.04381356e+00, -1.31156144e+00],
            [-1.37136749e+00,  4.92519220e-01],
            [-1.39971730e+00,  7.17087967e-01],
            [ 8.20980337e-01, -6.42414553e-01],
            [-3.36131139e-01,  7.00851190e-02],
            [ 3.13937306e-01, -9.30382084e-01],
            [ 1.36645155e+00, -1.17511616e+00],
            [-1.23440877e+00,  9.18865622e-01],
            [-2.06265475e-01, -7.51501347e-01],
            [-1.23976016e+00,  6.46810596e-01],
            [ 1.86615353e-01, -8.97781675e-01],
            [-4.13471925e-01,  1.48529165e+00],
            [ 6.30077234e-01,  9.47041197e-01],
            [ 9.91895301e-01,  8.15248367e-01],
            [ 4.75216895e-01, -5.98209203e-01],
            [ 7.78394789e-02, -7.01906969e-03],
            [-4.74103111e-01,  1.45080164e+00],
            [-8.43915056e-01,  9.84889320e-01],
            [-1.09512126e+00,  1.39814592e+00],
            [ 1.84790065e-01,  8.09187767e-01],
            [-5.81420621e-01,  9.53756460e-01],
            [ 9.23344479e-01,  1.40583171e+00],
            [ 1.37771425e+00, -1.90746328e+00],
            [-1.61330589e+00,  7.73611540e-01],
            [-1.41193788e+00,  9.35015143e-01],
            [-2.54636671e+00, -1.67681290e+00],
            [ 3.27931149e-01, -8.91101689e-01],
            [-2.80980552e+00, -1.86423658e+00],
            [-1.15497421e+00, -8.91835682e-01],
            [ 1.89958618e+00, -2.34238057e+00],
            [-2.08135729e+00, -1.65688671e-01],
            [-3.06273612e+00, -1.58071556e+00],
            [-2.93523916e+00, -1.92467560e+00],
            [-1.82616569e+00, -1.94067438e-01],
            [-2.00604407e+00, -1.19465263e+00],
            [ 8.66741026e-01,  5.14826357e-01],
            [-1.43290429e+00, -1.19035322e+00],
            [-1.48851057e-01,  1.78282332e+00],
            [ 5.72617280e-01,  1.85023072e-01],
            [-1.21880873e+00,  9.16596045e-01],
            [-8.21819137e-01,  1.02159353e+00],
            [-1.52970752e+00,  1.03024785e+00],
            [-1.02867354e+00,  9.85208133e-01],
            [ 2.39227649e-01,  1.83728158e+00],
            [-1.99222570e+00,  8.20218244e-01],
            [-1.29958710e+00, -1.06997824e+00],
            [ 5.34402217e-01,  5.89022169e-01],
            [ 1.52450386e+00,  9.57831612e-01],
            [ 1.67156300e+00,  1.63992275e+00],
            [-7.16296283e-01,  1.27138388e+00],
            [ 7.47388009e-01,  1.01437060e+00],
            [-1.26071964e+00,  1.45029384e+00],
            [-1.52831543e-01,  1.38050029e+00],
            [ 3.05088977e-01,  1.55217990e+00],
            [-3.19610659e+00, -1.75682154e+00],
            [ 1.11933094e+00,  6.07581766e-01],
            [-2.38312621e+00, -7.70315613e-03],
            [-1.12454579e+00, -1.00583610e+00],
            [ 1.47349499e+00, -2.12086856e+00],
            [ 1.49080144e-01,  1.30148888e+00],
            [ 3.07284634e-01, -3.68718526e-01],
            [-1.27654721e+00, -9.59064821e-01],
            [-1.41941934e+00,  2.06475067e-01],
            [ 1.30403595e-01,  1.76474207e+00],
            [-6.15598237e-01, -9.55955993e-01],
            [-5.44625318e-01, -8.97500829e-01],
            [ 2.31923621e+00, -1.63123121e+00],
            [-2.11086058e-01, -3.67605211e-01],
            [ 9.06820725e-01,  1.19578459e+00],
            [ 1.85519833e+00, -1.33314413e+00],
            [-3.97280332e-01,  2.08588424e+00],
            [-2.98106416e-01,  1.65851821e+00],
            [-1.65114761e+00,  5.66143709e-01],
            [-5.79756612e-01,  1.50776283e+00],
            [-1.04973434e+00,  1.14231457e+00],
            [ 5.12752574e-01,  8.79195444e-01],
            [ 1.91709261e+00,  1.59369074e+00],
            [-1.15617366e+00,  6.59801490e-01],
            [ 6.60727788e-01,  8.88719975e-01],
            [-4.00284320e-01,  5.99802716e-01],
            [ 9.82040010e-01,  9.04459762e-01],
            [-2.44717034e+00, -7.15115078e-01],
            [ 2.14922604e+00,  1.29404175e+00],
            [ 1.47064507e+00,  1.13021381e+00],
            [-4.02866759e-01, -1.33657464e+00],
            [ 1.42731978e+00,  8.15212047e-01],
            [-3.65291672e-01, -1.09310906e+00],
            [ 4.57465866e-01,  6.07575985e-01],
            [-1.07247394e+00,  1.56500751e+00],
            [ 3.61428746e-01,  7.15781425e-02],
            [-1.09109854e+00,  8.45530367e-01],
            [ 1.34517149e+00, -1.04156360e+00],
            [-1.89640221e+00, -1.10590583e+00],
            [-1.02173487e+00, -7.36518323e-01],
            [-7.32379084e-01,  9.87215918e-01],
            [-1.13601324e-01,  2.15642836e+00],
            [ 1.58308065e+00, -1.18173323e+00],
            [ 2.00067623e-01, -1.80875972e+00],
            [-9.82302389e-01, -1.35716559e+00],
            [-2.06792920e+00,  7.03769019e-02],
            [ 1.00452937e+00,  6.76318738e-01],
            [-1.63772727e+00, -1.17359059e+00],
            [ 1.72079973e+00, -1.86600963e+00],
            [ 1.22903349e+00,  5.53848809e-01],
            [-1.74306039e+00, -1.22346488e+00],
            [-1.47590019e-01, -5.64347346e-01],
            [ 7.11147838e-01,  1.13448735e+00],
            [ 5.14143665e-01,  2.15632458e-01],
            [-3.70147611e+00, -2.20760905e+00],
            [-7.82112525e-01,  1.41861071e+00],
            [-1.44406214e-01, -5.11133682e-01],
            [-1.19333504e+00, -9.80837715e-01],
            [ 7.28622714e-01, -1.04327039e+00],
            [-1.89622071e+00, -1.54609738e+00],
            [-1.45501339e+00,  4.46370274e-01],
            [ 2.28824771e-01, -2.26603143e-01],
            [-4.41809929e-01,  1.23122936e+00],
            [ 9.90716920e-01,  1.17235321e+00],
            [ 9.73184736e-01,  1.16375861e+00],
            [ 6.08104093e-01,  9.66531594e-01],
            [ 4.16452806e-01,  2.65003369e+00],
            [ 1.59070351e+00, -1.18076864e+00],
            [ 9.14213952e-01,  7.38000737e-01],
            [ 1.16565229e+00, -9.49759592e-01],
            [ 1.37596892e+00, -7.14633032e-01],
            [-1.09065056e+00, -9.95464098e-01],
            [-1.06400670e+00,  9.50770217e-01],
            [-1.06059837e+00,  8.50786581e-01],
            [ 1.56638944e+00,  8.17499445e-01],
            [-1.94913831e+00, -1.90601147e+00],
            [ 1.53440506e+00, -1.16872375e-01],
            [-3.92435991e-01,  1.39209018e+00],
            [-1.18812487e-01,  9.69737392e-01]]))

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x=X[:,0], y=X[:,1], hue=y);
```

```python
# First extract features x1 and x2 from x and reshape them to x1 vector arrays
x1 = x[:,1]
x2 = x[:,2]
x1 = x1.reshape(x1.shape[0], 1)
x2 = x2.reshape(x2.shape[0], 1)
print(x[:5,:]) # For visualization of the first 5 values
print(x1[:5,:]) # For visualization of the first 5 values
print(x2[:5,:]) # For visualization of the first 5 values
```

    [[ 1.          2.25698215 -1.34710915]
     [ 1.          1.43699308  1.28420453]
     [ 1.          0.57927295  0.23690172]
     [ 1.          0.42538132 -0.24611145]
     [ 1.          1.13485101 -0.61162683]]
    [[2.25698215]
     [1.43699308]
     [0.57927295]
     [0.42538132]
     [1.13485101]]
    [[-1.34710915]
     [ 1.28420453]
     [ 0.23690172]
     [-0.24611145]
     [-0.61162683]]

################# Do not write above this line #################

Your code here

```python
def is_in(label, cur_labels):
    na, nb = label.count("a"), label.count("b")
    for l in cur_labels:
        na_l, nb_l = l.count("a"), l.count("b")
        if na_l == na and nb_l == nb:
            return True
    return False

def new_features(x, degree):
    # 3nd degree features must have the order [x, x1x1, x1x2, x2x2, x1x1x1, x1x1x2, x1x2x2, x2x2x2]
    features = []
    # initialize grades to degree 0
    for triple in x:
        c_features = []
        a, b = triple[1], triple[2]
        cur_factors, cur_labels = [a, b], ["a", "b"]
        for degrees in range(2, degree + 1):
            # factors list
            new_factors, new_labels = [], []
            for index in range(len(cur_factors)):
                r_a, ra_label = a * cur_factors[index], "a" + cur_labels[index]
                r_b, rb_label = b * cur_factors[index], "b" + cur_labels[index]
                if not is_in(ra_label, new_labels):
                    new_factors.append(r_a)
                    new_labels.append(ra_label)
                if not is_in(rb_label, new_labels):
                    new_factors.append(r_b)
                    new_labels.append(rb_label)
            c_features += new_factors
            cur_factors, cur_labels = new_factors, new_labels
        features.append([*triple, *c_features])
    features = np.array(features, np.float64)
    return features
```

```python
f = new_features(x, 2)
print(f[0])
print(f[1])
```

    [ 1.          2.25698215 -1.34710915  5.09396844 -3.04040131  1.81470307]
    [1.         1.43699308 1.28420453 2.06494911 1.84539302 1.64918128]

################# Do not write below this line #################

```python
x_new_quad = new_features(x, degree=2)
x_new_cubic = new_features(x, degree=3)

#reordering output features
temp = np.copy(x_new_quad[:, -1])
x_new_quad[:, -1] = x_new_quad[:, -2]
x_new_quad[:, -2] = temp

temp = np.copy(x_new_cubic[:, -1])
x_new_cubic[:, -1] = x_new_cubic[:, -2]
x_new_cubic[:, -2] = x_new_cubic[:, -3]
x_new_cubic[:, -3] = temp

```

Now use the gradient ascent optimization algorithm to learn theta by maximizing the log-likelihood, both for the case of x_new_quad and x_new_cubic.

```python
# Initialize theta0, in case of quadratic features
theta0_quad = np.zeros(x_new_quad.shape[1])

theta_final_quad, log_l_history_quad = gradient_ascent(theta0_quad,x_new_quad,y,lr=0.5,num_steps=n_iter)

# Initialize theta0, in case of quadratic and cubic features
theta0_cubic = np.zeros(x_new_cubic.shape[1])

# Run Newton's method, in case of quadratic and cubic features
theta_final_cubic, log_l_history_cubic = gradient_ascent(theta0_cubic,x_new_cubic,y,lr=0.5,num_steps=n_iter)

# check and compare with previous results
print(theta_final_quad)
print(theta_final_cubic)
```

    [ 0.07548038  3.32398303  0.27106753 -0.51873505 -0.34088582 -0.04846172]
    [ 0.82395453  2.41957969  1.66049077 -1.13927962 -0.20657699 -0.96074035
      0.34384121 -0.64078458  0.95132383  1.40927121]

```python
# Plot the log likelihood values in the optimization iterations, in one of the two cases.
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history_quad)),log_l_history_quad,'b.')
```

#### Exercise 2.b **(3 Points)** Plot the computed non-linear boundary and discuss the questions

First, define a boundary_function to compute the boundary equation for the input feature vectors $x_1$ and $x_2$, according to estimated parameters theta, both in the case of quadratic (theta_final_quad) and of quadratic and cubic features (theta_final_cubic). Refer for the equation to the introductory part of Question 2.

################# Do not write above this line #################

Your code here

```python
def boundary_function(x1_vec, x2_vec, theta_final):

    x1_vec, x2_vec = np.meshgrid(x1_vec,x2_vec)
    if len(theta_final) == 6:
        # boundary function value for features up to quadratic
        c_0, c_1, c_2, c_3, c_4, c_5 = theta_final
        f = c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*x1_vec*x2_vec

    elif len(theta_final) == 10:
        # boundary function value for features up to cubic
        c_0, c_1, c_2, c_3, c_4, c_5, c_6, c_7, c_8, c_9 = theta_final
        f = c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*x1_vec*x2_vec + c_6*(x1_vec**3) + c_7*(x2_vec**3) + c_8*(x1_vec**2)*x2_vec + c_9*(x2_vec**2)*x1_vec

    else:
        raise("Number of Parameters is not correct")

    return x1_vec, x2_vec, f
```

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x1_vec*x1_vec
```

    array([2.21038776e+01, 2.17082715e+01, 2.13162376e+01, 2.09277759e+01,
           2.05428865e+01, 2.01615693e+01, 1.97838244e+01, 1.94096517e+01,
           1.90390512e+01, 1.86720230e+01, 1.83085670e+01, 1.79486833e+01,
           1.75923718e+01, 1.72396325e+01, 1.68904655e+01, 1.65448707e+01,
           1.62028481e+01, 1.58643978e+01, 1.55295197e+01, 1.51982138e+01,
           1.48704802e+01, 1.45463188e+01, 1.42257297e+01, 1.39087128e+01,
           1.35952681e+01, 1.32853957e+01, 1.29790955e+01, 1.26763676e+01,
           1.23772118e+01, 1.20816284e+01, 1.17896171e+01, 1.15011781e+01,
           1.12163114e+01, 1.09350168e+01, 1.06572945e+01, 1.03831445e+01,
           1.01125667e+01, 9.84556109e+00, 9.58212775e+00, 9.32226664e+00,
           9.06597778e+00, 8.81326115e+00, 8.56411676e+00, 8.31854460e+00,
           8.07654468e+00, 7.83811700e+00, 7.60326156e+00, 7.37197836e+00,
           7.14426739e+00, 6.92012866e+00, 6.69956216e+00, 6.48256791e+00,
           6.26914589e+00, 6.05929611e+00, 5.85301856e+00, 5.65031326e+00,
           5.45118019e+00, 5.25561935e+00, 5.06363076e+00, 4.87521440e+00,
           4.69037028e+00, 4.50909840e+00, 4.33139876e+00, 4.15727135e+00,
           3.98671618e+00, 3.81973324e+00, 3.65632255e+00, 3.49648409e+00,
           3.34021787e+00, 3.18752388e+00, 3.03840214e+00, 2.89285263e+00,
           2.75087536e+00, 2.61247032e+00, 2.47763753e+00, 2.34637697e+00,
           2.21868864e+00, 2.09457256e+00, 1.97402871e+00, 1.85705710e+00,
           1.74365773e+00, 1.63383059e+00, 1.52757569e+00, 1.42489303e+00,
           1.32578261e+00, 1.23024442e+00, 1.13827848e+00, 1.04988476e+00,
           9.65063290e-01, 8.83814053e-01, 8.06137053e-01, 7.32032291e-01,
           6.61499767e-01, 5.94539479e-01, 5.31151429e-01, 4.71335617e-01,
           4.15092041e-01, 3.62420703e-01, 3.13321603e-01, 2.67794739e-01,
           2.25840113e-01, 1.87457725e-01, 1.52647574e-01, 1.21409660e-01,
           9.37439831e-02, 6.96505439e-02, 4.91293421e-02, 3.21803776e-02,
           1.88036504e-02, 8.99916063e-03, 2.76690818e-03, 1.06893087e-04,
           1.01911534e-03, 5.50357494e-03, 1.35602719e-02, 2.51892062e-02,
           4.03903778e-02, 5.91637868e-02, 8.15094332e-02, 1.07427317e-01,
           1.36917438e-01, 1.69979796e-01, 2.06614392e-01, 2.46821225e-01,
           2.90600296e-01, 3.37951603e-01, 3.88875149e-01, 4.43370931e-01,
           5.01438951e-01, 5.63079208e-01, 6.28291703e-01, 6.97076434e-01,
           7.69433404e-01, 8.45362610e-01, 9.24864054e-01, 1.00793774e+00,
           1.09458365e+00, 1.18480181e+00, 1.27859220e+00, 1.37595483e+00,
           1.47688970e+00, 1.58139681e+00, 1.68947615e+00, 1.80112773e+00,
           1.91635155e+00, 2.03514760e+00, 2.15751589e+00, 2.28345642e+00,
           2.41296919e+00, 2.54605420e+00, 2.68271144e+00, 2.82294092e+00,
           2.96674263e+00, 3.11411659e+00, 3.26506278e+00, 3.41958120e+00,
           3.57767187e+00, 3.73933477e+00, 3.90456991e+00, 4.07337729e+00,
           4.24575691e+00, 4.42170876e+00, 4.60123285e+00, 4.78432918e+00,
           4.97099774e+00, 5.16123854e+00, 5.35505158e+00, 5.55243686e+00,
           5.75339437e+00, 5.95792412e+00, 6.16602611e+00, 6.37770034e+00,
           6.59294680e+00, 6.81176550e+00, 7.03415644e+00, 7.26011961e+00,
           7.48965503e+00, 7.72276268e+00, 7.95944256e+00, 8.19969469e+00,
           8.44351905e+00, 8.69091565e+00, 8.94188449e+00, 9.19642556e+00,
           9.45453887e+00, 9.71622442e+00, 9.98148221e+00, 1.02503122e+01,
           1.05227145e+01, 1.07986890e+01, 1.10782357e+01, 1.13613547e+01,
           1.16480459e+01, 1.19383094e+01, 1.22321450e+01, 1.25295530e+01,
           1.28305331e+01, 1.31350855e+01, 1.34432102e+01, 1.37549070e+01])

################# Do not write below this line #################

Now plot the decision boundaries corresponding to the theta_final_quad and theta_final_cubic solutions.

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

```python
x1_vec = np.linspace(X[:,0].min()-1,X[:,0].max()+1,200);
x2_vec = np.linspace(X[:,1].min()-1,X[:,1].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic)

sns.scatterplot(x=X[:,0], y=X[:,1], hue=y, data=X);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
plt.show()
```

#### Confusion Matrix

Here you can see the confusion matrices related to the three models you've implemented.

```python
from sklearn.metrics import confusion_matrix
```

```python
## logistic regression with linear buondary

z = np.dot(x,theta_final)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[218,  35],
           [ 22, 225]], dtype=int64)

```python
## logistic regression with non linear buondary - quadratic

z = np.dot(x_new_quad,theta_final_quad)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[220,  33],
           [ 15, 232]], dtype=int64)

```python
## logistic regression with non linear buondary - cubic

z = np.dot(x_new_cubic,theta_final_cubic)
probabilities = sigmoid(z)
y_hat = np.array(list(map(lambda x: 1 if x>0.5 else 0, probabilities)))
confusion_matrix(y, y_hat)
```

    array([[225,  28],
           [ 11, 236]], dtype=int64)

################# Do not write above this line #################

Write now your considerations. Discuss in particular:
1. Look back at the plots you have generated. What can you say about the differences between the linear, quadratic, and cubic decision boundaries? Can you say if the model is improving in performances, increasing the degree of the polynomial? Do you think you can incur in underfitting increasing more and more the degree?

- The _linear decision boundary_ shown in the first plot is an example of underfitting, in fact the model is not capable of capturing the underlying structure of the data.
- In the same way we can say that the _cubic decision boundary_ overfits the data, in fact the model captures too scrupulously the behaviour of these particular observations. This means that it may fail to fit additional data or predict future observations reliably.
- The _quadratic decision boundary_ fits the data in the most balanced way.

- We can notice that, the more the degree grows, the more the decision boundaries fit the data. This leads us to an improvement if we move from the _linear function_ to the _quadratic_ one. The model does not improve in performances if we move from the _quadratic function_ to the _cubic_ one, instead.

2. Let's now delve into some quantitative analysis. The three tables you have generated represent the confusion matrix for the model you have implemented in the first two questions. What can you say about actual performances? Does the increase of the degree have a high effect on the results?

################# Do not write below this line #################

## Question 3: Multinomial Classification (Softmax Regression) **(13 Points)**

### Code and Theory **(10 Points)**
### Report **(3 Points)**

#### Exercise 3.a **(4 Points)**

In the multinomial classification we generally have $K>2$ classes. So the label for the $i$-th sample $X_i$ is $y_i\in\{1,...,K\}$, where $i=1,...,N$. The output class for each sample is estimated by returning a score $s_i$ for each of the K classes. This results in a vector of scores of dimension K.
In this exercise we'll use the *Softmax Regression* model, which is the natural extension of *Logistic Regression* for the case of more than 2 classes. The score array is given by the linear model:

\begin{equation}
s_i =  X_i \theta
\end{equation}

Scores may be interpreted probabilistically, upon application of the function *softmax*. The position in the vector with the highest probability will be predicted as the output class. The probability of the class k for the $i$-th data sample is:

\begin{equation}
p_{ik} = \frac{\exp(X_i \theta_k)}{\sum_{j=1}^K(X_i \theta_j))}
\end{equation}

We will adopt the *Cross Entropy* loss and optimize the model via *Gradient Descent*.
In the first of this exercise we have to:
-    Write the equations of the Cross Entropy loss for the Softmax regression model;
-    Compute the equation for the gradient of the Cross Entropy loss for the model, in order to use it in the gradient descent algorithm.

#### A bit of notation

*  N: is the number of samples
*  K: is the number of classes
*  X: is the input dataset and it has shape (N, H) where H is the number of features
*  y: is the output array with the labels; it has shape (N, 1)
*  $\theta$: is the parameter matrix of the model; it has shape (H, K)

################# Do not write above this line #################

Your equations here.

\begin{equation}
L(\theta) = ...
\end{equation}

\begin{equation}
\nabla_{\theta_k} L(\theta) = ...
\end{equation}

################# Do not write below this line #################

#### Exercise 3.b **(4 Points)**

Now we will implement the code for the equations. Let's implement the functions:
-  softmax
-  CELoss
-  CELoss gradient
-  gradient descent

We generate a toy dataset with *sklearn* library. Do not change anything outside the parts provided of your own code (else the provided checkpoint will not work).

```python
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X.shape, y.shape
```

    ((300, 7), (300,))

As a hint for the implementations of your functions: consider the labels $y$ as one-hot vector. This will allow matrix operations (element-wise multiplication and summation).

```python
import scipy
import numpy as np

def class2OneHot(vec):
    out_sparse = scipy.sparse.csr_matrix((np.ones(vec.shape[0]), (vec, np.array(range(vec.shape[0])))))
    out_onehot = np.array(out_sparse.todense()).T
    return out_onehot

y_onehot = class2OneHot(y)
```

Let's visualize the generated dataset. We use as visualizzation method the *Principal Component Analysis* (PCA). PCA summarize the high-dimensional feature vectors of each sample into 2  features, which we can illustrate with a 2D plot. Look at the following plot, the 3 generated classes do not seem separable.

```python
from sklearn.decomposition import PCA
import pandas as pd

pca = PCA(n_components=2)
principalComponents = pca.fit_transform(X)
principalDf = pd.DataFrame(data = principalComponents, columns = ['pc1', 'pc2'])
finalDf = pd.concat([principalDf, pd.DataFrame(y, columns = ['target'])], axis = 1)
```

```python
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(x='pc1', y='pc2', hue='target', data=finalDf);
```

################# Do not write above this line #################

```python
def softmax(theta, X):
    '''
    Function to compute associated probability for each sample and each class.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)

    Output:
    softmax: it's the matrix containing probability for each sample and each class. The shape is (N, K)
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return softmax

def CELoss(theta, X, y_onehot):
    '''
    Function to compute softmax regression model and Cross Entropy loss.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    loss: The scalar that is the mean error for each sample.
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return loss

def CELoss_jacobian(theta, X, y_onehot):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)

    Output:
    jacobian: A matrix with the partial derivatives of the loss. The shape is (H, K)
    '''

    #####################################################
    ##                 YOUR CODE HERE                  ##
    #####################################################

    return jacobian

def gradient_descent(theta, X, y_onehot, alpha=0.01, iterations=100):
    '''
    Function to compute gradient of the cross entropy loss with respect the parameters.

    Input:
    theta: it's the model parameter matrix. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y_onehot: it's the label array in encoded as one hot vector. The shape is (N, K)
    alpha: it's the learning rate, so it determines the speed of each step of the GD algorithm
    iterations: it's the total number of step the algorithm performs

    Output:
    theta: it's the updated matrix of the parameters after all the iterations of the optimization algorithm. The shape is (H, K)
    loss_history: it's an array with the computed loss after each iteration
    '''

    # We initialize an empty array to be filled with loss value after each iteration
    loss_history = np.zeros(iterations)

    # With a for loop we compute the steps of GD algo
    for it in range(iterations):

        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

    return theta, loss_history
```

################# Do not write below this line #################

```python
# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 1000
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.01, iterations=n_iter)
```

```python
theta_final
```

```python
loss = CELoss(theta_final, X, y_onehot)
loss
```

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('loss')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)), log_l_history,'b.')
```

#### Exercise 3.c **(2 Points)**

Let's now evaluate the goodness of the learnt based on accuracy:

\begin{equation}
Accuracy = \frac{Number\ of\ correct\ predictions}{Total\ number\ of\ predictions}
\end{equation}

Implement the compute_accuracy function. You may compare the accuracy achieved with learnt model Vs. a random model (random $\Theta$) or one based on $\Theta$'s filled with zeros.

################# Do not write above this line #################

```python
def compute_accuracy(theta, X, y):
    '''
    Function to compute accuracy metrics of the softmax regression model.

    Input:
    theta: it's the final parameter matrix. The one we learned after all the iterations of the GD algorithm. The shape is (H, K)
    X: it's the input data matrix. The shape is (N, H)
    y: it's the label array. The shape is (N, 1)

    Output:
    accuracy: Score of the accuracy.
    '''

    return accuracy
```

################# Do not write below this line #################

```python
compute_accuracy(theta_final, X, y)
```

```python
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))
compute_accuracy(theta0, X, y)
```

```python
compute_accuracy(np.zeros((X.shape[1], len(np.unique(y)))), X, y)
```

### Report **(3 Points)**

Experiment with different values for the learning rate $\alpha$ and the number of iterations. Look how the loss plot changes the convergence rate and the resulting accuracy metric. Report also execution time of each run. For this last step you could you %%time at the beginning of the cell to display time needed for the algorithm.

```python
%%time

# Initialize a theta matrix with random parameters
theta0 = np.random.rand(X.shape[1], len(np.unique(y)))

print("Initial Loss with initialized theta is:", CELoss(theta0, X, y_onehot))

# Run Gradient Descent method
n_iter = 100
theta_final, log_l_history = gradient_descent(theta0, X, y_onehot, alpha=0.001, iterations=n_iter)
```

**Write your Report here**

| LR | Iter | Accuracy | Time |
|---|---|---|---|
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
| ... |  |  |  |

## Question 4: Multinomial Naive Bayes **(6 Points)**

### Code and Theory

The Naive Bayes classifier is a probabilistic machine learning model often used for classification tasks, e.g. document classification problems.

In the multinomial Naive Bayes classification you generally have $K>2$ classes, and the features are assumed to be generated from a multinomial distribution.

##### __*Example Data*__
General models consider input data as values. In the case of MultinomialNB, being used mainly in the field of document classification, these data consider how many features $X_i$ are present in the sample. Basically, it is a count of features within each document.

Taking into account $D=3$ documents and a vocabulary consisting of $N=4$ words, the data are considered as follows.

|  | $w_1$ | $w_2$ | $w_3$ | $w_4$ |
|---|---|---|---|---|
| $d_1$  | 3 | 0 | 1 | 1 |
| $d_2$ | 2 | 1 | 3 |0|
| $d_3$ | 2 | 2 | 0 |2|

By randomly generating the class to which each document belongs we have $y=[1,0,1]$

##### __*A bit of notation*__
- $Y =\{y_1, y_2, ... , y_{|Y|}\}$: set of classes

- $V =\{w_1, w_2, ... , w_{|V|}\}$: set of vocabulary

-  $D =\{d_1, d_2, ... , d_{|D|}\}$: set of documents

-  $N_{yi}$:  count of a specific word $w_i$ in each unique class, e.g. for $y=1$ you select $D_1$ and $D_3$, then for third column  you have $N_{y,3}=1$
-  $N_y$: total count of features for a specific class, e.g. for $y=1$ you sum all rows values which the correspondent label is 1, so $N_y=11$

-  $n$: total number of features (words in vocabulary)
-  $\alpha$: smoothing parameters

##### __*Task*__
Find the class $y$ to which the document is most likely to belong given the words $w$.
Use the Bayes formula and the posterior probability for it.

Bayes Formula:
\begin{equation}
P(A|B) = \frac{P(A)*P(B|A)}{P(B)}
\end{equation}

Where:
- P(A): Prior probability of A
- P(B): Prior probability of B
- P(B|A): Likelihood, multiplying posterior probability, that is multinomial Naive Bayes is:
\begin{equation}
P(B|A) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

**Reminder: do not change any part of this notebook outside the assigned work spaces**

#### Generate random dataset

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

X, y = make_classification(n_samples=300, n_features=7, n_informative=7, n_redundant=0, n_classes=3, random_state=1)
X = np.floor(X)-np.min(np.floor(X))
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
X_train.shape, y_train.shape
```

    ((240, 7), (240,))

#### Step0: $N_y$ and $N_{yi}$

```python
def feature_count(X, y, classes, n_classes, n_features):
        '''
        Function to compute the count of a specific word in each unique class and the total count.

        Input:
        X: it's the input data matrix.
        y: label array
        classes: unique values of y
        n_classes: number of classes
        n_features: it's the number of word in Vocabulary.

        Output:
        N_yi:   count of a specific word $w_i$ in each unique class
        N_y: total count of features for a specific class
        '''
        N_yi = np.zeros((n_classes, n_features)) # feature count
        N_y = np.zeros((n_classes)) # total count

        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return N_yi, N_y
```

```python
n_samples_train, n_features = X_train.shape
classes = np.unique(y_train)
n_classes = 3
alpha = 0.1

N_yi, N_y = feature_count(X_train, y_train, classes, n_classes, n_features)
```

#### Step1: Prior Probability
The probability of a document being in a specific category from the given set of documents.

######################################

Your equations here:
\begin{equation}
P(y_j) = ...
\end{equation}

######################################

```python
def prior_(X, y, n_classes, n_samples):
        """
        Calculates prior for each unique class in y.

        Input:
        X: it's the input data matrix.
        y: label array
        n_classes: number of classes
        n_samples: number of documents

        Output:
        P: prior probability for each class. Shape: (, n_classes)
        """
        classes = np.unique(y)
        P = np.zeros(n_classes)

        # Implement Prior Probability P(A)
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return P
```

```python
prior_prob = prior_(X_train, y_train, n_classes, n_samples_train)
print(prior_prob)
```

    [0.3        0.34583333 0.35416667]

#### Step2
Posterior Probability: The conditional probability of a word occurring in a document given that the document belongs to a particular category.

\begin{equation}
P(w_i|y_j) = \left(\frac{N_{yi}+\alpha}{N_{y}+\alpha*n\_features}\right)^{X_{doc,i}}
\end{equation}

Likelihood for a single document:
######################################

Your equations here:
\begin{equation}
P(w|y_j) = ...
\end{equation}

######################################

```python
def posterior_(x_i, i, h, N_y, N_yi, n_features, alpha):
        """
        Calculates posterior probability. aka P(w_i|y_j) using equation in the notebook.

        Input:
        x_i: feature x_i
        i: feature index.
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        posterior: P(xi | y). Float.
        """

        # Implement Posterior Probability
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################
        return posterior

def likelihood_(x, h, N_y, N_yi, n_features, alpha):
        """
        Calculates Likelihood P(w|j_i).

        Input:
        x: a row of test data. Shape(n_features,)
        h: a class in y
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        likelihood: Float.
        """

        tmp = []
        for i in range(x.shape[0]):
            tmp.append(posterior_(x[i], i, h, N_y, N_yi, n_features, alpha))

        # Implement Likelihood
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return likelihood
```

```python
# Example of likelihood for first document
likelihood_(X_test[0], 0, N_y, N_yi, n_features, alpha)
```

    2.7754694679413126e-53

#### Step3
Joint Likelihood that, given the words, the documents belongs to specific class
######################################

Your equations here:
\begin{equation}
P(y_i|w) = ...
\end{equation}

######################################

Finally, from the probability that the document is in that class given the words, take the argument correspond to max value.

\begin{equation}
y(D) = argmax_{y \in Y} \frac{P(y|w)}{\sum_{j}P(y_j|w)}
\end{equation}

```python
def joint_likelihood(X, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha):
        """
        Calculates the joint probability P(y_i|w) for each class and makes it probability.
        Then take the argmax.

        Input:
        X: test data
        prior_prob:
        classes:
        n_classes:
        N_yi:   count of a specific word in each unique class
        N_y: total count of features for a specific class
        n_features: it's the number of word in Vocabulary.
        alpha: smoothing parameter

        Output:
        predicted_class: Predicted class of the documents. Int. Shape: (,#documents)
        """
        samples, features = X.shape
        predict_proba = np.zeros((samples,n_classes))

        # Calculate Joint Likelihood of each row for each class, then normalize in order to make them probabilities
        # Finally take the argmax to have the predicted class for each document
        #####################################################
        ##                 YOUR CODE HERE                  ##
        #####################################################

        return predicted_class
```

```python
yhat = joint_likelihood(X_test, prior_prob, classes, n_classes, N_y, N_yi, n_features, alpha)
```

#### Step4: Calculate the Accuracy Score

```python
print('Accuracy: ', np.round(accuracy_score(yhat, y_test),3))
```

    Accuracy:  0.717

**Sanity Check**

Here we use a function from the sklearn library, one of the most widely used in machine learning. MultinomialNB() implements the required algorithm, so the result of your implementation should be equal to the output of the following function.

```python
from sklearn import naive_bayes
clf = naive_bayes.MultinomialNB(alpha=0.1)
clf.fit(X_train,y_train)
sk_y = clf.predict(X_test)
print('Accuracy: ', np.round(accuracy_score(sk_y, y_test),3))
```

    Accuracy:  0.717


## 3. Structure discovery for the Lorenz-96 problem

In this notebook, we will show how we can estimate the conditional independence structure of a random variable $\mathsf{X} \in \mathbb{R}^{N_x}$ with density $\pi$ based on i.i.d. samples $\{\boldsymbol{x}^i\}, \; i = 1, \ldots, N_e$ from $\pi$.

References:

[1] Lorenz, E.N., 1996, September. Predictability: A problem partly solved. In Proc. Seminar on predictability (Vol. 1, No. 1).

[2] Baptista, R., Zahm, O., & Marzouk, Y. (2020). An adaptive transport framework for joint and conditional density estimation. arXiv preprint arXiv:2009.10303.

[3] Baptista, R., Marzouk, Y., Morrison, R.E. and Zahm, O., 2021. Learning non-Gaussian graphical models via Hessian scores and triangular transport. arXiv preprint arXiv:2101.03093.

```julia
using Revise
using LinearAlgebra
using AdaptiveTransportMap
using Statistics
using Distributions
using OrdinaryDiffEq
using ProgressMeter
```

    ┌ Info: Precompiling AdaptiveTransportMap [bdf749b0-1400-4207-80d3-e689c0e3f03d]
    └ @ Base loading.jl:1278
    ┌ Warning: Type annotations on keyword arguments not currently supported in recipes. Type information has been discarded
    └ @ RecipesBase ~/.julia/packages/RecipesBase/92zOw/src/RecipesBase.jl:116
    ┌ Warning: Type annotations on keyword arguments not currently supported in recipes. Type information has been discarded
    └ @ RecipesBase ~/.julia/packages/RecipesBase/92zOw/src/RecipesBase.jl:116

Load some packages to make nice figures

```julia
using Plots
default(tickfont = font("CMU Serif", 9),
        titlefont = font("CMU Serif", 14),
        guidefont = font("CMU Serif", 12),
        legendfont = font("CMU Serif", 10),
        grid = false)
# Plots.font("sans-serif")
# clibrary(:colorbrewer)
# gr()
pyplot()

using LaTeXStrings
# PyPlot.rc("text", usetex = "true")
# rcParams = PyPlot.PyDict(PyPlot.matplotlib."rcParams")
# rcParams["text.usetex"] = true;
PyPlot.rc("font", family = "CMU Serif")
PyPlot.matplotlib[:rc]("mathtext",fontset="cm")        #computer modern font
PyPlot.matplotlib[:rc]("font",family="serif",size=12)

using ColorSchemes
```

The Lorenz-96 model [1] is a famous problem used in data assimilation and weather prediction. It was derived from first principles as a one-dimensional model for the response of the mid-latitude atmosphere to forcing input. For certain forcing input, it can exhibit a chaotic behavior: sensitivity to initial conditions, strong mixing. In this notebook, we consider the fifteen-dimensional version of the Lorenz-96 model.  The state $\boldsymbol{x} = (x_1, \ldots, x_{15})$ at time $t$ is governed by the following set of ordinary differential equations:

\begin{equation}
    \frac{\mathrm{d}x_i}{\mathrm{d}t} = (x_{i+1} - x_{i-2}) x_{i-1} -x_i + F,
\end{equation}
where it is assumed that $x_{-1} = x_{n − 1} , x_0 = x_{n}$ and $x_{n+1} = x_1$, setting the forcing input $F=8.0$ leads to chaos.

We integrate the Lorenz-96 problem with our favorite ODE library: OrdinaryDiffEq.jl.

```julia
function lorenz96!(du,u,p,t)
    F = 8.0
    n = size(u,1)
    du[1] = (u[2]-u[end-1])*u[end] - u[1] + F
    du[2] = (u[3]-u[end])*u[1] - u[2] + F
    du[end] = (u[1] - u[end-2])*u[end-1] - u[end] + F

    @inbounds for i=3:n-1
        du[i] = (u[i+1] - u[i-2])*u[i-1] - u[i] + F
    end
    return du
end
```

    lorenz96! (generic function with 1 method)

```julia
# Define the distribution for initial condition
Nx = 20
# Number of runs
Ne = 500
πx0 = MvNormal(zeros(Nx), ones(Nx))
# Zero vector of the size of the state
x0 = rand(πx0)

# Time stepp
tspan = (0.0, 500.0)
prob = ODEProblem(lorenz96!,x0, tspan)
```

    ODEProblem with uType Array{Float64,1} and tType Float64. In-place: true
    timespan: (0.0, 500.0)
    u0: 20-element Array{Float64,1}:
     -0.5901971392068536
      0.39478770854432377
     -0.9782033574150046
     -0.22024619340106624
     -0.8394683233148887
      0.7852932777661026
     -0.8201363233707767
      1.2932492486959453
     -1.363092127868417
     -0.5814537134272362
      0.9780347874070602
      0.5766977457297677
      0.6590533906875842
      0.6669303405271328
      1.6714289713704404
     -1.0307067482720573
     -0.07787630196744173
     -0.7693031317820128
      0.08699657661085415
      1.2810036267757885

Create cache to store the different final solutions

```julia
X = zeros(Nx, Ne);
```

Solve Lorenz-96 model for the different initial conditions and store the solutions as the columns of `X`

```julia
@inbounds for i=1:Ne
    prob = remake(prob; u0 = rand(πx0))
    sol = solve(prob, Tsit5(), dt = 0.01, adaptive = false, dense = false, saveat = tspan[end], save_start = false)
    X[:,i] = deepcopy(sol.u[1])
end
```

We can use the adaptive transport map procedure to estimate the underlying distribution for the state $\mathsf{X}$.

```julia
λ = 0.0
δ = 1e-8
γ = 2.0

@time S = SparseRadialMap(Nx, -1; λ = λ, δ = δ, γ = γ)
```

      0.000041 seconds (951 allocations: 84.469 KiB)

    Sparse Radial Map of dimension Nx = 20 and order p = [[-1], [-1, -1], [-1, -1, -1], [-1, -1, -1, -1], [-1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]]
            with parameters (γ, λ, δ, κ) = (2.0, 0.0, 1.0e-8, 10.0)

```julia
using BenchmarkTools
```

```julia
@btime begin
    S = SparseRadialMap(Nx, -1; λ = λ, δ = δ, γ = γ)
    optimize(S, X, 2, 1, "kfolds"; verbose = false)
end
```

      930.119 ms (11386774 allocations: 1.49 GiB)

    Sparse Radial Map of dimension Nx = 20 and order p = [[1], [2, 1], [2, 2, 1], [2, 2, 2, 1], [2, -1, 2, 2, 1], [2, 2, 2, 2, 2, 1], [-1, 2, 2, 2, 2, -1, 1], [2, 2, 2, 2, 2, 2, 2, 1], [2, 2, 2, -1, 2, 2, 2, 2, 1], [2, -1, 2, -1, 2, 2, 2, 2, 2, 1], [-1, -1, -1, -1, -1, -1, -1, -1, 2, -1, 1], [2, -1, 2, -1, -1, -1, 2, 2, 2, 2, -1, 1], [2, -1, 2, -1, -1, -1, -1, -1, -1, 2, 2, -1, 1], [2, -1, 2, -1, -1, -1, -1, -1, -1, -1, -1, 2, -1, 1], [2, -1, 2, -1, -1, -1, -1, -1, -1, 2, -1, -1, 2, 2, 1], [2, 2, 2, -1, 2, -1, -1, -1, -1, 2, -1, -1, -1, 2, -1, 1], [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 2, -1, 1], [2, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 2, -1, 1], [2, 2, 2, -1, -1, -1, -1, -1, -1, 2, -1, -1, -1, -1, -1, -1, 2, 2, 1], [2, 2, 2, -1, -1, -1, -1, -1, -1, 2, -1, -1, -1, -1, -1, -1, -1, 2, -1, 1]]
            with parameters (γ, λ, δ, κ) = (2.0, 0.0, 1.0e-8, 10.0)

```julia
order = fill(-1, (Nx, Nx))

for i=1:Nx
    view(order,i,1:i) .= S.p[i]
end
```

```julia

```

```julia
plot(S)
```

```julia
heatmap(order, yflip = true, ratio = 1, color = cgrad([:white, :teal, :navyblue, :purple]))
savefig("radialmap_Lorenz96_structure.pdf")
```

```julia
SpushX = S(X)
```

    20×500 Array{Float64,2}:
     -0.339433  -0.204719   -0.875541     …  -0.655298   0.243134    0.776783
     -1.23045   -1.89291    -0.17724          0.887493  -1.42605     0.440173
     -0.965185  -0.6666     -0.397047         1.20678   -0.129728    0.516596
     -0.991359   1.91401     1.99899         -0.793543   1.60683     0.0285819
      0.729749   0.0217032   1.00762          0.642489  -1.16443    -1.50505
      1.12028    1.26267    -0.795134     …  -0.34673    0.842495   -0.42077
      0.293599   1.82212     0.645063         0.474779  -0.760496   -0.846265
      2.03736   -0.786459   -0.107551        -0.357772  -0.0945134  -0.182125
      1.07523    0.757135   -0.0218839       -1.89061    0.563188    0.119089
     -0.111753  -2.92356     0.26566         -0.285016  -2.22325    -1.64374
      1.44674   -1.27796    -1.37543      …   1.9629     0.0765255   0.203902
     -0.621854   0.862049   -0.371249        -1.91671   -1.18463    -1.22939
     -0.927222  -0.23515    -0.826029        -0.438996   0.32424     1.26891
     -0.629207   0.546993    1.20183          0.335005   1.36567    -1.21427
      0.689672  -0.880906    0.701013         0.038981  -0.651341    0.179489
      0.239564   0.0570641   1.06666      …  -1.10708    0.728878   -1.40957
     -1.10269    0.0201533   0.000251823      0.410028   1.10083     0.133739
     -1.7381     0.399525   -1.30633         -0.143572  -1.51489     1.86041
     -0.94101   -0.551241   -1.5296           1.64214    0.210762   -0.389787
      1.21916   -1.6561      2.79903          2.4752     0.0725534   0.681785

```julia
plt = plot(layout = grid(4,4), legend = false, ratio = 1)

for i=1:4
    for j=1:i
    scatter!(plt[i,j], SpushX[i,:], SpushX[j,:])
    end
end
plt
```

Baptista et al.[3] showed that the conditional independence properties of a random variable $\mathsf{X} \in \mathbb{R}^n$ with density $\pi$ are specified by the *Hessian score matrix* $\boldsymbol{\Omega} \in \mathbb{R}^{n \times n}$ defined as:

\begin{equation}
\Omega_{i,j} = \mathrm{E}_{\pi} \left[ |\partial_i\partial_j \log \pi(\boldsymbol{x})|^2  \right] = \int |\partial_i\partial_j \log \pi(\boldsymbol{x})|^2 \pi(\boldsymbol{x}) \; \mathrm{d}\boldsymbol{x}.
\end{equation}

The pullback density $\boldsymbol{S}^{\sharp} \rho$ can be used to approximate the true density $\pi$, where $\rho$ denotes the standard Gaussian distribution of $\mathbb{R}^n$. Thus, the score matrix is estimated by [3]:
\begin{equation}
\hat{\Omega}_{i,j} = \mathrm{E}_{\pi} \left[ |\partial_i\partial_j \log \boldsymbol{S}^{\sharp} \rho|^2  \right]
\end{equation}

In practice, we use a sampled-based approximation of $\hat{\boldsymbol{\Omega}}$ using samples $\{\boldsymbol{x}^k\}, \; k=1,\ldots, N_e$ of $\pi$ [3]:

\begin{equation}
\hat{\Omega}_{i,j} \approx \sum_{k=1}^{N_e}|\partial_i\partial_j \log \boldsymbol{S}^{\sharp}\rho(\boldsymbol{x}^k )|^2
\end{equation}

The function `hess_x_log_pdf!` computes in-place the Hessian of the log pdf of the pullback density $\boldsymbol{S}^{\sharp} \rho$. You can also use `hess_x_log_pdf`.

```julia
Ωhat = zeros(Nx, Nx)
cache = zeros(1, Nx, Nx)
@showprogress for i=1:Ne
    hess_x_log_pdf!(cache, S, X[:,i:i])
    Ωhat .+= copy(cache[1,:,:]).^2
end
rmul!(Ωhat, 1/Ne)
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:12

    15×15 Array{Float64,2}:
         4.90708e5    441.481     11414.6        …  9622.74         631.448
       441.481          7.1244e5    409.046         1393.69           0.0
     11414.6          409.046         3.75106e5     2151.42         106.842
        12.1883     12859.4          50.4824           4.80565        0.0
        14.2867      1203.55      19012.1            251.675          0.0
        26.0031      3786.77         16.3586     …     8.64355      202.812
       841.125         24.8467      246.088            0.0            0.0
        51.2645         0.962193    520.471            1.56015        7.82402
       101.178         10.657         4.8528           9.50035      268.254
         1.84552        9.39884      22.1943         421.593        164.217
       521.527          1.89079       1.40687    …    11.0213        61.5223
        68.5322       161.799       170.027         7150.05           7.64101
        69.152          2.46953       0.881189      2074.26        9123.54
      9622.74        1393.69       2151.42             1.92134e5   2236.33
       631.448          0.0         106.842         2236.33       83583.5

```julia
plt = plot(size = (800, 800))

heatmap!(plt,log10.(Ωhat), ratio = 1, yflip = true,
        colorbar = true, color = :plasma, clim = (1.5, Inf), colorbar_title = L"\log_{10}(\hat{\Omega})",
        xlabel = "Index", ylabel = "Index", background_color_inside = palette(:plasma)[1],
        yticks = (reverse(collect(0:10:Nx))))
plt
```

```julia

```

notebooks/Structure discovery Lorenz 96 radialmap.ipynb