# **The Modeling and Control Of Manipulators**
## **Goals for today**

---

During today practice we will:
*   Revise the Dynamics of Mechanical Systems.
*   Implement the simple multidof PD controller.
*   Study the gravity compensation and inverse dynamics (fb)

## **Dynamics of Manipulators**

Equation of motion for most mechanical systems may be written in following form:
\begin{equation}
\mathbf{Q} - \mathbf{Q}_d = \mathbf{D}(\mathbf{q})\ddot{\mathbf{q}} + \mathbf{C}(\mathbf{q},\dot{\mathbf{q}})\dot{\mathbf{q}} + \mathbf{g}(\mathbf{q})  = \mathbf{D}(\mathbf{q})\ddot{\mathbf{q}} + \mathbf{c}(\mathbf{q},\dot{\mathbf{q}}) + \mathbf{g}(\mathbf{q}) = \mathbf{D}(\mathbf{q})\ddot{\mathbf{q}} + \mathbf{h}(\mathbf{q},\dot{\mathbf{q}})
\end{equation}

where:
* $\mathbf{Q} \in \mathbb{R}^n $ - generalized forces corresponding to generilized coordinates
* $\mathbf{Q}_d \in \mathbb{R}^n $ - generalized disippative forces (for instance friction)
* $\mathbf{q} \in \mathbb{R}^{n}$ - vector of generilized coordinates
* $\mathbf{D} \in \mathbb{R}^{n \times n}$ - positive definite symmetric inertia matrix
* $\mathbf{C} \in \mathbb{R}^{n \times n}$ - describe 'coefficients' of centrifugal and Coriolis forces
* $\mathbf{g} \in \mathbb{R}^{n}$ - describes effect of gravity and other position depending forces
* $\mathbf{h} \in \mathbb{R}^n $ - combined effect of $\mathbf{g}$ and $\mathbf{C}$

#### **Lagrange-Euler Equations**

In order to find the EoM we will use the Lagrange-Euler equations:
\begin{equation}
{\frac  {d}{dt}}\left({\frac  {\partial \mathcal{L}}{\partial {\dot{\mathbf{q}}}_{i}}}\right)-{\frac  {\partial \mathcal{L}}{\partial \mathbf{q}_{i}}}=\mathbf{Q}_i -\frac{\partial \mathcal{R}}{\partial \mathbf{\dot{q}}_i},\quad i = 1,2,\dots, n
\end{equation}

where:
* $\mathcal{L}(\mathbf{q},\dot{\mathbf{q}}) \triangleq E_K - E_\Pi \in \mathbb{R}$ Lagrangian of the system
* $\mathcal{R} \in \mathbb{R}$ Rayleigh function  (describes energy dissipation)

### **Example: Remotely Driven Two Link Manipulator**

---

For instance let's consider remotely driven rigid 2-DoF manipulator:

#### **Energy**
Kinetic and potential energy of this system is equal to:
\begin{equation}
E_K = \frac{1}{2}\sum_{j=1}^{2}  m_j\dot{\mathbf{r}}_j^T\dot{\mathbf{r}}_j  +  J_j \dot{\alpha}^2_i, \quad E_\Pi = \sum_{j=1}^{n}  m_i g y_i
\end{equation}

Where $\mathbf{r}_j$ is Cartesian coordinates of mass $m_j$ calculated via forward kinematics as:
\begin{equation}
\mathbf{r}_1
=
\begin{bmatrix}
x_1 \\
y_1
\end{bmatrix}
=
\begin{bmatrix}
l_1 \cos \alpha_1 \\
l_1 \sin \alpha_1
\end{bmatrix}, \quad
\mathbf{r}_2
=
\begin{bmatrix}
x_2 \\
y_2
\end{bmatrix}
=
\begin{bmatrix}
l_1 \cos \alpha_1 + l_2 \cos \alpha_2 \\
l_1 \sin \alpha_1 + l_2 \sin \alpha_2
\end{bmatrix}
\end{equation}

Apart from kinetic and potential energy there are two dissipative elements in this system, namely "dampers" with coefficients $b_1, b_2$ (viscous friction), their dissipation function is given as:
\begin{equation}
\mathcal{R} = \frac{1}{2}\sum_{j=1}^{n}  b_j \dot{\alpha}^2_j
\end{equation}

Defining generalized coordinates as $\mathbf{q} = [\alpha_1,\alpha_2]^T$ and forces as $\mathbf{Q} = [u_1,u_2]^T$ one can apply Lagrange formalism to obtain equations of motion as:

\begin{equation}
\begin{cases}
I_1\ddot{\alpha}_1 + l_1^2 (m_1 + m_2) \ddot{\alpha}_1 + l_1 l_2 m_2 \cos(\alpha_1 - \alpha_2)\ddot{\alpha}_2 +
l_1 l_2 m_2 \sin(\alpha_1 - \alpha_2)\dot{\alpha}^2_2
+l_1 m_1 g \cos \alpha_1 + l_1 m_2 g \cos \alpha_2 + b_1 \dot{\alpha}_1 =u_1 \\
l_1 l_2 m_2 \cos(\alpha_1 - \alpha_2)\ddot{\alpha}_1 + I_2 \ddot{\alpha} + l_2^2 m_2 \ddot{\alpha}_2 - l_2 m_2 l_1 \sin(\alpha_1 - \alpha_2)\dot{\alpha}^2_1 + l_2 m_2 g \cos \alpha_2+ b_2 \dot{\alpha}_2 = u_2
\end{cases}
\end{equation}

Now we can find the $\mathbf{D}, \mathbf{C}, \mathbf{g}$ to be:
\begin{equation}
\mathbf{D} =
\begin{bmatrix}
l_1^2 (m_1 + m_2)+ J_1 & l_1 l_2 m_2  \cos(\alpha_1 - \alpha_2)\\
l_1 l_2 m_2 \cos(\alpha_1 - \alpha_2) & l_2^2 m_2+ J_2
\end{bmatrix}, \quad
\mathbf{Q}_d =
\begin{bmatrix}
b_1 \dot{\alpha}_1 \\
b_2 \dot{\alpha}_2
\end{bmatrix}
\end{equation}

\begin{equation}
\mathbf{c} = \begin{bmatrix}
l_1 l_2 m_2 \sin(\alpha_1 - \alpha_2)\dot{\alpha}^2_2 \\
- l_2 m_2 l_1 \sin(\alpha_1 - \alpha_2)\dot{\alpha}^2_1
\end{bmatrix}
, \quad
\mathbf{g} = \begin{bmatrix}
l_1 m_1 g \cos \alpha_1 + l_1 m_2 g \cos \alpha_1 \\
 l_2 m_2 g \cos \alpha_2
\end{bmatrix}
\end{equation}

### **Dynamics in State Space Form**
Choosing state as $\mathbf{x} = [\mathbf{x}_1,\mathbf{x}_2]^T= [\mathbf{q},\dot{\mathbf{q}}]^T$ and control as $\mathbf{Q} = [u_1,u_2]^T$ we can easily rewrite equations above as:
\begin{equation}
\dot{\mathbf{x}}
=
\begin{bmatrix}
\dot{\mathbf{x}}_1 \\
\dot{\mathbf{x}}_2
\end{bmatrix}
=
\begin{bmatrix}
\dot{\mathbf{q}} \\
\ddot{\mathbf{q}}
\end{bmatrix}
=
\begin{bmatrix}
\mathbf{x}_2 \\
\mathbf{D}(\mathbf{x}_1)^{-1} \big(\mathbf{u} - \mathbf{Q}_d -  \mathbf{h}(\mathbf{x}_1, \mathbf{x}_2) \big)
\end{bmatrix}
\end{equation}

Once system is written in state space form - one can use conventional ODE solvers to simulate it.

### **Tasks:**

1. Implement the dynamics of two link manipulator and model the free response
2. Propose the PD regulator to regulate the state to different desired values
3. Introduce the gravity compensation
4. Change desired trajectory to be the harmonical signal with different frequancies and amplitudes, discuss the response
5. Implement the inverse dynamics controller. How did the response change?
6. Vary the load parameter for simulation while keeping the same value for controller, discuss the change in system response both for regulation and tracking.

### **System Parameters:**

```python
# Manipulator parameters
l = 0.3, 0.3
m = 0.5, 3.0
J = 0.01, 0.01
b = 0.02, 0.02
g = 9.81
params = l, m, J, b, g
```


# Fundamentals of Data Science
Winter Semester 2020

## Prof. Fabio Galasso, Alessandro Flaborea and Luca Franco
<galasso@di.uniroma1.it>, <flaborea.1841969@studenti.uniroma1.it>, <franco.luca.17@gmail.com>

## Exercise 2

In Exercise 2, you will re-derive and implement logistic regression and optimize the parameters with Gradient Descent and with the Newton's method. Also, in this exercise you will re-derive and implement Gassian Discriminant Analysis.
We will use the files logistic_x.txt and logistic_y.txt. The first contains the feature values $x^{(i)}_1$ and $x^{(i)}_2$ for the $i$-th data sample $x^{(i)}$. The second contains the ground truth label $y^{(i)}$ for each corresponding data sample.

Note that, differently from what seen in lecture, $y^{(i)}\in \{-1,1\}$.

The completed exercise should be handed in as a single notebook file. Use Markdown to provide equations. Use the code sections to provide your scripts and the corresponding plots.
Submit it by sending an email to galasso@di.uniroma1.it, flaborea.1841969@studenti.uniroma1.it and franco.luca.17@gmail.com by Wednesday November 25th, 23:59.

## Notation

- $x^i$ is the $i^{th}$ feature vector
- $y^i$ is the expected outcome for the $i^{th}$ training example
- $m$ is the number of training examples
- $n$ is the number of features

Let's start by setting up our Python environment and importing the required libraries:

## [15 points] Question 1: Logistic Regression with Gradient Ascent

### (a) [5 points] Equations for the log likelihood, its gradient, and the gradient ascent update rule.

Write and simplify the likelihood $L(\theta)$ and log-likelihood $l(\theta)$ of the parameters $\theta$.

Recall the probabilistic interpretation of the hypothesis $h_\theta(x)= P(y=1|x;\theta)$ and that $h_\theta(x)=\frac{1}{1+\exp(-\theta^T x)}$.

Also derive the gradient $\frac{\delta l(\theta)}{\delta \theta_j}$ of $l(\theta)$ and write the gradient update equation.

Question: in order to learn the optimal $\theta$, do you need to minimize or to maximize $l(\theta)$?

################# Do not write above this line #################

### Exercise one

\begin{align}
P(z|x^{(i)}, \theta) = {h_\theta(x^{(i)}))^{z^{(i)}}(1-h_\theta(x^{(i)}))^{1-{z^{(i)}}}}
\end{align}

Due to $h_\theta$ being Independent and Identically Distributed:

\begin{equation*}
L(\theta)= \prod_{i=1}^{m}{h_\theta(x^{(i)}))^{z^{(i)}}(1-h_\theta(x^{(i)}))^{1-{z^{(i)}}}}
\end{equation*}

\begin{equation*}
h_\theta(x^{(i)})= \frac{1}{1 + \exp(-\theta^{T}x^{(i)})}
\end{equation*}

\begin{align}
L(\theta)= \prod_{i=1}^{m}\bigg[\frac{1}{1 + \exp(-\theta^{T}x^{(i)})}\bigg]^{z^{(i)}}
\bigg[1- \frac{1}{1 + \exp(-\theta^{T}x^{(i)})}\bigg]^{1-z^{(i)}}
=\prod_{i=1}^{m}\bigg[\frac{1}{1 + \exp(-\theta^{T}x^{(i)})}\bigg]^{z^{(i)}}
\bigg[\frac{exp(-\theta^{T}x^{(i)})}{1 + \exp(-\theta^{T}x^{(i)})}\bigg]^{1-z^{(i)}}=
\end{align}

\begin{align}
=\prod_{i=1}^{m}\bigg[\frac{1}{1 + \exp(-\theta^{T}x^{(i)})}\bigg]^{z^{(i)}}
\bigg[\frac{exp(-\theta^{T}x^{(i)})}{1 + \exp(-\theta^{T}x^{(i)})}\bigg]
\bigg[\frac{1 + \exp(-\theta^{T}x^{(i)})}{exp(-\theta^{T}x^{(i)})}\bigg]^{z^{(i)}}=
\end{align}

\begin{align}
 =\prod_{i=1}^{m}\bigg[\frac{exp(-\theta^{T}x^{(i)})}{1 + \exp(-\theta^{T}x^{(i)})}\bigg]
 \bigg[\frac{1}{\exp(-\theta^{T}x^{(i)})}\bigg]^{z^{(i)}}
\end{align}

\begin{align}
=\prod_{i=1}^{m}\biggl(\frac{1}{1 + \exp(-\theta^{T}x^{(i)})}\biggl){(exp(-\theta^{T}x^{(i)}}))^{1-z^{(i)}}
\end{align}

### We can obtain $l_\theta$ following two ways

***_first_***

\begin{align}
l(\theta) = \log[L(\theta)] = \sum_{i=1}^{m}\log \biggl[ \frac{1}{1 + \exp(-\theta^{T}x^{(i)})}\biggl] +
(1-z^{(i)})\log(exp(-\theta^{T}{x^{(i)}}))=
\end{align}

\begin{align}
\sum_{i=1}^{m} - \log(1+exp(-\theta^{T}{x^{(i)}}))-({1-z^{(i)})(\theta^{T}{x^{(i)}}})
\end{align}

\begin{align}
=\sum_{i=1}^{m}\theta^{T}{x^{(i)}}({z^{(i)}-1)} - log(1 + exp(-\theta^{T}{x^{(i)}}))
\end{align}

***_second_***

\begin{align}
l(\theta) = \log[L(\theta)] = \log\prod_{i=1}^{m}{h_\theta(x^{(i)}))^{z^{(i)}}(1-h_\theta(x^{(i)}))^{1-{z^{(i)}}}}
\end{align}

\begin{align}
l(\theta) = \sum_{i=1}^{m}z^{(i)}\log(h_\theta{x^{(i)}}) + (1-{z^{(i)}})\log(1-h_\theta{x^{(i)}}))=
\end{align}

\begin{align}
= \sum_{i=1}^{m}\log (1-h_\theta{x^{(i)}})+ {z^{(i)}}[\log(h_\theta({x^{(i)}}))-
\log(1-h_\theta({x^{(i)}}))]=
\end{align}

\begin{align}
= \sum_{i=1}^{m}\log(1-h_\theta({x^{(i)}}))+ {z^{(i)}}\log\bigg[\frac{h_\theta({x^{(i)}})}{1-h_\theta({x^{(i)}})}\bigg]=
\end{align}

\begin{align}
= \sum_{i=1}^{m}\log\bigg[1- \frac{1}{1+exp(-\theta^{T}{x^{(i)})}}\bigg]+
{z^{(i)}}\log\bigg[\frac{1}{exp(-\theta^{T}{x^{(i)})}}\bigg]=
\end{align}

\begin{align}
= \sum_{i=1}^{m}\log\bigg[\frac{exp(-\theta^{T}{x^{(i)}})}{1+exp(-\theta^{T}{x^{(i)}})}\bigg]+
{z^{(i)}}(\theta^{T}{x^{(i)}})=
\end{align}

\begin{align}
= \sum_{i=1}^{m} -\theta^{T}{x^{(i)}} -log(1+exp(-\theta^{T}{x^{(i)}}) + {z^{(i)}}[\theta^{T}{x^{(i)}}]
\end{align}

\begin{align}
=\sum_{i=1}^{m}\theta^{T}{x^{(i)}}({z^{(i)}-1)} - log(1 + exp(-\theta^{T}{x^{(i)}}))
\end{align}

Both approaches lead to the same result

##### Since in our specific case z can take values {-1, 1}, and the above equation only holds for cases in which z takes values {0, 1} we would simply apply the following transformation of z:

\begin{align}
z^{(i)} = \frac{y^{(i)}}{2} + \frac{1}{2}
\end{align}

### Derivative of Log-Likelyhood

\begin{align}
l(\theta) = \sum_{i=1}^{m} [\theta^{T}{x^{(i)}}({z^{(i)}-1)}]
- \log (1+ exp(-\theta^{T}{x^{(i)}}))
\end{align}

Splitting up the derivatives into parts.

A.
\begin{align}
l(\theta)= \theta^{T}{x^{(i)}}({z^{(i)}-1})
\end{align}

\begin{align}
\frac{\partial l(\theta)}{\partial \theta_{j}} = {x^{(i)}_{j}}({z^{(i)}-1})
\end{align}

B.
\begin{align}
l(\theta)= \log(1+exp({-\theta^{T}{x^{(i)}}}))
\end{align}

\begin{align}
\frac{\partial l(\theta)}{\partial \theta_{j}} = \frac{1}{(1 + exp(-{\theta^{T}{x^{(i)}}}))}
(-x^{(i)}_{j}exp({-\theta^{T}x^{(i)}}))=
\end{align}

\begin{align}
=\frac{-x^{(i)}_{j}}{(exp({\theta^{T}{x^{(i)}}})+1)}
\end{align}

\begin{align}
\nabla l(\theta) = \sum_{i=1}^{m} \bigg[{x^{(i)}_{j}}({z^{(i)}}-1)+ \frac{{x^{(i)}_{j}}}
{1+exp({\theta^{T}x^{(i)}})}\bigg]
\end{align}

***We can show that the partial derivative of $l(\theta)$ is equal to the result observed in class:***

\begin{align}
\frac{\partial l(\theta)}{\partial \theta_j}= \sum_{i=1}^{m}\biggl[{x_j^{(i)}}({z^{(i)}}-1)+{x_j^{(i)}}
\biggl(\frac{1}{1 + exp({\theta^{T}x^{(i)}})}+1-1\biggl)\biggl]= \\
=\sum_{i=1}^{m}\biggl[{x_j^{(i)}} - {x_j^{(i)}}+ {x_j^{(i)}}{z^{(i)
}}+{x_j^{(i)}} \biggl( \frac{1}{1+exp({ \theta^{T}x^{(i)}})}-1 \biggl)\biggl]=
\end{align}

\begin{align}
=\sum_{i=1}^{m}\biggl[ {x_j^{(i)}}{z^{(i)
}}+{x_j^{(i)}} \biggl( \frac{1-1-exp({\theta^{T}x^{(i)}})}{1+exp({\theta^{T}x^{(i)}})} \biggl)\biggl]=
\end{align}

\begin{align}
=\sum_{i=1}^{m}\biggl[ {x_j^{(i)}}{z^{(i)
}}-{x_j^{(i)}} \biggl( \frac{1}{1+exp({-\theta^{T}x^{(i)}})} \biggl)\biggl]
\end{align}

#### Gradient Update Equation

\begin{align}
\theta_{( t+1)}:= \theta_{( t)}+ {\alpha}\cdot \nabla l(\theta)
\end{align}

The log-likelihood is proportional to the probability of 𝑃(y|𝑥(𝑖),𝜃) (probability of getting a target value 1 or 0 given a specific feature), which is a quantiy we want to maximize. Therefore, we look for a $ {\theta} $ that maximizes the log-likelihood.

################# Do not write below this line #################

### (b) [10 points] Implementation of logistic regression with Gradient Ascent

Code up the equations above to learn the logistic regression parameters. Consider as data the files logistic_x.txt and logistic_y.txt. The first contains the feature values $x^{(i)}_1$ and $x^{(i)}_2$ for the $i$-th data sample $x^{(i)}$. The second contains the ground truth label $y^{(i)}$ for each corresponding data sample.

```python
%matplotlib inline
import numpy as np # imports a fast numerical programming library
import scipy as sp # imports stats functions, amongst other things
import matplotlib as mpl # this actually imports matplotlib
import matplotlib.cm as cm # allows us easy access to colormaps
import matplotlib.pyplot as plt # sets up plotting under plt
import pandas as pd # lets us handle data as dataframes

# sets up pandas table display
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)

import seaborn as sns # sets up styles and gives us more plotting options
```

Let's start by loading the data into two pandas DataFrames:

```python
df_x = pd.read_csv("./data/logistic_x.txt", sep="\ +", names=["x1","x2"], header=None, engine='python')
df_y = pd.read_csv('./data/logistic_y.txt', sep='\ +', names=["y"], header=None, engine='python')
df_y = df_y.astype(int)
df_x.head()
```

      x1
      x2

      0
      1.343250
      -1.331148

      1
      1.820553
      -0.634668

      2
      0.986321
      -1.888576

      3
      1.944373
      -1.635452

      4
      0.976734
      -1.353315

```python
df_y.head()
```

      y

      0
      -1

      1
      -1

      2
      -1

      3
      -1

      4
      -1

Let's get the NumPy arrays from the dataset, and add a column of 1's to $X$ to take into account the zero intercept

```python
x = np.hstack([np.ones((df_x.shape[0], 1)), df_x[["x1","x2"]].values])
y = df_y["y"].values
```

Adjust y so that $y^{(i)}\in [0,1]$

################# Do not write above this line #################

```python
y[y == -1] = 0
```

################# Do not write below this line #################

```python
[x[:5,:],x[-5:,:]] # Plot the first and last 5 lines of x, now containing features x0 (constant=1), x1 and x2
```

    [array([[ 1.        ,  1.3432504 , -1.3311479 ],
            [ 1.        ,  1.8205529 , -0.6346681 ],
            [ 1.        ,  0.98632067, -1.8885762 ],
            [ 1.        ,  1.9443734 , -1.635452  ],
            [ 1.        ,  0.97673352, -1.3533151 ]]),
     array([[ 1.        ,  4.7748538 ,  0.09941521],
            [ 1.        ,  5.8274854 , -0.69005848],
            [ 1.        ,  2.2894737 ,  1.9707602 ],
            [ 1.        ,  2.494152  ,  1.4152047 ],
            [ 1.        ,  2.0847953 ,  1.3567251 ]])]

```python
[y[:5],y[-5:]] # Plot the first and last 5 lines of y
```

    [array([0, 0, 0, 0, 0]), array([1, 1, 1, 1, 1])]

Define the sigmoid function "sigmoid", the function to compute the gradient of the log likelihood  "grad_l" and the gradient ascent algorithm.

################# Do not write above this line #################

Your code here

```python
def sigmoid(x):
    return 1/(1 + np.exp(-x))

def log_likelihood(theta,x,y):
    theta_x=np.dot(x,theta)
    log_l=theta_x*(y-1)-np.log(1+np.exp(-theta_x))
    log_l=np.mean(log_l)
    return log_l

def grad_l(theta, x, y):

    grad1=np.dot(y,x)
    theta = np.array(theta)
    theta=theta[np.newaxis].T
    t=np.dot(x,theta)
    g=sigmoid(t).T
    grad2=np.dot(g,x)
    G=grad1-grad2
    return G[0]

def gradient_ascent(theta,x,y,G,alpha=0.01,iterations=100):

    m = len(y)
    log_l_history = np.zeros(iterations)
    theta_history = np.zeros((iterations,len(theta)))
    G=G(theta,x,y)
    prod=alpha*G
    opttheta=theta+prod.T

    for i in range(iterations):
        log_l=log_likelihood(opttheta,x,y)
        log_l_history[i]=log_l
        theta_history[i]=opttheta
        G=grad_l(opttheta,x,y)
        prod=alpha*G
        opttheta+=(prod.T)

    return opttheta,log_l_history,theta_history

```

################# Do not write below this line #################

Check your grad_l implementation:
grad_l applied to the theta_test (defined below) should provide a value for log_l_test close to the target_value (defined below); in other words the error_test should be 0, up to machine error precision.

```python
target_value = -0.4250958770469834
theta_test=np.array([-2,1,2])

log_l_test  = log_likelihood(theta_test,x,y)
error_test=np.abs(log_l_test-target_value)

print("{:f}".format(error_test))
```

    41.659396

Let's now apply the function gradient_ascent and print the final theta as well as theta_history

```python
# Initialize theta0
theta0 = np.zeros(x.shape[1])

# Run Gradient Ascent method
n_iter=1000
theta_final, log_l_history, theta_history = gradient_ascent(theta0,x,y,grad_l,alpha=0.5,iterations=n_iter)
print(theta_final)
print(theta_history)
```

    [-78.99906711  20.98908291  41.73644465]
    [[ -0.25        23.83452377  27.46411597]
     [-17.62210474 -34.99184238  45.56598762]
     [  5.43334214  83.44009046  60.61944202]
     ...
     [-81.14434649  22.73798727  57.18655423]
     [-80.16218377  24.97061918  51.20849424]
     [-80.03179503  21.7793511   46.30917205]]

Let's plot the log likelihood over iterations

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)),log_l_history,'b.')
```

Plot the data and the decision boundary:

```python
df_x_pl=df_x.copy()
df_y_pl=df_y.copy()
df_x_pl.insert(0, "y", df_y)
df_x_pl["y"] = pd.to_numeric(df_x_pl["y"], downcast='signed')
df_x_pl.head()
```

      y
      x1
      x2

      0
      0
      1.343250
      -1.331148

      1
      0
      1.820553
      -0.634668

      2
      0
      0.986321
      -1.888576

      3
      0
      1.944373
      -1.635452

      4
      0
      0.976734
      -1.353315

```python
# Generate vector to plot decision boundary
x1_vec = np.linspace(df_x_pl["x1"].min(),df_x_pl["x1"].max(),2);

# Plot raw data
sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

# Plot decision boundary
plt.plot(x1_vec,(-x1_vec*theta_final[1]-theta_final[0])/theta_final[2], color="red");

# Save the theta_final value for later comparisons
theta_GA = theta_final.copy()
```

################# Do not write above this line #################

Write now your considerations. Discuss in particular:
- what effects do the learning rate $\alpha$ and the number of iterations have on the computed $\theta$
- may you still achieve the same value if you decrease the learning rate $\alpha$ and increase the number of iterations n_iter up to an infinite number of them? Why?
- may you still achieve the same value if you increase the learning rate $\alpha$ substantially and decrease the number of iterations n_iter? Why?

Substantiate your answer with experiments and plots of the $l(\theta)$ and the decision boundary.

##### Answers

In order to answer these questions, we carry out some experiments and plot the results. In the experiments, we will compute the $\theta$ based on different values of $\alpha$ (0.5, 0.05 and 0.005) and different number of iterations (100, 1000 and 10,000).

```python
alphas=[.5,.1,.005]
iterations = [100, 1000, 10000]
fig, ax = plt.subplots(ncols=len(alphas), nrows=len(iterations), figsize=(15,15))
ax = ax.flatten()
for a in range(len(alphas)):
    for i in range(len(iterations)):
        theta_final, log_l_history, theta_history = gradient_ascent(theta0,x,y,grad_l,alpha=alphas[a],iterations=iterations[i])
        axes = ax[(a*3)+i]
        axes.plot(range(len(log_l_history)),log_l_history,'b.')
        axes.set_title('Alpha: ' + str(alphas[a]) + '\n' + 'Iterations: ' + str(iterations[i]))
        plt.tight_layout()

```

By observing the plots above, we can come to the conclusion that the smaller $\alpha$, the more accurate the algorithm is, therefore the better results for $l(\theta)$ we get. As we saw in class, this happens because a higher value of $\alpha$ may fail to identify the optimal value, as it bounces around the curve described by $l(\theta)$, while a smaller $\alpha$ slowly analyses this path little by little. Naturally, this guarantees finding a good value for $\theta$ but at the cost of a longer computing time, which is the reason why finding a high enough value of $\alpha$ that does not overshoot is so important. In other words, there is a loss in efficiency that comes with a lower value of $\alpha$, since more steps are required to reach the maximum.

Moreover, we observe that for the iterations it is the better to have a bigger number of them in order to reach a better result for $l(\theta)$, although this implies a longer computational time.

Finally, by looking at the plots it appears clear that no number of iterations could ever reach the efficiency of a good $\alpha$ (that is, a very small $\alpha$). This answers the second and third questions: a very small amount of iterations (100) when combined with a low value of $\alpha$ (0.005) gives way better results for $l(\theta)$ compared to those obtained with a large number of iterations (10,000) but a greater $\alpha$ (0.5).

```python
alphas=[.5,.1,.005]
iterations = [100, 1000, 10000]
fig, ax = plt.subplots(ncols=len(alphas), nrows=len(iterations), figsize=(15,15))
ax = ax.flatten()
for a in range(len(alphas)):
    for i in range(len(iterations)):
        theta_final, log_l_history, theta_history = gradient_ascent(theta0,x,y,grad_l,alpha=alphas[a],iterations=iterations[i])
        print(theta_final)
        x1_vec = np.linspace(df_x_pl["x1"].min(),df_x_pl["x1"].max(),2);

        # Plot raw data
        axes=ax[(a*len(iterations))+i]
        sns.scatterplot(ax = axes, x="x1", y="x2", hue="y", data=df_x_pl);

        # Plot decision boundary

        axes.plot(x1_vec,(-x1_vec*theta_final[1]-theta_final[0])/theta_final[2], color="red");
        axes.set_title('Alpha: ' + str(alphas[a]) + '\n' + 'Iterations: ' + str(iterations[i]))
        plt.tight_layout()
```

The same conclusions can be drawn by observing the graphs above.

Firstly, it is clear to see that the approximation of $\theta$ for a high value of $\alpha$ (0.5) is not at all satisfactory. Such a learning rate would probably necessitate of an infinite amount of iterations, which of course is not conceivable in practice and even then the final value may not be adequate.

On the contrary, for a smaller $\alpha$ (0.05) we can distinctly observe how more iterations lead to better and better results.

Finally, for the best value of $\alpha$ we found in the last experiment (0.005) we obtain the same approximation for 1,000 and 10,000 iterations. This implies that a lower value of $\alpha$, although slower to compute, ensures we will reach the optimal theta values.

################# Do not write below this line #################

## [15 points] Question 2: Logistic Regression with the Newton's method

### (a) [5 points] Equations for the Hessian of the log likelihood

Derive the equation for the Hessian matrix $H$ of the log-likelihood $l(\theta)$ of the parameters $\theta$.

Recall that: $H_{i,j} = \frac{\delta^2 l(\theta)}{\delta \theta_i \delta \theta_j}$

Also write the parameter update rule for the Newton's method, as a function of the Hessian and gradient of $l(\theta)$.

Question: in order to learn the optimal $\theta$, do you need to minimize or to maximize $\nabla_{\theta} l$?

################# Do not write above this line #################

### Hessian Matrix

\begin{align}
H_{kj} = \frac{\partial^{2} l(\theta)}{\partial \theta_{k} \partial \theta_{j}}
\end{align}

\begin{align}
\frac{\partial l(\theta)}{\partial \theta_{j}} = l'(\theta)=
\sum_{i=1}^{m}x_j^{(i)} \biggl[ z^{(i)}- \frac{1}{1+exp({ -\theta^{T} }x^{(i)})}\biggl]
\end{align}

\begin{align}
\frac{\partial^{2} l(\theta)}{\partial \theta_{j} \partial \theta_{k}}=
\frac{\partial }{\partial \theta_{k} }\biggl[\frac{\partial l(\theta)}{\partial \theta_j} \biggl]= \sum_{i=1}^{m}x_j^{(i)}\biggl [\frac{\partial}{\partial \theta_k}(1+exp({-\theta^{T}}x^{(i)}))^{-1} \biggl] =
\end{align}

\begin{align}
\sum_{i=1}^{m} x_j^{(i)} \biggl [(1+exp({ \theta^{T}x^{(i)}}))^{-2}(x_k^{(i)})exp({-\theta^{T}{x^{(i)}})})\biggl] = \sum_{i=1}^{m}x_j^{(i)}x_k^{(i)} \biggl [-\frac{exp({-\theta^{T}}x^{(i)})}{(1+exp({{-\theta^{T}}{x^{(i)}}))^2}}\biggl]
\end{align}

### Newton's Method

\begin{align}
\theta^{( t+1)}:= \theta^{( t)}- {H}^{(-1)}\nabla_{\theta} l
\end{align}

Maximizing or minimizing the gradient of $l(\theta)$ gives the same result since we want $\nabla_{\theta} l$  to be equal 0. Therefore, in order to learn optimal $\theta$, we can maximize or minimize $\nabla_{\theta} l$.

################# Do not write below this line #################

### (b) [10 points] Implementation of logistic regression with the Newton's method

Code up the equations above to learn the logistic regression parameters. Consider as data the files logistic_x.txt and logistic_y.txt. The first contains the feature values  $x^{(i)}_1$ and $x^{(i)}_2$  for the  𝑖 -th data sample  $x^{(i)}$ . The second contains the ground truth label  $y^{(i)}$  for each corresponding data sample.

Define the Hessian function hess_l and the Newton's method function "newton":

################# Do not write above this line #################

Your code here

```python
def hess_l(theta, x, y):
    hess = np.zeros((len(x[0]),len(x[0])))
    hess_j_k=0
    for j in range(len(x[0])):
        for k in range(len(x[0])):
            for i in range(len(x)):
                hess_j_k += x[i][j]*x[i][k]*((np.exp(np.dot(np.transpose(-theta),x[i]))/((1+np.exp(np.dot(np.transpose(-theta),x[i])))**2)))
            hess[j][k]= -hess_j_k
            hess_j_k=0
    return hess

def newton(theta0, x, y, G, H, eps):
    keep=0
    llikelihood = log_likelihood(theta0, x,y)
    log_l_history = [llikelihood]
    theta_history = [theta0]
    gradient = G(theta0, x, y)
    hessian = H(theta0, x, y)
    hessian_inv = np.linalg.inv(hessian)
    product = np.dot(hessian_inv,gradient)
    theta =theta0 - product
    keep +=1
    theta_history.append(theta)
    llikelihood = log_likelihood(theta, x,y)
    log_l_history.append(llikelihood)
    while abs(log_l_history[keep]-log_l_history[keep-1]) > eps:
        keep +=1
        gradient = G(theta, x, y)
        hessian_inv = np.linalg.inv(H(theta, x, y))
        product = np.dot(hessian_inv,gradient)
        theta = theta - product
        llikelihood = log_likelihood(theta, x,y)
        theta_history.append(theta)
        log_l_history.append(llikelihood)
    return theta, theta_history, log_l_history

```

################# Do not write below this line #################

Check your hess_l implementation: hess_l applied to the theta_test (defined below) should provide a value for hess_l_test close to the target_value (defined below); in other words the error_test should be 0, up to machine error precision.

```python
target_value = np.array([[-0.14357024, -0.43836134,  0.18767787], \
                         [-0.43836134, -1.72295801,  0.61817001], \
                         [ 0.18767787,  0.61817001, -0.51247901]])
theta_test=np.array([-2,1,2])

hess_l_test  = hess_l(theta_test,x,y)
error_test=np.sum(np.abs(hess_l_test-target_value),axis=(0,1))

print("{:f}".format(error_test))
```

    191.932253

Now run Newton's method to perform logistic regression:

```python
# Initialize theta0
theta0 = np.zeros(x.shape[1])

# Run Newton's method
theta_final, theta_history, log_l_history = newton(theta0,x,y,grad_l,hess_l,1e-6)
print(theta_final)
```

    [-2.62050954  0.76037096  1.17194549]

```python
theta_history
```

    [array([0., 0., 0.]),
     array([-1.50983811,  0.43509696,  0.62161752]),
     array([-2.21834632,  0.64372727,  0.95944716]),
     array([-2.55431051,  0.74137714,  1.13493588]),
     array([-2.61847133,  0.75979248,  1.1707512 ]),
     array([-2.62050954,  0.76037096,  1.17194549])]

Let's plot the log likelihood over iterations

```python
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history)),log_l_history,'b.')
```

Plot the data and our decision boundary:

```python
df_x_pl=df_x.copy()
df_y_pl=df_y.copy()
df_x_pl.insert(0, "y", df_y)
df_x_pl["y"] = pd.to_numeric(df_x_pl["y"],downcast='signed')
df_x_pl.head()
```

      y
      x1
      x2

      0
      0
      1.343250
      -1.331148

      1
      0
      1.820553
      -0.634668

      2
      0
      0.986321
      -1.888576

      3
      0
      1.944373
      -1.635452

      4
      0
      0.976734
      -1.353315

```python
# Generate vector to plot decision boundary
x1_vec = np.linspace(df_x_pl["x1"].min(),df_x_pl["x1"].max(),2);

# Plot raw data
sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

# Plot decision boundary
plt.plot(x1_vec,(-x1_vec*theta_final[1]-theta_final[0])/theta_final[2], color="red");
```

################# Do not write above this line #################

Write now your considerations. Discuss in particular:
- the different convergence speed of the Newton's method, compared to Gradient Ascent
- the incurred cost for the different convergence speed

```python
from sklearn.metrics import confusion_matrix
import time

# Initialize theta0
theta0 = np.zeros(x.shape[1])

time_start_ga = time.clock()

# Run Gradient Ascent method
n_iter=1000
theta_final_ga, log_l_history_ga, theta_history_ga = gradient_ascent(theta0,x,y,grad_l,alpha=0.5,iterations=n_iter)

time_elapsed_ga = (time.clock() - time_start_ga)

time_start_n = time.clock()

# Run Newton's method
theta_final_n, theta_history_n, log_l_history_n = newton(theta0,x,y,grad_l,hess_l,1e-6)

time_elapsed_n = (time.clock() - time_start_n)

print("Computation time of the Gradient Ascent method is: " + str(time_elapsed_ga))
print("Computation time of the Newton's method is: " + str(time_elapsed_n))
print("The Newton's method is "+ str(time_elapsed_ga/time_elapsed_n)+ " times faster than the Gradient Ascent method.")
```

    [-1.50983811  0.43509696  0.62161752]
    [-2.21834632  0.64372727  0.95944716]
    [-2.55431051  0.74137714  1.13493588]
    [-2.61847133  0.75979248  1.1707512 ]
    [-2.62050954  0.76037096  1.17194549]
    Computation time of the Gradient Ascent method is: 0.09631220000005669
    Computation time of the Newton's method is: 0.006668799999943076
    The Newton's method is 14.442208493413927 times faster than the Gradient Ascent method.

    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel_launcher.py:7: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead
      import sys
    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel_launcher.py:13: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead
      del sys.path[0]
    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel_launcher.py:15: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead
      from ipykernel import kernelapp as app
    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel_launcher.py:20: DeprecationWarning: time.clock has been deprecated in Python 3.3 and will be removed from Python 3.8: use time.perf_counter or time.process_time instead

```python
from sklearn.metrics import confusion_matrix
y_predict_gradient_ascent = sigmoid(np.dot(x,theta_final_ga))
y_predict_gradient_ascent = (y_predict_gradient_ascent >= 0.5).astype(int)
y_predict_newton = sigmoid(np.dot(x,theta_final_n))
y_predict_newton = (y_predict_newton >= 0.5).astype(int)
cnf_matrix_1 = confusion_matrix(y, y_predict_gradient_ascent)
cnf_matrix_2 = confusion_matrix(y, y_predict_newton)
print("Confusion matrix of the Gradient Ascent method is: ")
print("------------------------------------------------------")
print(cnf_matrix_1)
print("Confusion matrix of the Newton's method is: ")
print("------------------------------------------------------")
print(cnf_matrix_2)
```

    Confusion matrix of the Gradient Ascent method is:
    ------------------------------------------------------
    [[46  4]
     [ 6 43]]
    Confusion matrix of the Newton's method is:
    ------------------------------------------------------
    [[45  5]
     [ 6 43]]

According to the computational time results for the Newton's and the Gradient Ascent methods, Newton's method is found slightly faster for our example compared to the Gradient Ascent method. These calculations are made with the same initial theta values. While Gradient Ascent stops iterating once reached the total number of iterations, Newton's method terminates finding the absolute difference between log-likelihood values less than the epsilon. Although Newton's method reaches the desired value with epsilon 1e-6 in 5 iterations, Gradient Ascent computes until 1000 iterations.
The similar computational time is due to the fact that the Newton has to compute the inverse of the hessian matrix, an NP-Problem algorithm.
From the graphs shows 'l(Theta)' and 'Iterations', it can be observed that Gradient Ascent has some dispersions in spite of Newton's has always non-decreasing approach.
From the confusion matrix point of view, cost of computation saves 0.01 higher accuracy rate.

################# Do not write below this line #################

## [7 points] Question 3: Logistic Regression with non linear boundaries

### (a) Polynomial features for logistic regression

Define new features, e.g. of 2nd and 3rd degrees, and learn a logistic regression classifier by using the new features, by using the Newton's optimization algorithm (or the gradient ascent one, if you like).

In particular, we would consider a polynomial boundary with equation:

$f(x_1, x_2) = c_0 + c_1 x_1 + c_2 x_2 + c_3 x_1^2 + c_4 x_2^2 + c_5 x_1 x_2 + c_6 x_1^3 + c_7 x_2^3 + c_8 x_1^2 x_2 + c_9 x_1 x_2^2$

We would therefore compute 7 new features: 3 new ones for the quadratic terms and 4 new ones for the cubic terms.

Create new arrays by stacking x (from the previous questions) and the new 7 features (in the order x1x1, x2x2, x1x2, x1x1x1, x2x2x2, x1x1x2, x1x2x2). In particular create x_new_quad by additionally stacking with x the quadratic features, and x_new_cubic by additionally stacking with x the quadratic and the cubic features.

```python
# First extract features x1 and x2 from x and reshape them to x1 vector arrays
x1 = x[:,1]
x2 = x[:,2]
x1 = x1.reshape(x1.shape[0], 1)
x2 = x2.reshape(x2.shape[0], 1)
print(x[:5,:]) # For visualization of the first 5 values
print(x1[:5,:]) # For visualization of the first 5 values
print(x2[:5,:]) # For visualization of the first 5 values
```

    [[ 1.          1.3432504  -1.3311479 ]
     [ 1.          1.8205529  -0.6346681 ]
     [ 1.          0.98632067 -1.8885762 ]
     [ 1.          1.9443734  -1.635452  ]
     [ 1.          0.97673352 -1.3533151 ]]
    [[1.3432504 ]
     [1.8205529 ]
     [0.98632067]
     [1.9443734 ]
     [0.97673352]]
    [[-1.3311479]
     [-0.6346681]
     [-1.8885762]
     [-1.635452 ]
     [-1.3533151]]

################# Do not write above this line #################

Your code here

```python
# Define features: x1x1, x2x2, x1x2, x1x1x1, x2x2x2, x1x1x2, x1x2x2

x_new_quad = np.hstack([x, x1**2, x2**2, x1*x2])
x_new_cubic = np.hstack([x_new_quad, x1**3, x2**3, (x1**2)*x2, x1*x2**2])

assert x[0, 1]**2 == x_new_quad[0, 3]
assert x[0, 2]**2 == x_new_quad[0, 4]
assert x[0, 1]*x[0, 2] == x_new_quad[0, 5]
assert x[0, 1]**3 == x_new_cubic[0, 6]
assert x[0, 2]**3 == x_new_cubic[0, 7]
assert (x[0, 1]**2)*x[0,2] == x_new_cubic[0, 8]
assert x[0, 1]*x[0, 2]**2 == x_new_cubic[0, 9]
```

################# Do not write below this line #################

Now use the Newton's optimization algorithm to learn theta by maximizing the log-likelihood, both for the case of x_new_quad and x_new_cubic.

Note: you may alternatively use gradient ascent, as in Question 1, if you like.

```python
# Initialize theta0, in case of quadratic features
theta0_quad = np.zeros(x_new_quad.shape[1])

# Run Newton's method, in case of quadratic features
theta_final_quad, theta_history_quad, log_l_history_quad = newton(theta0_quad, x_new_quad, y, grad_l, hess_l, 1e-6)

# Initialize theta0, in case of quadratic and cubic features
theta0_cubic = np.zeros(x_new_cubic.shape[1])

# Run Newton's method, in case of quadratic and cubic features
theta_final_cubic, theta_history_cubic, log_l_history_cubic = newton(theta0_cubic,x_new_cubic,y,grad_l,hess_l,1e-6)

# check and compare with previous results
print(theta_final_quad)
print(theta_final_cubic)
```

    [-2.01513398  0.17762534  1.91642971  0.07051926  0.18229119 -0.1295329 ]
    [-15.80123422  11.8465765    8.30908501  -3.03962504   1.86524025
      -2.79099185   0.25914729   0.14781699   0.24456401  -0.30583232]

```python
# Plot the log likelihood values in the optimization iterations, in one of the two cases.
fig,ax = plt.subplots(num=2)

ax.set_ylabel('l(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(len(log_l_history_cubic)),log_l_history_cubic,'b.')
```

### (b) Plot the computed non-linear boundary and discuss the questions

First, define a boundary_function to compute the boundary equation for the input feature vectors $x_1$ and $x_2$, according to estimated parameters theta, both in the case of quadratic (theta_final_quad) and of quadratic and cubic features (theta_final_cubic). Refer for the equation to the introductory part of Question 3.

################# Do not write above this line #################

Your code here

```python
def boundary_function(x1_vec, x2_vec, theta_final):

    x1_vec, x2_vec = np.meshgrid(x1_vec,x2_vec)

    if len(theta_final) == 6:
        # boundary function value for features up to quadratic
        c_0, c_1, c_2, c_3, c_4, c_5 = theta_final
        f = c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*x1_vec*x2_vec
    elif len(theta_final) == 10:
        # boundary function value for features up to cubic
        c_0, c_1, c_2, c_3, c_4, c_5, c_6, c_7, c_8, c_9 = theta_final
        f = (c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*x1_vec*x2_vec + c_6*(x1_vec**3) +
             c_7*(x2_vec**3) + c_8*(x1_vec**2)*x2_vec + c_9*x1_vec*(x2_vec**2))
    elif len(theta_final) == 12:
        # boundary function value for features up to tetra
        c_0, c_1, c_2, c_3, c_4, c_5, c_6, c_7, c_8, c_9, c_10, c_11 = theta_final
        f = (c_0 + c_1*x1_vec + c_2*x2_vec + c_3*(x1_vec**2) + c_4*(x2_vec**2) + c_5*x1_vec*x2_vec + c_6*(x1_vec**3) +
             c_7*(x2_vec**3) + c_8*(x1_vec**2)*x2_vec + c_9*x1_vec*(x2_vec**2) + c_10*(x1_vec**4) + c_11*(x2_vec**4))
    else:
        raise("Number of Parameters is not correct")

    return x1_vec, x2_vec, f
```

################# Do not write below this line #################

Now plot the decision boundaries corresponding to the theta_final_quad and theta_final_cubic solutions.

```python
x1_vec = np.linspace(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1,200);
x2_vec = np.linspace(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

```python
x1_vec = np.linspace(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1,200);
x2_vec = np.linspace(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic)

sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

################# Do not write above this line #################

#### If we now use the gradient ascent instead of the newtons method

```python
# Initialize theta0, in case of quadratic features
theta0_quad = np.zeros(x_new_quad.shape[1])

# Run Newton's method, in case of quadratic features
theta_final_quad, theta_history_quad, log_l_history_quad = gradient_ascent(theta0_quad, x_new_quad, y, G=grad_l, alpha=0.005, iterations=1000)

# Initialize theta0, in case of quadratic and cubic features
theta0_cubic = np.zeros(x_new_cubic.shape[1])

# Run Newton's method, in case of quadratic and cubic features
theta_final_cubic, theta_history_cubic, log_l_history_cubic = gradient_ascent(theta0_cubic,x_new_cubic,y, G=grad_l, alpha=0.005, iterations=1000)

# check and compare with previous results
print(theta_final_quad)
print(theta_final_cubic)
```

    [-1.39810928 -5.81552694  7.85054492  1.51244836 -0.89673691  0.69058293]
    [ -0.22163512  -7.59378978   7.2119274  -34.67073139   8.27459625
      13.49462667  15.82542568  20.5664375    4.27466746  -3.58210197]

    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel_launcher.py:6: RuntimeWarning: overflow encountered in exp

    C:\Users\juanm\Anaconda3\envs\data_science_general\lib\site-packages\ipykernel_launcher.py:2: RuntimeWarning: overflow encountered in exp

```python
x1_vec = np.linspace(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1,200);
x2_vec = np.linspace(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

```python
x1_vec = np.linspace(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1,200);
x2_vec = np.linspace(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic)

sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

Write now your considerations. Discuss in particular:
- What happens if you further increase the degree of the polynomial. How would the boundary change? Would you incur underfitting or overfitting?

Generally, increasing the degree of your polynomial to fit training data leads to an overfitting of your model (increasing the degree of your function is similar to manually adjusting your model to the training data). If you then apply your overfitted model to rel data (testing data) you might find it have a lot of error since you adjusted your model too much to your training data.

We can try to show this by including another two features ($x_{1}^{4}$ and $x_{2}^{4}$).

```python
x_new_tetra = np.hstack([x_new_cubic, x1**4, x2**4])

# Define new theta vector
theta0_tetra = np.zeros(x_new_tetra.shape[1])

# Run Newton's method, in case of quadratic features
theta_final_tetra, theta_history_tetra, log_l_history_tetra = newton(theta0_tetra, x_new_tetra, y, grad_l, hess_l, 1e-6)

# Plot Results based on new theta values
x1_vec = np.linspace(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1,200);
x2_vec = np.linspace(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_tetra)

sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

As we can see from the plot before our boundary function has adjusted in order to correctly classify several new y=0 example data. This might not reflect the $real$ behaviour of the target variable y.

- Look at the boundary of the quadratic and cubic polynomial features. Do they meet your expectations? Why? Consider in particular the classifier decision boundary in areas where there are fewer training data, i.e. where the classifier is supposed to generalize. Hint: Try changing the minimum and maximum limits in vectors x1_vec and x2_vec before plotting the boundaries.

In general the results obtained are not in line with our expectations. We are confident regarding the results obtained for the quadratic boundary function (function with a parabola like shape which seems to classify both classes in a very accurate manner), however the cubic function came as quite a surprise. Several of the data examples for y=0 which were properly classified in the quadratic boundary function are no longer classified correctly when using the cubic boundary function (please bear in mind that this might be an issue in the code).

In order to answer this question in more detail we will first change the limits over which x1_vec and x2_vec range (in one case focussing on more populated parts of the graphs, and secondly focussing on less populated parts of the graph)

##### x1_vec and x2_vec over low populated areas of graph

```python
x1_vec = np.linspace(df_x_pl["x1"].min(),df_x_pl["x1"].max(),200)
x2_vec = np.linspace(df_x_pl["x2"].min(),df_x_pl["x2"].max(),200)

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad)

sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

```python
x1_vec = np.linspace(df_x_pl["x1"].min(),df_x_pl["x1"].max(),200)
x2_vec = np.linspace(df_x_pl["x2"].min(),df_x_pl["x2"].max(),200)

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic)

sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

- Look at the values of the original features, the 2nd and 3rd degree ones. Do you expect that normalizing the input features (either by setting their range to [-1,1] or by subtracting their mean and dividing by their standard deviation) would improve the classifier? Why yes or why not?

```python
x1_standardized = ((x[:,1] - np.mean(x[:,1]))/np.std(x[:,1])).reshape((99, 1))
x2_standardized = ((x[:,2] - np.mean(x[:,2]))/np.std(x[:,2])).reshape((99, 1))

x_standardized_df = pd.DataFrame(columns=['y', 'x1', 'x2'])
x_standardized_df['y'] = y
x_standardized_df['x1'] = x1_standardized
x_standardized_df['x2'] = x2_standardized

x_standardized = np.hstack([x[:,0].reshape((99, 1)), x1_standardized, x2_standardized])

x_standardized_new_quad = np.hstack([x_standardized, x1_standardized**2,
                                     x2_standardized**2, x1_standardized*x2_standardized])
x_standardized_new_cubic = np.hstack([x_standardized_new_quad, x1_standardized**3,
                                      x2_standardized**3, (x1_standardized**2)*x2_standardized,
                                      x1_standardized*x2_standardized**2])

theta0_quad_standardized = np.zeros(x_standardized_new_quad.shape[1])

# Run Newton's method, in case of quadratic features
theta_final_quad_standardized, _, _ = newton(theta0_quad_standardized, x_standardized_new_quad, y, grad_l, hess_l, 1e-6)

# Initialize theta0, in case of quadratic and cubic features
theta0_cubic_standardized = np.zeros(x_standardized_new_cubic.shape[1])

# Run Newton's method, in case of quadratic and cubic features
theta_final_cubic_standardized, _, _ = newton(theta0_cubic_standardized,x_standardized_new_cubic,y,grad_l,hess_l,1e-6)
```

```python
x1_vec = np.linspace(x1_standardized.min()-1,x1_standardized.max()+1,200);
x2_vec = np.linspace(x2_standardized.min()-1,x2_standardized.max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_quad_standardized)

sns.scatterplot(x="x1", y="x2", hue="y", data=x_standardized_df);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

```python
x1_vec = np.linspace(x1_standardized.min()-1,x1_standardized.max()+1,200);
x2_vec = np.linspace(x2_standardized.min()-1,x2_standardized.max()+1,200);

x1_vec, x2_vec, f = boundary_function(x1_vec, x2_vec, theta_final_cubic_standardized)

sns.scatterplot(x="x1", y="x2", hue="y", data=x_standardized_df);

plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])
```

As one can observe from the two previous plots (in which we have normalized our features), the boundary function seems to have a very similar shape to the case which we did not normalize our data. This is probably due to the fact that the training data was not very far away from the standardized data. If instead we had had very different data (comparing training data before and after standardization), one would observer a higher impact on the final results.

In general it is always recommened to strandardize/normalize data so that all of the feature variables are "comparable".

################# Do not write below this line #################

## [10 points, extra, not compulsory] Question 4: Gaussian Discriminant Analysis

### (a) Review and implement classification with Gradient Discriminant Analysis (GDA)

Recall the assumptions for GDA

$\begin{aligned} y & \sim \operatorname{Bernoulli}(\phi) \\ x \mid y=0 & \sim \mathcal{N}\left(\mu_{0}, \Sigma\right) \\ x \mid y=1 & \sim \mathcal{N}\left(\mu_{1}, \Sigma\right) \end{aligned}$

Recall the probability distributions:

$\begin{aligned} p(y) &=\phi^{y}(1-\phi)^{1-y} \\ p(x \mid y=0) &=\frac{1}{(2 \pi)^{d / 2}|\Sigma|^{1 / 2}} \exp \left(-\frac{1}{2}\left(x-\mu_{0}\right)^{T} \Sigma^{-1}\left(x-\mu_{0}\right)\right) \\ p(x \mid y=1) &=\frac{1}{(2 \pi)^{d / 2}|\Sigma|^{1 / 2}} \exp \left(-\frac{1}{2}\left(x-\mu_{1}\right)^{T} \Sigma^{-1}\left(x-\mu_{1}\right)\right) \end{aligned}$

Overall, the parameters of the model are $\phi, \Sigma, \mu_{0}$ and $\mu_{1}$ and the log-likelihood is given by:

$\begin{aligned} \ell\left(\phi, \mu_{0}, \mu_{1}, \Sigma\right) &=\log \prod_{i=1}^{n} p\left(x^{(i)}, y^{(i)} ; \phi, \mu_{0}, \mu_{1}, \Sigma\right) \\ &=\log \prod_{i=1}^{n} p\left(x^{(i)} \mid y^{(i)} ; \mu_{0}, \mu_{1}, \Sigma\right) p\left(y^{(i)} ; \phi\right) \end{aligned}$

Parameters may be estimated by maximum likelihood estimate as follows:

$\begin{aligned} \phi &=\frac{1}{n} \sum_{i=1}^{n} 1\left\{y^{(i)}=1\right\} \\ \mu_{0} &=\frac{\sum_{i=1}^{n} 1\left\{y^{(i)}=0\right\} x^{(i)}}{\sum_{i=1}^{n} 1\left\{y^{(i)}=0\right\}} \\ \mu_{1} &=\frac{\sum_{i=1}^{n} 1\left\{y^{(i)}=1\right\} x^{(i)}}{\sum_{i=1}^{n} 1\left\{y^{(i)}=1\right\}} \\ \Sigma &=\frac{1}{n} \sum_{i=1}^{n}\left(x^{(i)}-\mu_{y^{(i)}}\right)\left(x^{(i)}-\mu_{y^{(i)}}\right)^{T} \end{aligned}$

Consider the data variables as from the previous questions, x and y.
For this question, we would select features $x^{(i)}_1$ and $x^{(i)}_2$ into the new varible x12, dropping the $x_0=1$ convention.

```python
# First extract the sole features x1 and x2 from x into an x12 array and reshape it to x2 vector arrays
x12 = x[:,1:]
print(x[:5,:]) # For visualization of the first 5 values
print(x12[:5,:]) # For visualization of the first 5 values
print(y[:5]) # For visualization of the first 5 values
```

    [[ 1.          1.3432504  -1.3311479 ]
     [ 1.          1.8205529  -0.6346681 ]
     [ 1.          0.98632067 -1.8885762 ]
     [ 1.          1.9443734  -1.635452  ]
     [ 1.          0.97673352 -1.3533151 ]]
    [[ 1.3432504  -1.3311479 ]
     [ 1.8205529  -0.6346681 ]
     [ 0.98632067 -1.8885762 ]
     [ 1.9443734  -1.635452  ]
     [ 0.97673352 -1.3533151 ]]
    [0 0 0 0 0]

Now code up the equations above to learn the GDA parameters from the training data.

################# Do not write above this line #################

Your code here

```python
def compute_phi(y):
    n = len(y)
    x = np.count_nonzero(y==1)/n
    return x

def compute_mu0(x, y):
    n = len(y)
    den = (y == 0).sum()
    partial_sum = 0
    for i in range(0,n):
        if (y[i]==0):
            partial_sum += x[i]
    return partial_sum/den

def compute_mu1(x, y):
    n = len(y)
    den = (y == 1).sum()
    partial_sum = 0
    for i in range(0,n):
        if (y[i]==1):
            partial_sum +=  x[i]

    return partial_sum/den

def compute_sigma(x, y, mu0, mu1):
    n = len(y)
    den0 = (y == 0).sum()
    den1 = (y == 1).sum()
    sigma_0 = np.zeros((len(x[0]), len(x[0])))
    sigma_1 = np.zeros((len(x[0]), len(x[0])))
    for i in range(n):
        if y[i] == 0:
            sigma_0 += np.dot((x[i]-mu0).reshape((1, 2)).T,(x[i]-mu0).reshape((1, 2)))
        else:
            sigma_1 += np.dot((x[i]-mu1).reshape((1, 2)).T,(x[i]-mu1).reshape((1, 2)))
    sigma = sigma_0 + sigma_1
    return sigma

# computing p(x|y) for the Bayes rule
def p_x_given_y(x, mu, sigma):
    dim = 2
    det = np.linalg.det(sigma)
    sigma_inv = np.linalg.inv(sigma)
    res = (1/(2*np.pi)**(dim/2)*np.sqrt(det)*np.exp(-0.5*np.dot(np.dot((x-mu).reshape((1, 2)),sigma_inv),(x-mu).reshape((1, 2)).T)))
    return res[0][0]
```

################# Do not write below this line #################

```python
# Support functions

# Function p(y) for applying the Bayes rule
def p_y(y,phi):
    if y==1: return phi
    else: return 1-phi
```

```python
# Now estimate the GDA parameters
phi = compute_phi(y)
mu0 = compute_mu0(x12,y)
mu1 = compute_mu1(x12,y)
sigma = compute_sigma(x12, y, mu0, mu1)

phi,mu0,mu1,sigma
```

    (0.494949494949495,
     array([ 2.93068144, -1.44476469]),
     array([4.93616668, 0.76771897]),
     array([[265.49277586, -45.94001206],
            [-45.94001206, 186.48414319]]))

Now that we know both the distributions of $p(y)$ and $p(x|y)$, we may estimate the posterior probability for an input $x$ via Bayes Rule:

\begin{equation}
p(y \mid x)=\frac{p(x \mid y) p(y)}{p(x)}
\end{equation}
where $p(x)$ is the same for both classes and may be disregarded.

This is used to classify each data point, by comparing $p(y=1|x) \propto p(x \mid y=1) p(y=1)$ and $p(y=0|x) \propto p(x \mid y=0) p(y=0)$.

```python
selected_example=1
print('Selected example =', x12[selected_example,:])

# compute p(y=0|x) ~ p(x|y=0)*p(y=0)  &  p(y=1|x) ~ p(x|y=1)*p(y=1)

# y=0
print('p(y=0|x) ~', p_x_given_y(x12[selected_example,:],mu0,sigma)*p_y(0,phi))

# y=1
print('p(y=1|x) ~', p_x_given_y(x12[selected_example,:],mu1,sigma)*p_y(1,phi))
```

    Selected example = [ 1.8205529 -0.6346681]
    p(y=0|x) ~ 17.440955513812025
    p(y=1|x) ~ 16.662676656260853

### (b) Plot the estimated likelihood functions, the decision boundary and write your remarks

```python
def boundary_gda(x1_vec, x2_vec, mu0, mu1, sigma, phi):

    x1_vec, x2_vec = np.meshgrid(x1_vec,x2_vec)

    f=np.zeros(x1_vec.shape)

    for i in range(f.shape[0]):
        for j in range(f.shape[1]):
            x12_ij = np.array( (x1_vec[i,j],x2_vec[i,j]) )
            f[i,j] = p_x_given_y(x12_ij,mu1,sigma)*p_y(1,phi) - p_x_given_y(x12_ij,mu0,sigma)*p_y(0,phi)

    return x1_vec, x2_vec, f

x1_vec = np.linspace(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1,200);
x2_vec = np.linspace(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1,200);

x1_vec, x2_vec, f = boundary_gda(x1_vec, x2_vec, mu0, mu1, sigma, phi)
```

```python
from matplotlib import cm
from scipy.stats import multivariate_normal

# Define Gaussian pdfs
rv0 = multivariate_normal(mu0, sigma)
rv1 = multivariate_normal(mu1, sigma)

pairx12 = np.dstack((x1_vec, x2_vec))

# plot pdf contours
plt.contourf(x1_vec, x2_vec, rv0.pdf(pairx12),alpha=0.5,cmap=cm.Reds)
plt.contourf(x1_vec, x2_vec, rv1.pdf(pairx12),alpha=0.35,cmap=cm.Blues)

# Add the GDA classification to the plot
sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl);
plt.contour(x1_vec, x2_vec, f, colors="red", levels=[0])

plt.ylim(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1)
plt.xlim(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1)
```

```python
# For comparison with the Logistic Regression boundary

# Generate vector to plot decision boundary
x1vec = np.linspace(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1,2)

# Plot raw data
sns.scatterplot(x="x1", y="x2", hue="y", data=df_x_pl)

# Plot decision boundary
log, = plt.plot(x1vec,(-x1vec*theta_GA[1]-theta_GA[0])/theta_GA[2], color="red")
gda = plt.contour(x1_vec, x2_vec, f, colors="green", levels=[0])

plt.legend([log, gda.collections[0]],['logistic regression','GDA'])

plt.ylim(df_x_pl["x2"].min()-1,df_x_pl["x2"].max()+1)
plt.xlim(df_x_pl["x1"].min()-1,df_x_pl["x1"].max()+1)

# plt.legend()
plt.show()
```

Write now your considerations. Discuss in particular:
- Gaussian Discriminant Analysis makes stronger modelling assumptions than logistic regression. When may this be helpful and where may this be harmful?

The approach of learning to distinguish and classify in a binary way, based on the two classes (y=1) and (y=0), can also be implemented with GDA.
So if we have two classes, which in our case correspond to (y=0) and (y=1), our aim is to construct a model that, for each class, is able to recognise new elements and assign it to one of the two class.
In this method, instead of p(y|x) used for the logistic regression, p(x|y) and p(y) are modelled.
Once known p(x|y) and p(y) we are able to understand p(y|x), using the Bayes rule.
In our classification problem, using GDA, the input features are continuous random variables and, as we have seen, p(x|y) is distributed as a multivariate normal distribution and p(y) is distributed as a Bernoulli.
From the analysis of the plot showing the two boundary decisions with GDA and LR, we can say that for this particular data training LR works better than GDA, even if the results are not totally inconsistent.
The explanation for this different behaviour lies in the fact that GDA makes more specific assumptions than the LR, in particular considering p(x|y) and  p(y).
These conditions, if they are true , allows GDA to work better than the LR, but we can observe
from the contour plots that our training set does not completely overlap whith the assumption that p(x|y) is guassian.
So the GDA assumptions are not completely verified.
In conclusion, we can say that GDA has better results than the LR if we know the nature of our data and their distribution matches the specific assumptions of this model.
As we have seen, if this aspect is neglected, we lose accuracy.
On the other hand, the LR model does not need restrictive assumptions like GDA and works better in situations where the probability distribution of the features is not Gaussian.

Credits for material: Andrew Ng, C. Combier


# Projet EDP : Ecoulement de gel hydroalcoolique avec l'équation de Stockes

( En cas de problème sur l'exécution du code ou avec les images, merci de me contacter : matthieu.briet@student-cs.fr

### Introduction

La crise sanitaire actuelle nous pousse à utiliser de plus en plus des gels hydroalcoolique parfois sous forme de bouteille avant de rentrer dans des magasins par exemple. Cependant, il arrive qu'une partie du gel hydroalcoolique durcisse et forme un dépôt solide dans l'embout de sortie ce qui projète une partie du gel un peu partout et parfois même sur nos vêtements ou nos chaussures. On se demande alors s'il est possible d'expliquer ou de prédire ces "jets" de gel hydroalcoolique non souhaités.

On se propose de simuler l'écoulement du gel dans cette partie de la bouteille :

### Modélisation

On s'intéresse ici à la résolution du problème de Stockes qui permet de décrire l'écoulement d'un fluide visqueux incompressible dans un lieu étoit où les effets visqueux prédominent sur les effets inertiels. Ces équations sont données par :

\begin{equation}
\begin {cases}
-\nu \Delta \vec u -\vec {grad  } ~p= \vec f \text {  sur le domaine  } \Omega \\
div ~\vec u =0 \text {  sur le domaine  } \Omega ~\text{(c'est la condition d'incompressibilité)}\\
\vec u=0 ~\text{sur la frontière} ~\partial \Omega ~\text{(conditions de bords de type Dirichlet homogène)}
\end{cases}
\end{equation}

avec
\begin{equation}
\begin{cases}
\Omega ~\text{le domaine d'étude qui sera spécifié plus tard}\\
\vec u ~\text{le champ de vitesse}\\
p ~\text{la pression}\\
\vec f ~\text{ la densité de force }\\
\nu >0 ~\text{la viscosité cinématique du fluide }
\end{cases}
\end{equation}

Grâce au cours d'EDP associé à quelques recherches (car ici on a deux équation à cause de la condition d'incompressibilité), on peut écrire la formulation faible du problème qui s'obtient en multipliant par une fonction test et en intégrant :

\begin{equation}
\begin{cases}
\int_\Omega \nabla u .\nabla \Phi +\frac{1}{\nu}.\int_\Omega \nabla p .\Phi = \frac{1}{\nu}. \int_\Omega f.\Phi \\
\int_\Omega div(u).q=0
\end{cases}
\end{equation}

avec $\Phi$ et $q$ des fonctions tests.

Ecrivons à présent la formulation variationnelle du problème avec :

\begin{equation}
\begin{array}{l|rcl}
a : & W \times W \longrightarrow \mathbf{R} \\
    & ((u,p),(\Phi,q)) & \longmapsto  \int_{\Omega} \nabla u .\nabla \Phi +\frac{1}{\nu}. \nabla p .\Phi+div(u).q
\end{array}
\end{equation}

avec $W= U \times P $ tel que $u\in U $ et $p\in P$

et
\begin{equation}
\begin{array}{l|rcl}
l: & W \longrightarrow \mathbf{R} \\
    & (\Phi,q) & \longmapsto  \frac{1}{\nu}.\int_{\Omega} f.\Phi
\end{array}
\end{equation}

On cherche alors à résoudre le problème :

Trouver $(u,p)\in W $ tel que $ \forall (\Phi,q) \in W , a((u,p),(\Phi,q)) =l((\Phi,q))$

On peut ensuite montrer que a est une forme bilinéaire continue et coercive et que l est une forme continue et donc qu'il existe une unique solution au problème.On se propose par la suite de résoudre ce problème numériquement avec FeniCS.

Il nous reste à présent à spécifier les valeurs prisent pour notre étude :

\begin{equation}
\begin{cases}
\nu= 3500 mm^{2}/s \text{  (viscosité cinématique du gel)}\\
u_{0} = 0.1 m/s   \text{  (vitesse initiale du gel)}\\
p_{sortie}= 1 bar \\
\end{cases}
\end{equation}

### Etude numérique avec FeniCS

     Chargement des modules

```python
from dolfin import *
from fenics import *
import matplotlib.pyplot as plt
from mshr import *
from __future__ import print_function
import random

```

  Définition des constantes

```python
# définir les constantes ici
nx=10
ny=10
X1=0
Y1=2
X2=1.5
Y2=3
X3=1.5
Y3=2
X4=3
Y4=2.5
nbre_pts=50
nu=0.035
```

 Définition du maillage

```python

rectangle1=Rectangle(Point(X1,Y1),Point(X2,Y2))
rectangle2=Rectangle((Point(X3,Y3)),Point(X4,Y4))
mon_maillage=generate_mesh(rectangle1+rectangle2,nbre_pts)
plt.figure(1)
plot(mon_maillage,title="mesh ")

```

 Définition de l'espace de travail

```python
U=VectorElement("Lagrange",mon_maillage.ufl_cell(),2) # pour des raisons de stabilité la vitesse ne peux pas être modélisés par des elements finis P1
P=FiniteElement("Lagrange",mon_maillage.ufl_cell(),1)
W=FunctionSpace(mon_maillage,U*P)  # W.sub(0) pour les vitesses et W.sub(1) pour les pressions
```

 Condition aux limites

```python
tol=1e-14

def condition_gauche(x,on_boundary):
    return  on_boundary and (x[0]<X1+tol and (x[1]<3+tol or x[1]>2-tol))

def condition_droite(x,on_boundary):
    return on_boundary and (x[0]>X4-tol)

def condition_haut_bas(x,on_boundary):
    return on_boundary and ((x[1]<tol+2) or (x[1]>3-tol and (x[0]<1.5+tol or x[0]>-tol)) or (x[1]>2.5-tol and (x[0]<3+tol or x[0]>1.5-tol)))

#le gel entre dans le bec du flacon avec une pression p

p_sortie=Constant(1e5)
bc_pression=DirichletBC(W.sub(1),p_sortie,condition_droite) #on applique la pression sur l'espace des pressions ici W(1)

#vitesse nulle sur les bords horizontaux et sur le bord vertical
v0=Expression(("0.0","0.0"),degree=2)

bcV0_haut_bas=DirichletBC(W.sub(0),v0,condition_haut_bas)

vitesse_entree=0.1
gel_entree=Expression((str(vitesse_entree),"0.0"),degree=2)
bc_gel_entree=DirichletBC(W.sub(0),gel_entree,condition_gauche)

bc_total=[bc_pression,bcV0_haut_bas,bc_gel_entree]
```

 Formulation variationnelle

```python
u,p=TrialFunctions(W)
v,q= TestFunctions(W)
f=Constant((0.0,0.0))
a= inner(grad(u),grad(v))*dx+1/nu*inner(v,grad(p))*dx+1/nu*q*div(u)*dx
l=1/nu*inner(f,v)*dx

```

 Résolution du système

```python
u=Function(W)
solve(a==l,u,bc_total)
u_f=u.split()[0]

plot(u_f,title="vitesse",scale=2)

```

On voit ici un écoulement normal du gel dans le bec distributeur avec une forte accélération de la vitesse du fluide dans le goulot. Ceci correspond à l'effet Venturi observé en mécanique des fluides et notre fluide à donc ici un comportement assez classique sans perturbation de l'écoulement par un quelconque dépot. On remarque bien sur la simulation précédente que la vitesse du fluide est nulle sur les bords et donc logiquement plus importante sur la ligne médiane de notre écoulement.

### Prise en compte d'un paramètre aléatoire

```python
Ox1=2.7
Oy1=2

Ox2=2.9
Oy2=random.uniform(2,2.5)
```

```python
rectangle1=Rectangle(Point(X1,Y1),Point(X2,Y2))
rectangle2=Rectangle((Point(X3,Y3)),Point(X4,Y4))
obstacle= Rectangle(Point(Ox1,Oy1),Point(Ox2,Oy2))
mon_maillage2=generate_mesh(rectangle1+rectangle2-obstacle,nbre_pts)
plt.figure(2)
plot(mon_maillage2,title="mesh2")

```

```python
U=VectorElement("Lagrange",mon_maillage2.ufl_cell(),2) # pour des raisons de stabilité la vitesse ne peux pas être modélisés par des elements finis P1
P=FiniteElement("Lagrange",mon_maillage2.ufl_cell(),1)
W=FunctionSpace(mon_maillage2,U*P)  # W.sub(0) pour les vitesses et W.sub(1) pour les pressions

tol=1e-14

def condition_gauche(x,on_boundary):
    return  on_boundary and (x[0]<X1+tol)

def condition_droite(x,on_boundary):
    return on_boundary and (x[0]>X4-tol)

def condition_haut_bas(x,on_boundary):
    return on_boundary and ((x[1]<tol+2) or (x[1]>3-tol and (x[0]<1.5+tol or x[0]>-tol)) or (x[1]>2.5-tol and (x[0]<3+tol or x[0]>1.5-tol)))

def condition_obstacle(x,on_boundary):
    return on_boundary and x[0]<Ox2+tol and x[0]>Ox1-tol and x[1]<Oy2+tol and x[1]>Oy1-tol

#le gel entre dans le bec du flacon avec une pression p

p_sortie=Constant(1e5)
bc_pression=DirichletBC(W.sub(1),p_sortie,condition_droite) #on applique la pression sur l'espace des pressions ici W(1)

#vitesse nulle sur les bords horizontaux et sur le bord vertical et sur l'obstacle
v0=Expression(("0.0","0.0"),degree=2)

bcV0_haut_bas=DirichletBC(W.sub(0),v0,condition_haut_bas)
bc_obstacle=DirichletBC(W.sub(0),v0,condition_obstacle)

#vecteur vitesse pour le gel en sortie

vitesse_entree=0.1
gel_sortie=Expression((str(vitesse_entree),"0.0"),degree=2)
bc_gel_entree=DirichletBC(W.sub(0),gel_sortie,condition_gauche)

bc_total=[bc_pression,bcV0_haut_bas,bc_obstacle,bc_gel_entree]
u,p=TrialFunctions(W)
v,q= TestFunctions(W)
f=Constant((0.0,0.0))
a= inner(grad(u),grad(v))*dx+1/nu*inner(v,grad(p))*dx+1/nu*q*div(u)*dx
l=1/nu*inner(f,v)*dx

u=Function(W)
solve(a==l,u,bc_total)
u_f=u.split()[0]
p=u.split()[1]

plot(u_f,title="vitesse",scale=2)

```

Dans le cas avec un dépôt solide de taille variable( composante aléatoire ici ou on fait varier la position et donc la taille du dépôt solide), on remarque la perturbation de notre écoulement et des vecteurs vitesses qui partent dans une direction  " non colinéaire" à l'écoulement (je vous conseille de lancer le code plusieurs fois afin de voir les changements liés à la taille du dépôt solide). Le comportement de notre fluide a donc été perturbé et ceci peut expliquer la projection de fluide dans des directions non souhaitées.
On pourrait ensuite s'intéresser dans un autre projet à la force exercée sur ce dépôt solide et à son évolution au fur et à mesure des utilisations ( evacuation de ce dernier peut être) et donc déterminer le moment où l'écoulement redevient "normal" c'est à dire sans projection. Pour avoir fait l'expérience de mon côté, le dépot solide s'évacuait avec environ 5 utilisations de gel hydroalcoolique(Ce gel ne fut pas gâché pour l'expérience). De plus un  début de dépôt solide pouvait être observé dans ma bouteille en environ 3h30. Evidemment toutes ces données peuvent varier en fonction du type de gel, sa composition, les conditions de température et de pression et la force exercée sur la bouteille pour faire sortir le gel.

### Bibliographie:

https://fr.wikipedia.org/wiki/Écoulement_de_Stokes

http://www.iecl.univ-lorraine.fr/~Jean-Francois.Scheid/Enseignement/polyNS2017_18.pdf

https://courses.ex-machina.ma/downloads/gm-2/s7/M_E_F/EFM_Stokes.pdf

https://fenicsproject.org/olddocs/dolfin/1.3.0/python/demo/documented/stokes-iterative/python/documentation.html

```python

```


# Tutorial on motion energy model implementation

This notebook demonstrates the components underlying a spatiotemporal energy model of motion perception. The model was originally introduced in 1985 by EH Adelson and JR Bergen. The basic idea is to think of motion velocity as an orientation in space-time. The model works by constructing filters with joint selectivity for speed and direction (velocity), which are then convolved with a movie of the motion stimulus.

The specific Python implemenation here was written by ML Waskom based on an earlier MATLAB implementation by R Kiani. The notebook is intended to provide some intuition for the parameters of the model and to show how the high-level interface can be used for extracting motion energy estimates from a dynamic stimulus.

This notebook is best viewed locally, because it has some interactive demos and movies of the stimulus that are not reproduced in the static HTML representation.

### References:

Adelson EH & Bergen JR (1985). [Spatiotemporal energy models for perception of motion](https://www.ncbi.nlm.nih.gov/pubmed/3973762). *J Opt Soc Am A* 2(2):284-99.

Kiani R, Hanks TD, Shadlen MN (2008). [Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment.
](https://www.ncbi.nlm.nih.gov/pubmed/18354005). *J Neurosci* 28(12):3017-29.

Waskom ML, Asfour JW, Kiani R (2018). [Perceptual insensitivity to higher-order statistical moments of coherent random dot motion](https://www.biorxiv.org/content/early/2018/04/26/261370). *J Vision*

```python
import numpy as np
from numpy.fft import fftn, fftshift, fftfreq
from scipy.ndimage import rotate
import seaborn as sns
import matplotlib.pyplot as plt
from ipywidgets import interact
```

```python
import motionenergy as me
import stimulus as stim
```

```python
%matplotlib inline
sns.set(font_scale=1.3, color_codes=True)
```

----

## Components of the spatiotemporal energy model

The spatiotemporal filters are constructed from space-time seperable components. There are two functions each for the spatial and temporal components. Using pairs of quadrature filters will make the model invariant to phase. The full spatial filters are defined as follows, letting $\alpha = \tan^{-1}(x/\sigma_c)$:

$$
\begin{align}
f_1(x,y) &= \cos^4(\alpha)\cos(4\alpha)\exp\big(-\frac{y^2}{2\sigma_g^2}\big) \\
f_2(x,y) &= \cos^4(\alpha)\sin(4\alpha)\exp\big(-\frac{y^2}{2\sigma_g^2}\big),
\end{align}
$$

On one spatial axis, the filters are fourth-order Cauchy functions (similar to a Gabor pattern) controlled by a parameter $\sigma_c$. The demo below shows how this parameter determines the width of the filter; together with the temporal components (see below), it is part of what gives the filter speed selectivity.

```python
@interact
def cauchy_param_tutorial(σ_c=(.1, .6, .05)):
    """Interactive widget to demo spatial component of filters."""
    x = me.filter_grid(256, .005, center=True)
    f1, f2 = me.cauchy(x, σ_c)
    f, ax = plt.subplots()
    ax.plot(x, f1, x, f2)
    ax.set(xlim=(-.65, .65),
           xlabel="Selective axis (deg)",
           ylabel="Amplitude")
```

    interactive(children=(FloatSlider(value=0.30000000000000004, description='σ_c', max=0.6, min=0.1, step=0.05), …

On the orthogonal axis, the filters are windowed with a Gaussian envelope; the two-dimensional filter is then oriented in space. The orientation of the filter determines its direction selectivity. The width of the Gaussian envelope influences how narrowly or broadly it is tuned around its preferred direction.

```python
def centered_colormap(data):
    """Set vmin and vmax to put center of colormap at 0."""
    lim = np.abs([data.min(), data.max()]).max()
    return dict(vmin=-lim, vmax=+lim, cmap="icefire")
```

```python
@interact
def spatial_param_tutorial(σ_c=(.05, .65), σ_g=(.02, .2, .02), θ=(-90, 90, 10)):
    """Interactive widget to demo full spatial component of filters."""
    x = me.filter_grid(125, .02, True)
    xx, yy = np.meshgrid(x, x)

    c1, c2 = me.cauchy(x, σ_c, 4)
    g = me.gaussian_envelope(x, σ_g)

    qx1, qy = np.meshgrid(c1, g)
    qx2, qy = np.meshgrid(c2, g)
    filters = ("Even filter ($f_1$)", qx1), ("Odd filter ($f_2$)", qx2)

    f, axes = plt.subplots(1, 2, figsize=(6, 3), sharey=True)
    for ax, (title, filt) in zip(axes, filters):

        filt = rotate(filt * qy, θ, reshape=False)
        ax.pcolormesh(xx, yy, filt, **centered_colormap(filt))
        ax.set(title=title)

    f.subplots_adjust(0, 0, 1, 1, .02)
    axes[0].set(xlabel="Selective axis (deg)",
                ylabel="Non-selective axis (deg)")
```

    interactive(children=(FloatSlider(value=0.35, description='σ_c', max=0.65, min=0.05), FloatSlider(value=0.1, d…

----

The temporal filters model the impulse response of direction-selective neurons in cortex. They are implemented as a pair of difference-of-poisson functions:

$$
\begin{align}
g_1(t) &= (kt)^3\exp(-kt)\Bigg[\frac{1}{3!} - \frac{(kt)^2}{(3 + 2)!}\Bigg] \\
g_2(t) &= (kt)^5\exp(-kt)\Bigg[\frac{1}{5!} - \frac{(kt)^2}{(5 + 2)!}\Bigg]
\end{align}
$$

The main parameter is $k$, which controls the latency of the filter. Together with $\sigma_c$, this will determine the speed selectivity.

```python
@interact
def temporal_param_tutorial(k=(30, 70, 5)):
    """Interactive widget to demo temporal component of filters."""
    t = me.filter_grid(128, .005)
    f1 = me.temporal_impulse_response(t, 3, k)
    f2 = me.temporal_impulse_response(t, 5, k)
    f, ax = plt.subplots()
    lines = ax.plot(t, f1, t, f2)
    ax.legend(lines, ["$g_1$", "$g_2$"])
    ax.set(xlabel="Time (sec)", ylabel="Amplitude")
```

    interactive(children=(IntSlider(value=50, description='k', max=70, min=30, step=5), Output()), _dom_classes=('…

---

## Putting the components together

These spatial and temporal functions are combined to create full three-dimensional spatiotemporal filters. When the model is used, each stimulus is convolved with four different filters. There are two filters oriented to be selective for motion in a given direction (preferred) and then two filters oriented to be selective for motion in the reverse direction (null). This is done because motion energy is defined as the difference between the energy in the preferred and null directions (i.e. opponent motion energy). For each direction, there is an even filter and an odd filter. The responses of this quadrature pair are combined so that the system is invariant to phase.

When plotting the filters on a two-dimensional axes (selective spatial dimension and temporal dimension), they look like schmeared Gabors. This emphasizes how the model characterizes motion as an orientation in space-time. Just as an oriented Gabor filter is selective for a spatial orientation, an oriented Gabor schmear is selective for a velocity.

```python
# Determine the size and resolution of the filters
nx, dx = 235, .01
nt, dt = 300, .001
size = nx, 1, nt
res = dx, dx, dt

# Define the mesh that the filters are sampled on
x = me.filter_grid(nx, dx, center=True)
t = me.filter_grid(nt, dt)
xx, tt = np.meshgrid(x, t, indexing="ij")

# Create a set of motion energy filters
filters = me.motion_filters(size, res)

# Show each filter in the set
f, axes = plt.subplots(1, 4, sharey=True, figsize=(10, 4))
titles = ["Even preferred", "Odd preferred", "Even null", "Odd null"]

for ax, title, filt in zip(axes, titles, filters):
    filt = filt.squeeze()[:, -nt:]
    ax.pcolormesh(xx, tt, filt, **centered_colormap(filt))
    ax.set_title(title)

axes[0].set(xlabel="Selective axis (deg)", ylabel="Time (sec)")
f.subplots_adjust(0, 0, 1, 1, .02, .02)
```

We can show the spectral representation of the filter to get a better sense for how the spatial and temporal parameters control the speed selectivity. What we want is to find a ratio between the spatial and temporal frequencies so that the filter has maximum power at the speed of the stimulus we want to characterize:

```python
@interact
def spatiotemporal_tutorial(filter=titles, k=(30, 80, 5), σ_c=(.05, .65, .05)):
    """Interactive widget to demo spatiotemporal selectivity."""
    # Determine the size and resolution of the filters
    nx, nt = 41, 41
    dx, dt = .0436, .0133
    size = nx, 1, nt
    res = dx, .01, dt

    # Create the filter set and select the one we want to view
    filters = me.motion_filters(size, res, k=k, csigx=σ_c)
    filt = filters[titles.index(filter)]

    # Take the fourier transform of the filter to show its spectral representation
    filt_fft = np.abs(fftshift(fftn(filt)).squeeze()).real

    # The filter is causal in time, so crop off the first half
    filt = filt.squeeze()[:, nt:]

    # Define the spatial mesh
    xs = me.filter_grid(nx, dx, center=True)
    ts = me.filter_grid(nt, dt)
    xx, tt = np.meshgrid(xs, ts, indexing="ij")

    # Define the spectral mesh
    fxs = fftshift(fftfreq(nx, dx))
    fts = fftshift(fftfreq(nt * 2 - 1, dt))
    fxx, ftt = np.meshgrid(fxs, fts, indexing="ij")

    # Plot the spatial and spectral representations of the filter
    gridspec = {"width_ratios": (4, 5)}
    f, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 5), gridspec_kw=gridspec)

    ax1.pcolormesh(xx, tt, filt, **centered_colormap(filt))
    ax2.pcolormesh(fxx, ftt, filt_fft)

    ax1.set(xlabel="Selective axis ($\circ$)", ylabel="Time (s)")
    ax2.set(xlabel="$\omega_x$ (cycle / sec)", ylabel="$\omega_t$ (cycle / deg)")
    f.tight_layout()
```

    interactive(children=(Dropdown(description='filter', options=('Even preferred', 'Odd preferred', 'Even null', …

----

## Using the filters to model motion perception

Having gone through the components of the model, the next section demonstrates how it might actually be used with a perceptual stimulus.

First, we are going to generate a random dot motion stimulus as a 3D movie.

```python
# Determine the parameters of the display

radius = 3  # radius of the stimulus in degrees
ppd = 40  # display spatial resolution (pixels per degree)
framerate = 1 / 60  # display temporal resolution

# Determine the parameters of the stimulus

density = 100.2  # density of dots in dots per deg^2 per sec
size = 3  # size of each dot in pixels
speed = 5  # speed of coherent motion in deg per sec
coherence = .9  # proportion of dots displaced coherently
moments = 0, 0.001, 0, 3  # mean, sd, skew, and variance of dot motion
duration = 1  # length of the movit, in seconds
```

```python
dots = stim.dot_movie(radius, density, size, speed, coherence, ppd, framerate, duration, moments)
stim.play_movie(dots, framerate)
```

    /home/adrian/Git/GitHub/work/Waskom_JVision_2018/stimulus.py:416: RuntimeWarning: invalid value encountered in true_divide
      p /= integrate.trapz(p, x_orig)
    /home/adrian/Git/GitHub/work/Waskom_JVision_2018/stimulus.py:439: RuntimeWarning: invalid value encountered in less
      return rs.choice(x, replace=True, p=p, size=size)

To extract the motion energy of this stimulus, we convolve the three dimensional arrays (the stimulus movie and each motion energy filter). The filters should have the same resolution as the movie, but they can have a different size.

```python
filter_shape = 64, 64, 25
filter_res = 1 / ppd, 1 / ppd, framerate
filters = me.motion_filters(filter_shape, filter_res)
```

Applying the filters to the stimulus will return a three dimensional array with the same shape as the stimulus movie. We can visualize it in the same way and see that the model has replaced each dot with a local burst of motion energy:

```python
%%time
dots_energy = me.apply_motion_energy_filters(dots, filters)
stim.play_movie(dots_energy, framerate, vmax=10, cmap="mako")
```

    CPU times: user 22.5 s, sys: 2.99 s, total: 25.5 s
    Wall time: 25.7 s

The motion energy can be summed across space and shown as a function of time. Shown this way, it is clear that there is an initial rise from zero to a steady state; this rise time is determined by the temporal latency parameter. Becauase the random dot stimulus is stochastic, the motion energy then fluctuates somewhat around its steady state value.

```python
f, ax = plt.subplots()
t = me.filter_grid(duration / framerate, framerate)
ax.plot(t, dots_energy.sum(axis=(0, 1)))
ax.set(xlim=(0, duration), ylim=(0, None),
       xlabel="Time (sec)",
       ylabel="Motion energy (a.u.)")
f.tight_layout()
```

We can use a bank of filters with different preferred orientations to model the response of a population of direction-selective cells to a particular stimulus. Alternatively, you can think of this as estimating the tuning curve of the filter system given the set of parameters. Constructing the motion energy profile involves a lot of convolution, so the next cell will take some time to execute.

```python
%%time
thetas = [-60, -45,  -30, -15, 0, +15, +30, +45, +60]
filter_bank = me.filter_bank(thetas, filter_shape, filter_res)
bank_energy = np.sum(me.apply_motion_energy_filters(dots, filter_bank), axis=(1, 2))
```

    CPU times: user 2min 56s, sys: 24.6 s, total: 3min 21s
    Wall time: 3min 22s

To plot the motion energy profile, we take the mean over time (after cropping to avoid the initial latency period) for each direction.

```python
t = me.filter_grid(duration / framerate, framerate)
energy_profile = bank_energy[:, t > .2].mean(axis=-1)
```

```python
f, ax = plt.subplots()
ax.plot(thetas, energy_profile, "o-")
ax.set(xlabel=r"Motion energy filter orientation ($\theta$)",
       ylabel="Motion energy (a.u.)")
f.tight_layout()
```

```python

```


**It would be nice to do a Judea Pearl-type DAG**

Let's say we're interested in predicting a college-football game. What are all the things that influence the outcome? Here's a list of things that come to mind:

* Team A's offensive strength ($A_o$).
* Team B's offensive strength ($B_o$).
* Team A's defensive strength ($A_d$).
* Team B's defensive strength ($B_d$).
* Team A's special-teams strength ($A_s$).
* Team B's special-teams strength ($B_s$).
* Team A's "heart and determination" ($A_h$).
* Team B's "heart and determination" ($B_h$).
* Home-field advantage ($H$).
* Referees ($R$).
* Other influences, which I'll call The X Factor.

Obviously, this list is incomplete: there are missing variables (perhaps each team's previous-week result) and some variables are aggregates of more finely grained variables (for example, offensive abilities is a combination of passing abilities and rushing abilities.) But to make things easy, pretend that only these variables determine the outcome of football games and they do so in the following way:

$MOV = (A_o − B_d) − (B_o − A_d) + (A_s − B_s) + (A_h − B_h) + H + R + X$,

where $MOV$ is Team A's margin of victory. $MOV$ can take positive and negative values. A negative $MOV$ means Team B wins.

We can use this equation to make predictions. For example, given two equal teams ($A_o = B_o$, $A_d = B_d$, $A_s = B_s$, and $A_h = B_h$) and unbiased refs ($R = 0$), A will win by $H + X$ points.

Equations require consistency of units:

1. To be equal, quantities must have the same units. Five oranges do not equal five apples, and five miles do not equal five miles per hour. This means the left- and right-hand side equations must have the same units. In our football equation, the left-hand side is expressed in units. Then the right-hand side must be measured in points too.
2. We can only add and subtract quantities with the same units. Therefore,   Because the left-hand side is measured in points, the right-hand side must be measured in points. And because the right-hand side is a sum, each part of the right-hand side must be measured in points.

We call this precisely defined relationship between the causes (on the right) and the effect (on the left) the **data-generating process**. This equation is easily interpeted:

1. A one-point change in any of these causes changes $MOV$ by one point; for example, increasing $H$ from 2 to 3, holding the other causes constant, increases $MOV$ by 1.
3. $(A_o − B_d)$: A's offensive contribution to $MOV$ depends not only on A's offensive strength but also on B's defensive strength. Each additional point of A's offensive strength increases $MOV$ by one, and each additional point of B's defensive strength decreases $MOV$ by one.
4. $− (B_o − A_d)$: B's offensive contribution to $MOV$ depends not only on B's offensive strength but also on A's defensive strength. Each additional point of B's offensive strength decreases $MOV$ by one, and each additional point of A's defensive strength increases $MOV$ by one.
5. $(A_s − B_s)$: A's special-teams contribution to $MOV$ depends not only on A's special-teams strength but also on B's special-teams strength. Each additional point of A's special-teams strength increases $MOV$ by one, and each additional point of B's special-teams strength decreases $MOV$ by one.
6. $(A_h − B_h)$: A's heart and determination contributes to $MOV$ only as much as it exceeds B's heart and determination. If B's heart exceeds A's heart, this term is negative.
7. $H$: conventional wisdom tells us that this term is positive if A is home and negative if A is away. An $H$ of 3 means that the home team gets the equivalent of an extra field goal by playing at home.
8. $R$: The refs can be biased. A positive $R$ means the refs make calls in favor of A, and a negative $R$ means the refs make calls in favor of B.
9. $X$: This is a catch-all. Many things can affect the outcome of the game, such as weather, injuries, and unlucky bounces. $X$ captures all of these influences. $X$ is positive if, in the aggregate, these things help A and negative if these things hurt A.

# Why Statistics
If we perfectly knew the values for each part of the right-hand side of the equation, we could perfectly predict the result of each game. Unfortunately,

# Random Variable

A random variable or stochastic variable is, roughly speaking, a variable whose value results from a measurement on some type of random process.

It is easy to confuse random variables with algebraic variables, but the two differ. The value of an algebraic variable is deterministic (i.e. the variable can take multiple values, but given inputs to the deterministic process there is only
one possible value that the algebraic variable can take) while the value of a random variable is at least partly determined by a random process (i.e. even if a deterministic process underlies a random variable, knowing inputs to the deterministic process is not good enough to know the value of the random variable with certainty.) Here are a few examples:

1. **algebraic variable** y: y = 2x + 3  If we know x, then we know y with certainty. If x = 2, y must equal 7. If x = 1, y cannot equal anything but 5. An algebraic variable like this has a two-way functional relationship: we can calculate x given y (x = (y − 3)) (?).
2. **random variable** y: Nature assigns y such that P (y = 1) = .5 and P (y = 0) = .5  In this example, y can take 0 or 1. There is not a deterministic process that determines whether y will equal 1 or 0, so knowing x (or any other potential inputs) does not tell us with certainty what value y will take; y could still take a 0 or 1. Unless the process assigns an outcome with probability 1, it is random.
3. **random variable** z: z = 2x + 3 + y  Assume y from the above example. Then knowing x does not tell us with certainty what value z will take. If x = 1, then z could equal 5 or 6 (with equal probability). If x = 4, then z could equal 11 or 12 (with equal probability).  Note that a variable that is a function of a random variable is also a random variable.

Often we treat deterministic processes as random because it is simpler to think of them that way. For example, if we knew the exact weight and measurement of a die and the speed, height, rotation, etc. at which it was tossed, we might be able to figure out exactly which side would come up (this has been demonstrated using the coin toss). But getting that information and doing those calculations is a burden, and treating it as random is simpler.

Formally, a random variable is a function from a probability space, typically to the real numbers, that is measurable. (For finite probability spaces, the measurable requirement is superfluous.) Random variables can be classified as either discrete
(a random variable that may assume either a finite number of values or an infinite sequence of values) or as continuous (a variable that may assume any numerical value in an interval or collection of intervals). A random variable's possible values
might represent the possible outcomes of a yet-to-be-performed experiment, or the potential values of a quantity whose already-existing value is uncertain (for example, as a result of incomplete information or imprecise measurements).

```python

```

## Power Distribution
$f_X(X=x|k) = cx^{-k}$
Note that $x$ and $k$ need constraints. For example, if $k = -2$ the distribution doesn't integrate:

```python
import numpy
import matplotlib.pyplot as plt
%matplotlib inline

x = numpy.linspace(0.1,10,99)
y = x**2
plt.plot(x,y)
plt.show()
```

To force large $x$ toward 0, $k$ needs to be positive. And if $k$ is positive, $x \geq 1$.

Let's find the normalizing constant ($c$):

$1 = \int_{1}^{\infty} c x^{-k} dx$

$1 = c \bigl[ \frac{1}{1-k} x^{1-k} + d \bigr]_{1}^{\infty}$

$1 = \frac{c}{1-k} \bigl[ 0 - 1 \bigr]$

$1 = \frac{c}{k-1}$

$c = k-1$

So the power law density function is
$\begin{equation}
  f_X(X=x | k)=\begin{cases}
    (k-1)x^{-k} & \text{if }1 \leq x < \infty \text{ and } k > 0 \\
    0 & \text{otherwise}.
  \end{cases}
\end{equation}$

Here's what $f_X(X=x | k=2)$ looks like:

```python
power_k2 = lambda x: x**-2 if x>=1 else 0
x = numpy.linspace(0.1,10,99)
y = [power_k2(z) for z in x]
plt.plot(x,y)
plt.show()
```

Nassim Taleb offered the following quiz that uses a power distribution. Note his typo ("$q=.07$" should be "$q=.007$"). Using our equation, what is $k$?

First integrate the distribution from some $y$ to infinity:
$1-F_X(X=x|k) = 1 - \int_1^y (k-1)x^{-k} dx$

$1-F_X(X=x|k) = 1 - \biggl[ (k-1) \bigl[ \frac{1}{1-k} x^{1-k} \bigr]_1^y \biggr]$

$1-F_X(X=x|k) = 1 - \biggl[ (-1) \bigl[ y^{1-k} - 1 \bigr] \biggr]$

$1-F_X(X=x|k) = 1 - \biggl[ 1 - y^{1-k} \biggr]$

$1-F_X(X=x|k) = y^{1-k}$

Then

$.45 = .007^{1-k}$

$\ln{.45} = (1-k) \ln{.007}$

$k = 1 - \frac{ \ln{.45} }{ \ln{.007} }$

$k = .84$

## Convex set of distributions
Is this a mixture distribution?

What happens as data move from simple to complex? We look at it using a convex set of a simple distribution (uniform) and a complex distribution (power).

First the uniform:

$\begin{equation}
  f_X(X=x)=\begin{cases}
    1 & \text{if }1 \leq x \leq 2 \\
    0 & \text{otherwise}.
  \end{cases}
\end{equation}$

Then the power:

$\begin{equation}
  g_X(X=x | k)=\begin{cases}
    (k-1)x^{-k} & \text{if }1 \leq x < \infty \text{ and } k \geq 0 \\
    0 & \text{otherwise}.
  \end{cases}
\end{equation}$

And the convex set:

$\begin{equation}
  h_X(X=x | \alpha, k)=\begin{cases}
    \alpha + (1-\alpha)(k-1)x^{-k} & \text{if }1 \leq x < 2 \text{ and } k > 0 \\
    (1-\alpha)(k-1)x^{-k} & \text{if }2 \leq x < \infty \text{ and } k > 0 \\
    0 & \text{otherwise}.
  \end{cases}
\end{equation}$

(No normalizing constant needed because those were included in the input distributions.)

```python
def convex_dist(x,alpha,k):
    if x>=1 and x<2 and k>0:
        return alpha + (1-alpha)*(k-1)*x**-k
    elif x>=2 and k>0:
        return (1-alpha)*(k-1)*x**-k
    else:
        return 0

x = numpy.linspace(0.1,10,99)
y0 = [convex_dist(z,0,2) for z in x]
y5 = [convex_dist(z,0.5,2) for z in x]
y1 = [convex_dist(z,1,2) for z in x]
plt.plot(x,y0)
plt.plot(x,y5)
plt.plot(x,y1)
plt.show()
```