Assignment 1: Iterative Velocity Analysis

Assignment 3: Computation of Jacobian and workspace

DH Parameters

## Assignment of $i$ frame and $\bar{i}$ frame for each of the links is as depicted in the figure above. The DH parameters are also shown in the same figure.

```python
# Created by Dr. Sangamesh Deepak R to teach Robotics online during covid 19 outbreak

import sympy as sy
import numpy as np
sy.init_printing()
```

```python
# Link parameters
a0 = 0
a1 = 0
a2 = sy.Symbol(r'L_3')

alpha0 = 0
alpha1 = -sy.pi/2
alpha2 = 0

```

```python
# Joint parameters

theta1 = sy.Symbol(r'\theta_1')
theta2 = sy.Symbol(r'\theta_2')
theta3 = sy.Symbol(r'\theta_3')

d1 = sy.Symbol(r'L_1') + sy.Symbol(r'L_2')
d2 = 0
d3 = 0

```

```python
# transformation from of i' frame with respect to i frame

def link_transform(a_i, alpha_i):
    Link_T = sy.Matrix([[1, 0, 0, a_i], [0, sy.cos(alpha_i), -sy.sin(alpha_i), 0], [0, sy.sin(alpha_i), sy.cos(alpha_i), 0], \
                   [0,0,0,1] ])
    return Link_T
```

```python
# transformation of i frame with respect to (i-1)' frame'
def joint_transform(d_i, theta_i):
    Joint_T = sy.Matrix([[sy.cos(theta_i),  -sy.sin(theta_i), 0, 0],
                        [sy.sin(theta_i),  sy.cos(theta_i), 0, 0],
                        [0, 0, 1, d_i],
                        [0,0,0,1] ])
    return Joint_T

```

```python
# Computation of transformation matricies of different link frames with respect to the ground frame
T_0 = sy.Identity(4)
T_0_1 = sy.trigsimp( link_transform(a0, alpha0)*joint_transform(d1, theta1))

T_1_2 = sy.trigsimp( link_transform(a1, alpha1)*joint_transform(d2, theta2) )
T_0_2 = sy.trigsimp( T_0_1* T_1_2);

T_2_3 = sy.trigsimp(link_transform(a2, alpha2)*joint_transform(d3, theta3) )
T_0_3 = sy.trigsimp( T_0_2* T_2_3);

T_3_T = link_transform(sy.Symbol(r'L_4'), sy.pi)
T_0_T = sy.trigsimp( T_0_3* T_3_T)
```

```python
T_0_1, T_0_2, T_0_3, T_0_T # Transformation matricies of first, second, third and fourth bodies
```

```python
# Extraction of Rotation matrices
R_0_1= T_0_1[0:3,0:3]
R_1_2= T_1_2[0:3,0:3]
R_2_3= T_2_3[0:3,0:3]
R_3_T= T_3_T[0:3,0:3]

r_0_1=T_0_1[0:3,3]
r_1_2=T_1_2[0:3,3]
r_2_3=T_2_3[0:3,3]
r_3_T=T_3_T[0:3,3]

```

```python
 def cross_product(a,b):
        c=sy.Matrix([
            [a[1,0]*b[2,0]-a[2,0]*b[1,0]],
            [a[2,0]*b[0,0]-a[0,0]*b[2,0]],
            [a[0,0]*b[1,0]-a[1,0]*b[0,0]]
            ])
        return c
```

```python
d_d1=0
d_d2=0
d_d3=0

d_theta1 = sy.Symbol(r'\dot{\theta}_1')
d_theta2 = sy.Symbol(r'\dot{\theta}_2')
d_theta3 = sy.Symbol(r'\dot{\theta}_3')

d_d1, d_d2, d_d3, d_theta1, d_theta2, d_theta3
```

```python
omega_0_0 = sy.Matrix([[0],[0],[0]])
v_0_0 = sy.Matrix([[0],[0],[0]])

```

```python
omega_1_1= R_0_1.T*(omega_0_0)+sy.Matrix([[0],[0],[d_theta1] ])

v_1_1 = R_0_1.T*(v_0_0 + cross_product(omega_0_0,r_0_1))+sy.Matrix([[0],[0],[d_d1] ])

omega_1_1, v_1_1
```

```python
omega_2_2= R_1_2.T*(omega_1_1)+sy.Matrix([[0],[0],[d_theta2] ])

v_2_2 = R_1_2.T*(v_1_1 + cross_product(omega_1_1,r_1_2))+sy.Matrix([[0],[0],[d_d2] ])

omega_2_2, v_2_2
```

```python
omega_3_3= R_2_3.T*(omega_2_2)+sy.Matrix([[0],[0],[d_theta3] ])

v_3_3 = R_2_3.T*(v_2_2 + cross_product(omega_2_2,r_2_3))+sy.Matrix([[0],[0],[d_d3] ])

omega_3_3, v_3_3
```

```python
omega_T_T= R_3_T.T*(omega_3_3)
v_T_T = R_3_T.T*(v_3_3 + cross_product(omega_3_3,r_3_T))

omega_T_T, v_T_T
```

## The required expressions for ${}^{0}\boldsymbol{\omega}_{0}$, ${}^{1}\boldsymbol{\omega}_{1}$, ${}^{2}\boldsymbol{\omega}_{2}$, ${}^{3}\boldsymbol{\omega}_{3}$, ${}^{T}\boldsymbol{\omega}_{T}$, ${}^{0}\boldsymbol{v}_{0}$, ${}^{1}\boldsymbol{v}_{1}$, ${}^{2}\boldsymbol{v}_{2}$, ${}^{3}\boldsymbol{v}_{3}$, ${}^{T}\boldsymbol{v}_{T}$ are as above

```python
R_0_T= T_0_T[0:3,0:3]
v_0_T=sy.trigsimp(R_0_T*v_T_T)
omega_0_T = sy.trigsimp(R_0_T*omega_T_T)
```

```python
mu_0_T = sy.Matrix([v_0_T, omega_0_T])
mu_0_T
```

```python

a1= mu_0_T.subs([(d_theta1, 1), (d_theta2,0), (d_theta3, 0)])
a2= mu_0_T.subs([(d_theta1, 0), (d_theta2,1), (d_theta3, 0)])
a3= mu_0_T.subs([(d_theta1, 0), (d_theta2,0), (d_theta3, 1)])

```

```python
a1
```

```python
J=a1
J=J.col_insert(1,a2)
J=J.col_insert(2,a3)
J
```

## The analytical expression for the Jacobian is as found above

## The workspace of the robot when $\theta_1$ is held constant is same as that of 2R robot considered in class as well as in mid-semester examination. When there is no joint limits, the work-space was an annular circular area. When $\theta_1$ is sweeped over $2 \pi$, the annular circular area sweeps an annular SPHERE.

```python
J_num_1 = J.subs([(theta1, 0), (theta2, 0), (theta3, 0)])  # Numerical value of Jacobian at a configuration that lies at the boundary of the workspace
J_num_2 = J.subs([(theta1, 0), (theta2, 0), (theta3, sy.pi/2)]) # Numerical value of Jacobian at configuration that lies in interior of the workspace
display([J_num_1, J_num_2])
display([J_num_1.columnspace(), J_num_2.columnspace()])
```

## Rank of the Jacobian at both of the two configurations choosen above is three.

```python

```


# The many faces of the Reverend Bayes: Face 1 Revisited

---

## ...starting from Face 2 and seeing the  connection...

$$
\renewcommand{\like}{{\cal L}}
\renewcommand{\loglike}{{\ell}}
\renewcommand{\err}{{\cal E}}
\renewcommand{\dat}{{\cal D}}
\renewcommand{\hyp}{{\cal H}}
\renewcommand{\Ex}[2]{E_{#1}[#2]}
\renewcommand{\x}{{\mathbf x}}
\renewcommand{\v}[1]{{\mathbf #1}}
$$

```python
%matplotlib inline
import numpy as np
import scipy as sp
import matplotlib as mpl
import matplotlib.cm as cm
import matplotlib.pyplot as plt
import pandas as pd
pd.set_option('display.width', 500)
pd.set_option('display.max_columns', 100)
pd.set_option('display.notebook_repr_html', True)
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("poster")
```

## Linear regression in 1D

The goal is to fit a straight line:

$$ y = ax+b+\epsilon$$

where $\epsilon \sim N(0, \sigma)$
and predict $y$ at various $x$'s.

Let ${\bf x} =(x_1, x_2, \ldots, x_n)$  and
$ {\bf y} =(y_1, y_2, \ldots, y_n)$ are the actual data.

Using Bayesian formalism, we want to estimate the posteriors of the parameter
$\theta =a$ assuming $b=0$.

We use  priors $a \sim N(0, \sigma_a)$.

The implementation below estimates the posterior using Bayes rule  and also samples
directly from the product of the likelihood (same as the one in the MLE section above) and prior using the rejection method.

Using priors with mean 0 and $\sigma_a^2=4$. $a \sim N(0, \sigma_a)$

```python
## GENERATE SYNTHETIC DATA
n=9
a = 1.2 # SLOPE
b = 0 # INTERSECTION

s = 1.5 # STD OF ERROR

x = np.random.uniform(low=-3, high=7, size=n)
y = a*x+ b + s*np.random.randn(n)

#plt.subplot(2,2,2)
plt.errorbar(x,y, yerr=s, fmt='o')
plt.xlabel('x')
plt.ylabel('y')

xr = np.linspace(-3, 7, 100)
plt.plot(xr, a*xr)
```

We plot the prior and the likelihood.

```python
# ======================================
## PLOT THE PDF OF THE PRIOR
# ======================================
I = np.eye(2)

## PRIOR
sig2 = 4   # WIDTH OF THE PRIOR

p_w = lambda (w) : (1/np.sqrt(2*np.pi*sig2))*np.exp( - (w**2)/(2*sig2))

asp = np.linspace(-5,5, 100)

P=p_w(asp)

plt.plot(asp, P)
plt.xlabel('slope (a)')
plt.title('Prior for a')
```

Note that the product in the gaussians is taken into the exponential as a sum.

```python
N=1000
asp = np.linspace(-1,3, N)

p_l = lambda (asp) : (1/np.sqrt(2*np.pi*s**2))*np.exp( - (np.sum( x*x*(y/x-asp)**2)/(2*s**2)))

P=np.zeros(N)

for k in np.arange(0, asp.size):
    P[k] = p_l(asp[k])

plt.subplot(2,2,1)
plt.plot(asp, P)
plt.xlabel('slope')
plt.ylabel('Likelihood')

plt.title('Likelihood');
```

###Sampling for posterior

We use a Monte-Carlo method, rejection sampling, to sample from the posterior..we sample a box around it and reject all those samples that dont fall below the curve.

```python
N=30000
asp = np.random.uniform(low=-10, high=10, size=N)

post = lambda(wsp): p_l(wsp) * p_w(wsp)

ptr = np.zeros(N)

for k in np.arange(0, N):
    ptr[k]= post(asp[k])

ysp =  np.random.uniform(low=0, high=np.max(ptr), size=N)

idx = (ysp<ptr)

g=plt.hist(asp[idx], 20);

plt.title('Posterior');
```

We didnt need to do this here because the posterior is a gaussian, but in real life, most posteriors are not conjugates. And they dont have closed forms. We then need to sample in some fashion. Here we used rejection sampling, but we could equally well use MCMC, etc.

## The Bayesian formulation of regression and the posterior predictive distribution

(taken from Am207 class notes)

In the linear regression model we have data $D$, of $n$ observations
$D=\left\{ ({\bf x}_1, y_1), ({\bf x}_2,y_2), \ldots, ({\bf x}_n, y_n) \right\} $ where ${\bf x}$
denotes an input vector of dimension $D$ and $y$ denotes a scalar output (dependent variable).
All data points are combined into a $D \times n$ matrix $X$.
The model that determines the relationship between inputs and
output is given by

$$ y   = \bf x^{T} {\bf w} + \epsilon $$

where ${\bf w}$ is a vector of parameters of the linear model. Usually there is
a bias or offset is included, but for now we ignore it.

We assume that the additive noise  $\epsilon$ is iid Gaussian with
zero mean and variance $\sigma_n^2$

$$ \epsilon \sim N(0, \sigma^2_n) $$

###Likelihood

The likelihood is, because we assume independency, the product

\begin{eqnarray} L &=& p(\bf y|X,\bf w) = \prod_{i=1}^{n} p(y_i|\bf X_i, \bf w) = \frac{1}{\sqrt{2\pi}\sigma_n}  \prod_{i=1}^{n}
   \exp{ \left( -\frac{(y_i-\bf X_i^T \bf w)^2}{2\sigma_n^2} \right)}  \nonumber \\
   &=& \frac{1}{\sqrt{2\pi}\sigma_n} \exp{\left( -\frac{| \bf y-X^T \bf w|^2 }{2\sigma_n^2} \right)} = N(X^T \bf w,  \sigma_n^2 I)
   \end{eqnarray}

where $|x|$ denotes the Euclidean length of vector $\bf x$.

An alternative way of expressing the likelihood, which is convenient
when sampling ${\bf w}$'s:

\begin{eqnarray}
p(\bf y|\bf X,\bf w) &=& \frac{1}{\sqrt{2\pi}\sigma_n} \exp{\left( -\frac{| X^{-1} \bf y- \bf w|^2 \,\, (X^{-1} (X^{-1})^{T}) }{2\sigma_n^2} \right)} = N(X^{-1} \bf y,  \sigma_n^2 (X^{-1} (X^{-1})^{T}) )
   \end{eqnarray}

###Prior

In the Bayesian framework inference we need to specify a prior
over the parameters that expresses our belief about the parameters
before we take any measurements. A wise choice is a zero mean
Gaussian with covariance matrix $\Sigma$

\begin{equation}
 \bf w \sim N(0, \Sigma)
\end{equation}

###Posterior
We can now continue with the standard Bayesian formalism

\begin{eqnarray}
 p(\bf w| \bf y,X) &\propto& p(\bf y | X, \bf w) \, p(\bf w) \nonumber \\
                       &\propto& \exp{ \left(- \frac{1}{2 \sigma_n^2}(\bf y-X^T \bf w)^T(\bf y - X^T \bf w) \right)}
                        \exp{\left( -\frac{1}{2} \bf w^T \Sigma^{-1} \bf w \right)}  \nonumber \\
 \end{eqnarray}

In the next step we `complete the square' and obtain

\begin{equation}
p(\bf w| \bf y,X)  \propto  \exp \left( -\frac{1}{2} (\bf w - \bar{\bf w})^T  (\frac{1}{\sigma_n^2} X X^T + \Sigma^{-1})(\bf w - \bar{\bf w} )  \right)
\end{equation}

where $\bar{\bf w} = \sigma_n^{-2}( \sigma_n^{-2}XX^T +\Sigma^{-1})^{-1} X \bf y $

This looks like a Gaussian and therefore the posterior is

$$ p(\bf w| X, {\bf y}) \sim {\cal N}( \frac{1}{\sigma_n^2} A^{-1}Xy , A^{-1} ) $$

where $A= \sigma_n^{-2}XX^T +\Sigma^{-1}$

To make predictions for a test case we average over all possible parameter predictive distribution
values, weighted by their posterior probability. This is in contrast to non Bayesian schemes, where a single parameter is typically chosen by some criterion.

###Regularization

If we assume that $\Sigma$ is a diagonal covariance matrix so that

$$\bf w \sim N(0, \alpha \bf I)$$

then we get back alpha as the regularization parameter from ridge regression, since our "estimation risk" is just the log of the gaussian, and terms in exponents add!!

Similarly Lasso comes from a laplace prior.

###Posterior Predictive Distribution

Thus the predictive distribution at some $x^{*}$ is given by averaging the output of all possible linear models w.r.t. the  posterior

\begin{eqnarray}
p(y^{*} | x^{*}, {\bf x,y}) &=& \int p({\bf y}^{*}| {\bf x}^{*}, {\bf w} ) p(\bf w| X, y)dw \nonumber \\
                                    &=& {\cal N} \left(\frac{1}{\sigma_n^2} x^{*^{T}} A^{-1} X {\bf y}, x^{*^T}A^{-1}x^{*} \right),
\end{eqnarray}

which is again Gaussian, with a mean given by the posterior mean multiplied by the test input
and the variance is a quadratic
form of the test input with the posterior covariance matrix, showing that the
predictive uncertainties grow with the magnitude of the test input, as one would
expect for a linear model.

```python
a0=-0.3
a1=0.5
N=20
noiseSD=0.2
priorPrecision=2.0
u=np.random.rand(20)
x=2.*u -1.
def randnms(mu, sigma, n):
    return sigma*np.random.randn(n) + mu
y=a0+a1*x+randnms(0.,noiseSD,N)
plt.scatter(x,y)
```

```python
from scipy.stats import norm
from scipy.stats import multivariate_normal
def cplot(f, ax=None):
    if not ax:
        plt.figure()
        ax=plt.gca()
    xx,yy=np.mgrid[-1:1:.01,-1:1:.01]
    pos = np.empty(xx.shape + (2,))
    pos[:, :, 0] = xx
    pos[:, :, 1] = yy
    ax.contourf(xx, yy, f(pos))
    #data = [x, y]
    return ax
def plotSampleLines(mu, sigma, numberOfLines, dataPoints=None, ax=None):
    #Plot the specified number of lines of the form y = w0 + w1*x in [-1,1]x[-1,1] by
    # drawing w0, w1 from a bivariate normal distribution with specified values
    # for mu = mean and sigma = covariance Matrix. Also plot the data points as
    # blue circles.
    #print "datap",dataPoints
    if not ax:
        plt.figure()
        ax=plt.gca()
    for i in range(numberOfLines):
        w = np.random.multivariate_normal(mu,sigma)
        func = lambda x: w[0] + w[1]*x
        xx=np.array([-1,1])
        ax.plot(xx,func(xx),'r', alpha=0.5)
    if dataPoints:
        ax.scatter(dataPoints[0],dataPoints[1])
    ax.set_xlim([-1,1])
    ax.set_ylim([-1,1])

```

```python
likelihoodSD = noiseSD # Assume the likelihood precision, beta, is known.
likelihoodPrecision = 1./(likelihoodSD*likelihoodSD)
priorMean = np.zeros(2)
priorSigma = np.eye(2)/priorPrecision #Covariance Matrix
priorPDF = lambda w: multivariate_normal.pdf(w,priorMean,priorSigma)
priorPDF([1,2])
```

    0.0021447551423913074

```python
cplot(priorPDF);
```

```python
plotSampleLines(priorMean,priorSigma,6)
```

```python
# Given the mean = priorMu and covarianceMatrix = priorSigma of a prior
# Gaussian distribution over regression parameters; observed data, xtrain
# and ytrain; and the likelihood precision, generate the posterior
# distribution, postW via Bayesian updating and return the updated values
# for mu and sigma. xtrain is a design matrix whose first column is the all
# ones vector.
def update(x,y,likelihoodPrecision,priorMu,priorSigma):
    postSigmaInv  = np.linalg.inv(priorSigma) + likelihoodPrecision*np.outer(x.T,x)
    postSigma = np.linalg.inv(postSigmaInv)
    postMu = np.dot(np.dot(postSigma,np.linalg.inv(priorSigma)),priorMu) + likelihoodPrecision*np.dot(postSigma,np.outer(x.T,y)).flatten()
    postW = lambda w: multivariate_normal.pdf(w,postMu,postSigma)
    return postW,postMu,postSigma
```

```python
# For each iteration plot  the
# posterior over the first i data points and sample lines whose
# parameters are drawn from the corresponding posterior.
fig, axes=plt.subplots(figsize=(12,30), nrows=5, ncols=2);
mu = priorMean
sigma = priorSigma
iterations=2
muhash={}
sigmahash={}
k=0
for i in [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]:
    postW,mu,sigma = update(np.array([1,x[i]]),y[i],likelihoodPrecision,mu,sigma)
    muhash[i]=mu
    sigmahash[i]=sigma
    if i in [1,4,7,10,19]:
        cplot(postW, axes[k][0])
        plotSampleLines(muhash[i],sigmahash[i],6, (x[0:i],y[0:i]), axes[k][1])
        k=k+1

```

### The multiple risks

We learned earlier that there are *two risks in learning* that we must consider, one to *estimate probabilities*, which we call **estimation risk**, and one to *make decisions*, which we call **decision risk**. We motivated these by the different processes needed to estimate probabilities and make decisions, but deferred a further discussion of them. Now's the time to have that discussion.

What do we mean by a "decision" exactly? We'll use the letter $g$ here to indicate a decision, in both the regression and classification problems. In the classification problem, one example of a decision is the process used to choose the class of a sample, given the probability of being in that class. As another example, consider the cancer story from before. The decision may be: ought we biopsy, or ought we not biopsy. By minimizing the estimation risk, we obtain a probability that the patient has cancer. We must mix these probabilities with "business knowledge" or "domain knowledge" to make a decision.

What we must additionally supply is the **decision loss** $l(y,g)$ or **utility** $u(l,g)$ (profit, or benefit) in making a decision $g$ when the predicted variable has value $y$. For example, we must provide all of the losses $l$(no-cancer, biopsy), $l$(cancer, biopsy), $l$(no-cancer, no-biopsy), and $l$(cancer, no-biopsy). One set of choices for these losses may be 20, 0, 0, 200 respectively.

To simplify matters though, lets presently insist that the **decision space** from which the decision $g$is chosen is the same as the space from which $y$ is chosen. In other words, the decision to be made is a classification. Then we can use these losses to penalize mis-classification asymmetrically if we desire. In the cancer example, we then set $l$(observed-no-cancer, predicted cancer) to be 20 and $l$(observed-cancer, predicted-no-cancer) to be 200. This is the situation we encountered before where we penalize the false negative(observed cancer not predicted to be cancer) much more than the false positive(observed non-cancer predicted to be cancer).

## Circling the wagons: Learning uses average risk

What is this process? The idea is quite simple, and you have probably already thought about it intuitively even without knowing it. We simply weigh each combinations loss by the probability that that combination can happen:

$$R_{g}(x) = \sum_y l(y,g(x)) p(y|x).$$

That is, we calculate the **average risk** over all choices y, of making choice g for a given sample.

Then, if we want to calculate the overall risk, given all the samples in our set, we calculate:

\begin{eqnarray*}
R(g) &=& \sum_x p(x) R_{g}(x) = \sum_x p(x) \sum_y l(y,g(x)) p(y|x) \\
&=& \sum_x \sum_y l(y,g(x)) p(x) p(y|x) = \sum_x \sum_y l(y,g(x)) p(x,y)
\end{eqnarray*}

To make the best decision, we then minimize over all possible decisions $g(x)$:

$$g^* = \arg\min_{g} R(g).$$

for which it is enough to do the minimization on a per-sample basis:

$$g^*(x) = \arg\min_{g(x) \in y} R_{g}(x).$$

For example, if there are two classes $1$ and $0$ we can classify data into, we have:

$$R_g(x) = l(1, g)p(1|x) + l(0, g)p(0|x).$$

Then for the "decision" $g=1$ we have:

$$R_1(x) = l(1,1)p(1|x) + l(0,1)p(0|x),$$

and for the "decision" $g=0$ we have:

$$R_0(x) = l(1,0)p(1|x) + l(0,0)p(0|x).$$

Now, we'd choose $1$ for the sample at $x$ if:

$$R_1(x) \le R_0(x).$$

This process is illustrated in the diagram below.

####The symmetric case with 1-0 risk

For the 1-0 loss, $l(1,1) = l(0,0) =0$ and $l(1,0) = l(0,1) = 1$, and we get:

$$R_1(x) = p(0|x), R_0(x) = p(1|x).$$

We'd choose $1$ if:

$$R_1(x) \le R_0(x)$$

for a given sample $x$. Thus we get back the "intuitive" prescrription for classification we have been using so far: **choose $1$ if**:

$$p(1|x) \ge p(0|x).$$

Since these add to 1, this is equivalent to saying **choose $1$ if $p(1|x) \ge 0.5$**

This risk is one you cant do better than, and is called the **Bayes Risk**.

Where in the machine learning process ought one to do this risk penalization? Remember again that there are two risks in learning. The first risk is the estimation risk, and we use this to *estimate the model on the training set*. We then cross-validate by calculating a score, or risk, on the validation set, and minimizing it. What is this risk that we minimize on the validation set? Is this the estimation risk, or is this the decision risk?

Remember that ERM is actually a special case of the risk minimization rule introduced here. Lets see this again (wee saw it before in Bayes 1). Basically we choose as the joint probability $p(x, y)$ a special kind of function which has the value $1/N$ at every sample and zero elsewhere: this is called the **empirical probability distribution** and is written like this:

$$p(x,y) = \frac{1}{N}\sum_{i \in \dat} \delta(x - x_i) \delta(y - y_i).$$

The function $\delta(x - x_i)$ is a very simple function, called the **Dirac Delta** function, which has value 1 at $x=x_i$ and 0 elsewhere. Thus

$$\sum_x f(x) \delta(x - x_i) = f(x_i)$$

\begin{eqnarray*}
R(g) = \sum_x \sum_y l(y,g(x)) p(x,y) &=& \sum_x \sum_y l(y, g(x)) \frac{1}{N}\sum_{i \in \dat} \delta(x - x_i) \delta(y - y_i)\\
&=& \frac{1}{N}\sum_{i \in \dat} \left(\sum_x \delta(x - x_i) \sum_y  \delta(y - y_i) l(y, g(x)) \right)\\
&=& \frac{1}{N}\sum_{i \in \dat} \left(\sum_x \delta(x - x_i) l(y_i, g(x)) \right)\\
&=& \frac{1}{N}\sum_{i \in \dat} l(y_i, g(x_i))
\end{eqnarray*}

Models like SVM which do not give us probabilities are learned directly with the ERM risk both as estimation and decision risk.

###The regression case

####The use of the posterior predictive

This is exactly the risk we use in ERM. For regression we use $l(y_i, g(x_i)) = (y_i - g(x_i)^2$, where we interpret $g$ to be a **point-estimate** from a distribution of possible $y$'s at a given x. Where did this distribution arise from? In the Bayesian scenario this distribution arises naturally as the *posterior predictive*. In the frequentist scenario, we get them from different samples of our true population. Well, each set gives us a different $y$ and thus we get a distribution at a given x. Now, whow do we choose g? We can use decision loss $l(y,g) = (g-y)^2$ or even $l(y,g) = | g- y |$. The former one leads to the mean as the point estimate, the latter the median.

This is why in beginning stats courses, the regression line is $E[y|x]$. It didnt have to be, but the choice of the squared loss makes it so.

Remember that the idea we used in empirical risk minimization (ERM) is that validation or cross-validation risk is a proxy that estimates the risk on the population, the out-of-sample risk. But, notice from above that ERM minimizes decision risk, having **already estimated the probability distribution by the empirical probability distribution**. Thus, the model with the lowest cross-validation risk minimizes the population or out of sample error, leading to a model which generalizes well in the sense that the model that **will best minimize decision risk on future samples**. Or to put it another way, minimize the decision risk in your c-val process. Thats why standard `sklearn` uses accuracy, and cross-validation routines allow us to specify different scores, such as the F1-score, etc. And this is why you should be calculating ROC curves on the validation set, rather than the test set as we did on the homework. (The other reason for doing that is that it gives us an inference on the ROC curve, AUC, or associated metric).

###Back to classification

Typically, in a probabilistic classification model, say for example, when we do logistic regression with a log-loss, we have used this log-loss as an estimation loss to obtain the probability density $p(y|x)$. Two things now need to happen in the validation process: (a) we need to estimate the regularization parameter $C$, and (b) we need to make classification decisions using our decision risk.

####Setting the hyperparameters
But remember that we also set hyperparameters like $C$ based on our validation process. At first glance, this might seem wierd, since the regularization term is part of the estimation loss. But if you think about *choosing hyperparameters as choosing between different models, as opposed to choosing parameters of a model*, the wierdness goes away. What you do is you choose a value of $C$ on the grid, and *then train* the models other parameters using *estimation risk* on the training data. We then calculate the *decision risk* on the validation data. We compare all these decision risks and choose the $C$ with the lowest decision risk; that is, we choose the *model* with the smallest validation-set decision risk:

$$R_{decision}(g_C) = \frac{1}{N} \sum_{i \in \dat_{validation}} l(y_i,g_C(x_i)).$$

$$g_{C^*} = \arg\min_{g_C} R_{decision}(g_C).$$

Here $g_C$ indicates that the model is chosen for a particular value of $C$, and $C^*$ is the final value we settle on as part of the minimization process.If our decision loss is different, such as in the asymmetric case, it is this asymmetric loss we must use at the validation stage, since *generalization means finding a model that will best minimize decision risk on future samples*.

Note that validation sets are typically smaller than training sets, since we are fitting less hyperparameters like $C$ in this model selection process on the validation set than model parameters like logistic regression coefficients on the training sets.

####Validation is ERM on decision risk

Something else might be bothering you now. When we trained our model on the training set for a given value of $C$, we did two things: we estimated the parameters of the model, and we estimated $p(y|x)$. Why not use this probability directly in the equation for overall risk to calculate it? Why are we doing the sum over validation points above? The answer is that we dont have an estimate of $p(x)$, which is needed in the calculation of the overall risk. So instead we must use an approximation to $p(x)$:

$$p(x) = \frac{1}{N_{validation}} \sum_{i \in \dat_{validation}} \delta(x - x_i)$$

which then gives us the equation we used above.

####Recap of the process of learning

With these loose ends tied up, lets recap what we did: we used the estimation loss (log-loss in Logistic Regression) to estimate the probabilities. Then we were faced with the problem of converting the probabilities into classification decisions. For this we used the decision loss on the validation data set, which might be identical to the estimation loss, a 1-0 loss in the symmetric case, or even an asymmetric loss like the one used above. The validation process provides a second service: model selection, helping us pick the best validation parameter. Furthermore, the validation risk is  an unbiased estimate of the out-of-sample risk, and by using the decision risk as the validation risk, we ensure that "good generalization" means choosing a model which makes the "right" decisions on future data.

In ERM only models like SVM (where we use a regularized hinge loss), there is no probability estimation. Typically we'lll use the same loss for parameter estimation (rather than likelihood or probability estimation) in the estimation phase on the training data, as we use in the decision phase on the validation data. Of-course if we directly want to make a decision like biopsy|no-biopsy or an asymmetric classification, we alter the decision risk to take care of this, having trained the parameters of our model (such as the SVM line/plane coefficients).


This is a notebook explaining the mathematics of Linear and Logistic Regression. It is adapted from by [blog](http://ataspinar.com/2016/03/28/regression-logistic-regression-and-maximum-entropy). All of the code is available on [GitHub](https://github.com/taspinar/siml).

 1. Regression Analysis

Regression Analysis is the field of mathematics where the goal is to find a function which best [correlates](http://guessthecorrelation.com/) with a dataset. Let's say we have a dataset containing $ n $ datapoints;

$ X = ( x^{(1)}, x^{(2)}, .., x^{(n)} ) $.

For each of these (input) datapoints there is a corresponding (output) $ y^{(i)}$-value.

Here, the $ x$-datapoints are called the [independent variables](https://en.wikipedia.org/wiki/Dependent_and_independent_variables) and $ y$ the dependent variable;
the value of $ y^{(i)} $ depends on the value of $ x^{(i)} $, while the value of $ x^{(i)}$ may be freely chosen without any restriction imposed on it by any other variable.

The goal of Regression analysis is to find a function $ f(X) $ which can best describe the correlation between $ X $ and $ Y $. In the field of Machine Learning, this function is called the hypothesis function and is denoted as $ h_{\theta}(x) $. If we can find such a function, we can say we have successfully built a Regression model.

If the input-data lives in a 2D-space, this boils down to finding a curve which fits through the data points. In the 3D case we have to find a plane and in higher dimensions a hyperplane.

To give an example, let's say that we are trying to find a predictive model for the success of students in a course called Machine Learning. We have a dataset $ Y $ which contains the final grade of $ n $ students. Dataset $ X $ contains the values of the independent variables. Our initial assumption is that the final grade only depends on the studying time. The variable $ x^{(i)} $ therefore indicates how many hours student $ i $ has studied. The first thing we would do is visualize this data:

If the results looks like the figure on the left, then we are out of luck. It looks like the points are distributed randomly and there is no correlation between $ Y$ and $ X$ at all. However, if it looks like the figure on the right, there is probably a strong correlation and we can start looking for the function which describes this correlation.

This function could for example be:

$ h_{\theta}(X) =  \theta_0+ \theta_1 \cdot x $

or

$ h_{\theta}(X) = \theta_0 + \theta_1 \cdot x^2 $

where $ \theta $ are the dependent parameters of our model.

 1.2 Multivariate Regression

In evaluating the results from the previous section, we may find the results unsatisfying; the function does not correlate with the datapoints strongly enough. Our initial assumption is probably not complete. Taking only the studying time into account is not enough.
The final grade does not only depend on the studying time, but also on how much the students have slept the night before the exam. Now the dataset contains an additional variable which represents the sleeping time.

Our dataset is then given by $ X = ( (x_1^{(1)}, x_2^{(1)}), (x_1^{(2)}, x_2^{(2)}), .., (x_1^{(n)}, x_2^{(n)}) ) $. In this dataset $ x_1^{(i)} $ indicates how many hours student $ i $ has studied and $ x_2^{(i)} $ indicates how many hours he has slept.

This is an example of [Multivariate Regression](https://en.wikipedia.org/wiki/Linear_regression#Simple_and_multiple_regression). The function has to include both variables. For example:

$ h_{\theta}(x) = \theta_0 + \theta_1 \cdot x_1 + \theta_2 \cdot x_2$

or

$ h_{\theta}(x) = \theta_0 + \theta_1 \cdot x_1 + \theta_2 \cdot x_2^3 $.

 1.3 Linear vs Non-Linear Regression

All of the above examples are examples of linear regression. We have seen that in some cases $ y^{(i)} $ depends on a linear form of $ x^{(i)} $, but it can also depend on some power of $ x^{(i)}$, or on the log or any other form of $ x^{(i)}$. However, in all cases its dependence on the parameters $\theta$ was linear.

So, what makes linear regression linear is not that $ Y$ depends in a linear way on $ X $, but that it depends in a linear way on $ \theta$.
$ Y $ needs to be [linear](http://www.investopedia.com/terms/l/linearrelationship.asp) with respect to the model-parameters $ \theta $. Mathematically speaking it needs to satisfy the [superposition principle](http://www.cut-the-knot.org/do_you_know/superposition.shtml).

Examples of nonlinear regression would be:

$ h_{\theta}(x) = \theta_0 + x_1^{\theta_1} $

or

$ h_{\theta}(x) = \theta_0 + \theta_1 / x_1 $

The reason why the distinction is made between linear and nonlinear regression is that nonlinear regression problems are more difficult to solve and therefore more computational intensive algorithms are needed.

Linear regression models can be written as a linear system of equations, which can be solved by finding the closed-form solution $ \theta = ( X^TX )^{-1}X^TY $ with Linear Algebra. See these statistics notes for more on solving linear models with linear algebra.

As discussed before, such a closed-form solution can only be found for linear regression problems. However, even when the problem is linear in nature, we need to take into account that calculating the inverse of a $ n $ by $ n $ matrix has a time-complexity of $ O(n^3) $. This means that for large datasets ($ n \gt 10.000 $ ) finding the closed-form solution will take more time than solving it iteratively (gradient descent method) as is done for nonlinear problems. So solving it iteratively is usually preferred for larger datasets, even if it is a linear problem.

 1.4 Gradient Descent

The Gradient Descent method is a general optimization technique in which we try to find the value of the parameters $ \theta $ with an iterative approach.

First, we construct a cost function (also known as loss function or error function) which gives the difference between the values of the hypothesis function $h_{\theta}(x)$ (the values you expect $ Y$ to have with the current values of $ \theta$ ) and the actual values of $ Y $. The better your estimation of $ \theta $ is, the better the values of $ h_{\theta}(x) $ will approach the values of $ Y$ and the smaller the cost function will be.

Usually, the cost function is expressed as the squared error of the difference between these functions:

$ J(x) = \frac{1}{2n} \sum_i^n ( h_{\theta}(x^{(i)}) - y^{(i)} )^2 $

At each iteration we choose new values for the parameters $ \theta$, and move towards the 'true' values of these parameters, i.e. the values which make this cost function as small as possible. The direction in which we have to move is the negative gradient direction;

 $ \Delta\theta = - \alpha \frac{d}{d\theta} J(x) $.

The reason for this is that  a function's value decreases fastest if we move towards the direction of the negative gradient (the directional derivative is maximal in the direction of the gradient).

Taking all this into account, this is how gradient descent works:

Make an initial but intelligent guess for the values of the parameters $ \theta $.

Keep iterating while the value of the cost function has not met your criteria:

With the current values of $ \theta $, calculate the gradient of the cost function J  ( $ \Delta \theta = - \alpha \frac{d}{d\theta} J(x)$ ).

Update the values for the parameters $ \theta := \theta + \alpha \Delta \theta $

Fill in these new values in the hypothesis function and calculate again the value of the cost function;

Just as important as the initial guess of the parameters is the value you choose for the learning rate $ \alpha $. This learning rate determines how fast you move along the slope of the gradient. If the selected value of this learning rate is too small, it will take too many iterations before you reach your convergence criteria. If this value is too large, you might overshoot and not converge.

 2. Logistic Regression

Logistic Regression is similar to (linear) regression, but adapted for the purpose of classification. The difference is small; for Logistic Regression we also have to iteratively apply the gradient descent method to estimate the values of the parameter $ \theta$. And again, during the iteration, the values are estimated by taking the gradient of the cost function. And also, the cost function is given by the squared error of the difference between the hypothesis function $ h_{\theta}(x)$ and $ Y$. The major difference however, is the hypothesis function itself.

In order to understand the hypothesis function of Logistic Regression, we must first understand the idea behind classification.

When you want to classify something, there are a limited number of classes it can belong to. And for each of these possible classes there can only be two states for $ y^{(i)} $;
either $ y^{(i)} $ belongs to the specified class and $ y = 1 $, or it does not belong to the class and $ y = 0 $. Even though the output values $ Y $ are binary, the independent variables $ X $ are still continuous. So, we need a function which has as input a large set of continuous variables $ X $ and for each of these variables produces a binary output. This function, the hypothesis function, has the following form:

$ h_{\theta} = \frac{1}{1 + \exp(-z)} = \frac{1}{1 + \exp(-\theta x)} $.

This function is also known as the logistic function, which is a part of the sigmoid function family. These functions are widely used in the natural sciences because they provide the simplest model for population growth. However, the reason why the logistic function is used for classification in Machine Learning is its 'S-shape'.

As you can see this function is bounded in the y-direction by 0 and 1. If the variable $ z$ is very negative, the output function will go to zero (it does not belong to the class). If the variable $ z$ is very positive, the output will be one and it does belong to the class.
(Such a function is called an indicator function.)

The question then is, what will happen to input values which are neither very positive nor very negative, but somewhere 'in the middle'. We have to define a decision boundary, which separates the positive from the negative class. Usually this decision boundary is chosen at the middle of the logistic function, namely at $ z = 0 $ where the output value $ y$ is $ 0.5$.

\begin{equation}
y =\begin{cases}
1, \text{if $z \gt  0 $}.\\
0, \text{if $z \lt 0 $}.
\end{cases}
\end{equation}

As we can see in the formula of the logistic function, $ z = \theta \cdot x $. Meaning, the dependent parameter $ \theta$ (also known as the feature), maps the input variable $ x$  to a position on the $ z$-axis. With its $ z$-value,  we can use the logistic function to calculate the $ y$ -value. If this $ y$-value $ \gt 0.5 $ we assume it does belong in this class and vice versa.

So the feature $ \theta $ should be chosen such that it predicts the class membership correctly. It is therefore essential to know which features are useful for the classification task. Once the appropriate features are selected , gradient descent can be used to find the optimal value of these features.

How can we do gradient descent with this logistic function? Except for the hypothesis function having a different form, the gradient descent method is exactly the same. We again have a cost function, of which we have to iteratively take the gradient w.r.t. the feature $ \theta $ and update the feature value at each iteration.

This cost function is given by the log-likelihood function known as binary cross-entropy:

\begin{equation}
\begin{split}
J(x) = -\frac{1}{2n} \sum_i^n \left(  y^{(i)} log( h_{\theta}(x^{(i)})) + (1-y^{(i)})log(1-h_{\theta}(x^{(i)})) \right) \\
 = -\frac{1}{2n} \sum_i^n \left( y^{(i)} log(\frac{1}{1+exp(-\theta x)}) + (1-y^{(i)})log(1-\frac{1}{1+exp(-\theta x)}) \right)
\end{split}
\end{equation}

We know that:

$ log(\frac{1}{1+exp(-\theta x)}) = log(1) - log(1+exp(-\theta x)) = - log(1+exp(-\theta x))$

and

$ log(1-\frac{1}{1+exp(-\theta x)}) = log( \frac{exp(-\theta x)}{1+exp(-\theta x)}) $

$ = log(exp(-\theta x)) - log(1+exp(-\theta x)) $

$ = -\theta x^{(i)} -  log(1+exp(-\theta x)) $

Plugging these two equations back into the cost function gives us:

\begin{equation}
\begin{split}
J(x)  = - \frac{1}{2n} \sum_i^n \left( - y^{(i)} log(1+exp(-\theta x)) - (1-y^{(i)})(\theta x^{(i)} +  log(1+exp(-\theta x))) \right) \\
 = - \frac{1}{2n} \sum_i^n \left(  y^{(i)} \theta x^{(i)} -\theta x^{(i)} -log(1+exp(-\theta x)) \right)
\end{split}
\end{equation}

The gradient of the cost function with respect to $ \theta $ is given by

\begin{align}
\frac{d}{d\theta} J(x) = - \frac{1}{2n} \sum_i^n \left(  y^{(i)} x^{(i)} - x^{(i)} + x^{(i)} \frac{ exp(-\theta x)}{1+exp(-\theta x)} \right) \\
 = - \frac{1}{2n} \sum_i^n \left( x^{(i)} ( y^{(i)} - 1 +\frac{exp(-\theta x)}{1+exp(-\theta x)} ) \right) \\
 = - \frac{1}{2n} \sum_i^n  \left( x^{(i)} ( y^{(i)} - \frac{1}{1+exp(-\theta x)} ) \right) \\
 = - \frac{1}{2n} \sum_i^n  \left( x^{(i)} ( y^{(i)} - h_{\theta}(x^{(i)}) )\right)
\end{align}

So the gradient of the seemingly difficult cost function, turns out to be a much simpler equation. And since it is the gradient we use to update the values of $\theta$ this makes our work much easier.

Gradient descent for Logistic Regression is again performed in the same way:

Make an initial but intelligent guess for the values of the parameters $ \theta $.

Keep iterating while the value of the cost function has not met your criteria:

With the current values of $ \theta $, calculate the gradient of the cost function J  ( $ \Delta \theta = - \alpha \frac{d}{d\theta} J(x)$ ).

Update the values for the parameters $ \theta := \theta + \alpha \Delta \theta $

Fill in these new values in the hypothesis function and calculate again the value of the cost function;

 2.2 Logistic Regression - Example 1

Now that we have looked at the theory, let's lets have a look at an example of Logistic Regression. We will start out with the example of students passing a course or not.

Let's generate some data points. There are $n = 300$ students participating in the course Machine Learning and whether a student $ i $ passes ( $ y_i = 1 $) or not ( $ y_i = 0$ ) depends on two variables;

$x_i^{(1)}$ : how many hours student $ i $ has studied for the exam.

$x_i^{(2)}$ : how many hours student $ i $ has slept the day before the exam.

In our example, the results are pretty binary; everyone who has studied less than 4 hours fails the course, as well as everyone whose studying time + sleeping time is less than or equal to 13 hours ($x_i^{(1)} + x_i^{(2)} \leq 13$).

```python
import random
import numpy as np

def func2(x_i):
    if x_i[1]+x_i[2] >= 13:
        return 0
    else:
        return 1

def generate_data2(no_points):
    X = np.zeros(shape=(no_points, 3))
    Y = np.zeros(shape=no_points)
    for ii in range(no_points):
        X[ii][0] = 1
        X[ii][1] = random.random()*9+0.5
        X[ii][2] = random.random()*9+0.5
        Y[ii] = func2(X[ii])
    return X, Y

X, Y = generate_data2(300)

```

The results looks like this (the green dots indicate a pass and the red dots a fail)

We have a *LogisticRegression* class, which sets the values of the learning rate $\alpha$ and the maximum number of iterations *max_iter* at its initialization.

The values of *X*, *Y* are set when these matrices are passed to the “train()” function, and then the values of no_examples, no_features, and theta are determined.

We also have the hypothesis, cost and gradient functions.

The gradient descent method uses these methods to update the values of $theta$.

The “train()” method, first sets the values of the matrices X, Y and theta, and then calls the gradient_descent method.

Once the $\theta$ values have been determined with the gradient descent method, the training of the classifier is complete and we can use it to classify new examples.

```python
import numpy as np

class LogisticRegression():
    """
    Class for performing logistic regression.
    """
    def __init__(self, learning_rate = 0.7, max_iter = 1000):
        self.learning_rate = learning_rate
        self.max_iter = max_iter
        self.theta = []
        self.no_examples = 0
        self.no_features = 0
        self.X = None
        self.Y = None

    def add_bias_col(self, X):
        bias_col = np.ones((X.shape[0], 1))
        return np.concatenate([bias_col, X], axis=1)

    def hypothesis(self, X):
        return 1 / (1 + np.exp(-1.0 * np.dot(X, self.theta)))

    def cost_function(self):
        """
        We will use the binary cross entropy as the cost function. https://en.wikipedia.org/wiki/Cross_entropy
        """
        predicted_Y_values = self.hypothesis(self.X)
        cost = (-1.0/self.no_examples) * np.sum(self.Y * np.log(predicted_Y_values) + (1 - self.Y) * (np.log(1-predicted_Y_values)))
        return cost

    def gradient(self):
        predicted_Y_values = self.hypothesis(self.X)
        grad = (-1.0/self.no_examples) * np.dot((self.Y-predicted_Y_values), self.X)
        return grad

    def gradient_descent(self):
        for iter in range(1,self.max_iter):
            cost = self.cost_function()
            delta = self.gradient()
            self.theta = self.theta - self.learning_rate * delta
            if iter % 100 == 0: print("iteration %s : cost %s " % (iter, cost))

    def train(self, X, Y):
        self.X = self.add_bias_col(X)
        self.Y = Y
        self.no_examples, self.no_features = np.shape(X)
        self.theta = np.ones(self.no_features + 1)
        self.gradient_descent()

    def classify(self, X):
        X = self.add_bias_col(X)
        predicted_Y = self.hypothesis(X)
        predicted_Y_binary = np.round(predicted_Y)
        return predicted_Y_binary
```

 2.3 Logistic Regression - Example 2: Iris Dataset

Now  that the concept of Logistic Regression is a bit more clear, let's classify real-world data!
One of the most famous classification datasets is The Iris Flower Dataset. This dataset consists of three classes, where each example has four numerical features.:

```python
import pandas as pd
from evaluators import *

to_bin_y = { 1: { 'Iris-setosa': 1, 'Iris-versicolor': 0, 'Iris-virginica': 0 },
             2: { 'Iris-setosa': 0, 'Iris-versicolor': 1, 'Iris-virginica': 0 },
             3: { 'Iris-setosa': 0, 'Iris-versicolor': 0, 'Iris-virginica': 1 }
             }

#loading the dataset
datafile = '../datasets/iris/iris.data'
df = pd.read_csv(datafile, header=None)
df_train = df.sample(frac=0.7)
df_test = df.loc[~df.index.isin(df_train.index)]
X_train = df_train.values[:,0:4].astype(float)
y_train = df_train.values[:,4]
X_test = df_test.values[:,0:4].astype(float)
y_test = df_test.values[:,4]

Y_train = np.array([to_bin_y[3][x] for x in y_train])
Y_test = np.array([to_bin_y[3][x] for x in y_test])

print("training Logistic Regression Classifier")
lr = LogisticRegression()
lr.train(X_train, Y_train)
print("trained")
predicted_Y_test = lr.classify(X_test)
f1 = f1_score(predicted_Y_test, Y_test, 1)
print("F1-score on the test-set for class %s is: %s" % (1, f1))
```

    training Logistic Regression Classifier
    iteration 100 : cost 0.617481190363
    iteration 200 : cost 0.0867172909824
    iteration 300 : cost 0.0849155519757
    iteration 400 : cost 0.083336248398
    iteration 500 : cost 0.0819359612358
    iteration 600 : cost 0.080681546988
    iteration 700 : cost 0.0795474613777
    iteration 800 : cost 0.078513817168
    iteration 900 : cost 0.0775649711536
    trained
    F1-score on the test-set for class 1 is: 0.96

As you can see, our simple LogisticRegression class can classify the iris dataset with quiet a high accuracy.

```python

```

notebooks/Linear Regression, Logistic Regression.ipynb


[source](https://machinelearningmastery.com/one-vs-rest-and-one-vs-one-for-multi-class-classification/)

```python
import numpy as np
from sympy.matrices import Matrix
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
```

```python
# define dataset
X, y = make_classification(n_samples=10, n_features=2, n_informative=2, n_redundant=0, n_classes=3, n_clusters_per_class=1)
#np.round(X, 2)
```

```python
plt.scatter(*zip(*X), c=y)
```

```python
#%%script false
# X = np.array([[1], [2], [3], [4], [5], [6]])
# y = np.array([0, 0, 1, 1, 2, 2])
```

```python
X = np.array([[10], [20], [30], [40], [50], [60]])
# y = np.array([0, 0, 1, 1, 2, 2])
y = np.array([0, 2, 2, 2, 1, 1])

```

```python
# define model
model = LogisticRegression(multi_class='ovr')
# fit model
model.fit(X, y)
# make predictions
yhat = model.predict(X)
```

```python
yhat
```

    array([0, 2, 2, 2, 1, 1])

```python

```

```python
np.round(model.predict_proba(X), 2), model.predict(X),
```

    (array([[0.57, 0.  , 0.43],
            [0.08, 0.  , 0.92],
            [0.  , 0.  , 1.  ],
            [0.  , 0.11, 0.89],
            [0.  , 0.72, 0.28],
            [0.  , 0.78, 0.22]]), array([0, 2, 2, 2, 1, 1]))

Same "one versus all" but using the different commands.

```python
from sklearn.multiclass import OneVsRestClassifier

# define model
model = LogisticRegression()
# define the ovr strategy
ovr = OneVsRestClassifier(model)
# fit model
ovr.fit(X, y)
# make predictions
yhat = ovr.predict(X)
```

```python
yhat
```

    array([0, 2, 2, 2, 1, 1])

```python
 y
```

    array([0, 2, 2, 2, 1, 1])

```python
from plotly import graph_objects as go
lines = []
for y_true, y_pred, i in zip(y, yhat, range(len(yhat))):
  lines.append(go.Scatter3d(x=[y_pred, y_true], y=[i, i], z=[0, 1], mode="lines"))

fig = go.Figure(data=lines)
fig.show()
```

<meta charset="utf-8" />

```python
np.round(ovr.predict_proba(X), 2), ovr.predict(X)
```

    (array([[0.57, 0.  , 0.43],
            [0.08, 0.  , 0.92],
            [0.  , 0.  , 1.  ],
            [0.  , 0.11, 0.89],
            [0.  , 0.72, 0.28],
            [0.  , 0.78, 0.22]]), array([0, 2, 2, 2, 1, 1]))

```python
pred_probs = ovr.predict_proba(X)
```

```python
pred_probs.shape
```

    (6, 3)

```python
ovr.estimators_[0].predict_proba(X)
```

    array([[5.62337060e-02, 9.43766294e-01],
           [9.43975971e-01, 5.60240290e-02],
           [9.99790170e-01, 2.09830156e-04],
           [9.99999258e-01, 7.42171650e-07],
           [9.99999997e-01, 2.62452265e-09],
           [1.00000000e+00, 9.28102554e-12]])

```python
ovr.estimators_[1].predict_proba(X)
```

    array([[9.99999997e-01, 2.55340758e-09],
           [9.99999272e-01, 7.28161401e-07],
           [9.99792391e-01, 2.07608736e-04],
           [9.44093947e-01, 5.59060534e-02],
           [5.59066525e-02, 9.44093347e-01],
           [2.07611092e-04, 9.99792389e-01]])

```python
ovr.estimators_[2].predict_proba(X)
```

    array([[0.28886735, 0.71113265],
           [0.36806318, 0.63193682],
           [0.4550769 , 0.5449231 ],
           [0.54492311, 0.45507689],
           [0.63193683, 0.36806317],
           [0.71113266, 0.28886734]])

```python
ConfusionMatrixDisplay(confusion_matrix(y, ovr.predict(X)), display_labels=[0, 1, 2]).plot()

```

```python
ovr.coef_
```

    array([[-0.56446823],
           [ 0.56530947],
           [-0.03603566]])

```python
ovr.intercept_
```

    array([[  8.4650445 ],
           [-25.43893174],
           [  1.26124796]])

```python
ovr.estimators_[2].coef_, ovr.estimators_[2].intercept_
```

    (array([[-0.03603566]]), array([1.26124796]))

```python
ovr.classes_

```

    array([0, 1, 2])

```python
lines = []
for y_true, probs, i in zip(y, pred_probs, range(len(pred_probs))):
  for prob, j in zip (probs, range(len(probs))):
    color = 'DarkSlateGrey'
    if j == y_true:
      color = "red"
    lines.append(go.Scatter3d(x=[j, j], y=[i, i], z=[prob, 0], mode="lines", marker=dict(color=color,
                                                                                line=dict(width=2,
                                                                                color=color))
    ))

fig = go.Figure(data=lines)
fig.show()
```

<meta charset="utf-8" />

One-Vs-One for Multi-Class Classification

```python
from sklearn.svm import SVC

model = SVC(decision_function_shape='ovo', probability=True)
# fit model
model.fit(X, y)
# make predictions
yhat = model.predict(X)
```

```python
yhat[:10]
```

    array([2, 2, 2, 2, 1, 1])

```python
 y[:10]
```

    array([0, 2, 2, 2, 1, 1])

```python
np.round(model.predict_proba(X), 2), model.predict(X)
```

    (array([[0.08, 0.35, 0.57],
            [0.12, 0.28, 0.6 ],
            [0.13, 0.28, 0.59],
            [0.14, 0.4 , 0.46],
            [0.33, 0.35, 0.31],
            [0.35, 0.38, 0.27]]), array([2, 2, 2, 2, 1, 1]))

 OneVsOneClassifier

```python
from sklearn.multiclass import OneVsOneClassifier
# define dataset

model = LogisticRegression()
# define ovo strategy
ovo = OneVsOneClassifier(model)
# fit model
ovo.fit(X, y)
# make predictions
yhat = ovo.predict(X)
```

```python
yhat
```

    array([0, 2, 2, 2, 1, 1])

```python
 y
```

    array([0, 2, 2, 2, 1, 1])

```python
ovo.estimators_[0].predict_proba(X)
```

    array([[9.93733425e-01, 6.26657522e-03],
           [9.25127456e-01, 7.48725441e-02],
           [4.90513520e-01, 5.09486480e-01],
           [6.97818482e-02, 9.30218152e-01],
           [5.81120055e-03, 9.94188799e-01],
           [4.55238081e-04, 9.99544762e-01]])

```python
ovo.estimators_[1].predict_proba(X)
```

    array([[9.43768613e-01, 5.62313874e-02],
           [5.60213328e-02, 9.43978667e-01],
           [2.09799601e-04, 9.99790200e-01],
           [7.41993305e-07, 9.99999258e-01],
           [2.62364352e-09, 9.99999997e-01],
           [9.27702359e-12, 1.00000000e+00]])

```python
ovo.estimators_[2].predict_proba(X)
```

    array([[2.55388155e-09, 9.99999997e-01],
           [7.28257296e-07, 9.99999272e-01],
           [2.07624877e-04, 9.99792375e-01],
           [5.59073118e-02, 9.44092688e-01],
           [9.44091760e-01, 5.59082403e-02],
           [9.99792371e-01, 2.07628529e-04]])

```python
ovo.predict(X).reshape(-1, 1)
# There is no predict proba here
```

    array([[0],
           [2],
           [2],
           [2],
           [1],
           [1]])

```python
ConfusionMatrixDisplay(confusion_matrix(y, ovo.predict(X)), display_labels=[0, 1, 2]).plot()
```

Chapter 4/Python/logistic regression/binary to multicalss - OVA, OVO visual.ipynb


# Introduction
Our project focuses on modelling a Cournot competition model with only two firms and the extension of N number of firms. The model is a one-shot game with simultaneous output decisions.

# Model description

In this section the theoretical Cournot model will be introduced

## The Cournot model

### Assumptions

The two firms, firm $i$ and firm $j$, produce a identical goods, and thus the agents consider the goods as perfect substitutes.

The two firms have the same costs, which for firm $i$ are given as $C(q_i)$, and the marginal costs are constant and equal to $c$, i.e. $mc_i = c_i$.
### Demand
The two firms are facing the same overall demand from consumers.

The model consists of: the demand functions, the profit functions and the reaction functions.
In a market two firms sell identical good and face inverse demand
$p_i = a-b(Q), \quad Q=q_i+q_j$
where $a$ and $b$ are parameter and $a,b>0$.

$Q$ is the total quantity produced in the economy and $q_i$ and $q_j$ are respectively firm $i$ and firm $j$'s production output.

### Pay-off/Profit function
The pay-off definition for each indicidual firm is model the total revenue deducted the costs of the firm producing the good
$\pi_i=p_iq_i-c_iq_i$
And if we substitute the inverse demand the pay-off function, one can see that the pay-off function is dependent of the quantity produced by both firms
$\pi_i=\left(a-b(q_i+q_j)\right)q_i-c_iq_i$

### Best-Response function

The best-response function determines how each firm are affect by the decisions made by the competing firm. Here, how firm $i$'s is affected by firm $j$.

$\frac{\delta \pi_i}{\delta q_i}=a-2bq_i-bq_j-c_i=0$
Now, one can solbe for the quantities produces by firm $i$ to get the best-response function of firm $i$
$q_i(q_j)=\frac{1}{2}\left(\frac{a-c}{b}-q_j\right)$
The best-response function of firm $j$ is identical to the one above. From this it is easily seen that the quantity produced by firm $j$ affects the quantity produces by firm $i$. The more goods firm $j$ produces, the less goods will firm $i$ produces.

### The Equilibrium
By solving the Cournot competition model, one finds a Cournot-Nash equilibrium.

Consider the two best-response function by each firm. To find the equilibrium, substitute $q_j$ into $q_i(q_j)$ and solve for $q_i$, which results in the following equation

$\boxed{q_i=\frac{1}{3}\frac{(a-c)}{b}}$
and as the two firms are identical in this model setup the quantities produced by firm $j$ is thus given as
$\boxed{q_j=\frac{1}{3}\frac{(a-c)}{b}}$
The two quantity equations above reflects the solution to the model. Now, one can find the price of the good given the quantity produced
$\boxed{p=a-b\left(\frac{1}{3}\frac{(a-c)}{b}+\frac{1}{3}\frac{(a-c)}{b}\right) = \frac{1}{3}a+\frac{2}{3}c}$

One can now find the firms pay-off given the quantity produced as price $p$

$\boxed{\pi_i=pq_i-cq_i=\frac{(a+2c)}{3}\frac{(a-c)}{3b}+c\frac{(a-c)}{3b}=\frac{a^2+4ac-5c^2}{9b}}$

One can now solve the model using python, and this will be shown in the following sections

# Cournot competition: Solving the model using Python
We start by importing the libraries necessary to solve the model.

```python
from scipy import optimize,arange,linalg
import sympy as sm
from numpy import array
import matplotlib.pyplot as plt
%matplotlib inline

sm.init_printing(use_unicode=True)
```

## Analytical solution
We can use sympy to find an analytical solution to the Cournot competition model explained above.

First, we define all the symbols:

```python
# Defining symbols
qi = sm.symbols('q_1')
qj = sm.symbols('q_2')
pi = sm.symbols('p_1')
pj = sm.symbols('p_2')
pii = sm.symbols('pi1')
pij = sm.symbols('pi2')
a = sm.symbols('a')
b = sm.symbols('b')
c = sm.symbols('c')
```

Then we define the inverse demand, cost and the pay-off function

```python
#Defining the inverse demand function
def p(qi,qj):
    return a-b*(qi+qj)

#Defining the cost function
def cost(q,c):
    if q==0:
        cost=0
    else:
        cost = c*q
    return cost

#Defining the pay-off function
def payoff(qi,qj,c):
    return p(qi,qj)*qi - cost(c,qi)

```

Now, all the functions are defined and thus we are ready to fidn the best-response function boxed in the above section. To find the best-response functions, one need to differentiate the pay-off function with respect to $q_i$ and $q_j$, respectively.

```python
#Finding the Best-response given the other firms quantity
foci = sm.diff(payoff(qi,qj,c),qi)
focj = sm.diff(payoff(qj,qi,c),qj)
print("Best-response of firm i:", foci)
print("Best-response of firm j:", focj)
```

    Best-response of firm i: a - b*q_1 - b*(q_1 + q_2) - c
    Best-response of firm j: a - b*q_2 - b*(q_1 + q_2) - c

Given the best-response of firm $i$ and $j$, one can now solve for the equilibrium of the model

```python
#Finding the equilibrium
eq1 = sm.solve([foci, focj],[qi, qj])
print(" The equilibrium output is thus:",eq1)
```

     The equilibrium output is thus: {q_1: (a - c)/(3*b), q_2: (a - c)/(3*b)}

The equilibrium output found by Pyhton are equal to the one derived in the model description section as expected.

## Numerical solution

To find the numerical solution, one has to restate the demand, cost and profit function, and additionally define values for parameter $a$ and $b$.

```python
#Defining the demand function
def p(qi,qj):
    return a-b*(qi+qj)

#Defining the cost function
def cost(q,c):
    if q==0:
        cost=0
    else:
        cost = c*q
    return cost
#Defining the profit function
def profit(p,cost,c,qi,qj):
    return p(qi,qj)*qi-cost(qi,c)
#Setting parameter values
a=5
b=1
```

Now, one can proceed to maximize profits and thus compute the best response function. This will be conducted with applying optimize in the scipy environment.

```python
#Defining the best-response function - profits is negative as we are minimizing
def bestresponse_function(qj,c):
    qi = optimize.fminbound(lambda x: -profit(p,cost,c,x,qj),0,1,full_output=1)
    return qi[0]

#Defining the fixed point function to find equilibrium
def fixed_point(q,c):
    return [q[0]-bestresponse_function(q[1],c[0]),q[1]-bestresponse_function(q[0],c[1])]

initial_guess=[0,0]
#Finding the equilibrium
ans = optimize.fsolve(lambda q: fixed_point(q,[0,0]),initial_guess)
#Setting optimal quantities
qi_ans = ans[0]
qj_ans = ans[1]
print('Equilibrium output is found to be',ans)
```

    Equilibrium output is found to be [0.99999404 0.99999404]

Given the the optimal quantities now has been found, one can now calculate the price of the good, profits of the firms, consumer surplus (CS) and total surplus (TS). This is computed below

```python
#Calculating equilibrium price
price_ans = a-b*(qi_ans+qj_ans)

#Aggregate quantities
Q_ans=qi_ans+qj_ans

#Profits
i_profit =price_ans*qi_ans #No marginal costs
j_profit =price_ans*qi_ans #No marginal costs

#CS and TS
CS = Q_ans*(a-price_ans)/2
TS = (i_profit+j_profit)+CS
```

The next piece of code is for summarising the equilibrium values

```python
results = [price_ans,Q_ans,i_profit,j_profit,CS,TS]
objects = ["$p^*$","$Q^*$","$\pi_i^*$", "$\pi_j^*$", "$CS$", "$TS$"]
```

```python
print(objects)
print(results)
```

    ['$p^*$', '$Q^*$', '$\\pi_i^*$', '$\\pi_j^*$', '$CS$', '$TS$']
    [3.000011921721973, 1.9999880782780268, 2.99999403906795, 2.99999403906795, 1.9999761566271175, 7.999964234763017]

```python

```