```python
import numpy as np
import numpy as np
from scipy import linalg
from scipy import optimize
import sympy as sm
from sympy import *
from scipy.optimize import minimize
%matplotlib inline
import matplotlib.pyplot as plt # baseline modul
from mpl_toolkits.mplot3d import Axes3D # for 3d figures
plt.style.use('seaborn-whitegrid') # whitegrid nice with 3d
from scipy import interpolate

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
from matplotlib import cm
from mpl_toolkits.mplot3d import Axes3D
```

# Linear regression

Consider the following **linear equation:**

$$y_i = \beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i} + \epsilon_i$$

Assume you have access to data of the **independent variables** ($x_{1,i}$, $x_{2,i}$) and the **dependent variable** ($y_i$) for $N$ individuals, where $i$ indexes individuals. The variable $\epsilon_i$ is a mean-zero **stochastic shock**.

Assume the **data generating process** is given by:

```python
def DGP(N):

    # a. independent variables
    x1 = np.random.normal(0,1,size=N)
    x2 = np.random.normal(0,1,size=N)

    # b. errors
    eps = np.random.normal(0,1,size=N)

    extreme = np.random.uniform(0,1,size=N)
    eps[extreme < 0.05] += np.random.normal(-5,1,size=N)[extreme < 0.05]
    eps[extreme > 0.95] += np.random.normal(5,1,size=N)[extreme > 0.95]

    # c. dependent variable
    y = 0.1 + 0.3*x1 + 0.5*x2 + eps

    return x1, x2, y
```

**The data you have access to is:**

```python
np.random.seed(2020)
x1,x2,y = DGP(10000)
```

**Question 1:** Estimate the vector of coefficients $\mathbf{\beta} = (\beta_0,\beta_1,\beta_2)$ using **ordinary least squares (OLS)** implemented with **matrix algebra** by

$$ \hat{\mathbf{\beta}} = (\mathbf{X}^{\prime}\mathbf{X})^{-1}\mathbf{X}^{\prime}\mathbf{y} $$

where $\mathbf{X}^{\prime}$ is the transpose of $\mathbf{X}$ and

$$\mathbf{y} =
\pmatrix{ y_1 \cr y_2 \cr  \vdots \cr y_N
}
, \quad \mathbf{X} = \pmatrix{
1 & x_{1,1} & x_{2,1} \cr
1 & x_{1,2} & x_{2,2} \cr
\vdots & \vdots \cr
1 & x_{1,N} & x_{2,N}
}$$

```python
ones = np.ones(10000)

# We'll now define the matrix for X
X = np.array(np.column_stack([ones,x1,x2]))

# We'll transpose our matrix
X_transposed = X.T
y1 = y[:,np.newaxis] # Our y
# We'll define the vector of coefficients
beta_hat = linalg.inv(X_transposed@X)@X_transposed@y1
beta_hat
```

    array([[0.0956821 ],
           [0.29294299],
           [0.50332771]])

We have now found our vector of coefficients.

```python
# We save our beta hats
beta0_hat = float(beta_hat[0])
beta1_hat = float(beta_hat[1])
beta2_hat = float(beta_hat[2])
```

**Question 2:** Construct a 3D plot, where the data is plotted as scattered points, and the prediction of the model is given by the plane

$$\hat{y}_i = \hat{\beta}_0 + \hat{\beta}_1 x_{1,i} + \hat{\beta}_2 x_{2,i}$$

```python
# Data for the graf
x_1 = np.array(x1)
x_2 = np.array(x2)
y_test = beta_hat[0]+beta_hat[1]*x_1+beta_hat[2]*x_2

# We plot the figure
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(x_1, x_2, y_test, color='red')

# We set the labels
ax.set_xlabel("X1")
ax.set_ylabel("X2")
ax.set_zlabel("y")
```

We see that they are very clustered, and it is hard to say how they hare correlated.

**Question 3:** Esimtate the vector of coefficients $\mathbf{\beta} = (\beta_0,\beta_1,\beta_2)$ using a **numerical solver** to solve the ordinary least square problem, shown below, directly. Compare your results with the matrix algebra results.

$$ \min_{\mathbf{\beta}} \sum^N_{i=1} (y_i - (\beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i}) )^2 $$

```python
# Definding the residuals

z=np.empty(len(x1))
def residuals(beta0_hat,beta1_hat,beta2_hat):

    for i in range(len(x1)):
        z[i] = (y[i] - beta0_hat - beta1_hat*x1[i] - beta2_hat*x2[i])
    return z

# Since we have OLS we sum our residuals and square them

def obj(x,beta0_hat,beta1_hat,beta2_hat):
    beta0_hat=x[0]
    beta1_hat=x[1]
    beta2_hat=x[2]
    x = np.sum(residuals(beta0_hat,beta1_hat,beta2_hat)**2)
    return x

# We now minimize our function

beta_test=np.array([1,1,1])

res = optimize.minimize(obj,beta_test,method='SLSQP',args=(beta0_hat,beta1_hat,beta2_hat))

print(res.x[0])
print(res.x[1])
print(res.x[2])

```

    0.09568210468227249
    0.2929429872141269
    0.5033277125253971

We can see that we get the same estimates as in question 1.

**Question 4:** Estimate the vector of coefficients $\mathbf{\beta} = (\beta_0,\beta_1,\beta_2)$ using **least absolute deviations (LAD)** using a numerical solver to solve the following problem directly:

$$  \min_{\beta} \sum^N_{i=1} |y_i - (\beta_0 + \beta_1 x_{1,i} + \beta_2 x_{2,i}) | $$

where $|z|$ is the absolute value of $z$.

```python
def u2(x,beta0_hat,beta1_hat,beta2_hat):
    beta0_hat=x[0]
    beta1_hat=x[1]
    beta2_hat=x[2]
    x2 = np.sum(np.abs(residuals(beta0_hat,beta1_hat,beta2_hat)))
    return x2

# We minimize
res = optimize.minimize(u2,beta_test,method='SLSQP',args=(beta0_hat,beta1_hat,beta2_hat))

print(res.x[0])
print(res.x[1])
print(res.x[2])

```

    0.09225971701166782
    0.30746837345387845
    0.51155249719207

We can see that we almost get the same estimates as in question 1 and 3. These estimates of beta are a bit bigger.

**Question 5:** Set $N = 50$. Repeat the estimation using the **OLS** and **LAD** methods $K=5000$ times, drawing a new random sample from the data generating process each time. Compare the estimates from each method using histograms. Which method do you prefer? Explain your choice.

```python
# We now simulate the OLS

beta0 = np.empty(5000)
beta1 = np.empty(5000)
beta2 = np.empty(5000)

for i in range(5000):
    x1_2,x2_2,y_2 = DGP(50)

    # Defining the residuals

    u_2=np.empty(len(x1_2))

    def residuals_new(beta0,beta1,beta2):

        for i in range(len(x1_2)):
            u_2[i] = (y_2[i] - beta0 - beta1*x1_2[i] - beta2*x2_2[i])
        return u_2

    def phy(x,beta0,beta1,beta2):
        beta0=x[0]
        beta1=x[1]
        beta2=x[2]
        sse_new = np.sum(residuals_new(beta0,beta1,beta2)**2)
        return sse_new

    res = optimize.minimize(phy,beta_test,method='SLSQP',args=(beta0,beta1,beta2))
    beta0[i-1]=res.x[0]
    beta1[i-1]=res.x[1]
    beta2[i-1]=res.x[2]
```

```python
# We now simulate the LAD

beta0_lad = np.empty(5000)
beta1_lad = np.empty(5000)
beta2_lad = np.empty(5000)

for i in range(5000):
    x1_2,x2_2,y_2 = DGP(50)

    # Defining the residuals

    u_2=np.empty(len(x1_2))

    def residuals_new(beta0_lad,beta1_lad,beta2_lad):

        for i in range(len(x1_2)):
            u_2[i] = (y_2[i] - beta0_lad - beta1_lad*x1_2[i] - beta2_lad*x2_2[i])
        return u_2

    def phy(x,beta0_lad,beta1_lad,beta2_lad):
        beta0_lad=x[0]
        beta1_lad=x[1]
        beta2_lad=x[2]
        sse1_new = np.sum(np.abs(residuals_new(beta0_lad,beta1_lad,beta2_lad)))
        return sse1_new

    res = optimize.minimize(phy,beta_test,method='SLSQP',args=(beta0_lad,beta1_lad,beta2_lad))
    beta0_lad[i-1]=res.x[0]
    beta1_lad[i-1]=res.x[1]
    beta2_lad[i-1]=res.x[2]
```

```python
fig1=plt.hist(beta0,bins=50)
fig2=plt.hist(beta0_lad,bins=50)
plt.show(fig1,fig2)
```

```python
fig3=plt.hist(beta1,bins=50)
fig4=plt.hist(beta1_lad,bins=50)
plt.show(fig3,fig4)
```

```python
fig5=plt.hist(beta2,bins=50)
fig6=plt.hist(beta2_lad,bins=50)
plt.show(fig5,fig6)
```

We prefer the OLS, because it is more concentrated around the estimate.

# Durable purchases

Consider a **household** living in two periods.

In the **second period** it gets utility from **non-durable consumption**, $c$, and **durable consumption**, $d+\chi x$:

$$
\begin{aligned}
v_{2}(m_{2},d)&= \max_{c}\frac{(c^{\alpha}(d+\chi x)^{1-\alpha})^{1-\rho}}{1-\rho}\\
\text{s.t.} \\
x &= m_{2}-c \\
c &\in [0,m_{2}]
\end{aligned}
$$

where

* $m_2$ is cash-on-hand in the beginning of period 2
* $c$ is non-durable consumption
* $d$ is pre-commited durable consumption
* $x = m_2 - c$ is extra durable consumption
* $\rho > 1$ is the risk aversion coefficient
* $\alpha \in (0,1)$ is the utility weight on non-durable consumption
* $\chi \in (0,1)$ implies that extra durable consumption is *less* valuable than pre-comitted durable consumption
* the second constraint ensures the household *cannot* die in debt

The **value function** $v_2(m_2,d)$ measures the household's value of having $m_2$ at the beginning of period 2 with precomitted durable consumption of $d$. The optimal choice of non-durable consumption is denoted $c^{\ast}(m_2,d)$. The optimal extra durable consumption function is $x^{\ast}(m_2,d) = m_2-c^{\ast}(m_2,d)$.

Define the so-called **end-of-period 1 value function** as:

$$
\begin{aligned}
w(a,d)&\equiv\beta\mathbb{E}_{1}\left[v_2(m_2,d)\right]
\end{aligned}
$$

where

$$
\begin{aligned}
m_2&= (1+r)a+y \\
y &= \begin{cases}
1-\Delta & \text{with prob. }\frac{1}{3}\\
1 & \text{with prob. }\frac{1}{3}\\
1+\Delta & \text{with prob. }\frac{1}{3}
\end{cases}\\
\end{aligned}
$$

and

* $a$ is assets at the end of period 1
* $\beta > 0$ is the discount factor
* $\mathbb{E}_1$ is the expectation operator conditional on information in period 1
* $y$ is income in period 2
* $\Delta \in (0,1)$ is the level of income risk (mean-preserving)
* $r$ is the return on savings

In the **first period**, the household chooses it's pre-comitted level of durable consumption for the next-period,

$$
\begin{aligned}
v_{1}(m_{1})&=\max_{d} w(a,d)\\&\text{s.t.}&\\
a&= m_{1}-d \\
d&\in [0,m_{1}]\\
\end{aligned}
$$

where $m_1$ is cash-on-hand in period 1. The second constraint ensures the household *cannot* borrow. The **value function** $v_1(m_1)$ measures the household's value of having $m_1$ at the beginning of period 1. The optimal choice of pre-committed durable consumption is denoted $d^{\ast}(m_1)$.

The **parameters** and **grids** for $m_1$, $m_2$ and $d$ should be:

```python
# a. parameters
rho = 2
alpha = 0.8
beta = 0.96
r = 0.04
Delta = 0.25
chi = 0.9

# b. grids
m1_vec = np.linspace(1e-8,10,100)
m2_vec = np.linspace(1e-8,10,100)
d_vec = np.linspace(1e-8,5,100)
```

**Question 1:** Find and plot the functions $v_{2}(m_{2},d)$, $c^{\ast}(m_2,d)$, and $x^{\ast}(m_2,d)$. Comment.

```python
# We first define our v2 function

def x(m2,c):
    x = m2 - c
    return x

def v2(c,m2,d,alpha,chi,rho):
    return ((c**alpha*(d + chi*x(m2,c))**(1-alpha))**(1-rho))*(1/(1-rho))

```

```python
# We now solve for the second period
def solve_period_2(alpha,chi,rho):

    # a. grids
    v2_grid = np.empty((100,100))
    c_grid = np.empty((100,100))
    m2_vec = np.linspace(1e-8,10,100)
    d_vec = np.linspace(1e-8,5,100)
    x_grid = np.empty((100,100))

    # b. solve for each m2 and d in grid
    for i,m2 in enumerate(m2_vec):
        for j,d in enumerate(d_vec):
            # i. objective
            obj = lambda c: -v2(c,d,m2,alpha,rho,chi)

            # ii. initial value (consume half)
            x0 = m2/2

            # iii. We then optimize the function
            result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=[1e-8,m2])

            # iv. We save
            v2_grid[i,j] = -result.fun
            c_grid[i,j] = result.x
            x_grid[i,j] = m2 - result.x

    return x_grid, v2_grid, c_grid
```

```python
# a. We use our solve function
x_grid,v2_grid,c_grid = solve_period_2(alpha,chi,rho)

# b. grids
m2_grid,d_grid = np.meshgrid(m2_vec,d_vec,indexing='ij')

# We plot x*(m2,d)
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(d_grid,m2_grid,x_grid,cmap=cm.jet)

ax.set_xlabel('$m_2$')
ax.set_ylabel('$d$')
ax.set_zlabel('$x$')

ax.invert_xaxis()

fig.colorbar(cs);

```

```python
# We now plot c*(m2,d)
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(m2_grid,d_grid,c_grid,cmap=cm.jet)

ax.set_xlabel('$m_2$')
ax.set_ylabel('$d$')
ax.set_zlabel('$c$')

ax.invert_xaxis()

fig.colorbar(cs);
```

```python
# We plot v2(m2,d)
fig = plt.figure()
ax = fig.add_subplot(1,1,1,projection='3d')
cs = ax.plot_surface(m2_grid,d_grid,v2_grid,cmap=cm.jet)

# d. add labels
ax.set_xlabel('$m_2$')
ax.set_ylabel('$d$')
ax.set_zlabel('$v2$')

# e. invert xaxis
ax.invert_xaxis()

# f. add colorbar
fig.colorbar(cs);
```

**Question 2:** Find and plot the functions $v_{1}(m_{1})$ and $d^{\ast}(m_1)$. Comment.

```python

#3 Define end of period one value function
def v1(m1,d,Delta,r,beta,v2_interp):
    a = m1-d
    # a. v2 value, if low income
    m2_low = (1+r)*a+(1-Delta)
    v2_low = v2_interp([m2_low,d])[0]

    #b. v2 value if medium income
    m2_medium = (1+r)*a+(1)
    v2_medium = v2_interp([m2_medium,d])[0]

    # c. v2 value, if high income
    m2_high = (1+r)*a+(1+Delta)
    v2_high = v2_interp([m2_high,d])[0]

    #d. expected v2 value
    v2 = 1/3*v2_low+ 1/3*v2_medium+ 1/3*v2_high

    v1 = beta*v2
    return v1

```

```python
# a. define solve function
def solve_period_1(alpha,rho,beta,r,Delta,v1,v2_interp):

    # a. grids
    m1_vec = np.linspace(1e-4,4,100)
    v1_vec = np.empty(100)
    d1_vec = np.empty(100)

    # b. solve for each m1 in grid
    for i,m1 in enumerate(m1_vec):

        # i. objective
        obj = lambda d: -v1(m1,d,Delta,r,beta,v2_interp)

        # ii. initial guess
        x0 = [m1/2]

         # iii. optimize

        result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=(1e-8,m1))
        # iv. save
        v1_vec[i] = -result.fun
        d1_vec[i] = result.x

    return v1_vec,d1_vec

# b. construct interpolator
v2_interp = interpolate.RegularGridInterpolator([m2_vec,d_vec], v2_grid,
                                                bounds_error=False,fill_value=None)

# c. solve period 1
v1_vec,d1_vec = solve_period_1(alpha,rho,beta,r,Delta,v1,v2_interp)

# d. plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(m1_vec,d1_vec,label='pre-committed durable consumption')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_ylabel('$d$')

fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(v1_vec,m1_vec,label='Households value in period 1')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_ylabel('$v1_vec$')

```

In figure one we see that the more money the household has, the more pre committed durable goods they can have. We also see that the households value increase as the amount of money increase in period one.

**Hint:** For interpolation of $v_2(m_2,d)$ consider using `interpolate.RegularGridInterpolator([GRID-VECTOR1,GRID-VECTOR2],VALUE-MATRIX,bounds_error=False,fill_value=None)`.

Next, consider an **extension** of the model, where there is also a **period 0**. In this period, the household makes a choice whether to stick with the level of durables it has, $z = 0$, or adjust its stock of durables, $z = 1$. If adjusting, the household loses a part of the value of its durable stock; more specificaly it incurs a proportional loss of $\Lambda \in (0,1)$.

Mathematically, the **household problem in period 0** is:

$$
\begin{aligned}
v_{0}(m_{0},d_{0}) &= \max_{z\in\{0,1\}} \begin{cases}
w(m_{0},d_{0}) & \text{if } z = 0\\
v_1(m_0+(1-\Lambda) d_{0}) & \text{if } z = 1\\
\end{cases}\\
\end{aligned}
$$

The **parameters** and **grids** for $m_0$ and $d_0$ should be:

```python
Lambda = 0.2
m0_vec = np.linspace(1e-8,6,100)
d0_vec = np.linspace(1e-8,3,100)
```

**Question 3:** For which values of $m_0$ and  $d_0$ is the optimal choice not to adjust, i.e. $z = 0$? Show this in a plot. Give an interpretion of your results.

```python
#We define our function for period 0
def v0(m0,d0,Delta,r,beta,v1_interp):
    a = m0-d0
    # a. v2 value, if low income
    m1_low = (1+r)*a+(1-Delta)
    v1_low = v1_interp([m1_low])[0]

    #b. v2 value if medium income
    m1_medium = (1+r)*a+(1)
    v1_medium = v1_interp([m1_medium])[0]

    # c. v2 value, if high income
    m1_high = (1+r)*a+(1+Delta)
    v1_high = v1_interp([m1_high])[0]

    #d. expected v2 value
    v2 = 1/3*v1_low+ 1/3*v1_medium+ 1/3*v1_high

    v0 = beta*v1
    return v0

```

```python
# a. define solve function
def solve_period_0(alpha,rho,beta,r,Delta,v0,v1_interp):

    # a. grids
    m0_vec = np.linspace(1e-4,4,100)
    v0_vec = np.empty(100)
    d0_vec = np.empty(100)

    # b. solve for each m1 in grid
    for i,m0 in enumerate(m0_vec):

        # i. objective
        obj = lambda d0: -v0(m0,d0,Delta,r,beta,v2_interp)

        # ii. initial guess
        x0 = [m0/2]

         # iii. optimize
        result = optimize.minimize_scalar(obj,x0,method='bounded',bounds=(1e-8,m0))
        # iv. save
        v0_vec[i] = -result.fun
        d0_vec[i] = result.x

    return v0_vec,d0_vec

# b. construct interpolator
v1_interp = interpolate.RegularGridInterpolator((m1_vec,), v1_vec,
                                                    bounds_error=False,fill_value=None)

# c. solve period 1
v0_vec,d0_vec = solve_period_0(alpha,rho,beta,r,Delta,v0,v1_interp)

# d. plot
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(m0_vec,d0_vec,label='pre-committed durable consumption')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_ylabel('$d$')

fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(v0_vec,m0_vec,label='Households value in period 1')
ax.legend(loc='upper left')
ax.set_xlabel('$m_1$')
ax.set_ylabel('$v1_vec$')
```

# Gradient descent

Let $\boldsymbol{x} = \left[\begin{array}{c}
x_1 \\
x_2\\
\end{array}\right]$ be a two-dimensional vector. Consider the following algorithm:

**Algorithm:** `gradient_descent()`

**Goal:** Minimize the function $f(\boldsymbol{x})$.

1. Choose a tolerance $\epsilon>0$, a scale factor $ \Theta > 0$, and a small number $\Delta > 0$
2. Guess on $\boldsymbol{x}_0$ and set $n=1$
3. Compute a numerical approximation of the jacobian for $f$ by

    $$
    \nabla f(\boldsymbol{x}_{n-1}) \approx \frac{1}{\Delta}\left[\begin{array}{c}
    f\left(\boldsymbol{x}_{n-1}+\left[\begin{array}{c}
    \Delta\\
    0
    \end{array}\right]\right)-f(\boldsymbol{x}_{n-1})\\
    f\left(\boldsymbol{x}_{n-1}+\left[\begin{array}{c}
    0\\
    \Delta
    \end{array}\right]\right)-f(\boldsymbol{x}_{n-1})
    \end{array}\right]
    $$

4. Stop if the maximum element in $|\nabla f(\boldsymbol{x}_{n-1})|$ is less than $\epsilon$
5. Set $\theta = \Theta$
6. Compute $f^{\theta}_{n} = f(\boldsymbol{x}_{n-1} - \theta \nabla f(\boldsymbol{x}_{n-1}))$
7. If $f^{\theta}_{n} < f(\boldsymbol{x}_{n-1})$ continue to step 9
8. Set $\theta = \frac{\theta}{2}$ and return to step 6
9. Set $x_{n} = x_{n-1} - \theta \nabla f(\boldsymbol{x}_{n-1})$
10. Set $n = n + 1$ and return to step 3

**Question:** Implement the algorithm above such that the code below can run.

**Optimizer function:**

```python
def step_6(x, o, Delta):
    return f(x - o * Delta * f(x))

def gradient_descent(f, x0, epsilon=1e-6, Theta=0.1, Delta=1e-8, max_iter=10_000):
    pass

    x = np.zeros((max_iter, 2))  # Initialize array of x's
    x[0] = x0  # Step 2

    for n in range(max_iter, 1):
        j_0 = f(x[n - 1] + np.array([Delta, 0])) - f(x[n - 1])  # Helper for step 3
        j_1 = f(x[n - 1] + np.array([0, Delta])) - f(x[n - 1])  # Helper for step 3

        jacob = (1 / Delta) * np.array([j_0, j_1])  # Step 3

        max_el = np.sqrt(np.dot(jacob, jacob))  # Helper for step 4

        if max_el < epsilon:  # Step 4
             return jacob, n

        o = Theta  # Step 5

        while true:
            fno = step_6(x[n - 1], o, Delta)  # Step 6

            if fno >= f(x[n - 0]):  # Reverse of Step 7 in order to jump correctly
                o = o / 2 # Step 8

                continue # Jump back to step 6

                x[n] = x[n - 1] - o * Delta * f(x[n - 1]) # Step 9
                break  # Step 10 is inferred here
                n += 1
                return [-1, -1], -1  # To show that it may fail
```

**Test case:**

```python
def rosen(x):
    return (1.0-x[0])**2+2*(x[1]-x[0]**2)**2

x0 = np.array([1.1,1.1])
try:
    x,it = gradient_descent(rosen,x0)
    print(f'minimum found at ({x[0]:.4f},{x[1]:.4f}) after {it} iterations')
    assert np.allclose(x,[1,1])
except:
    print('not implemented yet')
```

    not implemented yet


# Chapter 2 Exercises
In this notebook we will go through the exercises of chapter 2 of Introduction to Stochastic Processes with R by Robert Dobrow.

```python
import numpy as np
```

## 3.1
Consider a Markov chain with transition Matrix
$$P=\left(\begin{array}{cc}
1/2 & 1/4 & 0 & 1/4 \\
0 & 1/2 & 1/2 & 0\\
1/4 & 1/4 & 1/2 & 0 \\
0 & 1/4 & 1/2 & 1/4
\end{array}\right)$$
Find the stationary distribution without using technology

### Answer
We have 5 equations with 4 variables:
$$\begin{align}
1/2 \pi_1 +1/4\pi_3 &= \pi_1 \\
1/4 \pi_1 +1/2\pi_2+1/4\pi_3+1/4\pi_4 &= \pi_2 \\
1/2\pi_2+1/2\pi_3+1/2\pi_4 &= \pi_3 \\
1/4 \pi_1 +1/4\pi_4 &= \pi_4 \\
\pi_1+\pi_2+\pi_3+\pi_4=1
\end{align}$$
$=>$
$$\begin{align}
\pi_1  &= 3\pi_4 \\
\pi_3 &= 6\pi_4 \\
\pi_2 &= 5\pi_4 \\
\pi_4 &= 1/15 \\
\pi_2 &= 1/3 \\
\pi_3 &= 2/5 \\
\pi_1 &= 1/5 \\
\end{align}$$
then $\pi = (1/5, 1/3, 2/4, 1/15)$

```python
pi = np.array([1/5, 1/3, 2/5, 1/15])
P = np.matrix([[1/2 , 1/4 , 0 , 1/4],
[0 , 1/2 , 1/2 , 0],
[1/4 , 1/4 , 1/2 , 0],
[0 , 1/4 , 1/2 , 1/4]])
pi*P, pi
```

    (matrix([[0.2       , 0.33333333, 0.4       , 0.06666667]]),
     array([0.2       , 0.33333333, 0.4       , 0.06666667]))

## 3.2
A stochastic matrix is called *doubly stochastic* if its rows and columns sum to 1. Show that a Markov chain whose transition matrix is doubly stochastic has a stationary distribution, which is uniform on the state space.

### Answer
Let's remember that then the distribution at $X_n$ is $\alpha P^n$, let's see what happens at $n=1$

$n=1$
$$X_1=\alpha P = \left(\begin{array}{cc}
1/k &1/k&...& 1/k
\end{array}\right)\left(\begin{array}{cc}
p_{1,1} & p_{1,2} & ... & p_{1,k} \\
p_{2,1} & p_{2,2} & ... & p_{2,k} \\
...  \\
p_{k,1} & p_{k,2} & ... & p_{k,k} \\
\end{array}\right)=\left(\begin{array}{cc}
p_{1,1}*1/k + p_{2,1}*1/k + ... + p_{k,1}*1/k \\
p_{1,2}*1/k + p_{2,2}*1/k + ... + p_{k,2}*1/k \\
...  \\
p_{1,k}*1/k + p_{2,k}*1/k + ... + p_{k,k}*1/k \\
\end{array}\right)^T=\left(\begin{array}{cc}
(p_{1,1} + p_{2,1} + ... + p_{k,1})*1/k\\
(p_{1,2} + p_{2,2} + ... + p_{k,2})*1/k \\
...  \\
(p_{1,k} + p_{2,k} + ... + p_{k,k})*1/k \\
\end{array}\right)^T = \left(\begin{array}{cc}
(1)*1/k\\
(1)*1/k \\
...  \\
(1)*1/k \\
\end{array}\right)^T=\alpha
$$
This last step is due to the matrix being double stochastic

## 3.3
Consider a Markov chain with transition Matrix. Determine if they are regular.
$$P=\left(\begin{array}{cc}
0.4 & 0.6 & 0 & 0 \\
0 & 0 & 1 & 0\\
0 & 0 & 0 & 1 \\
1 & 0 & 0 & 0
\end{array}\right)$$

$$Q=\left(\begin{array}{cc}
0 & 1 \\
p & 1-p
\end{array}\right)$$

$$R=\left(\begin{array}{cc}
0 & 1 & 0 \\
0.25 & 0.5 & 0.25 \\
1 & 0 & 0
\end{array}\right)$$

### Answer
Since all the values have probability of existing in matrices P and R at all times, then these are regular. For $Q$ then it is regular for $0<p<1$ since when this happens the movement is certain and when $p=0$, it does not change from state 2.

## 3.4
Consider the Markov Chain with transition Matrix:
$$R=\left(\begin{array}{cc}
1-a & a & 0 \\
0 & 1-b & b \\
c & 0 & 1-c
\end{array}\right)$$
with $0< a,b,c <1$. Find the stationary distribution

### Answer
We have 4 equations with 3 variables:
$$\begin{align}
(1-a) \pi_1 + c \pi_3 &= \pi_1 \\
(1-b)\pi_2+a\pi_1 &= \pi_2 \\
b \pi_2 +(1-c)\pi_3 &= \pi_3 \\
\pi_1+\pi_2+\pi_3=1
\end{align}$$
$=>$
$$\begin{align}
\pi_3  &= a/c\pi_1 \\
\pi_2 &= a/b\pi_1 \\
\pi_1+a/b\pi_1+a/c\pi_1 &= 1 \\
=>  \\
\pi_1 &= \frac{bc}{ab+ac+bc} \\
\pi_2 &= \frac{ac}{ab+ac+bc} \\
\pi_3 &= \frac{ab}{ab+ac+bc} \\
\end{align}$$
then $\pi = (\frac{bc}{ab+ac+bc}, \frac{ac}{ab+ac+bc}, \frac{ab}{ab+ac+bc})$

```python
# An example with a=0.4, b=0.2, c=0.7
pi = np.array([.2*.7/(.4*.2+.2*.7+.4*.7), .4*.7/(.4*.2+.2*.7+.4*.7), .2*.4/(.4*.2+.2*.7+.4*.7)])
P = np.matrix([
[.6 , .4 , 0],
[0 , .8 , .2 ],
[.7 , 0 , .3]])
pi*P, pi
```

    (matrix([[0.28, 0.56, 0.16]]), array([0.28, 0.56, 0.16]))

## 3.5
Consider a Markov chain with transition Matrix
$$P=\left(\begin{array}{cc}
0 & 1/4 & 0 & 0 & 3/4 \\
3/4 & 0 & 0 & 0 & 1/4\\
0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1 & 0 \\
1/4 & 3/4 & 0 & 0 & 0
\end{array}\right)$$
(a) Describe the set of the stationary distributions for  the chain.

(b) Use technology to find the $lim_{n\to \infty}P^n$. Explain the long-term behavior of the chain.

(c) Explain why the chain does not have a limiting distribution, and why this does not contradict the existence of a limiting matrix as in (b)

### Answer
(a)

There is the same probability of catching into state 1,2 and 5 (because of the transition matrix is a double stochastic matrix), so the set of all stationary distributions are:
$(a,a,b,c,a)$ where 3a+b+c=1, although it is necessary to state that you can never reach any other state if you start at state b or c so if you start on this states then these are 1 since you do not move on those states.

```python
# (b)
P = np.matrix([[0 , 1/4 , 0 , 0 , 3/4],
[3/4 , 0 , 0 , 0 , 1/4],
[0, 0 , 1 , 0 , 0],
[0, 0 , 0 , 1 , 0],
[1/4 , 3/4 , 0 , 0 , 0]])
P**100
```

    matrix([[0.33333333, 0.33333333, 0.        , 0.        , 0.33333333],
            [0.33333333, 0.33333333, 0.        , 0.        , 0.33333333],
            [0.        , 0.        , 1.        , 0.        , 0.        ],
            [0.        , 0.        , 0.        , 1.        , 0.        ],
            [0.33333333, 0.33333333, 0.        , 0.        , 0.33333333]])

This means that if the chain starts at state 3 or 4, then you are stuck in there, whereas if you start on position 1,2 or 5, then you can be at any of these 3 states in a probability of 1/3 (since the matrix is a double stochastic matrix)

(c) There is no limit distribution since the limit distribution will bepend on which state you start on, but there is an stationary matrix because it shows you the limiting distribution for starting at state x (with x=1,2,3,4 or 5). This is because there exist 3 different commuinication classes.

## 3.6
Consider a Markov Chain with transition matirx
$$P=\left(\begin{array}{cc}
1/2 & 1/2 & 0 & 0 & .. \\
2/3 & 0 & 1/3 & 0 & ..\\
3/4 & 0 & 0 & 1/4 & .. \\
4/5 & 0 & 0 & 0 & .. \\
. & . & . & . & .
\end{array}\right)$$

defined by
    $$ P_{i,j}=   \left\{
\begin{array}{ll}
      i/(i+1) & \text{if j = 1} \\
      1/(i+1) & \text{if j = i+1} \\
      0& \text{otherwise} \\
\end{array}
\right.  $$
(a) Does the chain have an stationary distribution? If yes, exhibit its behavior, if no, explain why

(b) Classify the states of the chain

(c) Repeat part (a) with the row entries of $\textbf{P}$ switched. That is, let
    $$ P_{i,j}=   \left\{
\begin{array}{ll}
      1/(i+1) & \text{if j = 1} \\
      i/(i+1) & \text{if j = i+1} \\
      0& \text{otherwise} \\
\end{array}
\right.  $$

### Answer
(a) The chain does not have an stationary distribution, this because it does not matter how far it is down the line it would always be able to go back to state 0 and start over again. This means that after an infinity amount of jumps it can always go back to the beginning and start all over again

(b) The chain is an infinite, irreducible and recurrent chain.

(a) the same case as A applyies, eventhough the probability of going back to the beginning, this chain does not have a limiting distribution.

## 3.7
A Markov chain has n states. If the chain is at state k, a coin is flipped, whose heads probability is $p$. If the coin lands heads, the chain stays at $k$. if the coin lands tails, the chaiin moves to a different state uniformly at random. Exhibit the transition matrix and find the stationary distribution.

### Answer
Let's see the matrix:
$$P=\left(\begin{array}{cc}
p & q/n & q/n & q/n & .. \\
q/n & p & q/n & q/n & .. \\
q/n & q/n & p & q/n & .. \\
q/n & q/n & q/n & p & .. \\
.&.&.&.&.
\end{array}\right)$$
It is clear that the rows and columns are equal, hence, double stochastic. Then the stationary distribution is uniform.


```python
import unittest
import sympy as sym
from IPython.display import YouTubeVideo
```

```python
YouTubeVideo('MBhPaqKRehg', width=800, height=400)
```

```python
YouTubeVideo('NcIAd0ho43g', width=800, height=400)
```

```python
sym.init_printing(use_latex="mathjax")
```

```python
B = sym.Matrix([[1, 2, 0], [3, 4, 0], [7, 5, 0], [0,0, 0]])
```

```python
B
```

$$\left[\begin{matrix}1 & 2 & 0\\3 & 4 & 0\\7 & 5 & 0\\0 & 0 & 0\end{matrix}\right]$$

```python
foods = sym.Matrix([[260, 51, 70], [9, 5.4, 15], [5, 5.2, 0]])
b =sym.Matrix([40, 30, 10])
```

```python
B.echelon_form()
```

$$\left[\begin{matrix}1 & 2 & 0\\0 & -2 & 0\\0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$$

```python
B[0,:]
```

$$\left[\begin{matrix}1 & 2 & 0\end{matrix}\right]$$

```python
A = sym.diag(3, 4, 5)
```

```python
type(A)
```

    sympy.matrices.dense.MutableDenseMatrix

```python
B * A
```

$$\left[\begin{matrix}3 & 8 & 0\\9 & 16 & 0\\21 & 20 & 0\\0 & 0 & 0\end{matrix}\right]$$

```python
sym.ones(4) + sym.eye(4)
```

$$\left[\begin{matrix}2 & 1 & 1 & 1\\1 & 2 & 1 & 1\\1 & 1 & 2 & 1\\1 & 1 & 1 & 2\end{matrix}\right]$$

```python
sym.zeros(3)
```

$$\left[\begin{matrix}0 & 0 & 0\\0 & 0 & 0\\0 & 0 & 0\end{matrix}\right]$$

```python
sym.Matrix([[3,4],[5,7]])
```

$$\left[\begin{matrix}3 & 4\\5 & 7\end{matrix}\right]$$

```python
sym.eye(3)
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
YouTubeVideo('SNdzRI_yqIo', width=800, height=400)
```

```python
x, y = sym.symbols("x y")
```

```python
x**2
```

$$x^{2}$$

```python
a11, a12, a21, a22, b11, b12, b21, b22 = sym.symbols("a11, a12, a21, a22, b11, b12, b21, b22")
```

```python
A = sym.Matrix([[0, -2], [6, 1]])
B = sym.Matrix([[-1, 5, 5], [0,0,0]])
```

```python
A , B
```

$$\left ( \left[\begin{matrix}0 & -2\\6 & 1\end{matrix}\right], \quad \left[\begin{matrix}-1 & 5 & 5\\0 & 0 & 0\end{matrix}\right]\right )$$

```python
A * B
```

$$\left[\begin{matrix}0 & 0 & 0\\-6 & 30 & 30\end{matrix}\right]$$

```python
A * 3
```

$$\left[\begin{matrix}0 & -6\\18 & 3\end{matrix}\right]$$

```python
A + B
```

```python
A * B
```

$$\left[\begin{matrix}a_{11} b_{11} + a_{12} b_{21} & a_{11} b_{12} + a_{12} b_{22}\\a_{21} b_{11} + a_{22} b_{21} & a_{21} b_{12} + a_{22} b_{22}\end{matrix}\right]$$

```python

```

```python

```

```python

```

```python
YouTubeVideo('rqP1EWoAiVc', width=800, height=400)
```

```python
from sympy import Matrix
```

```python
A = Matrix([[1,2,3], [4,5,6], [10, 11, 9]])
A
```

$$\left[\begin{matrix}1 & 2 & 3\\4 & 5 & 6\\10 & 11 & 9\end{matrix}\right]$$

```python
L, U, _ = A.LUdecomposition()
```

```python
A == L*U
```

    True

```python
L
```

$$\left[\begin{matrix}1 & 0 & 0\\4 & 1 & 0\\10 & 3 & 1\end{matrix}\right]$$

```python
U
```

$$\left[\begin{matrix}1 & 2 & 3\\0 & -3 & -6\\0 & 0 & -3\end{matrix}\right]$$

```python
YouTubeVideo('vvq4VdYQOJY', width=800, height=400)
```

```python
from sympy import Matrix, init_printing
```

$$ A \underline{x} = \underline{b} $$

```python
A1 = Matrix([[3, 4, 1, 0], [4, 5, 0, 1]])
A1
```

$$\left[\begin{matrix}3 & 4 & 1 & 0\\4 & 5 & 0 & 1\end{matrix}\right]$$

```python
A1.rref()
```

$$\left ( \left[\begin{matrix}1 & 0 & -5 & 4\\0 & 1 & 4 & -3\end{matrix}\right], \quad \left ( 0, \quad 1\right )\right )$$

```python
A2 = Matrix([[3, 4], [4, 5]])
A2.inv()
```

$$\left[\begin{matrix}-5 & 4\\4 & -3\end{matrix}\right]$$

```python
A2.inv(method='LU')
```

$$\left[\begin{matrix}-5 & 4\\4 & -3\end{matrix}\right]$$

```python

```

```python
YouTubeVideo('FgXp28Abllw', width=800, height=400)
```

```python

```

```python

```

```python

```

```python
def add(a, b):
    return a + b
```

```python
class TestNotebook(unittest.TestCase):

    def test_add(self):
        self.assertEqual(add(2, 2), 4)
    def test_type(self):
        self.assertIsInstance(add(1, 1), int)

unittest.main(argv=[''], verbosity=2, exit=False)
```


# Recitation 9: Benders decomposition for stochastic facility location

```julia
using DataFrames, CSV
using JuMP, Gurobi
using LinearAlgebra, Random, Printf
using Plots

const GRB_ENV = Gurobi.Env()
```

## 1. Problem setup

We want to solve the following stochastic facility location problem:

$$\begin{align}
\min\quad & \sum_{i=1}^Nc_ix_i+\sum_{s=1}^Sp_s\left(\sum_{i=1}^N\sum_{j=1}^Mt_{ij}y_{ij}^s + \sum_{j=1}^Mq_jz_{j}^s\right)\\
\text{s.t.}\quad & \sum_{i=1}^Ny_{ij}^s+z_j^s\ge d_j^s & \forall j\in [M], s\in [S]\\
& \sum_{j=1}^My_{ij}^s \le C_ix_i&\forall i\in[N], s\in[S]\\
& \mathbf{y,z}\ge 0, \mathbf{x}\in\{0,1\}^N
\end{align}$$

## 2. Data Setup

We need to define quite a bit of data, so we're going to need to think a little more carefully than usual about how to store the data and parameters. First, we can use a NamedTuple to keep track of parameters.

```julia
function create_parameters(N, S)
    parameters = (
        seed = 1234,
        num_facilities = N,
        num_customers = N,
        num_scenarios = S,
        square_size = 100, # km
        unit_cost_per_km = 0.1,
        # demand generation
        min_demand = 1000 / N, max_demand = 2000 / N,
        # cost of unmet demand
        min_unmet_cost = 10.0, max_unmet_cost = 20.0,
        # cost of facility
        min_facility_cost = 20.0, max_facility_cost = 30.0,
        # capacity at each facility
        min_capacity = 10.0, max_capacity = 30.0,
    )
    return parameters
end
```

Let's create a parameters object to play with!

```julia
params = create_parameters(100, 10)
```

Now we can define a data type representing the facility location data. It's nice to have a single container with all the data we need. We could also have used a data frame, but the advantage of this approach is it can handle various shapes (e.g. a distance matrix and a cost vector).

```julia
"All dimensions assumed to be self-consistent"
struct FacilityLocationData
    "List of facility coordinates as n_facilities x 2 matrix"
    facilities::Matrix
    "List of customer coordinates"
    customers::Matrix

    "Facility capacities"
    capacity::Vector
    "Cost of unmet demand for each customer"
    cost_unmet_demand::Vector
    "Cost of building each facility"
    facility_cost::Vector
    "Transportation_cost (n_facilites x n_customers)"
    transportation_cost::Matrix

    "Demand scenarios"
    demand::Matrix
    "Probability of each scenario"
    prob::Vector
end

"Build from parameters"
function FacilityLocationData(p::NamedTuple)
    Random.seed!(p.seed)
    facilities = rand(p.num_facilities, 2) * p.square_size
    customers = rand(p.num_customers, 2) * p.square_size
    t_cost = [norm(facilities[i, :] .- customers[j, :])
              for i = 1:p.num_facilities, j=1:p.num_customers] .* p.unit_cost_per_km
    f_cost = rand(p.num_facilities) .* (p.max_facility_cost - p.min_facility_cost) .+ p.min_facility_cost
    capacity = rand(p.num_facilities) .* (p.max_capacity - p.min_capacity) .+ p.min_capacity
    unmet_cost = rand(p.num_customers) .* (p.max_unmet_cost - p.min_unmet_cost) .+ p.min_unmet_cost
    demand = rand(p.num_customers, p.num_scenarios) .* (p.max_demand - p.min_demand) .+ p.min_demand
    prob = 1/p.num_scenarios * ones(p.num_scenarios)
    return FacilityLocationData(facilities, customers, capacity, unmet_cost, f_cost, t_cost, demand, prob)
end
```

OK, let's now create a facility location data object using our parameters, and plot some characteristics.

```julia
data = FacilityLocationData(params)
```

This is a really ugly and useless printout! We can actually make this a bit nicer by defining a better print method for the `FacilityLocationData` type. To do this, we override the default print method, which is called `Base.show`.

```julia
function Base.show(io::IO, data::FacilityLocationData)
    @printf(io, "Facility location data with %d facilities, %d customers, and %d demand scenarios",
            size(data.facilities, 1), size(data.customers, 1), size(data.demand, 2))
end
data
```

And now let's plot the facilities and customers!

```julia
scatter(data.facilities[:, 1], data.facilities[:, 2], label="Facilities")
scatter!(data.customers[:, 1], data.customers[:, 2], label="Customers")
```

## 2. Direct optimization

```julia
TIME_LIMIT = 180;
OPTIMALITY_GAP = 0.01;
```

```julia
"Solve the optimization problem directly"
function direct_solve(data::FacilityLocationData; verbose::Bool = true)
    model = Model(() -> Gurobi.Optimizer(GRB_ENV))
    set_optimizer_attributes(model, "TimeLimit" => TIME_LIMIT, "MIPGap" => OPTIMALITY_GAP,
                             "OutputFlag" => ifelse(verbose, 1, 0))
    N = size(data.facilities, 1); M = size(data.customers, 1); S = size(data.demand, 2)
    @variable(model, x[1:N], Bin)
    @variable(model, y[1:N, 1:M, 1:S] >= 0)
    @variable(model, z[1:M, 1:S] >= 0)
    @objective(model, Min,
               sum(data.facility_cost[i] * x[i] for i=1:N) +
               sum(data.prob[s] * data.transportation_cost[i, j] * y[i, j, s] for i=1:N, j in 1:M, s in 1:S) +
               sum(data.prob[s] * data.cost_unmet_demand[j] * z[j, s] for j in 1:M, s in 1:S))
    @constraint(model, [j in 1:M, s in 1:S],
                sum(y[i, j, s] for i = 1:N) + z[j, s] >= data.demand[j,s])
    @constraint(model, [i in 1:N, s in 1:S],
                sum(y[i, j, s] for j=1:M) <= data.capacity[i] * x[i])
    solvetime = @elapsed optimize!(model)
    opt = objective_value(model)
    bound = objective_bound(model)
    return opt, bound, solvetime
end
```

```julia
direct_solve(data, verbose=true)
```

## 3. Benders decomposition

### 3.1. Multi-cut approach

We want to solve the problem using Benders decomposition. Consider the following multi-cut approach.

Main problem:
$$\begin{align}
\min\quad &\sum_{i=1}^N c_i x_i + \sum_{s=1}^Sp_s\theta_s\\
\text{s.t.}\quad & \mathbf{x}\in\{0,1\}^N, \mathbf{\theta}\ge 0
\end{align}$$

Subproblem $s$:
$$\begin{align}
\min\quad & \sum_{i=1}^N\sum_{j=1}^Mt_{ij}y_{ij}^s + \sum_{j=1}^Mq_jz_{j}^s\\
\text{s.t.}\quad & \sum_{i=1}^Ny_{ij}^s+z_j^s\ge d_j^s & \forall j\in [M]\\
& \sum_{j=1}^My_{ij}^s \le C_ix_i&\forall i\in[N]\\
& \mathbf{y,z}\ge 0
\end{align}$$

Dual subproblem $s$:
$$\begin{align}
\max\quad &  \sum_{j=1}^M \mu_j d_j^s - \sum_{i=1}^N\lambda_iC_ix_i\\
\text{s.t.}\quad & \mu_j -\lambda_i \le t_{ij} & \forall i\in[N], j\in[M]\\
& \mu_j \le q_j &\forall j\in [M]\\
& \mathbf{\mu, \lambda}\ge 0
\end{align}$$

- If the dual subproblem is unbounded, we obtain an extreme ray $(\mu^*, \lambda^*)$ and add a feasibility cut:
$$\sum_{j=1}^M\mu^*_j d_j^s - \sum_{i=1}^N \lambda^*_i C_i x_i \le 0$$

- If the dual subproblem solves to optimality, we obtain an extreme point $(\mu^*, \lambda^*)$ and add an optimality cut:
$$θ_s \ge \sum_{j=1}^M\mu^*_j d_j^s - \sum_{i=1}^N\lambda^*_i C_i x_i$$

We can implement this decomposition as follows:

```julia
"Solve problem using multi-cut Benders decomposition"
function solve_benders_multi(data::FacilityLocationData; verbose::Bool=false)
    # define main problem
    MP = Model(() -> Gurobi.Optimizer(GRB_ENV));
    set_optimizer_attributes(MP, "TimeLimit" => 60, "MIPGap" => 1e-4, "OutputFlag" => 0)
    N = size(data.facilities, 1); M = size(data.customers, 1); S = size(data.demand, 2)
    @variable(MP, x[1:N], Bin)
    @variable(MP, θ[1:S] >= 0)
    @objective(MP, Min, sum(data.facility_cost[i] * x[i] for i=1:N) + sum(data.prob[s] * θ[s] for s in 1:S))

    lower_bound_all = []; upper_bound_all = []
    MP_time = []; SP_max_time = []; SP_time = []
    while true
        # solve master problem
        push!(MP_time, @elapsed optimize!(MP))
        lower_bound_new = objective_value(MP)
        push!(lower_bound_all, lower_bound_new)
        x_MP = value.(MP[:x])
        # solve S subproblems
        obj_SP = zeros(S)
        SP_time_all = zeros(S)
        for s = 1:S
            SP_dual = Model(() -> Gurobi.Optimizer(GRB_ENV))
            set_optimizer_attributes(SP_dual, "OutputFlag" => 0)
            @variable(SP_dual, λ[1:N] >= 0);
            @variable(SP_dual, μ[1:M] >= 0);
            @objective(SP_dual, Max,
                       sum(μ[j] * data.demand[j,s] for j in 1:M) -
                       sum(λ[i] * data.capacity[i] * x_MP[i] for i in 1:N))
            @constraint(SP_dual, [i in 1:N, j in 1:M], μ[j] - λ[i] <= data.transportation_cost[i,j])
            @constraint(SP_dual, [j in 1:M], μ[j] <= data.cost_unmet_demand[j])
            SP_time_all[s] = @elapsed optimize!(SP_dual)
            obj_SP_dual = objective_value(SP_dual)
            λ_val = value.(SP_dual[:λ])
            μ_val = value.(SP_dual[:μ])
            if termination_status(SP_dual) == MOI.DUAL_INFEASIBLE # feasibility cut
                @constraint(MP, sum(μ_val[j] * data.demand[j, s] for j in 1:M) -
                            sum(λ_val[i] * data.capacity[i] * x[i] for i in 1:N) <= 0)
                obj_SP[s] = 999999999
            elseif termination_status(SP_dual) == MOI.OPTIMAL
                @constraint(MP, θ[s] >= sum(μ_val[j] * data.demand[j,s] for j in 1:M) -
                            sum(λ_val[i] * data.capacity[i] * x[i] for i in 1:N))
                obj_SP[s] = obj_SP_dual
            end
        end
        push!(SP_max_time, maximum(SP_time_all))
        push!(SP_time, sum(SP_time_all))
        upper_bound_new = sum(data.facility_cost[i] * x_MP[i] for i=1:N) + sum(data.prob[s] * obj_SP[s] for s in 1:S)
        push!(upper_bound_all, upper_bound_new)
        verbose && @printf("Sol: %.2f - Bound: %.2f\n", upper_bound_all[end], lower_bound_all[end])
        if sum(MP_time) + sum(SP_time) >= TIME_LIMIT ||
            (upper_bound_new-lower_bound_new)/lower_bound_new < OPTIMALITY_GAP
            break
        end
    end
    return upper_bound_all, lower_bound_all, MP_time, SP_time, SP_max_time
end
```

```julia
@time upper1, lower1, main_time1, subproblem_time1, subproblem_max_time1 = solve_benders_multi(data, verbose = true);
```

We can take a look at the convergence by iteration:

```julia
plot([upper1 lower1], label=["Upper bound" "Lower bound"], xlabel="Iteration")
```

And we can also look at it as a function of time:

```julia
plot(cumsum(main_time1 .+ subproblem_time1), [upper1 lower1], label=["Upper bound" "Lower bound"], xlabel="Time (s)")
```

In our implementation, we solved all the subproblems in series, but they are actually independent. We can estimate the runtime on a fully parallel system by using the maximum subproblem runtime instead of the sum of subproblem runtimes.

```julia
plot(cumsum(main_time1 .+ subproblem_max_time1),
    [upper1 lower1], label=["Upper bound" "Lower bound"], xlabel="Time in parallel (s)")
```

### 3.2. Single-cut approach

An alternative approach is the following formulation:

$$\begin{align}
\min\quad &\sum_{i=1}^N c_i x_i + \theta\\
\text{s.t.}\quad & \mathbf{x}\in\{0,1\}^N, \mathbf{\theta}\ge 0
\end{align}$$

We keep the same $S$ subproblems, but this time we add cuts as follows:

- If **any** dual subproblem is unbounded, we obtain an extreme ray $(\mu^*, \lambda^*)$ and add a feasibility cut:
$$\sum_{j=1}^M\mu^*_j d_j^s - \sum_{i=1}^N \lambda^*_i C_i x_i \le 0$$

- If **all** dual subproblems solve to optimality, we obtain extreme points $(\mu^*_s, \lambda^*_s)$ and add an optimality cut:
$$θ \ge \sum_{s=1}^Sp_s\left(\sum_{j=1}^M\mu^*_j d_j^s - \sum_{i=1}^N\lambda^*_i C_i x_i\right)$$

```julia
"Solve problem using multi-cut Benders decomposition"
function solve_benders_single(data::FacilityLocationData; verbose::Bool=false)
    MP = Model(() -> Gurobi.Optimizer(GRB_ENV));
    set_optimizer_attributes(MP, "TimeLimit" => 60, "MIPGap" => 1e-4, "OutputFlag" => 0)
    N = size(data.facilities, 1); M = size(data.customers, 1); S = size(data.demand, 2)
    @variable(MP, x[1:N], Bin)
    @variable(MP, θ >= 0)
    @objective(MP, Min, sum(data.facility_cost[i] * x[i] for i=1:N) + θ)
    lower_bound_all = []; upper_bound_all = []
    MP_time = []; SP_time = []; SP_max_time = []
    while true
        push!(MP_time, @elapsed optimize!(MP))
        lower_bound_new = objective_value(MP)
        push!(lower_bound_all, lower_bound_new)
        x_MP = value.(MP[:x])
        # set up subproblem loop
        obj_SP = zeros(S)
        SP_time_all = zeros(S)
        λ_all = zeros(N, S)
        μ_all = zeros(M, S)
        optimality_cut = true
        for s in 1:S
            SP_dual = Model(() -> Gurobi.Optimizer(GRB_ENV));
            set_optimizer_attributes(SP_dual, "TimeLimit" => 60, "MIPGap" => 1e-4, "OutputFlag" => 0)
            @variable(SP_dual, λ[1:N] >= 0)
            @variable(SP_dual, μ[1:M] >= 0)
            @objective(SP_dual, Max, sum(μ[j] * data.demand[j,s] for j in 1:M)-
                       sum(λ[i] * data.capacity[i] * x_MP[i] for i in 1:N))
            @constraint(SP_dual, [i in 1:N, j in 1:M], μ[j] - λ[i] <= data.transportation_cost[i, j])
            @constraint(SP_dual, [j in 1:M], μ[j] <= data.cost_unmet_demand[j])
            SP_time_all[s] = @elapsed optimize!(SP_dual)
            obj_SP_dual = objective_value(SP_dual)
            λ_all[:, s] = value.(SP_dual[:λ])
            μ_all[:, s] = value.(SP_dual[:μ])
            if termination_status(SP_dual) == MOI.DUAL_INFEASIBLE # feasibility cut
                optimality_cut = false
                @constraint(MP, sum(μ_val[j] * data.demand[j, s] for j in 1:M) -
                            sum(λ_val[i] * data.capacity[i] * x[i] for i in 1:N) <= 0)
                obj_SP[s] = 999999999
            elseif termination_status(SP_dual) == MOI.OPTIMAL
                obj_SP[s] = obj_SP_dual
            end
        end
        if optimality_cut
            @constraint(MP, θ >= sum(- data.prob[s] * λ_all[i, s] * data.capacity[i] * x[i] for i=1:N, s=1:S) +
                        sum(data.prob[s] * μ_all[j, s] * data.demand[j, s] for j = 1:M, s = 1:S))
        end
        push!(SP_max_time, maximum(SP_time_all))
        push!(SP_time, sum(SP_time_all))
        upper_bound_new = sum(data.facility_cost[i] * x_MP[i] for i=1:N) + sum(data.prob[s] * obj_SP[s] for s in 1:S);
        push!(upper_bound_all, upper_bound_new)
        verbose && @printf("Sol: %.2f - Bound: %.2f\n", upper_bound_all[end], lower_bound_all[end])
        if sum(MP_time) + sum(SP_time) >= TIME_LIMIT ||
            (upper_bound_new - lower_bound_new) / lower_bound_new < OPTIMALITY_GAP
            break
        end
    end
    return upper_bound_all, lower_bound_all, MP_time, SP_time, SP_max_time
end
```

Let's try this on our problem:

```julia
@time upper2, lower2, main_time2, subproblem_time2, subproblem_max_time2 = solve_benders_single(data, verbose = true);
```

And we can plot the progress, as before...

```julia
plot(cumsum(main_time2 .+ subproblem_time2), [upper2 lower2], label=["Upper bound" "Lower bound"], xlabel="Time (s)")
```

### 3.3 Pareto-optimal cuts

In lecture, we also discussed Pareto-optimal cuts, i.e. cuts that provide a tighter approximation to the recourse function. We mentioned that we can compute a Pareto-optimal cut by finding the best cut for a particular "core point" $x^0$. We first solve the dual subproblem:

$$\begin{align}
z^*=\max\quad &  \sum_{j=1}^M \mu_j d_j^s - \sum_{i=1}^N\lambda_iC_ix_i\\
\text{s.t.}\quad & \mu_j -\lambda_i \le t_{ij} & \forall i\in[N], j\in[M]\\
& \mu_j \le q_j& \forall j\in [M]\\
& \mathbf{\mu, \lambda}\ge 0
\end{align}$$

The, given the optimal objective, $x^*$, we solve the problem again with a new objective, i.e.

$$\begin{align}
\max\quad &  \sum_{j=1}^M \mu_j d_j^s - \sum_{i=1}^N\lambda_iC_ix^0_i\\
\text{s.t.}\quad & \mu_j -\lambda_i \le t_{ij} & \forall i\in[N], j\in[M]\\
& \mu_j \le q_j& \forall j\in [M]\\
& \sum_{j=1}^M \mu_j d_j^s - \sum_{i=1}^N\lambda_iC_ix_i=z^*\\
& \mathbf{\mu, \lambda}\ge 0
\end{align}$$

The core point we use is $x^0=\{0.5\}_{i=1}^N$.

```julia
"Solve problem using multi-cut Benders decomposition"
function solve_benders_pareto(data::FacilityLocationData; verbose::Bool=false)
    MP = Model(() -> Gurobi.Optimizer(GRB_ENV));
    set_optimizer_attributes(MP, "TimeLimit" => 60, "MIPGap" => 1e-4, "OutputFlag" => 0)
    N = size(data.facilities, 1); M = size(data.customers, 1); S = size(data.demand, 2)
    @variable(MP, x[1:N], Bin)
    @variable(MP, θ >= 0)
    @objective(MP, Min, sum(data.facility_cost[i] * x[i] for i=1:N) + θ)
    lower_bound_all = []; upper_bound_all = []
    MP_time = []; SP_time = []; SP_max_time = []
    while true
        push!(MP_time, @elapsed optimize!(MP))
        lower_bound_new = objective_value(MP)
        push!(lower_bound_all, lower_bound_new)
        x_MP = value.(MP[:x])
        # set up subproblem loop
        obj_SP = zeros(S)
        SP_time_all = zeros(S)
        λ_all = zeros(N, S)
        μ_all = zeros(M, S)
        optimality_cut = true
        for s in 1:S
            SP_dual = Model(() -> Gurobi.Optimizer(GRB_ENV));
            set_optimizer_attributes(SP_dual, "TimeLimit" => 60, "MIPGap" => 1e-4, "OutputFlag" => 0)
            @variable(SP_dual, λ[1:N] >= 0)
            @variable(SP_dual, μ[1:M] >= 0)
            @objective(SP_dual, Max, sum(μ[j] * data.demand[j,s] for j in 1:M)-
                       sum(λ[i] * data.capacity[i] * x_MP[i] for i in 1:N))
            @constraint(SP_dual, [i in 1:N, j in 1:M], μ[j] - λ[i] <= data.transportation_cost[i, j])
            @constraint(SP_dual, [j in 1:M], μ[j] <= data.cost_unmet_demand[j])
            SP_time_all[s] = @elapsed optimize!(SP_dual)
            obj_SP_dual = objective_value(SP_dual)
            λ_all[:, s] = value.(SP_dual[:λ])
            μ_all[:, s] = value.(SP_dual[:μ])
            if termination_status(SP_dual) == MOI.DUAL_INFEASIBLE # feasibility cut
                optimality_cut = false
                @constraint(MP, sum(μ_val[j] * data.demand[j, s] for j in 1:M) -
                            sum(λ_val[i] * data.capacity[i] * x[i] for i in 1:N) <= 0)
                obj_SP[s] = 999999999
            elseif termination_status(SP_dual) == MOI.OPTIMAL
                obj_SP[s] = obj_SP_dual
                @constraint(SP_dual, sum(μ[j] * data.demand[j,s] for j in 1:M)-
                            sum(λ[i] * data.capacity[i] * x_MP[i] for i in 1:N) == obj_SP[s])
                @objective(SP_dual, Max, sum(μ[j] * data.demand[j,s] for j in 1:M)-
                           sum(λ[i] * data.capacity[i] * 0.5 for i in 1:N))
                SP_time_all[s] += @elapsed optimize!(SP_dual)
                λ_all[:, s] = value.(SP_dual[:λ])
                μ_all[:, s] = value.(SP_dual[:μ])
            end
        end
        if optimality_cut
            @constraint(MP, θ >= sum(- data.prob[s] * λ_all[i, s] * data.capacity[i] * x[i] for i=1:N, s=1:S) +
                        sum(data.prob[s] * μ_all[j, s] * data.demand[j, s] for j = 1:M, s = 1:S))
        end
        push!(SP_max_time, maximum(SP_time_all))
        push!(SP_time, sum(SP_time_all))
        upper_bound_new = sum(data.facility_cost[i] * x_MP[i] for i=1:N) + sum(data.prob[s] * obj_SP[s] for s in 1:S);
        push!(upper_bound_all, upper_bound_new)
        verbose && @printf("Sol: %.2f - Bound: %.2f\n", upper_bound_all[end], lower_bound_all[end])
        if sum(MP_time) + sum(SP_time) >= TIME_LIMIT ||
            (upper_bound_new - lower_bound_new) / lower_bound_new < OPTIMALITY_GAP
            break
        end
    end
    return upper_bound_all, lower_bound_all, MP_time, SP_time, SP_max_time
end
```

Let's solve the problem using Pareto cuts:

```julia
@time upper3, lower3, main_time3, subproblem_time3, subproblem_max_time3 = solve_benders_pareto(data, verbose = true);
```

```julia
plot(cumsum(main_time3 .+ subproblem_time3), [upper3 lower3], label=["Upper bound" "Lower bound"], xlabel="Time (s)")
```

Instead of solving the dual subproblem twice, another way to obtain Pareto-optimal cuts is to add the core point objective with a small weight $\varepsilon$, to the original dual objective.

### 3.4 Benders cuts with lazy constraints

The main drawback of the approaches above is that we have to restart solving the master problem from scratch every time we add the cuts. One way we can change that is by using lazy constraints.

Instead of providing the solver with constraints, we give it a "callback function", i.e., a function which takes in a solution and provides a constraint that it violates. The solver will regularly call this function to see if there are new constraints that need to be added.

```julia
"Solve problem using multi-cut Benders decomposition - verbose can be 0, 1 or 2"
function solve_benders_lazy(data::FacilityLocationData; verbose::Int=0)
    # define main problem
    MP = Model(() -> Gurobi.Optimizer(GRB_ENV));
    set_optimizer_attributes(MP, "TimeLimit" => TIME_LIMIT, "MIPGap" => OPTIMALITY_GAP,
                             "OutputFlag" => ifelse(verbose > 0, 1, 0), "LazyConstraints" => 1)
    N = size(data.facilities, 1); M = size(data.customers, 1); S = size(data.demand, 2)
    @variable(MP, x[1:N], Bin)
    @variable(MP, θ >= 0)
    @objective(MP, Min, sum(data.facility_cost[i] * x[i] for i=1:N) + θ)

    "Define the callback function"
    function add_benders_cuts(cb_data)
        status = callback_node_status(cb_data, MP)
        if status == MOI.CALLBACK_NODE_STATUS_INTEGER
            verbose == 2 && println("Adding lazy constraints...")
            # get value of current solution
            x_MP = [round(callback_value(cb_data, x[i])) for i = 1:N]
            λ_all = zeros(N, S)
            μ_all = zeros(M, S)
            optimality_cut = true
            # solve dual subproblems
            for s = 1:S
                SP_dual = Model(() -> Gurobi.Optimizer(GRB_ENV))
                set_optimizer_attributes(SP_dual, "OutputFlag" => 0)
                @variable(SP_dual, λ[1:N] >= 0);
                @variable(SP_dual, μ[1:M] >= 0);
                @objective(SP_dual, Max,
                           sum(μ[j] * data.demand[j,s] for j in 1:M) -
                           sum(λ[i] * data.capacity[i] * x_MP[i] for i in 1:N))
                @constraint(SP_dual, [i in 1:N, j in 1:M], μ[j] - λ[i] <= data.transportation_cost[i,j])
                @constraint(SP_dual, [j in 1:M], μ[j] <= data.cost_unmet_demand[j])
                optimize!(SP_dual)
                λ_all[:,s] = value.(SP_dual[:λ])
                μ_all[:,s] = value.(SP_dual[:μ])
                if termination_status(SP_dual) == MOI.DUAL_INFEASIBLE # feasibility cut
                    optimality_cut = false
                    fea = @build_constraint(sum(μ_all[j, s] * data.demand[j, s] for j in 1:M) -
                                            sum(λ_all[i, s] * data.capacity[i] * x[i] for i in 1:N) <= 0)
                    MOI.submit(MP, MOI.LazyConstraint(cb_data), fea)
                elseif termination_status(SP_dual) == MOI.OPTIMAL
                    # do nothing
                end
            end
            opt = @build_constraint(θ >= sum(data.prob[s] * μ_all[j, s] * data.demand[j,s] for j = 1:M, s=1:S) -
                                    sum(data.prob[s] * λ_all[i, s] * data.capacity[i] * x[i] for i = 1:N, s=1:S))
            MOI.submit(MP, MOI.LazyConstraint(cb_data), opt)
        end
    end
    # set callback function and attach to model
    MOI.set(MP, MOI.LazyConstraintCallback(), add_benders_cuts)
    solvetime = @elapsed optimize!(MP)
    opt = objective_value(MP)
    bound = objective_bound(MP)
    return opt, bound, solvetime
end
```

We can solve the problem using lazy constraints:

```julia
solve_benders_lazy(data, verbose=0)
```

Note that here we are using JuMP's solver-independent solver callback syntax. That means we don't have as much control about what to do within the callback, and when to actually call it. For example, when we set `verbose=2` above, we see we generate a lot of lazy constraints even before the end of the presolve. Why might that happen?

If we want more fine-grained control over the solver, we can write a solver-dependent callback function, using the `Gurobi.jl` package directly - see [here](https://www.gurobi.com/documentation/9.1/refman/cb_codes.html) for more details!

Another drawback of using lazy constraints is that it is harder to parallelize, and we do not have full control over when the callback is run. There are many examples where using lazy constraints considerably speeds up the problem. A usual one is the traveling salesman problem, and related vehicle routing problems.


# 6. Internal dynamic factor

Based on:

[1] ISO 6336-1:2006 Calculation of load capacity of spur and helical gears -- Part 1: Basic principles, introduction and general influence factors

```python
from sympy import *
from matplotlib import pyplot
from numpy import arange
init_printing()

def symb(x, y, z = ''):
    return symbols('{0}_{1}{2}'.format(x, y, z), type = float)
```

### 6.4.2 Method B -- Factor $K_{v-B}$
This method is not recommended if:
\begin{align}
    \frac{v z_1}{100} \sqrt{u^2/(1 + u^2)} < 3 \mathrm{m/s}
\end{align}

## 6.5 Determination using Method B
### 6.5.3 Resonance running speed:
It is given by:

```python
u, z1, cga, mRed = symbols('u z_1 c_gamma_alpha m_red')
eq6 = 30000/(pi*z1)*sqrt(cga/mRed) # n_E1
eq6
```

where $m_{red}$ is the relative mass of a gear pair, i.e. of(?) the mass per face width of each gear, referred to its base radius or to the line of action. It is equal to:

```python
eq7 = symb('m',1,'^*')*symb('m',2,'^*')/(symb('m',1,'^*') + symb('m',2,'^*'))
eq7
```

The resonance ratio is the ratio of pinion speed to resonance speed, and it given by:

```python
simplify(symb('n', 1)/eq6)
```

```python
NS = lambda x: (0.85 if (x >= 100) else 0.5 + 0.35*sqrt(x/100))
X = arange(0, 200.0)
Y = array([NS(x) for x in X])
plotly(X, Y)
```

### 6.5.9
valid for gear pais with external teeth: spur, single and double helical.

```python
dm1, db1, rho1, q1, rho2, q2, rho = symbols('d_m1 d_b1 rho_1 q_1 rho_2 q_2 rho')
eq30r = dm1**2/(1/(rho1*(1 - q1**4)) + 1/(rho2*(1 - q2**4)*u**2))
eq30l = (pi/8)*(dm1/db1)**2
eq30 = eq30l*eq30r
eq30
```

for pinions and wheels of solid construction: $q_1 = q_2 = 0$.

```python
eq30 = eq30.subs([(q1, 0), (q2,0)])
simplify(eq30)
```

Considering that the gears have the same density, leads to:

```python
eq30 = eq30.subs([(rho1, rho), (rho2, rho)])
simplify(eq30)
```

Notice that altough this expression represents the reduced mass of a gear pair, it does not has units of mass.

```python

```


```python
%matplotlib inline
import numpy as np
from sympy.solvers import solve
from sympy import Symbol
import matplotlib.pyplot as plt
import pandas as pd
%config InlineBackend.figure_formats = {'png', 'retina'}
```

# Training Data: Loaded from a text file with labels.
## Two features, x and y

```python
train=pd.read_csv('synth.tr.txt', header=0, delim_whitespace=True, names=('xs', 'ys', 'yc'))

x0=np.array(train[train['yc']==0]['xs'])
y0=np.array(train[train['yc']==0]['ys'])

x1=np.array(train[train['yc']==1]['xs'])
y1=np.array(train[train['yc']==1]['ys'])

z0=np.array([x0,y0]).transpose()
z1=np.array([x1,y1]).transpose()
```

# Finding $\sigma$ and $\mu$ assuming Gaussian distribution for the training data.
# (Parametric classifier)

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

plt.figure(figsize=(10,10))
plt.plot(x0,y0,'r*',label='Class-0', markersize=9)
plt.plot(x1,y1,'go',label='Class-1', markersize=9)

plt.plot(mu0[0],mu0[1],'yX',label='Mean-0', markersize=15)
plt.plot(mu1[0],mu1[1],'cX',label='Mean-1', markersize=15)

plt.ylim(-0.25,1.4)
plt.xlim(-1.27,0.85)
plt.xlabel('X-feature', fontsize=17)
plt.ylabel('Y-feature', fontsize=17)
plt.title('Two-Class Two-Feature Classification', fontsize=20)
plt.legend(loc='upper right')
plt.legend(loc='upper right')
plt.show()
```

# Testing Data: Loaded from a text file. Two features, x and y. Labels are for accuracy claculation

```python
test=pd.read_csv('synth.te.txt', header=0, delim_whitespace=True, names=('xs', 'ys', 'yc'))
xtest=np.array(test['xs'])
ytest=np.array(test['ys'])
ztest=np.array([xtest,ytest]).transpose()
```

# Parameric classifier
## Case-1 Bayesian classifier : $\Sigma_1$ = $\Sigma_2$= $\sigma^2I$

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

sig=(sigma0[0,0]+sigma1[1,1])/2.0
sigma=sig*(np.identity(2, dtype=float))
sigmadet=np.linalg.det(sigma)
sigmainv=np.linalg.inv(sigma)

# Solving for the Decision Boundary in Case-1
x=Symbol('x')
y=Symbol('y')
zz=np.array([x,y])
z=zz.reshape(1,2)

Prior=0.5
g00=-0.5*(z-mu0).reshape(1,2).dot(sigmainv).dot((z-mu0).reshape(2,1))-0.5*np.log(sigmadet)+np.log(Prior)
g11=-0.5*(z-mu1).reshape(1,2).dot(sigmainv).dot((z-mu1).reshape(2,1))-0.5*np.log(sigmadet)+np.log(Prior)

DB1=solve(g00[0][0]-g11[0][0], y)

print('The linear equation for the decision boundary in case-1:\ny='+str(DB1[0])+'\n\n')

#------------------------------------------------------------------------------------------------------------------------------

#Categorizing based on the Discriminant Function

def g0(data, Prior):
    k,l=data.shape  # k = number of training samples, l = dimension (# of features)
    g0=np.zeros(k)
    for i in range(k):
        g0[i]=-0.5*(data[i]-mu0).reshape(1,l).dot(sigmainv).dot((data[i]-mu0).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g0

def g1(data, Prior):
    k,l=data.shape
    g1=np.zeros(k)
    for i in range(k):
        g1[i]=-0.5*(data[i]-mu1).reshape(1,l).dot(sigmainv).dot((data[i]-mu1).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g1

boolcase1=g0(ztest,0.5)>=g1(ztest,0.5)
boolcase1=np.array([0 if i else 1 for i in boolcase1])  # Converting the True to zero and False to 1.

#-----------------------------------Accuracy Calculation for Case-1------------------------------------------------------
testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.
accu_case1=testbool+boolcase1
unique1, counts1 = np.unique(accu_case1, return_counts=True)
case1=dict(zip(unique1, counts1))

# print('Case 1 = ' + str(case1))
accuracy_case1= (counts1[0]+counts1[2])/(counts1[0]+counts1[1]+counts1[2])
print('Case 1: Overall Accuracy = ' + str(accuracy_case1))
```

    The linear equation for the decision boundary in case-1:
    y=-0.832622948392766*x + 0.44378197841356

    Case 1: Overall Accuracy = 0.713

## Case-2 Bayesian classifier : $\Sigma_1$ $=$ $\Sigma_2$

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

sigma=(sigma0+sigma1)/2.
sigmadet=np.linalg.det(sigma)
sigmainv=np.linalg.inv(sigma)

# Solving for the Decision Boundary in Case-3
x=Symbol('x')
y=Symbol('y')
zz=np.array([x,y])
z=zz.reshape(1,2)

Prior=0.5
g00=-0.5*(z-mu0).reshape(1,2).dot(sigmainv).dot((z-mu0).reshape(2,1))-0.5*np.log(sigmadet)+np.log(Prior)
g11=-0.5*(z-mu1).reshape(1,2).dot(sigmainv).dot((z-mu1).reshape(2,1))-0.5*np.log(sigmadet)+np.log(Prior)

DB2=solve(g00[0][0]-g11[0][0], y)

print('The linear equation for the decision boundary in case-2:\ny='+str(DB2[0])+'\n\n')

#Categorizing based on the Discriminant Function

def g0(data, Prior):
    k,l=data.shape
    g0=np.zeros(k)
    for i in range(k):
        g0[i]=-0.5*(data[i]-mu0).reshape(1,l).dot(sigmainv).dot((data[i]-mu0).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g0

def g1(data, Prior):
    k,l=data.shape
    g1=np.zeros(k)
    for i in range(k):
        g1[i]=-0.5*(data[i]-mu1).reshape(1,l).dot(sigmainv).dot((data[i]-mu1).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g1

boolcase2=g0(ztest,0.5) >= g1(ztest,0.5)
boolcase2=np.array([0 if i else 1 for i in boolcase2])  # Converting the True to zero and False to 1.

#-----------------------------------Accuracy Calculation for Case-3------------------------------------------------------
testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.
accu_case2=testbool+boolcase2
unique2, counts2 = np.unique(accu_case2, return_counts=True)
case2=dict(zip(unique2, counts2))

# print('Case 2 = ' + str(case2))
accuracy_case2= (counts2[0]+counts2[2])/(counts2[0]+counts2[1]+counts2[2])
print('Case 2: Overall Accuracy = ' + str(accuracy_case2))
```

    The linear equation for the decision boundary in case-2:
    y=-0.134864086623903*x + 0.494549490884199

    Case 2: Overall Accuracy = 0.892

## Case-3 Bayesian classifier : $\Sigma_1$ $\neq$ $\Sigma_2$

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

sigma0det=np.linalg.det(sigma0)
sigma1det=np.linalg.det(sigma1)
sigma0inv=np.linalg.inv(sigma0)
sigma1inv=np.linalg.inv(sigma1)

# Solving for the Decision Boundary in Case-3
x=Symbol('x')
y=Symbol('y')
zz=np.array([x,y])
z=zz.reshape(1,2)

Prior=0.5
g00=-0.5*(z-mu0).reshape(1,2).dot(sigma0inv).dot((z-mu0).reshape(2,1))-0.5*np.log(sigma0det)+np.log(Prior)
g11=-0.5*(z-mu1).reshape(1,2).dot(sigma1inv).dot((z-mu1).reshape(2,1))-0.5*np.log(sigma1det)+np.log(Prior)

DB3=solve(g00[0][0]-g11[0][0], y)

print('The linear equation for the decision boundary in case-3:\ny='+str(DB3[0])+'\n\n')

#Categorizing based on the Discriminant Function

def g0(data, Prior):
    k,l=data.shape
    g0=np.zeros(k)
    for i in range(k):
        g0[i]=-0.5*(data[i]-mu0).reshape(1,l).dot(sigma0inv).dot((data[i]-mu0).reshape(l,1))-0.5*np.log(sigma0det)+np.log(Prior)
    return g0

def g1(data, Prior):
    k,l=data.shape
    g1=np.zeros(k)
    for i in range(k):
        g1[i]=-0.5*(data[i]-mu1).reshape(1,l).dot(sigma1inv).dot((data[i]-mu1).reshape(l,1))-0.5*np.log(sigma1det)+np.log(Prior)
    return g1

boolcase3=g0(ztest,0.5)>=g1(ztest,0.5) # True if the sample belongs to class-0, False otherwise
boolcase3=np.array([0 if i else 1 for i in boolcase3])  # Converting the True to zero and False to 1.

#-----------------------------------Accuracy Calculation for Case-3------------------------------------------------------
testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.
accu_case3=testbool+boolcase3
unique3, counts3 = np.unique(accu_case3, return_counts=True)
case3=dict(zip(unique3, counts3))

# print('Case 3 = ' + str(case3))
accuracy_case3= (counts3[0]+counts3[2])/(counts3[0]+counts3[1]+counts3[2])
print('Case 3: Overall Accuracy = ' + str(accuracy_case3))
```

    The linear equation for the decision boundary in case-3:
    y=-0.641052306096743*x - 8.91426865585982e-25*sqrt(-1.51514382948643e+45*x**2 - 1.95123755461517e+48*x + 3.25462873130337e+48) + 2.09052329406855

    Case 3: Overall Accuracy = 0.898

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

plt.figure(figsize=(10,10))
plt.plot(x0,y0,'r*',label='Class-0', markersize=9)
plt.plot(x1,y1,'go',label='Class-1', markersize=9)

plt.plot(mu0[0],mu0[1],'yX',label='Mean-0', markersize=15)
plt.plot(mu1[0],mu1[1],'cX',label='Mean-1', markersize=15)

xmin, xmax=np.min(np.array(train['xs'])), np.max(np.array(train['xs']))

xplot=np.arange(-1.27, 0.86, 0.01)
# xplot=np.arange(xmin, xmax, 0.01)

plt.plot(xplot, -0.832622948392766*xplot + 0.44378197841356, 'y-', linewidth=2, label='Case-1 Decision Boundary')
plt.plot(xplot, -0.134864086623903*xplot + 0.494549490884199, 'c-', linewidth=2, label='Case-2 Decision Boundary')
plt.plot(xplot, -0.641052306096743*xplot - 8.91426865585982e-25*np.sqrt(-1.51514382948643e+45*(xplot**2) - 1.95123755461517e+48*xplot + 3.25462873130337e+48) + 2.09052329406855, 'b-', linewidth=2, label='Case-3 Decision Boundary')

plt.ylim(-0.25,1.4)
plt.xlim(-1.27,0.85)
plt.xlabel('X-feature', fontsize=17)
plt.ylabel('Y-feature', fontsize=17)
plt.title('Two-Class Two-Feature Classification', fontsize=20)
plt.legend(loc='upper right')
plt.legend(loc='upper right')
plt.show()
```

```python
testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.

#------------------------------------------
xtest0=np.array(test[test['yc']==0]['xs'])#
ytest0=np.array(test[test['yc']==0]['ys'])#
                                          #
xtest1=np.array(test[test['yc']==1]['xs'])#
ytest1=np.array(test[test['yc']==1]['ys'])#
#------------------------------------------

plt.figure(figsize=(12,12))
plt.plot(x0,y0,'m*', alpha= 0.3, label='Train-0', markersize=5)
plt.plot(x1,y1,'yo', alpha= 0.3, label='Train-1', markersize=5)
plt.plot(xtest0,ytest0,'r*', label='Test-0', markersize=8)
plt.plot(xtest1,ytest1,'go', label='Test-1', markersize=8)

xmin, xmax=np.min(np.array(train['xs'])), np.max(np.array(train['xs']))

xplot=np.arange(-1.27, 0.86, 0.01)
# xplot=np.arange(xmin, xmax, 0.01)
plt.plot(xplot, -0.832622948392766*xplot + 0.44378197841356, 'y-', linewidth=2, label='Case-1 Decision Boundary')
plt.plot(xplot, -0.134864086623903*xplot + 0.494549490884199, 'c-', linewidth=2, label='Case-2 Decision Boundary')
plt.plot(xplot, -0.641052306096743*xplot - 8.91426865585982e-25*np.sqrt(-1.51514382948643e+45*(xplot**2) - 1.95123755461517e+48*xplot + 3.25462873130337e+48) + 2.09052329406855, 'b-', linewidth=2, label='Case-3 Decision Boundary')

plt.ylim(-0.25,1.4)
plt.xlim(-1.27,0.85)
plt.xlabel('X-feature', fontsize=20)
plt.ylabel('Y-feature', fontsize=20)
plt.title('Training and Testing Data comparison', fontsize=25)
plt.legend(loc='upper right')
plt.legend(loc='upper right')
plt.show()
```

## Overall Accuracy Calculation

```python
testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.

accu_case1=testbool+boolcase1
accu_case2=testbool+boolcase2
accu_case3=testbool+boolcase3

unique1, counts1 = np.unique(accu_case1, return_counts=True)
case1=dict(zip(unique1, counts1))

unique2, counts2 = np.unique(accu_case2, return_counts=True)
case2=dict(zip(unique2, counts2))

unique3, counts3 = np.unique(accu_case3, return_counts=True)
case3=dict(zip(unique3, counts3))

# print('Case 1 = ' + str(case1) + '\n' + 'Case 2 = ' + str(case2) + '\n'+ 'Case 3 = ' + str(case3))

accuracy_case1= (counts1[0]+counts1[2])/(counts1[0]+counts1[1]+counts1[2])
accuracy_case2= (counts2[0]+counts2[2])/(counts2[0]+counts2[1]+counts2[2])
accuracy_case3= (counts3[0]+counts3[2])/(counts3[0]+counts3[1]+counts3[2])

print('Case 1: Overall Accuracy = ' + str(accuracy_case1) + '\n' + 'Case 2: Overall Accuracy = ' + str(accuracy_case2) + '\n'+ 'Case 3: Overall Accuracy = ' + str(accuracy_case3))
```

    Case 1: Overall Accuracy = 0.713
    Case 2: Overall Accuracy = 0.892
    Case 3: Overall Accuracy = 0.898

## Classwise Accuracy Calculation

```python
testbool0=np.array(test['yc'])[:500]     # This is used to calculate the accuracy by comparing this with the different case discriminants.
testbool1=np.array(test['yc'])[500:1000] # This is used to calculate the accuracy by comparing this with the different case discriminants.

boolcase10=boolcase1[0:500]
boolcase11=boolcase1[500:1000]

accu_case10=testbool0+boolcase10
accu_case11=testbool1+boolcase11

boolcase20=boolcase2[0:500]
boolcase21=boolcase2[500:1000]

accu_case20=testbool0+boolcase20
accu_case21=testbool1+boolcase21

boolcase30=boolcase3[0:500]
boolcase31=boolcase3[500:1000]

accu_case30=testbool0+boolcase30
accu_case31=testbool1+boolcase31

unique10, counts10 = np.unique(accu_case10, return_counts=True)
case10=dict(zip(unique10, counts10))

unique11, counts11 = np.unique(accu_case11, return_counts=True)
case11=dict(zip(unique11, counts11))

unique20, counts20 = np.unique(accu_case20, return_counts=True)
case20=dict(zip(unique20, counts20))

unique21, counts21 = np.unique(accu_case21, return_counts=True)
case21=dict(zip(unique21, counts21))

unique30, counts30 = np.unique(accu_case30, return_counts=True)
case30=dict(zip(unique30, counts30))

unique31, counts31 = np.unique(accu_case31, return_counts=True)
case31=dict(zip(unique31, counts31))

# print('Case 01 = ' + str(case01) + '\n' + 'Case 2 = ' + str(case2) + '\n'+ 'Case 3 = ' + str(case3))

accuracy_case10= counts10[0]/(counts10[0]+counts10[1])
accuracy_case11= counts11[1]/(counts11[0]+counts11[1])

accuracy_case20= (counts20[0])/(counts20[0]+counts20[1])
accuracy_case21= (counts21[1])/(counts21[0]+counts21[1])

accuracy_case30= (counts30[0])/(counts30[0]+counts30[1])
accuracy_case31= (counts31[1])/(counts31[0]+counts31[1])

print('Total accuracy of case 1 = '+ str(accuracy_case1)+'\n')
print('Class-wise accuracy of case 1\n  '+ 'Class-0 accuracy of case 1 = '+ str(accuracy_case10)+ '\n  '  + 'Class-1 accuracy of case 1 = '+ str(accuracy_case11)+'\n')

print('Total accuracy of case 2 = '+ str(accuracy_case2)+'\n')
print('Class-wise accuracy of case 2\n  '+ 'Class-0 accuracy of case 2 = '+ str(accuracy_case20)+ '\n  '  + 'Class-1 accuracy of case 2 = '+ str(accuracy_case21)+'\n')

print('Total accuracy of case 3 = '+ str(accuracy_case3)+'\n')
print('Class-wise accuracy of case 3\n  '+ 'Class-0 accuracy of case 3 = '+ str(accuracy_case30)+ '\n  '  + 'Class-1 accuracy of case 3 = '+ str(accuracy_case31))
```

    Total accuracy of case 1 = 0.713

    Class-wise accuracy of case 1
      Class-0 accuracy of case 1 = 0.68
      Class-1 accuracy of case 1 = 0.746

    Total accuracy of case 2 = 0.892

    Class-wise accuracy of case 2
      Class-0 accuracy of case 2 = 0.9
      Class-1 accuracy of case 2 = 0.884

    Total accuracy of case 3 = 0.898

    Class-wise accuracy of case 3
      Class-0 accuracy of case 3 = 0.908
      Class-1 accuracy of case 3 = 0.888

# Performance Analysis of Bayesian Classifier for Three Cases
## Case-1

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

sig=(sigma0[0,0]+sigma1[1,1])/2.0
sigma=sig*(np.identity(2, dtype=float))
sigmadet=np.linalg.det(sigma)
sigmainv=np.linalg.inv(sigma)

#------------Categorizing based on the Discriminant Function-----------

def g0(data, Prior):
    k,l=data.shape  # k = number of training samples, l = dimension (# of features)
    g0=np.zeros(k)
    for i in range(k):
        g0[i]=-0.5*(data[i]-mu0).reshape(1,l).dot(sigmainv).dot((data[i]-mu0).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g0

def g1(data, Prior):
    k,l=data.shape
    g1=np.zeros(k)
    for i in range(k):
        g1[i]=-0.5*(data[i]-mu1).reshape(1,l).dot(sigmainv).dot((data[i]-mu1).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g1

#------------------------------------Accuracy as a function of prior values------------------------------------------------
perfo_case1=[]
pri=np.arange(0.1, 1., 0.05)
for prior in pri:
    boolcase1=g0(ztest,prior)>=g1(ztest,1.0-prior)
    boolcase1=np.array([0 if i else 1 for i in boolcase1])  # Converting the True to zero and False to 1.

#-----------------------------------Accuracy Calculation for Case-1------------------------------------------------------
    testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.
    accu_case1=testbool+boolcase1
    unique1, counts1 = np.unique(accu_case1, return_counts=True)
    case1=dict(zip(unique1, counts1))
    accuracy_case1= (counts1[0]+counts1[2])/(counts1[0]+counts1[1]+counts1[2])
    perfo_case1.append(accuracy_case1)

plt.figure(1)
plt.figure(figsize=(7,5))
plt.plot(pri, perfo_case1,'g.-', label='Accuracy')
plt.plot(pri[np.argmax(perfo_case1)], np.max(perfo_case1),'r*', label='Maximum Accuracy', markersize=12)
plt.ylim(0,1.)
plt.xlim(0,1.)
plt.grid(True)
plt.xlabel('Prior Probability for Class-0', fontsize=15)
plt.ylabel('Accuracy of Classification', fontsize=15)
plt.title('Performance Analysis \nfor Case-1 Classifier', fontsize=18)
plt.legend(loc='upper right')
plt.show()
```

## Case-2

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

sigma=(sigma0+sigma1)/2.
sigmadet=np.linalg.det(sigma)
sigmainv=np.linalg.inv(sigma)

#Categorizing based on the Discriminant Function

def g0(data, Prior):
    k,l=data.shape
    g0=np.zeros(k)
    for i in range(k):
        g0[i]=-0.5*(data[i]-mu0).reshape(1,l).dot(sigmainv).dot((data[i]-mu0).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g0

def g1(data, Prior):
    k,l=data.shape
    g1=np.zeros(k)
    for i in range(k):
        g1[i]=-0.5*(data[i]-mu1).reshape(1,l).dot(sigmainv).dot((data[i]-mu1).reshape(l,1))-0.5*np.log(sigmadet)+np.log(Prior)
    return g1

#------------------------------------Accuracy as a function of prior values------------------------------------------------
perfo_case2=[]
pri=np.arange(0.1, 1., 0.05)
for prior in pri:
    boolcase2=g0(ztest, prior) >= g1(ztest, 1.0-prior)
    boolcase2=np.array([0 if i else 1 for i in boolcase2])  # Converting the True to zero and False to 1.

#-----------------------------------Accuracy Calculation for Case-3------------------------------------------------------
    testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.
    accu_case2=testbool+boolcase2
    unique2, counts2 = np.unique(accu_case2, return_counts=True)
    case2=dict(zip(unique2, counts2))
    accuracy_case2= (counts2[0]+counts2[2])/(counts2[0]+counts2[1]+counts2[2])
    perfo_case2.append(accuracy_case2)

plt.figure(1)
plt.figure(figsize=(7,5))
plt.plot(pri, perfo_case2,'g.-', label='Accuracy')
plt.plot(pri[np.argmax(perfo_case2)], np.max(perfo_case2),'r*', label='Maximum Accuracy', markersize=12)
plt.ylim(0.,1.05)
plt.xlim(0,1.)
plt.grid(True)
plt.xlabel('Prior Probability for Class-0', fontsize=15)
plt.ylabel('Accuracy of Classification', fontsize=15)
plt.title('Performance Analysis \nfor Case-2 Classifier', fontsize=18)
plt.legend(loc='upper right')
plt.show()
```

## Case-3

```python
def EstGauss(data):
    mu=np.mean(data, axis=0)
    m,n=data.shape
    sigma=np.zeros((n,n))

    for i in range(m):
        sigma = sigma + (data[i] - mu).reshape(n,1).dot((data[i] - mu).reshape(1,n))

    sigma=sigma*(1.0/m)

    return mu, sigma

mu0,sigma0=EstGauss(z0)
mu1,sigma1=EstGauss(z1)

sigma0det=np.linalg.det(sigma0)
sigma1det=np.linalg.det(sigma1)
sigma0inv=np.linalg.inv(sigma0)
sigma1inv=np.linalg.inv(sigma1)

#Categorizing based on the Discriminant Function

def g0(data, Prior):
    k,l=data.shape
    g0=np.zeros(k)
    for i in range(k):
        g0[i]=-0.5*(data[i]-mu0).reshape(1,l).dot(sigma0inv).dot((data[i]-mu0).reshape(l,1))-0.5*np.log(sigma0det)+np.log(Prior)
    return g0

def g1(data, Prior):
    k,l=data.shape
    g1=np.zeros(k)
    for i in range(k):
        g1[i]=-0.5*(data[i]-mu1).reshape(1,l).dot(sigma1inv).dot((data[i]-mu1).reshape(l,1))-0.5*np.log(sigma1det)+np.log(Prior)
    return g1

#------------------------------------Accuracy as a function of prior values------------------------------------------------
perfo_case3=[]
pri=np.arange(0.1, 1., 0.05)
for prior in pri:
    boolcase3=g0(ztest, prior)>=g1(ztest, 1.0-prior) # True if the sample belongs to class-0, False otherwise
    boolcase3=np.array([0 if i else 1 for i in boolcase3])  # Converting the True to zero and False to 1.

#-----------------------------------Accuracy Calculation for Case-3------------------------------------------------------
    testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.
    accu_case3=testbool+boolcase3
    unique3, counts3 = np.unique(accu_case3, return_counts=True)
    case3=dict(zip(unique3, counts3))
    accuracy_case3= (counts3[0]+counts3[2])/(counts3[0]+counts3[1]+counts3[2])
    perfo_case3.append(accuracy_case3)

plt.figure(1)
plt.figure(figsize=(7,5))
plt.plot(pri, perfo_case3,'g.-', label='Accuracy')
plt.plot(pri[np.argmax(perfo_case3)], np.max(perfo_case3),'r*', label='Maximum Accuracy', markersize=12)
plt.ylim(0.,1.05)
plt.xlim(0,1.)
plt.grid(True)
plt.xlabel('Prior Probability for Class-0', fontsize=15)
plt.ylabel('Accuracy of Classification', fontsize=15)
plt.title('Performance Analysis \nfor Case-3 Classifier', fontsize=18)
plt.legend(loc='upper right')
plt.show()
```

## Assuming a Two-Modal Gaussian to estimate parameters

```python
mu01,sigma01= np.array([-0.75,  0.2]), np.array([[0.25, 0.], [0., 0.3]])
mu02,sigma02= np.array([0.3,  0.3]), np.array([[0.1, 0.], [0., 0.15]])
mu11,sigma11= np.array([-0.3,  0.78]), np.array([[0.1, 0.], [0., 0.15]])
mu12,sigma12= np.array([0.4,  0.65]), np.array([[0.25, 0.], [0., 0.3]])

sigma01det=np.linalg.det(sigma01)
sigma11det=np.linalg.det(sigma11)
sigma01inv=np.linalg.inv(sigma01)
sigma11inv=np.linalg.inv(sigma11)

sigma02det=np.linalg.det(sigma02)
sigma12det=np.linalg.det(sigma12)
sigma02inv=np.linalg.inv(sigma02)
sigma12inv=np.linalg.inv(sigma12)

#Categorizing based on the Discriminant Function

def gmodal_0(data, Prior):
    k,l=data.shape
    gmodal_0=np.zeros(k)
    for i in range(k):
        modal_01=(1/(2.0*np.pi))*(1/sigma01det)**0.5*np.exp(-0.5*(data[i]-mu01).reshape(1,2).dot(sigma01inv).dot((data[i]-mu01).reshape(2,1)))
        modal_02=(1/(2.0*np.pi))*(1/sigma02det)**0.5*np.exp(-0.5*(data[i]-mu02).reshape(1,2).dot(sigma02inv).dot((data[i]-mu02).reshape(2,1)))
        twomodal_0=modal_01+modal_02
        gmodal_0[i]=np.log(twomodal_0)+np.log(Prior)
    return gmodal_0

def gmodal_1(data, Prior):
    k,l=data.shape
    gmodal_1=np.zeros(k)
    for i in range(k):
        modal_11=(1/(2.0*np.pi))*(1/sigma11det)**0.5*np.exp(-0.5*(data[i]-mu11).reshape(1,2).dot(sigma11inv).dot((data[i]-mu11).reshape(2,1)))
        modal_12=(1/(2.0*np.pi))*(1/sigma12det)**0.5*np.exp(-0.5*(data[i]-mu12).reshape(1,2).dot(sigma12inv).dot((data[i]-mu12).reshape(2,1)))
        twomodal_1=modal_11+modal_12
        gmodal_1[i]=np.log(twomodal_1)+np.log(Prior)
    return gmodal_1

bimodalboolcase=gmodal_0(ztest,0.5)>=gmodal_1(ztest,0.5) # True if the sample belongs to class-0, False otherwise
bimodalboolcase=np.array([0 if i else 1 for i in bimodalboolcase])  # Converting the True to zero and False to 1.

#--------------------------------------------Accuracy Testing-------------------------------------------------------------------
testbool=np.array(test['yc']) # This is used to calculate the accuracy by comparing this with the different case discriminants.

accu_bimodal=testbool+bimodalboolcase
uniqueb, countsb = np.unique(accu_bimodal, return_counts=True)
bimodal=dict(zip(uniqueb, countsb))

# print('Bimodal Case = ' + str(bimodal))
accuracy_bimodal= (countsb[0]+countsb[2])/(countsb[0]+countsb[1]+countsb[2])

print('Bimodal Case: Overall Accuracy = ' + str(accuracy_bimodal))

```

    Bimodal Case: Overall Accuracy = 0.832


# Aufgabe

Eine Wasserleitung CU 35$\times$1.2 soll einen Volumenstrom von $\dot V = 4.0\,\frac{\mathrm{m}^3}{\mathrm{h}}$ transportieren.

Zulässig ist eine Geschwindigkeit von $v = 1.0\,\frac{\mathrm{m}}{\mathrm{s}}$, bei höheren Geschwindigkeiten können Strömungsgeräusche auftreten.

1. Berechnen Sie die Geschwindigkeit, die sich in dem Rohr einstellt.
2. Welcher Durchmesser ist mindestens zu wählen, um die zugelassene Geschwindigkeit nicht zu überschreiten.

Rechnen Sie schrittweise und kommentieren Sie Ihre Lösung ausführlich.

Die Zusammenhänge zwischen Volumenstrom, Durchmesser und Geschwindigkeit sind durch die folgenden Formeln gegeben:

\begin{align}
  \dot V &= A\,v,
  &
  A &= \dfrac{\pi\,d^2}{4}
  \\
  v &= \dfrac{\dot V}{A}
  \\
  A &= \dfrac{\dot V}{v},
  &
  d &= \sqrt{\dfrac{4\,A}{\pi}}
\end{align}

Das Modul `math` sollten Sie importieren. Das ist in der folgenden Zelle voreingestellt.

Sie finden die Zahl $\pi$ als `math.pi`.

Die Wurzel einer Zahl `x` können Sie als `math.sqrt(x)` berechnen.

```python
import math
```

```python
# Ihre Lösung beginnt hier
```


---
# Section 2.5: Roundoff Errors and Backward Stability

---

## Base-10 floating point numbers

Consider the following 6 digit numbers written using base-10 floating-point notation:

$$
\begin{align}
1.23456 \times 10^3    &= 1234.56 \\
1.23456 \times 10^{-2} &= 0.0123456 \\
\end{align}
$$

We can use this floating-point notation to write down both large and small numbers very compactly:

$$
1.23 \times 10^{50}, \qquad -1.234 \times 10^{-100}.
$$

The first is a number with 3 significant digits, and the second has 4 significant digits.

---

## Roundoff error

Base-10 with 4 significant digits:

$$
(1.112 \times 10^1) \times (1.112 \times 10^2) = \fbox{1.236}544 \times 10^3 \xrightarrow{\text{roundoff}} 1.237 \times 10^3
$$

Therefore, we have made the following error:

$$
\delta x = 1.237 \times 10^3 - 1.236544 \times 10^3 = 0.456
$$

The relative error is:

$$
\frac{\delta x}{x} = \frac{.456}{1.236544 \times 10^3} \approx 4 \times 10^{-4} = 0.04 \%
$$

---

## Machine epsilon `eps(Float64)`

`1.0 + eps(Float64)` is the first `Float64` that is larger than `1.0`:

$$\mathtt{eps(Float64)} = 2^{-52} \approx 2.2 \times 10^{-16}$$

---

## The unit roundoff $u$

$u = $ `eps(Float64)/2.0` is the largest possible relative error due to roundoff:

$$u = 2^{-53} \approx 1.1 \times 10^{-16}$$

It is for this reason that `Float64` accuracy is limited to **16 decimal places**.

```julia
u = eps()/2.0
```

---

## Floating-point error bounds

Let $\varepsilon =$ relative error computing $C$ using floats.

Then

$$
\frac{\mathrm{fl}(C) - C}{C} = \varepsilon
\qquad \iff \qquad
\mathrm{fl}(C) = C(1 + \varepsilon).
$$

IEEE floating-point standard guarantees:

$$
\begin{split}
\mathrm{fl}(x \pm y) &= (x \pm y)(1 + \varepsilon_1) \\
\mathrm{fl}(x \times y) &= (x \times y)(1 + \varepsilon_2) \\
\mathrm{fl}(x \div y) &= (x \div y)(1 + \varepsilon_3) \\
\end{split}
$$

where $|\varepsilon_i| \leq u$, for $i = 1,2,3$, where $u$ is the unit roundoff.

---

## Roundoff error accumulation

Suppose we already have errors in $x$ and $y$:

$$
\hat{x} = x(1 + \varepsilon_1), \qquad |\varepsilon_1| \ll 1
$$

$$
\hat{y} = y(1 + \varepsilon_2), \qquad |\varepsilon_2| \ll 1
$$

**Multiplication:**

$$
\begin{split}
\hat{x}\hat{y}
&= x(1 + \varepsilon_1) y(1 + \varepsilon_2)\\
&= xy (1 + \varepsilon_1 + \varepsilon_2 + \varepsilon_1 \varepsilon_2)\\
&= xy (1 + \hat{\varepsilon}),\\
\end{split}
$$

where $\hat{\varepsilon} = \varepsilon_1 + \varepsilon_2 + \varepsilon_1 \varepsilon_2$, so $|\hat{\varepsilon}| \ll 1$.

Now let's compare the true value of $xy$ with the computed value of $\mathrm{fl}(\hat{x}\hat{y})$.

We have

$$
\begin{split}
\mathrm{fl}(\hat{x}\hat{y})
&= \hat{x}\hat{y}(1 + \varepsilon_3) \qquad \qquad (|\varepsilon_3| \leq u) \\
&= xy (1 + \hat{\varepsilon})(1 + \varepsilon_3) \\
&= xy (1 + \hat\varepsilon + \varepsilon_3 + \hat\varepsilon \varepsilon_3)\\
&= xy (1 + \varepsilon),\\
\end{split}
$$

where $\varepsilon = \hat\varepsilon + \varepsilon_3 + \hat\varepsilon \varepsilon_3$, so $|\varepsilon| \ll 1$.

**Division:**

$$
\frac{\hat{x}}{\hat{y}} = \frac{x(1 + \varepsilon_1)}{y(1 + \varepsilon_2)}
\approx \frac{x}{y}(1 + \varepsilon_1)(1 - \varepsilon_2)
= \frac{x}{y} (1 + \varepsilon_1 - \varepsilon_2 - \varepsilon_1 \varepsilon_2)\\
= \frac{x}{y} (1 + \hat{\varepsilon}),\\
$$

where $\hat{\varepsilon} = \varepsilon_1 - \varepsilon_2 - \varepsilon_1 \varepsilon_2$, so $|\hat{\varepsilon}| \ll 1$.

Therefore, using a similar argument as above, we have

$$
\mathrm{fl}\left(\frac{\hat{x}}{\hat{y}}\right) = \frac{x}{y} (1 + \varepsilon), \qquad \text{where $|\varepsilon| \ll 1$}.
$$

**Addition and subtraction:**

$$
\begin{split}
\hat{x} + \hat{y}
&= x(1 + \varepsilon_1) + y(1 + \varepsilon_2) \\
&= (x + y) + x \varepsilon_1 + y \varepsilon_2 \\
&= (x + y)\left(1 + \frac{x}{x+y} \varepsilon_1 + \frac{y}{x+y} \varepsilon_2 \right) \\
&= (x + y)\left(1 + \hat\varepsilon \right), \\
\end{split}
$$

where $\hat\varepsilon = \frac{x}{x+y} \varepsilon_1 + \frac{y}{x+y} \varepsilon_2$.

If $x + y$ is very small compared to $x$ or $y$, then $\hat\varepsilon$ could be very large.

---

### Example:

Suppose

$$
\begin{align}
x &= 1.23450, & \hat{x} &= 1.23451, \\
y &= -1.23460, & \hat{y} &= -1.23459. \\
\end{align}
$$

Then:

$$
\hat{x} = x(1 + \varepsilon_1), \qquad \text{where $|\varepsilon_1| \approx 8 \times 10^{-6}$};
$$

$$
\hat{y} = y(1 + \varepsilon_2), \qquad \text{where $|\varepsilon_2| \approx 8 \times 10^{-6}$}.
$$

```julia
x = 1.23450
xhat = 1.23451
ɛ₁ = (x - xhat)/x
abs(ɛ₁)
```

```julia
y = -1.23460
yhat = -1.23459
ɛ₂ = (y - yhat)/y
abs(ɛ₂)
```

```julia
xhat + yhat
```

```julia
x + y
```

```julia
ɛ = ((x + y) - (xhat + yhat))/(x + y)
```

Therefore,

$$
\hat{x} + \hat{y} = (x+y)(1 + \varepsilon), \qquad \text{where $|\varepsilon| \approx 2 \times 10^{-1}$}!!!
$$

This is called **catastrophic cancellation** and can lead to a sudden loss of accuracy in a calculation.

---

## Backward error analysis

We say that a computation $C(x_1,\ldots,x_n)$ is **backwards stable** if

$$
\mathrm{fl}(C(x_1,\ldots,x_n)) = C(\bar{x_1},\ldots,\bar{x_n}),
$$

where the error in $\bar{x_1},\ldots,\bar{x_n}$ is a **small multiple** of the unit roundoff $u$.

---

## Example:

Suppose the computation $C(A, b)$ returns the solution $x$ to $Ax = b$ and

$$
\hat{x} = \mathrm{fl}(C(A, b)) = C(\hat{A}, \hat{b}).
$$

Then $\hat{A} \hat{x} = \hat{b}$.

If $C(A, b)$ is **backwards stable**, then $(A + \delta A)\hat{x} = b + \delta b$, where

$$
\frac{\lVert \delta A \rVert}{\lVert A \rVert} \quad \text{and} \quad
\frac{\lVert \delta b \rVert}{\lVert b \rVert} \quad \text{are small multiples of $u$}.
$$

If $A$ is also **well-conditioned**, then

$$
\frac{\lVert \delta x \rVert}{\lVert x \rVert}
\leq
\kappa(A) \left( \frac{\lVert \delta A \rVert}{\lVert A \rVert} + \frac{\lVert \delta b \rVert}{\lVert b \rVert} \right)
$$

implies that $\frac{\lVert \delta x \rVert}{\lVert x \rVert}$ is also a small multiple of $u$, so $\hat{x}$ is an **accurate approximation** of the true solution $x$.

---

## Residual test

After computing $\hat{x} = \mathrm{fl}(C(A, b))$, we can check that the computation is **backward stable** by finding the **residual**,

$$
\hat{r} = b - A \hat{x}.
$$

Then $A \hat{x} = b + \delta b$, where $\delta b = -\hat{r}$.

If $\frac{\lVert \delta b \rVert}{\lVert b \rVert}$ is a small multiple of $u$, then computation of $\hat{x}$ was **backward stable**.

---

## Summary

To verify accuracy of $\hat{x}$, we need to check **two things**:

1. The computation $\hat{x} = \mathrm{fl}(C(A, b))$ is **backward stable**.
2. $A$ is **well-conditioned**.

---

Section 2.5 - Roundoff Errors and Backward Stability.ipynb