# Exploring infectious disease dynamics with SIR models

## Numerically solving the SIR equations

As we saw in lecture, SIR-type models offer a powerful and general way of modeling infectious disease dynamics. Let's take another look at the differential equations for the standard SIR model:

\begin{eqnarray}
	\frac{dS}{dt} & = & -\beta S I  \\
	\frac{dI}{dt} & = & \beta S I - \gamma I \\
	\frac{dR}{dt} & = & \gamma I
\end{eqnarray}

Unfortunately, even though they look simple, we cannot analytically solve these equations for $S(t)$, $I(t)$ or $R(t)$ at a given time $t$. The incidence term $\beta S I$ makes this a ***nonlinear system*** of equations, which in general do not have simple analytical solutions. So intead, we need to ***numerically integrate (solve)*** these differential equations.

To integrate the equations, we need to consider the change in each variable over a small time step $\Delta t$. This turns out to be quite easy since the differential equations provide the derivative of each variable with respect time, or in other words, that variable's rate of change. To get the change in a variable, we just need to multiply the rate of change by $\Delta t$. For example, we can integrate the differential equation for $I$ from time $t$ to time $t + \Delta t$ as follows:

%%latex
$$
    I(t + \Delta t) \approx I(t) + \frac{dI}{dt} \times \Delta t
$$

The problem with this basic scheme is that the amount of error we introduce will depend heavily on $\Delta t$. Modern numerical integration packages therefore use much more sophisticated integration schemes that can adaptively choose $\Delta t$ based on the rates of change. We will use [SciPy's integrate](https://docs.scipy.org/doc/scipy/reference/integrate.html) package, which has very efficient methods for solving ODEs.

### Coding the SIR model in Python

We will implement the SIR model in Python. First we need to import some standard Python packages such as NumPy, SciPy and MatPlotLib. Make sure you hit ***Shift + Return*** in the following cell so that the lines of Python code are run through the interpreter.

```python
import numpy as np
import scipy.integrate as spi
import matplotlib.pyplot as plt
%matplotlib inline
```

Next, we'll define the parameters of our model, including the transmission rate $\beta$, the recovery rate $\gamma$ and the overall host population size $N$:

```python
beta = 0.3
gamma = 0.1
N = 1000.0
```

Again make sure you hit ***Shift + Return*** after each block of code so that the commands are evaluated. We will also create a grid of time points at which we will solve for each variable using NumPy's *linspace* function:

```python
# A grid of time points (in days)
t = np.linspace(0, 100, 21)
print(t)
```

    [  0.   5.  10.  15.  20.  25.  30.  35.  40.  45.  50.  55.  60.  65.
      70.  75.  80.  85.  90.  95. 100.]

This creates a time point at zero and 20 evenly spaced time points between 0 and 100. Now we will set the initial conditions for each of the state variables, which we'll treat as frequencies. We will store all of these in one variable called *y_init*:

```python
# Initial conditions"
S_init = 999.0 / N
I_init = 1.0 / N
R_init = 0 / N
y_init = S_init, I_init, R_init
print("S = " + f'{S_init:.3f}; ' + "I = " + f'{I_init:.3f}; ' + "R = " + f'{R_init:.3f}')
```

    S = 0.999; I = 0.001; R = 0.000

In order to numerically solve the ODE's in SciPy, we must first define a function that will compute and return the rate of change (i.e. the derivatives) for each variable. We do this in Python by using the keyword *def* to define the function and its inputs and the keyword *return* to set which variables are returned:

```python
def deriv(y, t, N, beta, gamma):
    S, I, R = y
    dS = -beta * S * I
    dI = beta * S * I - gamma * I
    dR = gamma * I
    return dS, dI, dR
```

With all of that, we can now use SciPy's *odeint* function to numerically integrate/solve our ODEs. Note the order of the input arguments here: first we list the function we are integrating *deriv*, then the initial conditions *y_init*, then the times at which we are solving *t*, and then a final tuple called *args* which supplies the other parameters in our model:

```python
# Integrate the SIR equations over the time grid t.
ret = spi.odeint(deriv, y_init, t, args=(N, beta, gamma))
```

Now we can plot the SIR dynamics!

```python
# Set up plot
import seaborn as sns # We will use seaborn to make the plots look slightly nicer
sns.set()
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
S, I, R = ret.T
ax.plot(t, S, 'o-', mew=1, ms=8, mec='w', label='S')
ax.plot(t, I, 'o-', mew=1, ms=8, mec='w', label='I')
ax.set_xlabel('Time')
ax.set_ylabel('Frequency')
ax.legend()
```

What happens if we vary the transmission rate $\beta$ and therefore also $R_{0}$? We can use a *for* loop to iterate through different values of $\beta$, recompute $R_{0}$ and then plot the  epidemic curve of prevalence over time:

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
betas = np.linspace(0.1, 0.5, 5) # lets' try beta values of 0.1, 0.2, 0.3, 0.4 and 0.5
for beta in betas:
    R0 = beta / gamma
    ret = spi.odeint(deriv, y_init, t, args=(N,beta,gamma))
    S, I, R = ret.T
    ax.plot(t, I, 'o-', mew=1, ms=8, mec='w', label=f'R0={R0:.1f}')
ax.set_xlabel('Time')
ax.set_ylabel('Prevalence')
ax.legend()
```

As expected, the epidemic only takes off if $R_{0} > 1$ and we get larger epidemics as we increase $\beta$.

### Adding host demography

As we discussed in lecture, nothing really interesting can happen in the basic SIR model. The epidemic just grows until eventually it burns out due to a depletion of susceptible hosts. Let's add host births and deaths to the model so that we get a replenishment of susceptible hosts. Recall the SIR model with demography:

%%latex
\begin{eqnarray}
    \frac{dS}{dt} & = & \nu -\beta S I - \mu S  \\
    \frac{dI}{dt} & = & \beta S I - (\gamma + \mu) I \\
    \frac{dR}{dt} & = & \gamma I - \mu R
\end{eqnarray}

Here, $\nu$ is the host birth rate and $\mu$ the death rate. We'll set this to reasonable values for humans. We will continue to specify time in days so rates are per day:

```python
beta = 520/365 # per day
gamma = 1/7 # per day - so average infectious period is 1 week
N = 1000.0 # pop size
nu = 1/(70*365) # birth rate per day
mu = nu # death rate per day
```

The host lifespan is 70 years, but since we are working in time units of days, we multply this by 365 days to get the birth rate per day. We'll set $\mu=\nu$ so that the host population remains in demographic equilibrium. We will set our grid of time points over a much longer time scale (20 years) so we can see how the long-term dynamics unfold:

```python
# A grid of time points (in days)
t = np.linspace(0, 365*20, 100)
```

Now we need to set the initial conditions. We will continue work with frequencies for S, I and R. I've set *S_init* to $1/R_{0}$ so that we start close to the equilbrium number of susceptibles.

```python
# Initial conditions"
R0 = beta / (gamma+mu)
S_init = 1. / R0
I_init = 1. / N
R_init = 1. - S_init - I_init
y_init = S_init, I_init, R_init
print("S = " + f'{S_init:.3f}; ' + "I = " + f'{I_init:.3f}; ' + "R = " + f'{R_init:.3f}')
```

    S = 0.100; I = 0.001; R = 0.899

Now I'm going to make you do a little work. I've set up the #deriv# function for the SIR model with host births ***BUT*** I have not added deaths yet. See if you can modify the equations for the rate of change of each variable to include deaths:

```python
def deriv(y, t, N, beta, gamma):
    S, I, R = y
    dS = nu - beta * S * I # Add deaths
    dI = beta * S * I - gamma * I # Add deaths
    dR = gamma * I # Add deaths
    return dS, dI, dR
```

As before, we can integrate our system of differential equations using *odeint*:

```python
ret = spi.odeint(deriv, y_init, t, args=(N, beta, gamma))
```

And then plot the prevalence of the disease over a 20 year time span:

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
S, I, R = ret.T
ax.plot(t/365, I, 'o-', mew=1, ms=8, mec='w')
ax.set_xlabel('Time (years)')
ax.set_ylabel('Prevalence')
```

As expected, we get fluctuations with damped oscillations as the system converges to equilbrium. How long do we need to run the dynamics before we hit equlibrium? You can vary the  number of years we run this for by changing the number of years in cell 27 above, then resolve the ODEs in cell 30 and plot again in cell 31.

You may also want to play around with changing the birth rate $\nu$. What happens if we increase or decrease the birth rate? The following code scales $\nu$ by 0.5, 1 and 2x.

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 4))
nu = 1/(70*365) # birth rate per day
nus = [0.5*nu, 1.0*nu, 2*nu] # let's scale nu by 0.5X, 1X and 2X
for nu in nus:
    mu = nu # reset death rate equal to nu
    ret = spi.odeint(deriv, y_init, t, args=(N,beta,gamma))
    S, I, R = ret.T
    ax.plot(t, I, 'o-', mew=1, ms=8, mec='w', label=f'nu={365*nu:.3f}')
ax.set_xlabel('Time')
ax.set_ylabel('Prevalence')
ax.legend()
```

As you can see, changing the birth rate changes the periodicity of the oscillations. Higher birth rates lead to more frequent outbreaks with a smaller amplitude.

### Advanced: Koala chlamydia

Chlamydia is a major sexually transmitted disease in koalas which can lead to population declines through increased mortality and decreased fertility. Here we will consider a simple SI-type model with both male and female koalas. Since koalas are prodiguous herbivores, we will assume koalas compete with one another for food and that koala mortality naturally increases with greater population densities. The differential equations for our koala SI model are:

%%latex
\begin{eqnarray}
    \frac{dS_{f}}{dt} & = & r (S_{f} + (1-\alpha) I_{f}) - r S_{f} \frac{N}{K} - \beta_{mf} \frac{S_{f}}{N} I_{m} \\
    \frac{dS_{m}}{dt} & = & r (S_{f} + (1-\alpha) I_{f}) - r S_{m} \frac{N}{K} - \beta_{fm} \frac{S_{m}}{N} I_{f} \\
    \frac{dI_{f}}{dt} & = & \beta_{mf} \frac{S_{f}}{N} I_{m} - r I_{f} \frac{N}{K} \\
    \frac{dI_{m}}{dt} & = & \beta_{fm} \frac{S_{m}}{N} I_{f} - r I_{m} \frac{N}{K} \\
    N & = & S_{f} + S_{m} + I_{f} + I_{f}
\end{eqnarray}

$S_{f}$ and $S_{m}$ are the number of susceptible female and male koalas. Females reproduce at rate $r$, so the term $r(S_{f} + (1-\alpha) I_{f})$ is the rate at which both susceptible and infected females reproduce. The $\alpha$ parameter captures the reduced fertility of infected females, such that if $\alpha=1$, all infected females are completely infertile. We will assume that all koalas die at a rate $r \frac{N}{K}$, where $N$ is the total population size and $K$ is the carrying capacity, or maximum size, of the population that can be supported given the amount of food resources. Thus, the higher $N$ is relative to $K$, the more density-dependent mortality there is.

Now we need to choose some parameter values for our model:

```python
r = 1.0 # koala reproductive rate
alpha = 0.4 # reduction in fertility due to infection
beta = np.array([[0.0,4.0],[4.6,0.0]]) # Transmission rates from Keeling & Rohani (pg 75)
K = 200 # carrying capacity
```

Notice that $\beta$ is a transmission rate matrix because we now have a structured host population with both males and females. I've set $\beta_{fm} = 4.0$ and $\beta_{mf} = 4.6$ such that male to female transmission is a little higher than female to male.

Next we will set the initial conditions. For now we will not include any infected males or females so that we can explore the host population dynamics in the absence of disease:

```python
t = np.linspace(0, 20, 20) # grid of time points to simulate
Sf = 60
Sm = 80
If = 0
Im = 0
y_init = Sf, Sm, If, Im # initial conditions
```

The following code cell defines the #deriv# function for computeing the derivatives of the differential equation model:

```python
def deriv(y, t, r, beta, alpha, K):
    Sf, Sm, If, Im = y
    N = Sf + Sm + If + Im
    dSf = r*(Sf + (1-alpha) * If) - r*Sf*(N/K) - beta[1,0] * Sf * Im / N
    dSm = r*(Sf + (1-alpha) * If) - r*Sm*(N/K) - beta[0,1] * Sm * If / N
    dIf = beta[1,0] * Sf * Im / N - r*If*(N/K)
    dIm = beta[0,1] * Sm * If / N - r*Im*(N/K)
    return dSf, dSm, dIf, dIm
```

Now we can numerically integrate the ODEs and plot the population dynamics:

```python
ret = spi.odeint(deriv, y_init, t, args=(r,beta,alpha,K)) # Integrate the SIR equations over the time grid, t.
Sf, Sm, If, Im = ret.T
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.plot(t, Sf, 'o-', mew=1, ms=8, mec='w', label="S Females")
ax.plot(t, Sm, 'o-', mew=1, ms=8, mec='w', label="S Males")
ax.plot(t, If, 'o-', mew=1, ms=8, mec='w', label="I Females")
ax.plot(t, Im, 'o-', mew=1, ms=8, mec='w', label="I Males")
ax.set_xlabel('Time')
ax.set_ylabel('Density')
ax.legend()
```

Remember we set $I_{f} = 0$ and $I_{m} = 0$, so we have not yet introduced the disease. In the absence of disease, the population quickly equilibrates to its carrying capacity ($K=200$). Now let's go back and seed the epidemic by including one infected male and female pair:

```python
Sf = 60
Sm = 80
If = 1
Im = 1
y_init = Sf, Sm, If, Im # initial conditions
```

```python
ret = spi.odeint(deriv, y_init, t, args=(r,beta,alpha,K)) # Integrate the SIR equations over the time grid, t.
Sf, Sm, If, Im = ret.T
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
ax.plot(t, Sf, 'o-', mew=1, ms=8, mec='w', label="S Females")
ax.plot(t, Sm, 'o-', mew=1, ms=8, mec='w', label="S Males")
ax.plot(t, If, 'o-', mew=1, ms=8, mec='w', label="I Females")
ax.plot(t, Im, 'o-', mew=1, ms=8, mec='w', label="I Males")
ax.set_xlabel('Time')
ax.set_ylabel('Density')
ax.legend()
```

Now we see that the number of infected males and females quickly rises to an equilbrium value where about half of the population is infected. What happens if we vary $\alpha$, the reduction in fertility due to disease? Here, we will vary $\alpha$ between zero and one and then plot the total prevalence of the disease in both males and females for different $\alpha$ values:

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
alphas = np.linspace(0.0, 1.0, 6) # lets' try beta values of 0.1, 0.2, 0.3, 0.4 and 0.5
t = np.linspace(0, 40, 21)
for alpha in alphas:
    ret = spi.odeint(deriv, y_init, t, args=(r,beta,alpha,K))
    Sf, Sm, If, Im = ret.T
    ax.plot(t, If+Im, 'o-', mew=1, ms=8, mec='w', label=f'alpha={alpha:.2f}')
ax.set_xlabel('Time')
ax.set_ylabel('Total prevalence')
ax.legend()
```

We can see that high values of $\alpha$ where infected females have low fertility eventually causes disease prevalence to decline because high amounts of infertility causes the overall host population size to decline. Thus it would be benificial for the pathogen to reduce its virulence in terms of the infertility it causes.

However, for many infectious diseases there is a tradeoff between maximizing virulence and maximizing transmissibility. Maximizing transmissability generally requires high within host pathogen growth rates and population sizes, which will also tend to increase virulence. We can add such a tradeoff in our model by rescaling the transmission rates in $\beta$ by $\alpha$ such that we get a new transmission rate $\beta^{\ast}$ that linearly covaries with the virulence $\alpha$:

%%latex
\begin{equation}
    \beta^{\ast} = \beta \times \alpha
\end{equation}

Now if we vary $\alpha$, we will also vary the transmission rates in $\beta$:

```python
fig, ax = plt.subplots(1, 1, figsize=(8, 5))
alphas = np.linspace(0.0, 1.0, 6) # lets' try beta values of 0.1, 0.2, 0.3, 0.4 and 0.5
t = np.linspace(0, 40, 21)
for alpha in alphas:
    beta_star = beta * alpha
    ret = spi.odeint(deriv, y_init, t, args=(r,beta_star,alpha,K))
    Sf, Sm, If, Im = ret.T
    ax.plot(t, If+Im, 'o-', mew=1, ms=8, mec='w', label=f'alpha={alpha:.2f}')
ax.set_xlabel('Time')
ax.set_ylabel('Total prevalence')
ax.legend()
```

Now we can clearly see that there is an intermediate value of $\alpha$ about 0.6 that maximizes the long-term or equilibrium prevalence of chlamydia in the koala population. Do you think that the pathogen will evolve reduced virulence to maximize its long term fitness? Or would a less virulent strain of chlamydia always be outcompeted by a strain with greater virulence and therefore transmission potential?

```python

```


## 10. Solve Equations

[](https://www.youtube.com/watch?v=c40z75JnT44&list=PLLBUgWXdTBDg1Qgmwt4jKtVn9BWh5-zgy "Python Data Science")

Equations are at the root of data science. It is what turns data into actionable information by developing mathematical expressions that mimic physical systems. Some math expressions are simple and can be calculated sequentially such as

$x=1 \quad y=x^2+2x-4$

The solution is $x=1$ and $y=1+2-4=-1$. Consider the case where $x$ also depends on $y$.

$x=y \quad y=x^2+2x-4$

There are two solutions that are calculated from the quadratic formula $y=\frac{-b\pm\sqrt{b^2-4ac}}{2a}$.

$0=y^2+(2y-y)-4 \quad y^2+y-4 = 0$ with $a=1$, $b=1$ and $c=-4$.

$y = \frac{-1 \pm \sqrt{17}}{2} = {1.56,-2.56}$

There are two primary ways to solve this problem. The first method is a **numeric solution** where the computer uses trial and error methods to get to a solution. Numeric methods are best when the number of equations is large and there is no analytic solution. The second method is a **symbolic solution** that produces an exact solution.

### Numeric Solution

Large-scale and complex problems require a numeric solution approach such as with `fsolve` or `gekko`. It requires a function that returns the equation error residual. This residual is $f(y)=y^2+y-4$ and is not equal to zero when the value of $y$ is not at the correct solution. An initial guess of `1` or `-2` give a different solution because we are starting close to one or the other.

#### Solution with Scipy fsolve

```python
from scipy.optimize import fsolve
def f(y):
    return y**2+y-4
z = fsolve(f,1); print(z)
z = fsolve(f,-2); print(z)
```

**Solution with Python Gekko**

```python
from gekko import GEKKO
m = GEKKO(remote=False)
y = m.Var(1); m.Equation(y**2+y-4==0)
m.solve(disp=False); print(y.value)
y.value = -2
m.solve(disp=False); print(y.value)
```

### Solve 2 Equations

It is similar when there are two equations instead of one.

$y=x^2+2x-4$

$x=y$

The function returns the error residual for each equation as a list. Two initial guesses are needed. This same method extends to more equations as well. Equation solvers can find solutions to problems with thousands or millions of variables.

**Solution with Scipy fsolve**

```python
from scipy.optimize import fsolve
def f(z):
    x,y = z
    return [x-y,y-x**2-2*x+4]
z = fsolve(f,[1,1]); print(z)
z = fsolve(f,[-2,-2]); print(z)
```

**Solution with Python Gekko**

```python
m = GEKKO(remote=False)
x=m.Var(); y = m.Var(1);
m.Equations([y==x**2+2*x-4, x==y])
m.solve(disp=False)
print(x.value, y.value)

x.value=-2; y.value=-2
m.solve(disp=False)
print(x.value, y.value)
```

### Solve 3 Equations

$x^2+y^2+z^2=1$

$x-2y+3z=0.5$

$x+y+z=0$

Solve the problem with 3 variables and 3 equations.

```python

```

### Symbolic Solution

Small problems may have an analytic solution that can be expressed symbolically. A symbolic math package in Python is `sympy`. The `display` function is also available to print the equations in Jupyter notebooks. It requires the import `from IPython.display import display`.

```python
from IPython.display import display
import sympy as sym
x = sym.Symbol('x')
y = sym.Symbol('y')
ans = sym.nonlinsolve([x-y, y-x**2-2*x+4], [x,y])
display(ans)
```

### Solve 3 Equations Symbolically

$x\,y\,z=0$

$x\,y=0$

$x+5\,y+z$

Solve the problem with 3 variables and 3 equations symbolically. The problem is degenerate (underspecified) so one of the variables will appear in the solution because there are an infinite set.

```python

```

### Linear Equations

Linear equations are also solved in Python but have efficient methods such as `x = np.linalg.solve(A,b)` to solve $A x = b$ equations with matrix $A$ and vectors $x$ and $b$.

$A = \begin{bmatrix}3 & 2\\ 1 & 2 \end{bmatrix} \quad b = \begin{bmatrix}1 \\ 0 \end{bmatrix}$

```python
import numpy as np
A = np.array([[3,2],[1,2]])
b = np.array([1,0])

x = np.linalg.solve(A,b)
print(x)
```

A symbolic solution to this set of linear equations is also available using the `sympy` `linsolve` function. If the problem is linear then `linsolve` is preferred because it is more efficient than `nonlinsolve` but it can solve both.

```python
import sympy as sym
x, y = sym.symbols('x y')
ans = sym.linsolve([3*x + 2*y - 1, x + 2*y], (x, y))
sym.pprint(ans)
```

### Optimization

When there are more variables than equations, the problem is underspecified and can't be solved with an equation solver such as `fsolve` (for linear or nonlinear) or `linalg.solve` (just for linear problems). Additional information is needed to guide the selection of the extra variables. An objective function $J(x)$ is one way to specify the problem so that a unique solution exists. The objective is to minimize $x_1 x_4 \left(x_1 + x_2 + x_3\right) + x_3$. The two equations guide the selection of two variables with inequality $\left(x_1 x_2 x_3 x_4 \ge 25\right)$ and equality $\left(x_1^2 + x_2^2 + x_3^2 + x_4^2 = 40\right)$ constraints. All four variables must be between `1` (lower bound) and `5` (upper bound).

$\quad \min x_1 x_4 \left(x_1 + x_2 + x_3\right) + x_3$

$\quad \mathrm{s.t.} \quad x_1 x_2 x_3 x_4 \ge 25$

$\quad x_1^2 + x_2^2 + x_3^2 + x_4^2 = 40$

$\quad 1\le x_1, x_2, x_3, x_4 \le 5$

with initial guess:

$\quad x_0 = (1,5,5,1)$

Additional information on optimization is given in the [Design Optimization Course](https://apmonitor.com/me575) and in the [Design Optimization Book](https://apmonitor.com/me575/index.php/Main/BookChapters). The first solution method is with `scipy.optimize.minimize`. Solvers in this package work well for moderate sized problems with black box models where an objective function is available through a function call.

```python
import numpy as np
from scipy.optimize import minimize

def objective(x):
    return x[0]*x[3]*(x[0]+x[1]+x[2])+x[2]

def constraint1(x):
    return x[0]*x[1]*x[2]*x[3]-25.0

def constraint2(x):
    sum_eq = 40.0
    for i in range(4):
        sum_eq = sum_eq - x[i]**2
    return sum_eq

# initial guesses
n = 4
x0 = np.zeros(n)
x0[0] = 1.0
x0[1] = 5.0
x0[2] = 5.0
x0[3] = 1.0

# optimize
b = (1.0,5.0)
bnds = (b, b, b, b)
con1 = {'type': 'ineq', 'fun': constraint1}
con2 = {'type': 'eq', 'fun': constraint2}
cons = ([con1,con2])
solution = minimize(objective,x0,method='SLSQP',\
                    bounds=bnds,constraints=cons)
x = solution.x

# show final objective
print('Final Objective: ' + str(objective(x)))

# print solution
print('Solution')
print('x1 = ' + str(x[0]))
print('x2 = ' + str(x[1]))
print('x3 = ' + str(x[2]))
print('x4 = ' + str(x[3]))
```

### Optimization with Gekko

[Python Gekko](https://gekko.readthedocs.io/en/latest/) also solves the problem and uses automatic differentiation and gradient-based solvers such as `APOPT` or `IPOPT` to find a solution. This solution method is better for large-scale problems. [Additional tutorials on Gekko](https://apmonitor.com/wiki/index.php/Main/GekkoPythonOptimization) show how to solve other types of optimization problems.

```python
from gekko import GEKKO
m = GEKKO(remote=False)

# initialize variables
x1,x2,x3,x4 = [m.Var(lb=1, ub=5) for i in range(4)]

# initial values
x1.value = 1
x2.value = 5
x3.value = 5
x4.value = 1

# Equations
m.Equation(x1*x2*x3*x4>=25)
m.Equation(x1**2+x2**2+x3**2+x4**2==40)

# Objective
m.Obj(x1*x4*(x1+x2+x3)+x3)

# Solve
m.solve(disp=False)

# Final objective
print('Final Objective: ' + str(m.options.objfcnval))

# Print solution
print('Solution')
print('x1: ' + str(x1.value))
print('x2: ' + str(x2.value))
print('x3: ' + str(x3.value))
print('x4: ' + str(x4.value))
```

### TCLab Activity

### Data Collection

Turn on heater 1 to 100% and record $T_1$ every 10 seconds for 3 minutes. The data should include a total of 19 data points for each temperature sensor and the recording time, starting at zero. Make a note of the temperature points at 0, 90, and 180 seconds.

```python

```

### Linear Equations

Three points are required to specify a quadratic polynomial of the form $y =a_0 + a_1 \; x + a_2 \; x^2$. Create a quadratic regression of $T_2$ by using only the first, middle, and last data points. Suppose these were the following data points for $T_2$:

| Time (sec) | Temperature (°C)  |
|------|------|
| 0    | 23.0 |
| 90    | 33.0 |
| 180    | 43.0 |

Solve the linear regression as a set of three equations that are derived by plugging in the three data points to the polynomial equation to create three separate equations with $y=T_2$ and $x=time$.

$\quad a_0 + a_1 \; 0 + a_2 \; 0^2 = 23.0$

$\quad a_0 + a_1 \; 90 + a_2 \; 90^2 = 33.0$

$\quad a_0 + a_1 \; 180 + a_2 \; 180^2 = 43.0$

In matrix form, the set of linear equations become:

$\quad \begin{bmatrix}1 & 0 & 0 \\ 1 & 90 & 90^2 \\ 1 & 180 & 180^2 \end{bmatrix}\begin{bmatrix}a_0\\a_1\\a_2\end{bmatrix} = \begin{bmatrix}23.0\\33.0\\43.0\end{bmatrix}$

Solve this set of equations for the quadratic parameters $a_0$, $a_1$, and $a_2$ with the data collected at the beginning of the TCLab activity. Plot the quadratic fit with the data to ensure that the curve goes through the three specified data points.

```python

```

### Nonlinear Equations

Fit the $T_1$ data to a nonlinear correlation using only three data points.

$\quad T_1 = a + b \exp{(c \, time)}$

Three points are required to uniquely specify a model with three parameters. When there are more than the minimum required number of points, a least squares regression is typically performed to minimize the squared error between the measured and predicted values. For this exercise, use only 3 points (first, middle, last) of the $T_1$ data. Suppose these were the following data points for $T_1$:

| Time (sec) | Temperature (°C)  |
|------|------|
| 0    | 22.0 |
| 90    | 42.0 |
| 180    | 52.0 |

Solve for the three parameters from the three equations that exactly intersect the required data points.

$\quad 22.0 = a + b \exp{(c \, 0)}$

$\quad 42.0 = a + b \exp{(c \, 90.3)}$

$\quad 52.0 = a + b \exp{(c \, 180.5)}$

Solve this set of equations for the unknown parameters $a$, $b$, and $c$ with the data collected at the beginning of this notebook. Use guess values of $a=100$, $b=-100$, and $c=-0.01$. Plot the nonlinear fit with the data to ensure that the curve goes through the three specified data points. Add appropriate labels to the plot.

```python

```


# Gradient Descent Optimizations

Mini-batch and stochastic gradient descent is widely used in deep learning, where the large number of parameters and  limited memory make the use of more sophisticated optimization methods impractical. Many methods have been proposed to accelerate gradient descent in this context, and here we sketch the ideas behind some of the most popular algorithms.

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
```

## Smoothing with exponentially weighted averages

```python
n = 50
x = np.arange(n) * np.pi
y = np.cos(x) * np.exp(x/100) - 10*np.exp(-0.01*x)
```

### Exponentially weighted average

The exponentially weighted average adds a fraction $\beta$ of the current value to a leaky running sum of past values. Effectively, the contribution from the $t-n$th value is scaled by

$$
\beta^n(1 - \beta)
$$

For example, here are the contributions to the current value after 5 iterations (iteration 5 is the current iteration)

| iteration | contribution |
| --- | --- |
| 1 | $\beta^4(1 - \beta)$ |
| 2 | $\beta^3(1 - \beta)$ |
| 3 | $\beta^2(1 - \beta)$ |
| 4 | $\beta^1(1 - \beta)$ |
| 5 | $(1 - \beta)$ |

Since $\beta \lt 1$, the contribution decreases exponentially with the passage of time. Effectively, this acts as a smoother for a function.

```python
def ewa(y, beta):
    """Exponentially weighted average."""

    n = len(y)
    zs = np.zeros(n)
    z = 0
    for i in range(n):
        z = beta*z + (1 - beta)*y[i]
        zs[i] = z
    return zs
```

### Exponentially weighted average with bias correction

Since the EWA starts from 0, there is an initial bias. This can be corrected by scaling with

$$
\frac{1}{1 - \beta^t}
$$

where $t$ is the iteration number.

```python
def ewabc(y, beta):
    """Exponentially weighted average with bias correction."""

    n = len(y)
    zs = np.zeros(n)
    z = 0
    for i in range(n):
        z = beta*z + (1 - beta)*y[i]
        zc = z/(1 - beta**(i+1))
        zs[i] = zc
    return zs
```

```python
beta = 0.9

plt.plot(x, y, 'o-')
plt.plot(x, ewa(y, beta), c='red', label='EWA')
plt.plot(x, ewabc(y, beta), c='orange', label='EWA with bias correction')
plt.legend()
pass
```

## Momentum in 1D

Momentum comes from physics, where the contribution of the gradient is to the velocity, not the position. Hence we create an accessory variable $v$ and increment it with the gradient. The position is then updated with the velocity in place of the gradient. The analogy is that we can think of the parameter $x$ as a particle in an energy well with potential energy $U = mgh$ where $h$ is given by our objective function $f$. The force generated is a function of the rat of change of potential energy $F \propto \nabla U \propto \nabla f$, and we use $F = ma$ to get that the acceleration $a \propto \nabla f$. Finally, we integrate $a$ over time to get the velocity $v$ and integrate $v$ to get the displacement $x$. Note that we need to damp the velocity otherwise the particle would just oscillate forever.

We use a version of the update that simply treats the velocity as an exponentially weighted average popularized by Andrew Ng in his Coursera course. This is the same as the momentum scheme motivated by physics with some rescaling of constants.

```python
def f(x):
    return x**2
```

```python
def grad(x):
    return 2*x
```

```python
def gd(x, grad, alpha, max_iter=10):
    xs = np.zeros(1 + max_iter)
    xs[0] = x
    for i in range(max_iter):
        x = x - alpha * grad(x)
        xs[i+1] = x
    return xs
```

```python
def gd_momentum(x, grad, alpha, beta=0.9, max_iter=10):
    xs = np.zeros(1 + max_iter)
    xs[0] = x
    v = 0
    for i in range(max_iter):
        v = beta*v + (1-beta)*grad(x)
        vc = v/(1+beta**(i+1))
        x = x - alpha * vc
        xs[i+1] = x
    return xs
```

### Gradient descent with moderate step size

```python
alpha = 0.1
x0 = 1
xs = gd(x0, grad, alpha)
xp = np.linspace(-1.2, 1.2, 100)
plt.plot(xp, f(xp))
plt.plot(xs, f(xs), 'o-', c='red')
for i, (x, y) in enumerate(zip(xs, f(xs)), 1):
    plt.text(x, y+0.2, i,
             bbox=dict(facecolor='yellow', alpha=0.5), fontsize=14)
pass
```

### Gradient descent with large step size

When the step size is too large, gradient descent can oscillate and even diverge.

```python
alpha = 0.95
xs = gd(1, grad, alpha)
xp = np.linspace(-1.2, 1.2, 100)
plt.plot(xp, f(xp))
plt.plot(xs, f(xs), 'o-', c='red')
for i, (x, y) in enumerate(zip(xs, f(xs)), 1):
    plt.text(x*1.2, y, i,
             bbox=dict(facecolor='yellow', alpha=0.5), fontsize=14)
pass
```

### Gradient descent with momentum

Momentum results in cancellation of gradient changes in opposite directions, and hence damps out oscillations while amplifying consistent changes in the same direction. This is perhaps clearer in the 2D example below.

```python
alpha = 0.95
xs = gd_momentum(1, grad, alpha, beta=0.9)
xp = np.linspace(-1.2, 1.2, 100)
plt.plot(xp, f(xp))
plt.plot(xs, f(xs), 'o-', c='red')
for i, (x, y) in enumerate(zip(xs, f(xs)), 1):
    plt.text(x, y+0.2, i,
             bbox=dict(facecolor='yellow', alpha=0.5), fontsize=14)
pass
```

## Momentum and RMSprop in 2D

```python
def f2(x):
    return x[0]**2 + 100*x[1]**2
```

```python
def grad2(x):
    return np.array([2*x[0], 200*x[1]])
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
pass
```

```python
def gd2(x, grad, alpha, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0,:] = x
    for i in range(max_iter):
        x = x - alpha * grad(x)
        xs[i+1,:] = x
    return xs
```

```python
def gd2_momentum(x, grad, alpha, beta=0.9, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0, :] = x
    v = 0
    for i in range(max_iter):
        v = beta*v + (1-beta)*grad(x)
        vc = v/(1+beta**(i+1))
        x = x - alpha * vc
        xs[i+1, :] = x
    return xs
```

### Gradient descent with large step size

We get severe oscillations.

```python
alpha = 0.01
x0 = np.array([-1,-1])
xs = gd2(x0, grad2, alpha, max_iter=75)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Vanilla gradient descent')
pass
```

### Gradient descent with momentum

The damping effect is clear.

```python
alpha = 0.01
x0 = np.array([-1,-1])
xs = gd2_momentum(x0, grad2, alpha, beta=0.9, max_iter=75)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Gradieent descent with momentum')
pass
```

### Gradient descent with RMSprop

RMSprop scales the learning rate in each direction by the square root of the exponentially weighted sum of squared gradients. Near a saddle or any plateau, there are directions where the gradient is very small - RMSporp encourages larger steps in those directions, allowing faster escape.

```python
def gd2_rmsprop(x, grad, alpha, beta=0.9, eps=1e-8, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0, :] = x
    v = 0
    for i in range(max_iter):
        v = beta*v + (1-beta)*grad(x)**2
        x = x - alpha * grad(x) / (eps + np.sqrt(v))
        xs[i+1, :] = x
    return xs
```

```python
alpha = 0.1
x0 = np.array([-1,-1])
xs = gd2_rmsprop(x0, grad2, alpha, beta=0.9, max_iter=10)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Gradient descent with RMSprop')
pass
```

### ADAM

ADAM (Adaptive Moment Estimation) combines the ideas of momentum, RMSprop and bias correction. It is probably the most popular gradient descent method in current deep learning practice.

```python
def gd2_adam(x, grad, alpha, beta1=0.9, beta2=0.999, eps=1e-8, max_iter=10):
    xs = np.zeros((1 + max_iter, x.shape[0]))
    xs[0, :] = x
    m = 0
    v = 0
    for i in range(max_iter):
        m = beta1*m + (1-beta1)*grad(x)
        v = beta2*v + (1-beta2)*grad(x)**2
        mc = m/(1+beta1**(i+1))
        vc = v/(1+beta2**(i+1))
        x = x - alpha * m / (eps + np.sqrt(vc))
        xs[i+1, :] = x
    return xs
```

```python
alpha = 0.1
x0 = np.array([-1,-1])
xs = gd2_adam(x0, grad2, alpha, beta1=0.9, beta2=0.9, max_iter=10)
```

```python
x = np.linspace(-1.2, 1.2, 100)
y = np.linspace(-1.2, 1.2, 100)
X, Y = np.meshgrid(x, y)
levels = [0.1,1,2,4,9, 16, 25, 36, 49, 64, 81, 100]
Z = x**2 + 100*Y**2
c = plt.contour(X, Y, Z, levels)
plt.plot(xs[:, 0], xs[:, 1], 'o-', c='red')
plt.title('Gradient descent with RMSprop')
pass
```

## Implementing a custom optimization routine for `scipy.optimize`

Gradient descent is not one of the methods available in `scipy.optimize`. However we can implement our own version by following the API of the `minimize` function.

```python
import scipy.optimize as opt
import scipy.linalg as la
```

```python
def custmin(fun, x0, args=(), maxfev=None, alpha=0.0002,
        maxiter=100000, tol=1e-10, callback=None, **options):
    """Implements simple gradient descent for the Rosen function."""
    bestx = x0
    bestf = fun(x0)
    funcalls = 1
    niter = 0
    improved = True
    stop = False

    while improved and not stop and niter < maxiter:
        niter += 1
        # the next 2 lines are gradient descent
        step = alpha * rosen_der(bestx)
        bestx = bestx - step

        bestf = fun(bestx)
        funcalls += 1

        if la.norm(step) < tol:
            improved = False
        if callback is not None:
            callback(bestx)
        if maxfev is not None and funcalls >= maxfev:
            stop = True
            break

    return opt.OptimizeResult(fun=bestf, x=bestx, nit=niter,
                              nfev=funcalls, success=(niter > 1))
```

```python
def reporter(p):
    """Reporter function to capture intermediate states of optimization."""
    global ps
    ps.append(p)
```

### Test on Rosenbrock banana function

We will use the [Rosenbrock "banana" function](http://en.wikipedia.org/wiki/Rosenbrock_function) to illustrate unconstrained multivariate optimization. In 2D, this is
$$
f(x, y) = b(y - x^2)^2 + (a - x)^2
$$

The function has a global minimum at (1,1) and the standard expression takes $a = 1$ and $b = 100$.

#### Conditioning of optimization problem

With these values for $a$ and $b$, the problem is ill-conditioned. As we shall see, one of the factors affecting the ease of optimization is the condition number of the curvature (Hessian). When the condition number is high, the gradient may not point in the direction of the minimum, and simple gradient descent methods may be inefficient since they may be forced to take many sharp turns.

For the 2D version, we have

$$
f(x) = 100(y - x^2)^2 + (1 - x)^2
$$

and can calculate the Hessian to be

$$
\begin{bmatrix}
802 & -400 \\
-400 & 200
\end{bmatrix}
$$

```python
H = np.array([
    [802, -400],
    [-400, 200]
])
```

```python
np.linalg.cond(H)
```

```python
U, s, Vt = np.linalg.svd(H)
s[0]/s[1]
```

#### Function to minimize

```python
def rosen(x):
    """Generalized n-dimensional version of the Rosenbrock function"""
    return sum(100*(x[1:]-x[:-1]**2.0)**2.0 +(1-x[:-1])**2.0)
```

```python
def rosen_der(x):
    """Derivative of generalized Rosen function."""
    xm = x[1:-1]
    xm_m1 = x[:-2]
    xm_p1 = x[2:]
    der = np.zeros_like(x)
    der[1:-1] = 200*(xm-xm_m1**2) - 400*(xm_p1 - xm**2)*xm - 2*(1-xm)
    der[0] = -400*x[0]*(x[1]-x[0]**2) - 2*(1-x[0])
    der[-1] = 200*(x[-1]-x[-2]**2)
    return der
```

#### Why is the condition number so large?

```python
x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)
Z = rosen(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
```

```python
# Note: the global minimum is at (1,1) in a tiny contour island
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.text(1, 1, 'x', va='center', ha='center', color='red', fontsize=20)
pass
```

#### Zooming in to the global minimum at (1,1)

```python
x = np.linspace(0, 2, 100)
y = np.linspace(0, 2, 100)
X, Y = np.meshgrid(x, y)
Z = rosen(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
```

```python
plt.contour(X, Y, Z, [rosen(np.array([k, k])) for k in np.linspace(1, 1.5, 10)], cmap='jet')
plt.text(1, 1, 'x', va='center', ha='center', color='red', fontsize=20)
pass
```

#### We will use our custom gradient descent to minimize the banana function

#### Helpful Hint

One of the most common causes of failure of optimization is because the gradient  or Hessian function is specified incorrectly. You can check for this using `check_grad` which compares the analytical gradient with one calculated using finite differences.

```python
from scipy.optimize import check_grad

for x in np.random.uniform(-2,2,(10,2)):
    print(x, check_grad(rosen, rosen_der, x))
```

```python
# Initial starting position
x0 = np.array([4,-4.1])
ps = [x0]
opt.minimize(rosen, x0, method=custmin, callback=reporter)
```

```python
x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)
Z = rosen(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
```

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T))
pass
```

### Comparison with standard algorithms

Note that all these methods take far fewer function iterations and function evaluations to find the minimum compared with vanilla gradient descent.

Many of these are based on estimating the Newton direction. Recall Newton's method for finding roots of a univariate function

$$
x_{K+1} = x_k - \frac{f(x_k)}{f'(x_k)}
$$

When we are looking for a minimum, we are looking for the roots of the *derivative* $f'(x)$, so

$$
x_{K+1} = x_k - \frac{f'(x_k}{f''(x_k)}
$$

Newton's method can also be seen as a Taylor series approximation

$$
f(x+h) = f(x) + h f'(x) + \frac{h^2}{2}f''(x)
$$

At the function minimum, the derivative is 0, so
\begin{align}
\frac{f(x+h) - f(x)}{h} &= f'(x) + \frac{h}{2}f''(x) \\
0 &= f'(x) + \frac{h}{2}f''(x)
\end{align}

and letting $\Delta x = \frac{h}{2}$, we get that the Newton step is

$$
\Delta x = - \frac{f'(x)}{f''(x)}
$$

The multivariate analog replaces $f'$ with the Jacobian and $f''$ with the Hessian, so the Newton step is

$$
\Delta x = -H^{-1}(x) \nabla f(x)
$$

Slightly more rigorously, we can optimize the quadratic multivariate Taylor expansion

$$
f(x + p) = f(x) + p^T\nabla f(x) + \frac{1}{2}p^TH(x)p
$$

Differentiating with respect to the direction vector $p$ and setting to zero, we get

$$
H(x)p = -\nabla f(x)
$$

giving

$$
p = -H(x)^{-1}\nabla f(x)
$$

```python
from scipy.optimize import rosen, rosen_der, rosen_hess
```

#### Nelder-Mead

There are some optimization algorithms not based on the Newton method, but on other heuristic search strategies that do not require any derivatives, only function evaluations. One well-known example is the Nelder-Mead simplex algorithm.

```python
ps = [x0]
opt.minimize(rosen, x0, method='nelder-mead', callback=reporter)
```

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T));
```

#### BFGS

As calculating the Hessian is computationally expensive, sometimes first order methods that only use the first derivatives are preferred. Quasi-Newton methods use functions of the first derivatives to approximate the inverse Hessian. A well know example of the Quasi-Newoton class of algorithjms is BFGS, named after the initials of the creators. As usual, the first derivatives can either be provided via the `jac=` argument or approximated by finite difference methods.

```python
ps = [x0]
opt.minimize(rosen, x0, method='Newton-CG', jac=rosen_der, hess=rosen_hess, callback=reporter)
```

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T))
pass
```

#### Newton-CG

Second order methods solve for $H^{-1}$ and so require calculation of the Hessian (either provided or approximated using finite differences). For efficiency reasons, the Hessian is not directly inverted, but solved for using a variety of methods such as conjugate gradient. An example of a second order method in the `optimize` package is `Newton-GC`.

```python
ps = [x0]
opt.minimize(rosen, x0, method='Newton-CG', jac=rosen_der, hess=rosen_hess, callback=reporter)
```

```python
ps = np.array(ps)
plt.figure(figsize=(12,4))
plt.subplot(121)
plt.contour(X, Y, Z, np.arange(10)**5, cmap='jet')
plt.plot(ps[:, 0], ps[:, 1], '-ro')
plt.subplot(122)
plt.semilogy(range(len(ps)), rosen(ps.T))
pass
```


# An intuitive introduction to the Entropy

Let $X$ be a discrete random variable (RV) taking values from set $\mathcal{X}$ with probability mass function $P(X)$.

*Definition* the entropy $H(X)$ of the discrete random variable $X$ is
\begin{equation}
H(X) = \sum_{x\in\mathcal{X}}P(X)\log \frac{1}{P(X)} = -\sum_{x\in\mathcal{X}}P(X)\log P(X).
\end{equation}

How to make sense out of this definition? We'll, rather informally, argue below that the entrpy of an RV provides a lower bound on the amount of information provided by the RV, which we'll dfine as the average number of bits required to transmit the value the RV has taken.

As a motivating example consider asking your friend for advice. The probabilities of his answers are given in the table below:

| $x$      | $P(x)$ |
|----------|--------|
| OK       | $1/2$  |
| Average  | $1/4$  |
| Bad      | $1/8$  |
| Terrible | $1/8$  |

To transmit the answer of your friend you must introduce an *encoding*, e.g.:

| $x$      | $P(x)$ | Code 1 |
|----------|--------|--------|
| OK       | $1/2$  | 00     |
| Average  | $1/4$  | 01     |
| Bad      | $1/8$  | 10     |
| Terrible | $1/8$  | 11     |

Under this encoding, we spend 2 bits per answer.

However, we could also consider a variable length code, that uses shorter codewords for more frequent symbols:

| $x$      | $P(x)$ | Code 2 |
|----------|--------|--------|
| OK       | $1/2$  | 0      |
| Average  | $1/4$  | 10     |
| Bad      | $1/8$  | 110    |
| Terrible | $1/8$  | 111    |

Under this encoding the average number of bits to encode an answer is:
\begin{equation}
\mathbb{E}[L] = \frac{1}{2} \cdot 1 + \frac{1}{4} \cdot 2 + \frac{1}{8} \cdot 3 + \frac{1}{8} \cdot 3 = \frac{7}{8}
\end{equation}

Thus, the new code is more efficient. Is it the best we can do?

### The code space

We'll now try formalize the coding task, i.e. the assignment of code lengths to possible values of the RV.

Let's first observe an important property of our code: in a variable length coding, no codeword can be the prefix of another one. Otherwise, decoding is not deterministic. Therefore, whenever a value is assigned a symbol of length $L$, $1/2^L$ of the code space is reserved and not available to other codes.

This can be visualised as a code space. Below, we indicate the codes assigned to symbols in the example and  grey-out codes that are not available because the shorter codes are used:

We can observe, that the length 1 code for "OK" uses $1/2$ of the available codes, the langth 2 code for "Average" uses $1/4$ and the two length 3 codes for "Bad" and "Terrible" each use $1/8$ of the code space.

In general, a code of length $L$ uses $1/2^L$ of the code space. Equivalently, assignign a fraction $f$ of the code space to a symbol makes it use a symbol of length $L=\log_2(1/f)$.

Assuming that we assign a fraction of a bit to a symbol, our optmal coding problem can be formulated as partitioning the code space into four regions (one for each value of the RV) such that the average length of the code is minised.

Formally, let $p_1, p_2, p_3, p_4$ be the proibabilities asigned to the 4 symbols and let $f_1, f_2, f_3, f_4$ be the coding space fractoins assigned to them.

We want to:
\begin{align}
\text{minimize } &p_1 \log_2 \frac{1}{f_1} +  p_2 \log_2 \frac{1}{f_2} + p_3 \log_2 \frac{1}{f_3} + p_4 \log_2 \frac{1}{f_4} \\
\text{subject to: } & f_1 + f_2 + f_3 + f_4 = 1
\end{align}

For simplicity, we will solve this problem for the case of only two symbols:
\begin{align}
\text{minimize } &p_1 \log_2 \frac{1}{f_1} +  p_2 \log_2 \frac{1}{f_2} \\
\text{subject to: } & f_1 + f_2 = 1
\end{align}

Notice first that $p_2 = 1-p_1$ and likewise $f_2 = 1-f_1$. Then our minimization objective becomes
\begin{equation}
\text{minimize } C = p_1 \log_2 \frac{1}{f_1} +  (1-p_1) \log_2 \frac{1}{1-f_1}
\end{equation}

To get the minimum over $f_1$ we compute the derivative of the expression with respect to $f_1$ and set it to zero:
\begin{equation}
\frac{\partial C}{\partial f_1} = \frac{p_1}{\log 2}\frac{-1}{f_1} + \frac{1 - p_1}{\log 2}\frac{1}{1 - f_1}
\end{equation}

Multiplying both sides by $\log 2 f_1 (1- f_1)$ we obtain:
\begin{align}
p_1(1-f_1) &= (1-p_1)f_1 \\
p_1 - p_1f_1 & =f_1 - p_1f_1 \\
f_1 &= p_1
\end{align}

Thus the optimal fraction of code space allocated to symbol 1 is $p_1$, the probability assigned to this symbol and the optimal code length is $\log_2(\frac{1}{p_1})$!

We now see, that the entropy
\begin{equation}
H(X) = \sum_{x\in\mathcal{X}}P(X)\log \frac{1}{P(X)}
\end{equation}
is simply the average code length!

### A note about logarithm basis

It is custommary to copute the entropy using natural logarithms, which gives its value in "nats". IF $\log_2$ were used, the entrpy has units of bits and corresponds and lowerbounds the average amount of bits needed to transmit a value of the RV.

# Further Reading
1. Chris Olah "Visual Information theory": https://colah.github.io/posts/2015-09-Visual-Information/
2. JA Thomas ad TM Cover, "Elements of Information Theory", chapter 2


# Rate of Change
Functions are often visualized as a line on a graph, and this line shows how the value returned by the function changes based on changes in the input value.

## Linear Rate of Change

For example, imagine a function that returns the number of meters travelled by a cyclist based on the number of seconds that the cyclist has been cycling.

Here is such a function:

\begin{equation}q(x) = 2x + 1\end{equation}

We can plot the output for this function for a period of 10 seconds like this:

```python
%matplotlib inline

def q(x):
    return 2*x + 1

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.xticks(range(0,11, 1))
plt.yticks(range(0, 22, 1))
plt.grid()

# Plot x against q(x)
plt.plot(x,q(x), color='green')

plt.show()
```

It's clear from the graph that ***q*** is a *linear* function that describes a slope in which distance increases at a constant rate over time. In other words, the cyclist is travelling at a constant speed.

But what speed?

Speed, or more technically, velocity is a measure of change - it measures how the distance travelled changes over time (which is why we typically express it as a unit of distance per a unit of time, like *miles-per-hour* or *meters-per-second*). So we're looking for a way to measure the change in the line created by the function.

The change in values along the line define its *slope*, which we know from a previous lesson is represented like this:

\begin{equation}m = \frac{\Delta{y}}{\Delta{x}} \end{equation}

We can calculate the slope of our function like this:

\begin{equation}m = \frac{q(x)_{2} - q(x)_{1}}{x_{2} - x_{1}} \end{equation}

So we just need two ordered pairs of ***x*** and ***q(x)*** values from our line to apply this equation.

- After 1 second, ***x*** is 1 and ***q***(1) = **3**.
- After 10 seconds, ***x*** is 10 and ***q***(10) = 21.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{21 - 3}{10 - 1} \end{equation}

This is the same as:

\begin{equation}m = \frac{18}{9} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{2}{1} \end{equation}

So our rate of change is 2/1 or put another way, the cyclist is travelling at 2 meters-per-second.

## Average Rate of Change
OK, let's look at another function that calculates distance travelled for a given number of seconds:

\begin{equation}r(x) = x^{2} + x\end{equation}

Let's take a look at that using Python:

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

plt.show()
```

This time, the function is not linear. It's actually a quadratic function, and the line from 0 seconds to 10 seconds shows an exponential increase; in other words, the cyclist is *accelerating*.

Technically, acceleration itself is a measure of change in velocity over time; and velocity, as we've already discussed, is a measure of change in distance over time. So measuring accelleration is pretty complex, and requires *differential calculus*, which we're going to cover shortly. In fact, even just measuring the velocity at a single point in time requires differential calculus; but we can use algebraic methods to calculate an *average* rate of velocity for a given period shown in the graph.

First, we need to define a *secant* line that joins two points in our exponential arc to create a straight slope. For example, a secant line for the entire 10 second time span would join the following two points:

- 0, ***r***(0)
- 10, ***r***(10)

Run the following Python code to visualize this line:

```python
%matplotlib inline

def r(x):
    return (x)**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([0,10])

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.show()
```

Now, because the secant line is straight, we can apply the slope formula we used for a linear function to calculate the average velocity for the 10 second period:

- At 0 seconds, ***x*** is 0 and ***r***(0) = **0**.
- At 10 seconds, ***x*** is 10 and ***r***(10) = 110.

So we can meassure the rate of change like this:

\begin{equation}m = \frac{110 - 0}{10 - 0} \end{equation}

This is the same as:

\begin{equation}m = \frac{110}{10} \end{equation}

Which simplifies to:

\begin{equation}m = \frac{11}{1} \end{equation}

So our rate of change is 11/1 or put another way, the cyclist is travelling at an average velocity of 11 meters-per-second over the 10-second period.

Of course, we can measure the average velocity between any two points on the exponential line. Use the following Python code to show the secant line for the period between 2 and 7 seconds, and calculate the average velocity for that period

```python
%matplotlib inline

def r(x):
    return x**2 + x

# Plot the function
import numpy as np
from matplotlib import pyplot as plt

# Create an array of x values from 0 to 10
x = np.array(range(0, 11))

# Create an array for the secant line
s = np.array([2,7])

# Calculate rate of change
x1 = s[0]
x2 = s[-1]
y1 = r(x1)
y2 = r(x2)
a = (y2 - y1)/(x2 - x1)

# Set up the graph
plt.xlabel('Seconds')
plt.ylabel('Meters')
plt.grid()

# Plot x against r(x)
plt.plot(x,r(x), color='green')

# Plot the secant line
plt.plot(s,r(s), color='magenta')

plt.annotate('Average Velocity =' + str(a) + ' m/s',((x2+x1)/2, (y2+y1)/2))

plt.show()

```

```python

```

Basics of Calculus by Hiren/02-01-Rate of Change.ipynb


## Geometric Distribution

Suppose we have a biased coin with $\mathbb{P}(H) = p$ and we toss it repeatedly, then the sample space will look like

$$\Omega = \{H, TH, TTH, \ldots \}$$
Then
$$\mathbb{P}(TH) = \mathbb{P}(T)\cdot \mathbb{P}(H) = (1-p)p$$

Let $X$ is the random variable defined as

$$X = \#\text{ tosses untill first heads }$$
Then
$$\begin{align}\mathbb{P}(X=x) &= \mathbb{P}(x-1 \text{ tails followed by heads}) \\
&= (1-p)^{x-1}p \end{align}
$$
This is probability mass function of random variable $X$. If we normalize we will get the probability distribution.

### Practice Problem: The Geometric Distribution

Let $X \sim \text {Geo}(p)$ so that

$$p_ X(x) = (1-p)^{x-1} p\qquad \text {for }x=1, 2, \dots $$

**Question:** Show that each of the table entries $p_ X(x)$ is nonnegative for $x = 1, 2, \dots.$

**Solution:** Since $p$ is a probabiltiy hence $p \in [0,1]$ then $1-p \in [0,1]$, also $x-1 \geq 0$. Hence

$$p_X(x) = \underbrace{(1-p)^{x-1}}_{\geq 0}\cdot \underbrace{p}_{\geq 0} \geq 0.$$

**Question:**Show that the sum of all the table entries is 1, i.e., $\sum _{x=1}^\infty p_ X(x) = 1.$

**Hint:** You may find the following result from calculus helpful: For $r\in (-1,1),$

$$\sum _{i=0}^{\infty }r^{i}=\frac{1}{1-r}.$$

Solution: For $p\in (0,1),$

$$\begin{align}
\sum _{x=1}^{\infty }p_{X}(x)&=\sum _{x=1}^{\infty }(1-p)^{x-1}p &\\
&= p \cdot \sum _{i=0}^{\infty }(1-p)^{i}&\text{where } i = x-1\\
&= p \cdot \frac{1}{1-(1-p)}&\\
&= p \cdot \frac{1}{p}=1.&
\end{align}.$$

### Exercise: The Expectation of a Geometric Distribution

In this exercise, we use the law of total expectation to find the expected value of a geometric random variable. The law of total expectation says that for a random variable $X$ (with alphabet $\mathcal{X}$) and a partition $\mathcal{B}_1,\dots ,\mathcal{B}_ n$ of the sample space,

$$\mathbb {E}[X]=\sum _{i=1}^{n}\mathbb {E}[X\mid \mathcal{B}_{i}]\mathbb {P}(\mathcal{B}_{i}),$$

where

$$\mathbb {E}[X\mid \mathcal{B}_{i}] = \sum _{x\in \mathcal{X}}xp_{X\mid \mathcal{B}_{i}}(x) = \sum _{x\in \mathcal{X}}x\frac{\mathbb {P}(X=x,\mathcal{B}_{i})}{\mathbb {P}(\mathcal{B}_{i})}.$$

Let $X \sim \text {Geo}(p)$ be the number of tosses until we get heads for the first time, where the probability of heads is $p$. Let $\mathcal{B}$ be the event that we get heads in 1 try. Let $\mathcal{B}^c$ be the event that we get heads in more than 1 try. Note that $\mathcal{B}$ and $\mathcal{B}^c$ form a partition of the sample space.

**Question:** What is $\mathbb {P}(\mathcal{B})?$

**Solution:** $\mathcal{B}$ is the event of getting "head" in one coin toss. Hence, $\mathbb {P}(\mathcal{B})=p.$

**Question:** What is $\mathbb {E}[X \mid \mathcal{B}]?$

**Solution:**

$$\begin{align}\require{cancel}
\mathbb{E}[X\mid \mathcal{B}] &= \sum_{x=1}^{\infty} xp_{X\mid \mathcal{B}}(x)\\
&= \sum_{x=1}^{\infty} x\frac{\mathbb {P}(X=x,\mathcal{B})}{\mathbb{P}(\mathcal{B})}\\
&= 1\times \frac{\cancelto{\mathbb{P}(\mathcal{B})}{\mathbb {P}(X=1,\mathcal{B})}}{\mathbb{P}(\mathcal{B})} + \sum_{x=2}^{\infty} x\frac{\cancelto{0}{\mathbb {P}(X=x,\mathcal{B})}}{\mathbb{P}(\mathcal{B})}\\
&= 1.
\end{align}$$

**Question:** What is $\mathbb{E}[X \mid \mathcal{B}^c]?$ Write your answer in terms of $m \triangleq \mathbb {E}[X].$ Note that we do not know $\mathbb {E}[X]$ for right now, but we can still relate $\mathbb {E}[X \mid \mathcal{B}^ c]$ to $\mathbb {E}[X].$

Hint: If you do not get heads the first time, then starting from the second toss, the distribution for the number of tosses remaining is still geometric!

**Solution:** So we tossed the coin and it was tails, so this took up 1 toss. The number of tosses that remains is just another $\text {Geo}(p)$ random variable (remember: the tosses are all independent so that initial toss doesn't affect any of the future tosses)!

Thus, the expectation of $X$ given that the first toss was tails (i.e., it takes more than 1 try) is

$$\mathbb {E}[X \mid \mathcal{B}^ c] = 1 + \mathbb {E}[X].$$

The 1 appears because that's the first toss where we got tails.

Using the law of total expectation,

$$\mathbb {E}[X] = \mathbb {E}[X \mid \mathcal{B}]\mathbb {P}(\mathcal{B}) + \mathbb {E}[X \mid \mathcal{B}^ c](1 - \mathbb {P}(\mathcal{B})).$$

Using your answers to the previous part, you should now have a recursive equation, meaning that the unknown quantity $\mathbb {E}[X]$ appears on both sides of the equation, and so you can solve for it.

**Question:** What is $\mathbb {E}[X]?$

In this part, please provide your answer as a mathematical formula (and not as Python code). Use ^ for exponentiation, e.g., x^2 denotes x^2. Explicitly include multiplication using *, e.g. x*y is xy.

**Solution:** Putting together the pieces from the previous parts,

$$\begin{align}\mathbb {E}[X] &= 1 \cdot p + (1 + \mathbb {E}[X]) \cdot (1 - p)\\
&= p + 1 - p + \mathbb {E}[X] - \mathbb {E}[X] p\\
&= 1 + \mathbb {E}[X] - \mathbb {E}[X] p.\end{align}$$

Rearranging terms yields

$$\mathbb {E}[X] = \boxed {\frac1p}.$$

```python

```


# Задание 2 Вариант 29

Хомутов Евений Васильевич, БПМ-151

```python
from sympy.solvers import solve
from sympy import Symbol, diff, sqrt,cos,sin
from scipy.misc import derivative
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
```

Определим необходимые для решения задач методы

```python
def BisecMethod(f,a,b,epselon):
    n = 0
    A = a
    B = b
    while (B-A) >= 2*epselon:
        n = n + 1
        x = (A+B)/2
        if f(x)*f(A) < 0:
            B = x
        elif f(x)*f(B) < 0:
            A = x
    return x, n
```

```python
def NewtoonMethod(f,x,epselon):
    n = 1
    x_old = x
    x_new = x_old - f(x_old)/derivative(f,x_old,dx = epselon)
    while abs(x_new - x_old) >= epselon:
        x_old = x_new
        x_new = x_old - f(x_old)/derivative(f,x_old,dx = epselon)
        n = n + 1
    return x_new, n
```

```python
def SimpleIterationMethod(phi,f,x,q,epselon):
    n = 1
    x_old = x
    x_new = phi(x_old,f)
    while abs(x_new - x_old) >= (1 - q)/q*epselon:
        x_old = x_new
        x_new = phi(x_old,f)
        n = n + 1
    return x_new,q, n
```

Чтобы найти ответы встроеными методами:

```python
x = Symbol('x')
```

## Задание № 1

```python
def f1(x):
    return x**4 - 26/5 * x**2 + 1
def g1(x):
    return x**4 - 10 * x**2 + 25
epselon1 = 10**(-10)
a1 = 0
b1 = 3
```

Найдем корни уравнения $f(x) = 0$ встроеными методами:

```python
ExactAnswer1 = solve(f1(x),x)
ExactAnswer1
```

    [-2.23606797749979, -0.447213595499958, 0.447213595499958, 2.23606797749979]

График функции $f(x)$:

```python
X = np.arange(a1, b1, 0.01)

fig = plt.figure()
plt.plot(X, f1(X))
plt.show()
```

Из графика поулчаем что отрезко локализации, например [0;1] и [2;3]

```python
x11 = BisecMethod(f1,0,1,0.1*epselon1)
x12 = BisecMethod(f1,2,3,0.1*epselon1)
print(x11, x12)
```

    (0.447213595485664, 36) (2.2360679774865275, 36)

Построим график функции $g(x)$:

```python
fig = plt.figure()
plt.plot(X, g1(X))
plt.show()
```

Имеем корень четной кратности, следовательно не можем применить метод бисекции

Вывод: Мы нашли корни правильно!

## Задание №2

```python
def f2(x):
    return np.sqrt(x)-np.cos(x)
epselon2 = 10**(-6)
```

Найдем корни уравнения $f(x) = 0$ встроеными методами Wolfram:

Построим график функции:

```python
X = np.arange(0, 2,0.1)
fig = plt.figure()
plt.plot(X, f2(X))
plt.show()
```

Из графика: возьмем отрезок локализации как [0;1]

```python
print(NewtoonMethod(f2,0.75,epselon2))
print(BisecMethod(f2,0,1,epselon2))
```

    (0.6417143708728962, 3)
    (0.6417140960693359, 19)

Вывод: в пределах точности наши корни совпали, при это число итераций у метода Ньютона в 6 раз меньше чем у метода бисекции

## Задание №3

```python
def f3(x):
    return x - np.e**(-x**(2))
epselon3 = 10**(-5)
```

График:

```python
X = np.arange(0.1, 1, 0.01)

fig = plt.figure()
plt.plot(X, f3(X))
plt.show()
```

Из графика возьмем отрезок локализации как [0.6;0.7]

```python
a3 = .6
b3 = .7
```

Определим константы m и M из графика производной:

Определим производную функции:

```python
def ff3(x):
    return 1 + 2*x*np.e**(-x**2)
```

График производной:

```python
X = np.arange(a3, b3, 0.01)

fig = plt.figure()
plt.plot(X, ff3(X))
plt.show()
```

Производная функции монотонно возрастает.

Следовательно для m и M берем значение производной в начале и конце отрезка соответсвенно

```python
m = ff3(a3)
M = ff3(b3)
```

Определим q

```python
q = (M-m)/(M+m)
```

Определим функцию $\phi$

```python
 def phi(x,f):
    return x - 2/(M+m)*f(x)
```

Ответ первым способом:

```python
SimpleIterationMethod(phi,f3,a3+b3/2,q,epselon3)
```

    (0.6529186597988018, 0.005538830287837204, 3)

Второй способ;

Сделаем из метода простой итерации метод Ньютона - потому что все любят Ньютона

```python
def gamma(x,f):
    return x - f(x)/derivative(f,x,dx = 10**(-5))

def Gamma(x):
    return (-np.e**(-(x)**2)+x)*(2*np.e**(-x**2)-4*x**2*np.e**(-x**2))/(1+2*x*np.e**(-x**2))**2
```

График модуля производной функции $\gamma$

```python
X = np.arange(a3, b3, 0.01)

fig = plt.figure()
plt.plot(X, abs(Gamma(X)))
plt.show()
```

Следовательно Q равен значению гамма в начале отрезка

```python
Q = abs(Gamma(a3))
```

Сравним овтеты метода Ньютона и метода простой итерации, превращеного в метод Ньютона:

```python
SimpleIterationMethod(gamma,f3,a3+b3/2,Q,epselon3)
```

    (0.6529186404213283, 0.011306103997355802, 3)

```python
NewtoonMethod(f3,a3+b3/2,epselon3)
```

    (0.6529186404213283, 3)

Вывод: все ответы совпали


Linear Algebra Examples
====

This just shows the machanics of linear algebra calculations with python. See Lecture 5 for motivation and understanding.

```python
import numpy as np
import scipy.linalg as la
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
plt.style.use('ggplot')
```

Resources
----

- [Tutorial for `scipy.linalg`](http://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html)

Exact solution of linear system of equations
----

\begin{align}
x + 2y &= 3 \\
3x + 4y &= 17
\end{align}

```python
A = np.array([[1,2],[3,4]])
A
```

    array([[1, 2],
           [3, 4]])

```python
b = np.array([3,17])
b
```

    array([ 3, 17])

```python
x = la.solve(A, b)
x
```

    array([11., -4.])

```python
np.allclose(A @ x, b)
```

    True

```python
A1 = np.random.random((1000,1000))
b1 = np.random.random(1000)
```

### Using solve is faster and more stable numerically than using matrix inversion

```python
%timeit la.solve(A1, b1)
```

    437 ms ± 115 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

```python
%timeit la.inv(A1) @ b1
```

    584 ms ± 127 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

### Under the hood (Optional)

The `solve` function uses the `dgesv` fortran function to do the actual work. Here is an example of how to do this directly with the `lapack` function. There is rarely any reason to use `blas` or `lapack` functions directly becuase the `linalg` package provides more convenient functions that also perfrom error checking, but you can use Python to experiment with `lapack` or `blas` before using them in a language like C or Fortran.

- [How to interpret lapack function names](http://www.netlib.org/lapack/lug/node24.html)
- [Summary of BLAS functions](http://cvxopt.org/userguide/blas.html)
- [Sumary of Lapack functions](http://cvxopt.org/userguide/lapack.html)

```python
import scipy.linalg.lapack as lapack
```

```python
lu, piv, x, info = lapack.dgesv(A, b)
x
```

Basic information about a matrix
----

```python
C = np.array([[1, 2+3j], [3-2j, 4]])
C
```

    array([[1.+0.j, 2.+3.j],
           [3.-2.j, 4.+0.j]])

```python
C.conjugate()
```

    array([[1.-0.j, 2.-3.j],
           [3.+2.j, 4.-0.j]])

#### Trace

```python
def trace(M):
    return np.diag(M).sum()
```

```python
trace(C)
```

    (5+0j)

```python
np.allclose(trace(C), la.eigvals(C).sum())
```

    True

#### Determinant

```python
la.det(C)
```

    (-8-5j)

#### Rank

```python
np.linalg.matrix_rank(C)
```

    2

#### Norm

```python
la.norm(C, None) # Frobenius (default)
```

    6.557438524302

```python
la.norm(C, 2) # largest sinular value
```

    6.389028023601217

```python
la.norm(C, -2) # smallest singular value
```

    1.4765909770949925

```python
la.svdvals(C)
```

Least-squares solution
----

```python
la.solve(A, b)
```

```python
x, resid, rank, s = la.lstsq(A, b)
x
```

```python
A1 = np.array([[1,2],[2,4]])
A1
```

```python
b1 = np.array([3, 17])
b1
```

```python
try:
    la.solve(A1, b1)
except la.LinAlgError as e:
    print(e)
```

```python
x, resid, rank, s = la.lstsq(A1, b1)
x
```

```python
A2 = np.random.random((10,3))
b2 = np.random.random(10)
```

```python
try:
    la.solve(A2, b2)
except ValueError as e:
    print(e)
```

```python
x, resid, rank, s = la.lstsq(A2, b2)
x
```

### Normal equations

One way to solve least squares equations $X\beta = y$ for $\beta$ is by using the formula $\beta = (X^TX)^{-1}X^Ty$ as you may have learnt in statistical theory classes (or can derive yourself with a bit of calculus). This is implemented below.

Note: This is not how the `la.lstsq` function solves least square problems as it can be inefficent for large matrices.

```python
def least_squares(X, y):
    return la.solve(X.T @ X, X.T @ y)
```

```python
least_squares(A2, b2)
```

Matrix Decompositions
----

```python
A = np.array([[1,0.6],[0.6,4]])
A
```

    array([[1. , 0.6],
           [0.6, 4. ]])

### LU

```python
p, l, u = la.lu(A)
```

```python
p
```

    array([[1., 0.],
           [0., 1.]])

```python
l
```

    array([[1. , 0. ],
           [0.6, 1. ]])

```python
u
```

    array([[1.  , 0.6 ],
           [0.  , 3.64]])

```python
np.allclose(p@l@u, A)
```

    True

### Choleskey

```python
U = la.cholesky(A)
U
```

    array([[1.       , 0.6      ],
           [0.       , 1.9078784]])

```python
np.allclose(U.T @ U, A)
```

    True

```python
# If workiing wiht complex matrices
np.allclose(U.T.conj() @ U, A)
```

    True

### QR

```python
Q, R = la.qr(A)
```

```python
Q
```

    array([[-0.85749293, -0.51449576],
           [-0.51449576,  0.85749293]])

```python
np.allclose((la.norm(Q[:,0]), la.norm(Q[:,1])), (1,1))
```

    True

```python

```

```python
np.allclose(Q@R, A)
```

    True

### Spectral

When matrix is symmetric, you can use la.eigh

```python
u, v = la.eig(A)
```

```python
u
```

    array([0.88445056+0.j, 4.11554944+0.j])

```python
v
```

    array([[-0.98195639, -0.18910752],
           [ 0.18910752, -0.98195639]])

```python
np.allclose((la.norm(v[:,0]), la.norm(v[:,1])), (1,1))
```

    True

```python
np.allclose(v @ np.diag(u) @ v.T, A)
```

    True

#### Inverting A

```python
np.allclose(v @ np.diag(1/u) @ v.T, la.inv(A))
```

#### Powers of A

```python
np.allclose(v @ np.diag(u**5) @ v.T, np.linalg.matrix_power(A, 5))
```

### SVD

```python
U, s, V = la.svd(A)
```

```python
U
```

```python
np.allclose((la.norm(U[:,0]), la.norm(U[:,1])), (1,1))
```

```python
s
```

```python
V
```

```python
np.allclose((la.norm(V[:,0]), la.norm(V[:,1])), (1,1))
```

```python
np.allclose(U @ np.diag(s) @ V, A)
```

```python

```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

Let's compute the area under $\sin x$ from $x=0$ to $x=\pi$.

We use the same technique as in how to write and evaluate indefinite integrals,
except that we add the lower and upper bounds together with $x$, as shown below.

```python
var( 'x' )
formula = sin(x)
Integral( formula, (x,0,pi) )
```

$\displaystyle \int\limits_{0}^{\pi} \sin{\left(x \right)}\, dx$

The above code just displays the definite integral.
To evaluate it, use the `integrate` command.

```python
integrate( formula, (x,0,pi) )
```

$\displaystyle 2$

database/tasks/How to write and evaluate definite integrals/Python, using SymPy.ipynb


# Quadratic programming examples

A quadratic programming problem can be expressed in the following form:

$$
\begin{align}
\text{minimize} & \quad \frac{1}{2}x^T Q x + p^T x \\
\text{subject to} & \quad
\begin{cases}
G x \le h \\
A x = b
\end{cases}
\end{align}
$$

where:
- $p$ is a real-valued, $n$-dimensional vector;
- $Q$ is an $n \times n$-dimensional real symmetric matrix;
- $G$ is an $m \times n$-dimensional real matrix;
- $h$ is an $m$-dimensional real vector;
- $A$ is a $p \times n$-dimensional real matrix;
- $b$ is a $p$-dimensional real vector.

We will use the [qpsolvers](https://github.com/stephane-caron/qpsolvers) library, together with [numpy](https://numpy.org/), to formulate and solve a few examples of quadratic programming problems. Both libraries can be installed with `pip` as explained in the README file; once installed, we can import them as follows:

```python
import numpy as np

from qpsolvers import solve_qp
```

## Example 1

This example is taken from [CVXOPT](https://cvxopt.org/examples/tutorial/qp) and is formulated as follows:

$$
\begin{align}
\text{minimize} & \quad f(x) = 2x_1^2 + x_2^2 + x_1x_2 + x_1 + x_2 \\
\text{subject to} & \quad
\begin{cases}
x_1 \ge 0 \\
x_2 \ge 0 \\
x_1 + x_2 = 1
\end{cases}
\end{align}
$$

We need to extract the $Q$ matrix first. We have 2 variables, so for the quadratic term we have:

$
\begin{align}
q(x) & = \dfrac{1}{2}x^T Q x \\
& = \dfrac{1}{2} \begin{bmatrix}x_1 & x_2\end{bmatrix}\begin{bmatrix}a_{11} & a_{12} \\ a_{21} & a_{22}\end{bmatrix}\begin{bmatrix}x_1 \\ x_2\end{bmatrix} \\
& = \dfrac{1}{2} \sum\limits_{i=1}^2 \sum\limits_{j=1}^2 a_{ij} x_i x_j \\
& = \dfrac{1}{2} (a_{11} x_1 x_1 + a_{12} x_1 x_2 + a_{21} x_2 x_1 + a_{22} x_2 x_2) \\
& = \dfrac{1}{2} (a_{11} x_1^2 + a_{22} x_2^2 + (a_{12} + a_{21}) x_1 x_2)
\end{align}
$

Since $Q$ is assumed to be symmetric we have that $a_{12} = a_{21}$, so that:

$
q(x) = \dfrac{1}{2} a_{11} x_1^2 + \dfrac{1}{2} a_{22} x_2^2 + a_{12} x_1 x_2
$

Now we can find the values for the elements $a_{ij}$:

$
\begin{cases}
\dfrac{1}{2} a_{11} & = 2 \implies a_{11} = 4 \\
\dfrac{1}{2} a_{22} & = 1 \implies a_{22} = 2 \\
a_{12} & = 1
\end{cases}
$

and write the $Q$ matrix as:

$Q = \begin{bmatrix}4 & 1 \\ 1 & 2\end{bmatrix}$

or, in order to show the quadratic coefficients more clearly, as:

$Q = 2 * \begin{bmatrix}2 & \dfrac{1}{2} \\ \dfrac{1}{2} & 1\end{bmatrix}$

```python
Q = 2 * np.array([[2, 0.5], [0.5, 1]])
```

The linear term is straightforward:

$
p(x) = p^T x = \begin{bmatrix}x_1 & x_2\end{bmatrix}\begin{bmatrix}a_1 \\ a_2\end{bmatrix} = a_1 x_1 + a_2 x_2
$

resulting in:

$
\begin{cases}
a_1 = 1 \\
a_2 = 1
\end{cases}
$

from which the $p$ vector can be written as:

$
p = \begin{bmatrix}1 & 1\end{bmatrix}
$

```python
p = np.array([1.0, 1.0])
```

We have two inequality constraints, $x_1 \ge 0$ and $x_2 \ge 0$. We need to write them in the form $G x \le h$, so we have:

$
\begin{align}
\begin{cases}
-x_1 &+ 0x_2 & \le 0 \\
0x_2 &- x_2 & \le 0
\end{cases}
\end{align}
$

from which we derive:

$
\begin{bmatrix}-1 & 0 \\ 0 & -1\end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \end{bmatrix} \le \begin{bmatrix}0 \\ 0 \end{bmatrix} \implies
G = \begin{bmatrix}-1 & 0 \\ 0 & -1\end{bmatrix} , h = \begin{bmatrix}0 \\ 0 \end{bmatrix}
$

```python
G = np.array([[-1.0, 0.0], [0.0, -1.0]])
h = np.array([0.0, 0.0])
```

Finally, we have an equality constraint $x_1 + x_2 = 1$, that we can represent as:

$
\begin{bmatrix}1 & 1\end{bmatrix} \begin{bmatrix}x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix}1\end{bmatrix} \implies
A = \begin{bmatrix}1 & 1\end{bmatrix} , b = \begin{bmatrix}1\end{bmatrix}
$

```python
A = np.array([1.0, 1.0])
b = np.array([1.0])
```

We now have everything we need to run the solver, which will return the optimal solution $x^* = \begin{bmatrix}x^*_1 & x^*_2\end{bmatrix}$.

```python
x_star = solve_qp(Q, p, G, h, A, b)
print(x_star)
```

    [0.25 0.75]

The `solve_qp` method uses the [quadprog](https://pypi.python.org/pypi/quadprog/) solver by default, but it can be used with other solvers as well; take a look at the [project's page](https://github.com/stephane-caron/qpsolvers#solvers) to find the list of supported solvers.

## Example 2

This example is taken from the [Northwestern University website](https://optimization.mccormick.northwestern.edu/index.php/Quadratic_programming#Numerical_example) and is formulated as follows:

$$
\begin{align}
\text{minimize} & \quad f(x) = 3x_1^2 + x_2^2 + 2x_1x_2 + x_1 + 6x_2 + 2 \\
\text{subject to} & \quad
\begin{cases}
2x_1 + 3x_2 \ge 4 \\
x_1 \ge 0 \\
x_2 \ge 0
\end{cases}
\end{align}
$$

The constant term $2$ in the objective function can be discarded. Since there are no equality constraints, we will only have $G$ and $h$; let's keep in mind that the inequality contraints have to be expressed as $Gx \le h$.

```python
Q = 2 * np.array([[3.0, 1.0], [1.0, 1.0]])
p = np.array([1.0, 6.0])
G = np.array([[-2.0, -3.0], [-1.0, 0.0], [0.0, -1.0]])
h = np.array([-4.0, 0.0, 0.0])

x_star = solve_qp(Q, p, G, h)
print(x_star)
```

    [0.5 1. ]

Since the objective function can be expressed as $f(x) = \frac{1}{2}x^T Q x + p^T x$, its value in $x^*$ can be found as follows:

```python
f = 0.5 * x_star.T @ Q @ x_star + p.T @ x_star + 2
print(f)
```

    11.25

## Example 3

This example is taken from the [Matlab website](https://uk.mathworks.com/help/optim/ug/quadprog.html) and is formulated as follows:

$$
\begin{align}
\text{minimize} & \quad f(x) = \dfrac{1}{2} x_1^2 + x_2^2 − x_1 x_2 − 2 x_1 − 6 x_2 \\
\text{subject to} & \quad
\begin{cases}
x_1 + x_2 \le 2 \\
-x_1 + 2 x_2 \le 2 \\
2 x_1 + x_2 \le 3
\end{cases}
\end{align}
$$

This time, instead of using float numbers, we will declare the type of each variable explicitly with `dtype`:

```python
Q = np.array([[1, -1], [-1, 2]], dtype=np.float)
p = np.array([-2, -6], dtype=np.float)

G = np.array([[1, 1], [-1, 2], [2, 1]], dtype=np.float)
h = np.array([2, 2, 3], dtype=np.float)

x_star = solve_qp(Q, p, G, h)
print(x_star)
```

    [0.66666667 1.33333333]

Value of the objective function in $x^*$:

```python
f = 0.5 * x_star.T @ Q @ x_star + p.T @ x_star
print(f)
```

    -8.222222222222218


# Optimization in Python

You might have noticed that we didn't do anything related to sparsity with scikit-learn models. A lot of the work we covered in the machine learning class is very recent research, and as such is typically not implemented by the popular libraries.

If we want to do things like sparse regression, we're going to have to roll up our sleeves and do it ourselves. For that, we need to be able to solve optimization problems. In Julia, we did this with JuMP. In Python, we'll use a similar library called *pyomo*.

# Installing pyomo

You can run the following command to install pyomo if you haven't already.

```python
!pip install pyomo --user
```

# Intro to pyomo

Let's see how we translate a simple, 2 variable LP to pyomo code.

$$
\begin{align*}
\max_{x,y} \quad& x + 2y \\
\text{s.t.}\quad& x + y \leq 1 \\
& x, y \geq 0.
\end{align*}
$$

First thing is to import the pyomo functions:

```python
from pyomo.environ import *
from pyomo.opt import SolverFactory
```

Next, we construct a model object. This is a container for everything in our optimization problem: variables, constraints, solver options, etc.

```python
m = ConcreteModel()
```

Next, we define the two decision variables in our optimization problem. We use the ``Var`` function to create the variables. The `within` keyword is used to specify the bounds on the variables, or equivalently the `bounds` keyword. The variables are added to the model object with names `x` and `y`.

```python
m.x = Var(within=NonNegativeReals)
m.y = Var(bounds=(0, float('inf')))
```

We now add the single constraint of our problem using the ``Constraint`` function. We write it algebraically, and save the result to the model.

```python
m.con = Constraint(expr=m.x + m.y <= 1)
```

We specify the objective function with the `Objective` function:

```python
m.obj = Objective(sense=maximize, expr=m.x + 2 * m.y)
```

We solve the optimization problem by first specifying a solver using `SolverFactory` and then using this solver to solve the model:

```python
solver = SolverFactory('gurobi')
solver.solve(m)
```

We can now inspect the solution values and optimal cost.

```python
m.obj()
```

```python
m.x.value
```

```python
m.y.value
```

Let's put it all together to compare with Julia/JuMP

```python
# Create model
m = ConcreteModel()
# Add variables
m.x = Var(within=NonNegativeReals)
m.y = Var(bounds=(0, float('inf')))
# Add constraint
m.con = Constraint(expr=m.x + m.y <= 1)
# Add objective
m.obj = Objective(sense=maximize, expr=m.x + 2 * m.y)
# Solve model
solver = SolverFactory('gurobi')
solver.solve(m)
# Inspect solution
print(m.obj())
print(m.x.value)
print(m.y.value)
```

```julia
# Create model
m = Model(solver=GurobiSolver())
# Add variables
@variable(m, x >= 0)
@variable(m, y >= 0)
# Add constraint
@constraint(m, x + y <= 1)
# Add objective
@objective(m, Max, x + 2y)
# Solve model
solve(m)
# Inspect solution
@show getobjectivevalue(m)
@show getvalue(x)
@show getvalue(y)
```

### Exercise

Code and solve the following optimization problem:

$$
\begin{align*}
\min_{x,y} \quad& 3x - y \\
\text{s.t.}\quad& x + 2y \geq 1 \\
& x \geq 0 \\
& 0 \leq y \leq 1.
\end{align*}
$$

```python
# Create the model
m = ConcreteModel()
# Add the variables
m.x = Var(within=NonNegativeReals)
m.y = Var(bounds=(0, 1))
# Add the constraint
m.con = Constraint(expr=m.x + 2 * m.y >= 1)
# Add the objective
m.obj = Objective(sense=minimize, expr=3 * m.x - m.y)

solver = SolverFactory('gurobi')
solver.solve(m)

print(m.x.value, m.y.value)
```

```python
for v in m.component_data_objects(Var, active=True):
    print(v, value(v)) # doctest: +SKIP
```

```python
m.pprint()
```

# Index sets

Let's now move to a more complicated problem. We'll look at a transportation problem:

$$
\begin{align}
\min & \sum\limits_{i = 1}^{m} \sum\limits_{j = 1}^{n} c_{ij} x_{ij}\\
& \sum\limits_{j = 1}^{n} x_{ij} \leq b_i && i = 1, \ldots, m\\
& \sum\limits_{i = 1}^{m} x_{ij} = d_j && j = 1, \ldots, n\\
& x_{ij} \ge 0 && i = 1, \ldots, m, j = 1, \ldots, n
\end{align}
$$

And with some data:

```python
import numpy as np

m = 2  # Number of supply nodes
n = 5  # Number of demand nodes
# Supplies
b = np.array([1000, 4000])
# Demands
d = np.array([500, 900, 1800, 200, 700])
# Costs
c = np.array([[2, 4, 5, 2, 1],
              [3, 1, 3, 2, 3]])
```

Now we can formulate the problem with pyomo

```python
model = ConcreteModel()
```

First step is adding variables. We can add variables with indices by passing the relevant index sets to the `Var` constructor. In this case, we need a $m$-by$n$ matrix of variables:

```python
model.x = Var(range(m), range(n), within=NonNegativeReals)
```

Now to add the constraints. We have to add one supply constraint for each factory, so we might try something like:

```python
for i in range(m):
    model.supply = Constraint(expr=sum(model.x[i, j] for j in range(n)) <= b[i])
```

Can you see the problem? We are overwriting `model.supply` in each iteration of the loop, and so only the last constraint is applied.

Luckily, pyomo has a (not-so-easy) way to add multiple constraints at a time. We first define a *rule* that takes in the model and any required indices, and then returns the expression for the constraint:

```python
def supply_rule(model, i):
    return sum(model.x[i, j] for j in range(n)) <= b[i]
```

We then add the constraint by referencing this rule along with the index set we want the constraint to be defined over:

```python
model.supply2 = Constraint(range(m), rule=supply_rule)
```

We then apply the same approach for the demand constraints

```python
def demand_rule(model, j):
    return sum(model.x[i, j] for i in range(m)) == d[j]
model.demand = Constraint(range(n), rule=demand_rule)
```

Finally, we add the objective:

```python
model.obj = Objective(sense=minimize,
    expr=sum(c[i, j] * model.x[i, j]
    for i in range(m) for j in range(n)))
```

Now we can solve the problem

```python
solver = SolverFactory('gurobi')
solver.solve(model)
```

It solved, so we can extract the results

```python
flows = np.array([[model.x[i, j].value for j in range(n)] for i in range(m)])
flows
```

We can also check the objective value for the cost of this flow

```python
model.obj()
```

For simplicity, here is the entire formulation and solving code together:

```python
model = ConcreteModel()
# Variables
model.x = Var(range(m), range(n), within=NonNegativeReals)
# Supply constraint
def supply_rule(model, i):
    return sum(model.x[i, j] for j in range(n)) <= b[i]
model.supply2 = Constraint(range(m), rule=supply_rule)
# Demand constraint
def demand_rule(model, j):
    return sum(model.x[i, j] for i in range(m)) == d[j]
model.demand = Constraint(range(n), rule=demand_rule)
# Objective
model.obj = Objective(sense=minimize,
    expr=sum(c[i, j] * model.x[i, j]
    for i in range(m) for j in range(n)))
# Solve
solver = SolverFactory('gurobi')
solver.solve(model)
# Get results
flows = np.array([[model.x[i, j].value for j in range(n)] for i in range(m)])
model.obj()
```

# Machine Learning

Now let's put our pyomo knowledge to use and implement some of the same methods we saw in the machine learning class

First, specify your solver executable location:

```python
executable='C:/Users/omars/.julia/v0.6/Ipopt/deps/usr/bin/ipopt.exe'
```

To use the version left over from Julia
### On MacOS and Linux

`executable="~/.julia/v0.6/Homebrew/deps/usr/Cellar/ipopt/3.12.4_1/bin/ipopt")`

### On Windows

The path is probably under WinRPM:

`executable='%HOME%/.julia/v0.6/WinRPM/...')")`

# Linear Regression

Let's just try a simple linear regression

```python
def linear_regression(X, y):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))

    # Add constraints

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)

    return [m.beta[j].value for j in range(p)]
```

Let's load up some data to test it out on:

```python
from sklearn.datasets import load_boston
data = load_boston()
X = data.data
y = data.target
```

Try our linear regression function:

```python
print(linear_regression(X, y))
```

We can compare with sklearn to make sure it's right:

```python
from sklearn.linear_model import LinearRegression
m = LinearRegression(fit_intercept=False)
m.fit(X, y)
m.coef_
```

Just for reference, let's look back at how we do the same thing in JuMP!

```julia
using JuMP, Gurobi
function linear_regression(X, y)
    n, p = size(X)
    m = Model(solver=GurobiSolver())
    @variable(m, beta[1:p])
    @objective(m, Min, sum((y[i] - sum(X[i, j] * beta[j] for j = 1:p)) ^ 2 for i = 1:n))
    solve(m)
    getvalue(beta)
end
```

or even

```julia
using JuMP, Gurobi
function linear_regression(X, y)
    n, p = size(X)
    m = Model(solver=GurobiSolver())
    @variable(m, beta[1:p])
    @objective(m, Min, sum((y - X * beta) .^ 2))
    solve(m)
    getvalue(beta)
end
```

Much simpler!

### Exercise

Modify the linear regression formulation to include an intercept term, and compare to scikit-learn's LinearRegression with `fit_intercept=False` to make sure it's the same

```python
def linear_regression_intercept(X, y):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.b0 = Var()

    # Add constraints

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)) - m.b0, 2)
        for i in range(n)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)

    return [m.beta[j].value for j in range(p)]

linear_regression_intercept(X, y)
```

```python
m = LinearRegression(fit_intercept=True)
m.fit(X, y)
m.coef_
```

# Robust Regression

We saw in the class that both ridge and lasso regression were robust versions of linear regression. Both of these are provided by `sklearn`, but we need to know how to implement them if we want to extend regression ourselves

```python
def ridge_regression(X, y, rho):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)),2)
        for i in range(n)) + rho * sum(pow(m.beta[j], 2) for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

```python
ridge_regression(X, y, 100000)
```

```python
def lasso(X, y, rho):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)),2)
        for i in range(n)) + rho * sum(pow(m.beta[j], 2) for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

### Exercise

Implement Lasso regression

```python
def lasso_regression(X, y, rho):
    n, p = X.shape

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.absb = Var(range(p))

    # Add constraints
    def absbeta1(m, j):
        return m.beta[j] <= m.absb[j]
    m.absb1 = Constraint(range(p), rule=absbeta1)
    def absbeta2(m, j):
        return -m.beta[j] <= m.absb[j]
    m.absb2 = Constraint(range(p), rule=absbeta2)

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)) + rho * sum(m.absb[j] for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

```python
lasso_regression(X, y, 1000)
```

# Sparse Regression

```python
def sparse_regression(X, y, k):
    n, p = X.shape
    M = 1000

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.z = Var(range(p), within=Binary)

    # Add constraints
    def bigm1(m, j):
        return m.beta[j] <= M * m.z[j]
    m.bigm1 = Constraint(range(p), rule=bigm1)
    def bigm2(m, j):
        return m.beta[j] >= -M * m.z[j]
    m.bigm2 = Constraint(range(p), rule=bigm2)

    m.sparsity = Constraint(expr=sum(m.z[j] for j in range(p)) <= k)

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
    # results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)
    return [m.beta[j].value for j in range(p)]
```

```python
sparse_regression(X, y, 10)
```

```python
import numpy as np
l = np.array([1,2,3,4])

print(l**2)
print([sqrt(i) for i in l])
```

### Exercise

Try implementing the algorithmic framework for linear regression:
- sparsity constraints
- lasso regularization
- restrict highly correlated pairs of features
- nonlinear transformations (just $\sqrt(x)$ and $x^2$)

```python
import numpy as np
from sklearn.preprocessing import normalize

def all_regression(X_orig, y, k, rho):
    n, p_orig = X_orig.shape
    M = 10

    X = np.concatenate(
        [X_orig, np.sqrt(X_orig), np.square(X_orig)], axis=1
    )
    p = X.shape[1]

    # Normalize data
    X = normalize(X, axis=0)
    y = (y - np.mean(y)) / np.linalg.norm(y)

    # Create model
    m = ConcreteModel()

    # Add variables
    m.beta = Var(range(p))
    m.z = Var(range(p), within=Binary)
    m.absb = Var(range(p))

    # Sparsity constraints
    def bigm1(m, j):
        return m.beta[j] <= M * m.z[j]
    m.bigm1 = Constraint(range(p), rule=bigm1)
    def bigm2(m, j):
        return m.beta[j] >= -M * m.z[j]
    m.bigm2 = Constraint(range(p), rule=bigm2)
    m.sparsity = Constraint(expr=sum(m.z[j] for j in range(p)) <= k)

    # Lasso constraints
    def absbeta1(m, j):
        return m.beta[j] <= m.absb[j]
    m.absb1 = Constraint(range(p), rule=absbeta1)
    def absbeta2(m, j):
        return -m.beta[j] <= m.absb[j]
    m.absb2 = Constraint(range(p), rule=absbeta2)

    # Correlation constraints
    corX = np.corrcoef(np.transpose(X))
    def cor_rule(m, i, j):
        if i > j and abs(corX[i, j]) > 0.8:
            return (sum(m.z[k] for k in range(i, p, p_orig)) +
                    sum(m.z[k] for k in range(j, p, p_orig)) <= 1)
        else:
            return Constraint.Skip
    m.cor = Constraint(range(p_orig), range(p_orig), rule=cor_rule)

    # Nonlinear constraints
    def nl_rule(m, i):
        return sum(m.z[k] for k in range(i, p, p_orig)) <= 1
    m.nl = Constraint(range(p_orig), rule=nl_rule)

    # Add objective
    m.obj = Objective(sense=minimize, expr=sum(
        pow(y[i] - sum(X[i, j] * m.beta[j] for j in range(p)), 2)
        for i in range(n)) + rho * sum(m.absb[j] for j in range(p)))

    solver = SolverFactory('ipopt', executable=executable)

    ## tee=True enables solver output
#     results = solver.solve(m, tee=True)
    results = solver.solve(m, tee=False)

    return np.array([m.beta[j].value for j in range(p)]).reshape(-1, p_orig)
```

```python
all_regression(X, y, 6, 0)
```

# Logistic Regression

Like JuMP, we need to use a new solver for the nonlinear problem. We can use Ipopt as before, except we have to set it up manually. You'll need to download Ipopt and add it to the PATH.

On Mac, you can do this with Homebrew if you have it:

The other way is to download a copy of ipopt and specify the path to it exactly when creating the solver. For example, I have a copy of Ipopt left over from JuMP, which I can use by modifying the SolverFactory line as indicated below:

```python
def logistic_regression(X, y):
    n, p = X.shape

    # Convert y to (-1, +1)
    assert np.min(y) == 0
    assert np.max(y) == 1
    Y = y * 2 - 1
    assert np.min(Y) == -1
    assert np.max(Y) == 1

    # Create the model
    m = ConcreteModel()

    # Add variables
    m.b = Var(range(p))
    m.b0 = Var()

    # Set nonlinear objective function
    m.obj = Objective(sense=maximize, expr=-sum(
        log(1 + exp(-Y[i] * (sum(X[i, j] * m.b[j] for j in range(p)) + m.b0)))
        for i in range(n)))

    # Solve the model and get the optimal solutions
    solver = SolverFactory('ipopt', executable=executable)

    solver.solve(m)
    return [m.b[j].value for j in range(p)], m.b0.value
```

Load up some data

```python
from sklearn.datasets import load_breast_cancer
data = load_breast_cancer()
X = data.data
y = data.target
```

```python
logistic_regression(X, y)
```

### Exercise

Implement the regularized versions of logistic regression that scikit-learn provides:

```python
def logistic_regression_l1(X, y, C):
    n, p = X.shape

    # Convert y to (-1, +1)
    assert np.min(y) == 0
    assert np.max(y) == 1
    Y = y * 2 - 1
    assert np.min(Y) == -1
    assert np.max(Y) == 1

    # Create the model
    m = ConcreteModel()

    # Add variables
    m.b = Var(range(p))
    m.b0 = Var()

    # Lasso constraints
    m.absb = Var(range(p))
    def absbeta1(m, j):
        return m.b[j] <= m.absb[j]
    m.absb1 = Constraint(range(p), rule=absbeta1)
    def absbeta2(m, j):
        return -m.b[j] <= m.absb[j]
    m.absb2 = Constraint(range(p), rule=absbeta2)

    # Set nonlinear objective function
    m.obj = Objective(sense=minimize, expr=sum(m.absb[j] for j in range(p)) + C * sum(
        log(1 + exp(-Y[i] * (sum(X[i, j] * m.b[j] for j in range(p)) + m.b0)))
        for i in range(n)))

    # Solve the model and get the optimal solutions
    solver = SolverFactory('ipopt', executable=executable)

    solver.solve(m)
    return [m.b[j].value for j in range(p)], m.b0.value
```

```python
logistic_regression_l1(X, y, 100)
```

```python
def logistic_regression_l2(X, y, C):
    n, p = X.shape

    # Convert y to (-1, +1)
    assert np.min(y) == 0
    assert np.max(y) == 1
    Y = y * 2 - 1
    assert np.min(Y) == -1
    assert np.max(Y) == 1

    # Create the model
    m = ConcreteModel()

    # Add variables
    m.b = Var(range(p))
    m.b0 = Var()

    # Set nonlinear objective function
    m.obj = Objective(sense=minimize, expr=0.5 * sum(pow(m.b[j], 2) for j in range(p)) + C * sum(
        log(1 + exp(-Y[i] * (sum(X[i, j] * m.b[j] for j in range(p)) + m.b0)))
        for i in range(n)))

    # Solve the model and get the optimal solutions
    solver = SolverFactory('ipopt', executable=executable)

    solver.solve(m)
    return [m.b[j].value for j in range(p)], m.b0.value
```

```python
logistic_regression_l2(X, y, 1000)
```

```python

```

ML3 - Optimization Modeling (Complete).ipynb