# Using optimization routines from `scipy` and `statsmodels`

```python
%matplotlib inline
```

```python
import scipy.linalg as la
import numpy as np
import scipy.optimize as opt
import matplotlib.pyplot as plt
import pandas as pd
```

```python
np.set_printoptions(precision=3, suppress=True)
```

Using `scipy.optimize`
----

One of the most convenient libraries to use is `scipy.optimize`, since it is already part of the Anaconda installation and it has a fairly intuitive interface.

```python
from scipy import optimize as opt
```

#### Minimizing a univariate function $f: \mathbb{R} \rightarrow \mathbb{R}$

```python
def f(x):
    return x**4 + 3*(x-2)**3 - 15*(x)**2 + 1
```

```python
x = np.linspace(-8, 5, 100)
plt.plot(x, f(x));
```

The [`minimize_scalar`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize_scalar.html#scipy.optimize.minimize_scalar) function will find the minimum, and can also be told to search within given bounds. By default, it uses the Brent algorithm, which combines a bracketing strategy with a parabolic approximation.

```python
opt.minimize_scalar(f, method='Brent')
```

```python
opt.minimize_scalar(f, method='bounded', bounds=[0, 6])
```

### Local and global minima

```python
def f(x, offset):
    return -np.sinc(x-offset)
```

```python
x = np.linspace(-20, 20, 100)
plt.plot(x, f(x, 5));
```

```python
# note how additional function arguments are passed in
sol = opt.minimize_scalar(f, args=(5,))
sol
```

```python
plt.plot(x, f(x, 5))
plt.axvline(sol.x, c='red')
pass
```

#### We can try multiple random starts to find the global minimum

```python
lower = np.random.uniform(-20, 20, 100)
upper = lower + 1
sols = [opt.minimize_scalar(f, args=(5,), bracket=(l, u)) for (l, u) in zip(lower, upper)]
```

```python
idx = np.argmin([sol.fun for sol in sols])
sol = sols[idx]
```

```python
plt.plot(x, f(x, 5))
plt.axvline(sol.x, c='red');
```

#### Using a stochastic algorithm

See documentation for the [`basinhopping`](http://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.optimize.basinhopping.html) algorithm, which also works with multivariate scalar optimization. Note that this is heuristic and not guaranteed to find a global minimum.

```python
from scipy.optimize import basinhopping

x0 = 0
sol = basinhopping(f, x0, stepsize=1, minimizer_kwargs={'args': (5,)})
sol
```

```python
plt.plot(x, f(x, 5))
plt.axvline(sol.x, c='red');
```

### Constrained optimization with `scipy.optimize`

Many real-world optimization problems have constraints - for example, a set of parameters may have to sum to 1.0 (equality constraint), or some parameters may have to be non-negative (inequality constraint). Sometimes, the constraints can be incorporated into the function to be minimized, for example, the non-negativity constraint $p \gt 0$ can be removed by substituting $p = e^q$ and optimizing for $q$. Using such workarounds, it may be possible to convert a constrained optimization problem into an unconstrained one, and use the methods discussed above to solve the problem.

Alternatively, we can use optimization methods that allow the specification of constraints directly in the problem statement as shown in this section. Internally, constraint violation penalties, barriers and Lagrange multipliers are some of the methods used used to handle these constraints. We use the example provided in the Scipy [tutorial](http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html) to illustrate how to set constraints.

We will optimize:

$$
f(x) = -(2xy + 2x - x^2 -2y^2)
$$
subject to the constraint
$$
x^3 - y = 0 \\
y - (x-1)^4 - 2 \ge 0
$$
and the bounds
$$
0.5 \le x \le 1.5 \\
1.5 \le y \le 2.5
$$

```python
def f(x):
    return -(2*x[0]*x[1] + 2*x[0] - x[0]**2 - 2*x[1]**2)
```

```python
x = np.linspace(0, 3, 100)
y = np.linspace(0, 3, 100)
X, Y = np.meshgrid(x, y)
Z = f(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
plt.contour(X, Y, Z, np.arange(-1.99,10, 1), cmap='jet');
plt.plot(x, x**3, 'k:', linewidth=1)
plt.plot(x, (x-1)**4+2, 'k:', linewidth=1)
plt.fill([0.5,0.5,1.5,1.5], [2.5,1.5,1.5,2.5], alpha=0.3)
plt.axis([0,3,0,3])
```

To set constraints, we pass in a dictionary with keys `type`, `fun` and `jac`. Note that the inequality constraint assumes a $C_j x \ge 0$ form. As usual, the `jac` is optional and will be numerically estimated if not provided.

```python
cons = ({'type': 'eq',
         'fun' : lambda x: np.array([x[0]**3 - x[1]]),
         'jac' : lambda x: np.array([3.0*(x[0]**2.0), -1.0])},
        {'type': 'ineq',
         'fun' : lambda x: np.array([x[1] - (x[0]-1)**4 - 2])})

bnds = ((0.5, 1.5), (1.5, 2.5))
```

```python
x0 = [0, 2.5]
```

Unconstrained optimization

```python
ux = opt.minimize(f, x0, constraints=None)
ux
```

Constrained optimization

```python
cx = opt.minimize(f, x0, bounds=bnds, constraints=cons)
cx
```

```python
x = np.linspace(0, 3, 100)
y = np.linspace(0, 3, 100)
X, Y = np.meshgrid(x, y)
Z = f(np.vstack([X.ravel(), Y.ravel()])).reshape((100,100))
plt.contour(X, Y, Z, np.arange(-1.99,10, 1), cmap='jet');
plt.plot(x, x**3, 'k:', linewidth=1)
plt.plot(x, (x-1)**4+2, 'k:', linewidth=1)
plt.text(ux['x'][0], ux['x'][1], 'x', va='center', ha='center', size=20, color='blue')
plt.text(cx['x'][0], cx['x'][1], 'x', va='center', ha='center', size=20, color='red')
plt.fill([0.5,0.5,1.5,1.5], [2.5,1.5,1.5,2.5], alpha=0.3)
plt.axis([0,3,0,3]);
```

## Some applications of optimization

### Finding paraemeters for ODE models

This is a specialized application of `curve_fit`, in which the curve to be fitted is defined implicitly by an ordinary differential equation
$$
\frac{dx}{dt} = -kx
$$
and we want to use observed data to estimate the parameters $k$ and the initial value $x_0$. Of course this can be explicitly solved but the same approach can be used to find multiple parameters for $n$-dimensional systems of ODEs.

[A more elaborate example for fitting a system of ODEs to model the zombie apocalypse](http://adventuresinpython.blogspot.com/2012/08/fitting-differential-equation-system-to.html)

```python
from scipy.integrate import odeint

def f(x, t, k):
    """Simple exponential decay."""
    return -k*x

def x(t, k, x0):
    """
    Solution to the ODE x'(t) = f(t,x,k) with initial condition x(0) = x0
    """
    x = odeint(f, x0, t, args=(k,))
    return x.ravel()
```

```python
# True parameter values
x0_ = 10
k_ = 0.1*np.pi

# Some random data genererated from closed form solution plus Gaussian noise
ts = np.sort(np.random.uniform(0, 10, 200))
xs = x0_*np.exp(-k_*ts) + np.random.normal(0,0.1,200)

popt, cov = opt.curve_fit(x, ts, xs)
k_opt, x0_opt = popt

print("k = %g" % k_opt)
print("x0 = %g" % x0_opt)
```

```python
import matplotlib.pyplot as plt
t = np.linspace(0, 10, 100)
plt.plot(ts, xs, 'r.', t, x(t, k_opt, x0_opt), '-');
```

### Another example of fitting a system of ODEs using the `lmfit` package

You may have to install the [`lmfit`](http://cars9.uchicago.edu/software/python/lmfit/index.html) package using `pip` and restart your kernel. The `lmfit` algorithm is another wrapper around `scipy.optimize.leastsq` but allows for richer model specification and more diagnostics.

```python
! pip install lmfit
```

```python
from lmfit import minimize, Parameters, Parameter, report_fit
import warnings
```

```python
def f(xs, t, ps):
    """Lotka-Volterra predator-prey model."""
    try:
        a = ps['a'].value
        b = ps['b'].value
        c = ps['c'].value
        d = ps['d'].value
    except:
        a, b, c, d = ps

    x, y = xs
    return [a*x - b*x*y, c*x*y - d*y]

def g(t, x0, ps):
    """
    Solution to the ODE x'(t) = f(t,x,k) with initial condition x(0) = x0
    """
    x = odeint(f, x0, t, args=(ps,))
    return x

def residual(ps, ts, data):
    x0 = ps['x0'].value, ps['y0'].value
    model = g(ts, x0, ps)
    return (model - data).ravel()

t = np.linspace(0, 10, 100)
x0 = np.array([1,1])

a, b, c, d = 3,1,1,1
true_params = np.array((a, b, c, d))

np.random.seed(123)
data = g(t, x0, true_params)
data += np.random.normal(size=data.shape)

# set parameters incluing bounds
params = Parameters()
params.add('x0', value= float(data[0, 0]), min=0, max=10)
params.add('y0', value=float(data[0, 1]), min=0, max=10)
params.add('a', value=2.0, min=0, max=10)
params.add('b', value=2.0, min=0, max=10)
params.add('c', value=2.0, min=0, max=10)
params.add('d', value=2.0, min=0, max=10)

# fit model and find predicted values
result = minimize(residual, params, args=(t, data), method='leastsq')
final = data + result.residual.reshape(data.shape)

# plot data and fitted curves
plt.plot(t, data, 'o')
plt.plot(t, final, '-', linewidth=2);

# display fitted statistics
report_fit(result)
```

#### Optimization of graph node placement

To show the many different applications of optimization, here is an example using optimization to change the layout of nodes of a graph. We use a physical analogy - nodes are connected by springs, and the springs resist deformation from their natural length $l_{ij}$. Some nodes are pinned to their initial locations while others are free to move. Because the initial configuration of nodes does not have springs at their natural length, there is tension resulting in a high potential energy $U$, given by the physics formula shown below. Optimization finds the configuration of lowest potential energy given that some nodes are fixed (set up as boundary constraints on the positions of the nodes).

$$
U = \frac{1}{2}\sum_{i,j=1}^n ka_{ij}\left(||p_i - p_j||-l_{ij}\right)^2
$$

Note that the ordination algorithm Multi-Dimensional Scaling (MDS) works on a very similar idea - take a high dimensional data set in $\mathbb{R}^n$, and project down to a lower dimension ($\mathbb{R}^k$) such that the sum of distances $d_n(x_i, x_j) - d_k(x_i, x_j)$, where $d_n$ and $d_k$ are some measure of distance between two points $x_i$ and $x_j$ in $n$ and $d$ dimension respectively, is minimized. MDS is often used in exploratory analysis of high-dimensional data to get some intuitive understanding of its "structure".

```python
from scipy.spatial.distance import pdist, squareform
```

- P0 is the initial location of nodes
- P is the minimal energy location of nodes given constraints
- A is a connectivity matrix - there is a spring between $i$ and $j$ if $A_{ij} = 1$
- $L_{ij}$ is the resting length of the spring connecting $i$ and $j$
- In addition, there are a number of `fixed` nodes whose positions are pinned.

```python
n = 20
k = 1 # spring stiffness
P0 = np.random.uniform(0, 5, (n,2))
A = np.ones((n, n))
A[np.tril_indices_from(A)] = 0
L = A.copy()
```

```python
L.astype('int')
```

```python
def energy(P):
    P = P.reshape((-1, 2))
    D = squareform(pdist(P))
    return 0.5*(k * A * (D - L)**2).sum()
```

```python
D0 = squareform(pdist(P0))
E0 = 0.5* k * A * (D0 - L)**2
```

```python
D0[:5, :5]
```

```python
E0[:5, :5]
```

```python
energy(P0.ravel())
```

```python
# fix the position of the first few nodes just to show constraints
fixed = 4
bounds = (np.repeat(P0[:fixed,:].ravel(), 2).reshape((-1,2)).tolist() +
          [[None, None]] * (2*(n-fixed)))
bounds[:fixed*2+4]
```

```python
sol = opt.minimize(energy, P0.ravel(), bounds=bounds)
```

#### Visualization

Original placement is BLUE
Optimized arrangement is RED.

```python
plt.scatter(P0[:, 0], P0[:, 1], s=25)
P = sol.x.reshape((-1,2))
plt.scatter(P[:, 0], P[:, 1], edgecolors='red', facecolors='none', s=30, linewidth=2);
```

Optimization of standard statistical models
---

When we solve standard statistical problems, an optimization procedure similar to the ones discussed here is performed. For example, consider multivariate logistic regression - typically, a Newton-like algorithm known as iteratively reweighted least squares (IRLS) is used to find the maximum likelihood estimate for the generalized linear model family. However, using one of the multivariate scalar minimization methods shown above will also work, for example, the BFGS minimization algorithm.

The take home message is that there is nothing magic going on when Python or R fits a statistical model using a formula - all that is happening is that the objective function is set to be the negative of the log likelihood, and the minimum found using some first or second order optimization algorithm.

```python
import statsmodels.api as sm
```

### Logistic regression as optimization

Suppose we have a binary outcome measure $Y \in {0,1}$ that is conditinal on some input variable (vector) $x \in (-\infty, +\infty)$. Let the conditioanl probability be $p(x) = P(Y=y | X=x)$. Given some data, one simple probability model is $p(x) = \beta_0 + x\cdot\beta$ - i.e. linear regression. This doesn't really work for the obvious reason that $p(x)$ must be between 0 and 1 as $x$ ranges across the real line. One simple way to fix this is to use the transformation $g(x) = \frac{p(x)}{1 - p(x)} = \beta_0 + x.\beta$. Solving for $p$, we get
$$
p(x) = \frac{1}{1 + e^{-(\beta_0 + x\cdot\beta)}}
$$
As you all know very well, this is logistic regression.

Suppose we have $n$ data points $(x_i, y_i)$ where $x_i$ is a vector of features and $y_i$ is an observed class (0 or 1). For each event, we either have "success" ($y = 1$) or "failure" ($Y = 0$), so the likelihood looks like the product of Bernoulli random variables. According to the logistic model, the probability of success is $p(x_i)$ if $y_i = 1$ and $1-p(x_i)$ if $y_i = 0$. So the likelihood is
$$
L(\beta_0, \beta) = \prod_{i=1}^n p(x_i)^y(1-p(x_i))^{1-y}
$$
and the log-likelihood is
\begin{align}
l(\beta_0, \beta) &= \sum_{i=1}^{n} y_i \log{p(x_i)} + (1-y_i)\log{1-p(x_i)} \\
&= \sum_{i=1}^{n} \log{1-p(x_i)} + \sum_{i=1}^{n} y_i \log{\frac{p(x_i)}{1-p(x_i)}} \\
&= \sum_{i=1}^{n} -\log 1 + e^{\beta_0 + x_i\cdot\beta} + \sum_{i=1}^{n} y_i(\beta_0 + x_i\cdot\beta)
\end{align}

Using the standard 'trick', if we augment the matrix $X$ with a column of 1s, we can write $\beta_0 + x_i\cdot\beta$ as just $X\beta$.

```python
df_ = pd.read_csv("binary.csv")
df_.columns = df_.columns.str.lower()
df_.head()
```

```python
# We will ignore the rank categorical value

cols_to_keep = ['admit', 'gre', 'gpa']
df = df_[cols_to_keep]
df.insert(1, 'dummy', 1)
df.head()
```

### Solving as a GLM with IRLS

This is very similar to what you would do in R, only using Python's `statsmodels` package. The GLM solver uses a special variant of Newton's method known as iteratively reweighted least squares (IRLS), which will be further desribed in the lecture on multivarite and constrained optimizaiton.

```python
model = sm.GLM.from_formula('admit ~ gre + gpa',
                            data=df, family=sm.families.Binomial())
fit = model.fit()
fit.summary()
```

### Or use R

```python
%load_ext rpy2.ipython
```

```r
%%R -i df
m <- glm(admit ~ gre + gpa, data=df, family="binomial")
summary(m)
```

### Home-brew logistic regression using a generic minimization function

This is to show that there is no magic going on - you can write the function to minimize directly from the log-likelihood equation and run a minimizer. It will be more accurate if you also provide the derivative (+/- the Hessian for second order methods), but using just the function and numerical approximations to the derivative will also work. As usual, this is for illustration so you understand what is going on - when there is a library function available, you should probably use that instead.

```python
def f(beta, y, x):
    """Minus log likelihood function for logistic regression."""
    return -((-np.log(1 + np.exp(np.dot(x, beta)))).sum() + (y*(np.dot(x, beta))).sum())
```

```python
beta0 = np.zeros(3)
opt.minimize(f, beta0, args=(df['admit'], df.loc[:, 'dummy':]), method='BFGS', options={'gtol':1e-2})
```

### Optimization with `sklearn`

There are also many optimization routines in the `scikit-learn` package, as you already know from the previous lectures. Many machine learning problems essentially boil down to the minimization of some appropriate loss function.

### Resources

- [Scipy Optimize reference](http://docs.scipy.org/doc/scipy/reference/optimize.html)
- [Scipy Optimize tutorial](http://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html)
- [LMFit - a modeling interface for nonlinear least squares problems](http://cars9.uchicago.edu/software/python/lmfit/index.html)
- [CVXpy- a modeling interface for convex optimization problems](https://github.com/cvxgrp/cvxpy)
- [Quasi-Newton methods](http://en.wikipedia.org/wiki/Quasi-Newton_method)
- [Convex optimization book by Boyd & Vandenberghe](http://stanford.edu/~boyd/cvxbook/)
- [Nocedal and Wright textbook](http://www.springer.com/us/book/9780387303031)


# The Laplace Approximation
The laplace approximation is a widely used framework that finds a Gaussian approximation to a probability density definted over a set of continuous variables. It is especially useful when applying Bayesian principles to logistic regression where computing integral of posterior distributions becomes intractable.

## Basic Idea
Consider a continuous random variable $z \in \mathcal{R}^D$ with probability distribution given by $p(z) = \frac{1}{Z}f(z)$ where $Z = \int{f(z) dz}$ is the normalizing constant and need not be known.

In the Laplace approximation, the goal is to `find a Gaussian distribution q(z) centered on a mode of the p(z)`. The mode can be computed by determining the value of $z=z_0$ where $\frac{dp(z)}{dz} = 0$.

Note that if $p(z)$ is `multi-modal`, the laplace approximation is only precise in the neighborhood of one of its many modes.

Let $q(z) \sim \mathcal{N}(z_0,A^{-1})$ where $A$ is the precision matrix. Note: Precision matrix is the inverse of covariance matrix and is often employed for computational reasons.

$$ \begin{align} q_z &= \frac{\sqrt{|A|}}{(2\pi)^{D/2}} \exp \{-\frac{1}{2}(z-z_0)^T A (z-z_0)\} \\ \Rightarrow \ln{q_z} &= \frac{1}{2} \left(\ln{|A|} - D \ln{2\pi}\right) - \frac{1}{2}(z-z_0)^T A(z-z_0) \\
&= \ln{f_{z0}} - \frac{1}{2}A(z-z_0)^2\end{align}$$

Note that this is a Taylor series expansion for $p_z$ at a mode where $\frac{d \ln p(z)}{dz} = 0$ and $\frac{d^2 \ln p(z)}{dz^2} = -A < 0 \Rightarrow A > 0$.

In summary, the laplace approximation involves evaluating the mode $z_0$ and the Hessian $A$ at $z_0$. So if f(z) has an intractable but analytical form, the mode can be found by some form of numerical optimization algorithm. Note that the normalization constant $Z$ does not need to be known to apply this method.

## Example
This is an example to demonstrate the Laplace approximation and adapted from Figure 4.14 in [1].

Suppose $p(z) \propto \sigma(20z+4) \exp{\left(\frac{-z^2}{2}\right)}$ where $\sigma(\cdot)$ is the sigmoid function. This form is very common in classification problems and serves as a good practical example.

To compute the mode $z_0$ & Hessian $-A$,

$$ \begin{align} \frac{d}{dz}\ln p_z &\propto \frac{d}{dz}\ln \sigma(\cdot) + \frac{d}{dz}\ln \exp{\left(\frac{-z^2}{2}\right)} \\
&= 20 (1-\sigma(\cdot)) - z \\
&= 0 \text{  iff  } z_0 = 20(1-\sigma(20 z_0 + 4))\end{align}$$

The above expression to determine $z_0$ is nonlinear and can be solved by Newton's method.
Let $y(z_0) = z_0 - 20(1-\sigma(20 z_0 + 4))$. To find $z_0$ such that $y=0$, we start with an initial guess $z_{0,0}$ and iterate the following equation till convergence.
$z_{0,k+1} = z_{0,k} - \left(y'(z_{0,k})\right)^{-1} y(z_{0,k})$. The convergence criteria can be either set to a fixed maximum number of iterations or till $|z_{0,k+1} - z_{0,k}| \le \epsilon$ for some small $\epsilon$.

The Hessian is expressed as:

$$ \begin{align} \frac{d^2}{dz^2}\ln p_z &\propto \frac{d}{dz}\frac{d}{dz}\ln p_z  \\
&= -400\sigma(\cdot)(1-\sigma(\cdot)) - 1 \\
\Rightarrow A &= -\Bigg(\frac{d^2}{dz^2}\ln p_z\Bigg)\Bigg\vert_{z=z_0} = 400\sigma(20 z_0 + 4)(1-\sigma(20 z_0 + 4)) + 1\end{align}$$

```python
import numpy as np
from scipy.integrate import trapz
from scipy.stats import norm
import matplotlib.pyplot as plt
import matplotlib
# matplotlib.rcParams['text.usetex'] = True
# matplotlib.rcParams['text.latex.unicode'] = True
%matplotlib inline

def sigmoid(x):
    den = 1.0+np.exp(-x)
    return 1.0/den

def p_z(z):
    p = np.exp(-np.power(z,2)/2)*sigmoid(20*z+4)
    sum_p = trapz(p,z) ## normalize for plotting
    return p,p/sum_p

def findMode(z_init,max_iter = 25,tol = 1E-6):
    iter = 0
    z_next = np.finfo('d').max
    z_cur = z_init
    while (iter < max_iter and np.abs(z_next-z_cur) > tol):
        if iter > 0:
            z_cur = z_next
        y     = z_cur - 20*(1-sigmoid(20*z_cur+4))
        der_y = 1 + 400*sigmoid(20*z_cur+4)*(1-sigmoid(20*z_cur+4))
        z_next = z_cur - y/der_y
        iter = iter+1
#         print("Iter-"+str(iter)+":"+str(z_next))
    return z_next

def getHessian(z):
    sig_x = sigmoid(20*z+4)
    return 400*sig_x*(1-sig_x) + 1
```

```python
z = np.linspace(-10,10,10000)
pz,pzn = p_z(z)

## Mode & Precision matrix
z0 = findMode(0)
A = getHessian(z0)
z0_idx = np.where(np.abs(z-z0) == np.min(np.abs(z-z0)))[0]
p_z0 = pzn[z0_idx]

dp = np.gradient(pzn,z[1]-z[0])
d2p = np.gradient(dp,z[1]-z[0])

## Get approx Gaussian distribution
q_z = norm.pdf(z, z0, 1/np.sqrt(A))
fig,ax = plt.subplots(1,1,figsize=(4,3))
ax.cla()
ax.plot(z,pzn,color="orange")
ax.fill_between(z,pzn, 0,
                 facecolor="orange", # The fill color
                 color='orange',       # The outline color
                 alpha=0.2)          # Transparency of the fill
#ax.axvline(x=z0)#,ylim=0,ymax=0.7)
ax.vlines(z0, ymin=0, ymax=p_z0,linestyles='dotted')
ax.plot(z,q_z,'r')
ax.set_xlim([-2,4]);
ax.set_ylim([0,0.8]);
ax.set_yticks([0,0.2,0.4,0.6,0.8]);
ax.legend(['p_z','N('+str(np.round(z0,4))+','+str(np.round(1/np.sqrt(A),3))+')'])
ax.set_title('p(z) with its Laplace Approximation');
```

## References
[1]: Bishop, Christopher M. 2006. Pattern Recognition and Machine Learning. Springer.


Consider a markovian process with state spaces {0,1} with the following state-transition matrix:
$\pi =$ \begin{bmatrix}
Pr(s_{t+1} = 0|s_{t}= 0) & Pr(s_{t+1} = 1|s_{t}= 0) \\
Pr(s_{t+1} = 0|s_{t}= 1) & Pr(s_{t+1} = 1|s_{t}= 1)
\end{bmatrix}.

Assume that $Pr(s_{t+1} = 0|s_{t}= 0) = Pr(s_{t+1} = 1|s_{t}= 1) = 2/3$ and $Pr(s_{t+1} = 0|s_{t}= 1) =
Pr(s_{t+1} = 1|s_{t} = 0) = 1/3$. Consider that the initial distribution for the states as: $\Pi_{0} = [5/6 1/6]'$
and answer the following itens:

a) Right a Python function that iterates the states until the stationary state distribution for a given initial condition.
Like: $\Pi_{t+1} = \pi'\Pi_{t}$

Answer:

Importing modules

```python
import numpy as np
from scipy import linalg
```

Defining the state-transition matrix and the initial condition:

```python
T = np.array([[2/3, 1/3], [1/3 , 2/3]])
pi_0 = np.array([[5/6, 1/6]])
```

Creating the function:

```python
#Returns the stationary distribution
def markov_int(T, pi_0):
    pi_t = np.copy(pi_0)
    norm, tol = 1, 1e-6
    while norm > tol:
        pi_t1 = np.dot(pi_t, T)
        norm = np.max(abs(pi_t1 - pi_t))
        pi_t = np.copy(pi_t1)
    return pi_t
```

```python
pi_t = markov_int(T, pi_0)
print(f'The stationary distribution is {pi_t}')
```

    The stationary distribution is [[0.50000021 0.49999979]]

b) Create a function in Python that returns the stationary distribution using the
left normalized eigenvector of $\pi$ associated to the eignvalue $1$.

I will deactivate the warnings because sometimes the function to find the eignvalues/vectors divides by zero

```python
import warnings
warnings.filterwarnings('ignore')
```

```python
# returns the normalized eignvectors and eignvalues
# the first line calculates the eignvector(its normalized already) and eignvalues
# the second line copies the eignvectors
# the third line normalizes it(just because the exercise asks to do it)
def markov_matr(T):
    w, v = linalg.eig(T, left = True, right = False)
    pi = v[:, :]
    pi /= pi.sum(axis = 0)
    return pi, w
```

```python
v, w = markov_matr(T)
print(v)
print(abs(w))
print(v[:,0].reshape(2,1))
print('The first column of the eignvectors matrix is the eignvector associated to the first eignvalue and so on.')
print('The normalized eignvectors matrix is:')
print(f'{v}')
print(f'The eignvalues associated are:')
print(f'{abs(w)}')
```

    [[ 0.5 -inf]
     [ 0.5  inf]]
    [1.         0.33333333]
    [[0.5]
     [0.5]]
    The first column of the eignvectors matrix is the eignvector associated to the first eignvalue and so on.
    The normalized eignvectors matrix is:
    [[ 0.5 -inf]
     [ 0.5  inf]]
    The eignvalues associated are:
    [1.         0.33333333]

Consider a vector of possible incomes $y = [y_{1} \space y_{2}]$ and a vector of three possible values for the assets
$a = [a_{1} \space a_{2} \space a_{3}]$. The joint stationary distribution of endowments of assets and income is given by:
$\phi_{\infty} = [Pr(y_{1}, a_{1}) \space Pr(y_{1}, a_{2}) \space Pr(y_{1}, a_{3}) \space Pr(y_{2}, a_{1})
 \space Pr(y_{2}, a_{2}) \space Pr(y_{2}, a_{3})]$.

Suppose that the policy function is given by $a'(a,y) = a$. Suppose that $Pr(y_{1}, a_{i}) = 0.1$ for all
$i \in {1,2,3}$ and $Pr(y_{2}, a_{1}) = Pr(y_{2}, a_{2}) = 0.2$ and that $Pr(y_{1}, a_{3}) = 0.3$.
Compute the aggregate demand for assets, that is:

\begin{equation}
E[a] = \int a'(a,y)d\phi_{\infty}
\end{equation}

Defining the vectors:

```python
phi = np.array([[.1, .1, .1], [.2, .2, .3]])
```

Note that the initial income(y) and asset(a) distribution were not defined.We need to do it.

Defining state dimensions:

```python
n_y = 2
n_a = 3
```

Defining state space:

```python
y_grid = np.linspace(0,1, n_y)
a_grid = np.linspace(0,1, n_a)
```

Creating a policy function as defined by the exercise:

```python
def f_poli(a_grid):
    a_line = np.zeros((n_y, n_a))
    a_line[0,:] = a_grid
    a_line[1,:] = a_grid

    return a_line
a_line = f_poli(a_grid)
a_line
```

    array([[0. , 0.5, 1. ],
           [0. , 0.5, 1. ]])

Now we need loop through all income and assets(a line) states defined by the policy functio,
but first we will define the initial expected asset distribution as zero and the phi given.

```python
Ea = 0 #Expected asset demand distribution
phi = np.array([.1, .1, .1, .2, .2, .3]) #Creating phi
#Now looping
for i_y in range(n_y):
    for i_a in range(n_a):
        idx = i_y * n_a + i_a
        print(idx)
        Ea += a_line[i_y, i_a]*phi[idx]
print(Ea)
```

    0.55

Markov Chain and Asset Distribution.ipynb


# Graphs and Linear Algebra

Recall that the adjacency matrix of an undirected graph has $A_{i,j} = 1$ if and only if nodes $i$ and $j$ are adjacent. Also, recall that a graph is **regular** with degree $k$ if every node has $k$ neighbors. We also say that the graph is $k$-regular. Finally, for shorthand, we say that the eigenvalues of a graph are the eigenvalues of its adjacency matrix.

**a)** Find three graphs with more than five nodes that are 2-regular, 3-regular, and 4-regular. Represent these in `networkx`, draw them, and find their adjacency matrices. These will be running examples for this problem.

**b)** Find the eigenvalues of the three examples, along with the multiplicities of the eigenvalues.

**c)** Show that if $G$ is $k$-regular, then $k$ is an eigenvalue of $G$.

**d)** Show that $G$ is $k$-regular and connected, then the eigenvalue $k$ of $G$ has multiplicity one.

**e)** Show that $G$ is $k$-regular then $|\lambda|\leq k$ for any eigenvalue $\lambda$ of $G$.

**f)** Let $J$ be the matrix of all ones and $A$ be the adjacency matrix of a $k$-regular graph. Show that $AJ = JA=kJ$.

**g)** Show by construction that there exists regular graph with least eigenvalue equal to $-2$.

**h)** Show that the following graph, called the Petersen Graph, is $3$-regular by finding its eigenvalues.

**i)** Show that if both $n \geq k+1$ and $nk$ is even, then there exists a $k$-regular graph of size $n$.

```python
import networkx as nx
import math
import scipy
import scipy.integrate as spi
import numpy as np
import sympy as sm
sm.init_printing(use_latex='mathjax')
import matplotlib.pyplot as plt
import itertools
import random
%matplotlib inline
```

**a)** Find three graphs with more than five nodes that are 2-regular, 3-regular, and 4-regular. Represent these in networkx, draw them, and find their adjacency matrices. These will be running examples for this problem.

```python
#Ans.a) Representing three graphs with more than five nodes
#that are 2-regular, 3-regular, and 4-regular and drawing them.

#Graph 1 parameters
G = nx.Graph()
G.add_nodes_from([1,2,3,4,5,6,7,8])
G.add_edges_from([(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),(7,8),(8,1)])

#Graph 2 parameters
H = nx.Graph()
H.add_nodes_from([1,2,3,4,5,6,7,8])
H.add_edges_from([(1,2),(2,3),(3,4),(4,5),(5,6),(6,7),
                  (7,8),(8,1),(1,3),(2,4),(5,7),(6,8)])

#Graph 3 parameters
I = nx.Graph()
I.add_nodes_from([1,2,3,4,5,6,7,8])
I.add_edges_from([(1,2),(1,3),(1,7),(1,8),(2,3),(2,4),(2,8),(3,4),
                  (3,5),(4,5),(4,6),(5,6),(5,7),(6,7),(6,8),(7,8)])

#Plotting parameters
basic_graph,ax = plt.subplots(1,3, figsize= (15,5))

nx.draw(G,
        ax=ax[0],
        pos=nx.kamada_kawai_layout(G),
        with_labels=True,
        node_color='#444444',
        font_color="white",
        )
ax[0].set_title('2-regular graph with 8 nodes');

nx.draw(H,
        ax=ax[1],
        pos=nx.kamada_kawai_layout(H),
        with_labels=True,
        node_color='#444444',
        font_color="white",
        )
ax[1].set_title('3-regular graph with 8 nodes');

nx.draw(I,
        ax=ax[2],
        pos=nx.kamada_kawai_layout(I),
        with_labels=True,
        node_color='#444444',
        font_color="white",
        )
ax[2].set_title('4-regular graph with 8 nodes');
```

**Ans.a) To find the adjacency matrix of the graphs drawn:**

Adjacency matrix for the 2-regular graph with 8 nodes:
$$
A_2 = \begin{pmatrix}
0 & 1 & 0 & 0 & 0 & 0 & 0 & 1\\
1 & 0 & 1 & 0 & 0 & 0 & 0 & 0\\
0 & 1 & 0 & 1 & 0 & 0 & 0 & 0\\
0 & 0 & 1 & 0 & 1 & 0 & 0 & 0\\
0 & 0 & 0 & 1 & 0 & 1 & 0 & 0\\
0 & 0 & 0 & 0 & 1 & 0 & 1 & 0\\
0 & 0 & 0 & 0 & 0 & 1 & 0 & 1\\
1 & 0 & 0 & 0 & 0 & 0 & 1 & 0
\end{pmatrix}
$$

Adjacency matrix for the 3-regular graph with 8 nodes:
$$
A_3 = \begin{pmatrix}
0 & 1 & 1 & 0 & 0 & 0 & 0 & 1\\
1 & 0 & 1 & 1 & 0 & 0 & 0 & 0\\
1 & 1 & 0 & 1 & 0 & 0 & 0 & 0\\
0 & 1 & 1 & 0 & 1 & 0 & 0 & 0\\
0 & 0 & 0 & 1 & 0 & 1 & 1 & 0\\
0 & 0 & 0 & 0 & 1 & 0 & 1 & 1\\
0 & 0 & 0 & 0 & 1 & 1 & 0 & 1\\
1 & 0 & 0 & 0 & 0 & 1 & 1 & 0
\end{pmatrix}
$$

Adjacency matrix for the 4-regular graph with 8 nodes:
$$
A_4 = \begin{pmatrix}
0 & 1 & 1 & 0 & 0 & 0 & 1 & 1\\
1 & 0 & 1 & 1 & 0 & 0 & 0 & 1\\
1 & 1 & 0 & 1 & 1 & 0 & 0 & 0\\
0 & 1 & 1 & 0 & 1 & 1 & 0 & 0\\
0 & 0 & 1 & 1 & 0 & 1 & 1 & 0\\
0 & 0 & 0 & 1 & 1 & 0 & 1 & 1\\
1 & 0 & 0 & 0 & 1 & 1 & 0 & 1\\
1 & 1 & 0 & 0 & 0 & 1 & 1 & 0
\end{pmatrix}
$$

**b)** Find the eigenvalues of the three examples, along with the multiplicities of the eigenvalues

```python
#Ans.b) Eigenvalues of the 3 example graphs:
#Representing the adjacent matrix
A_2 = np.matrix([[0 , 1 , 0 , 0 , 0 , 0 , 0 , 1],
               [1 , 0 , 1 , 0 , 0 , 0 , 0 , 0],
               [0 , 1 , 0 , 1 , 0 , 0 , 0 , 0],
               [0 , 0 , 1 , 0 , 1 , 0 , 0 , 0],
               [0 , 0 , 0 , 1 , 0 , 1 , 0 , 0],
               [0 , 0 , 0 , 0 , 1 , 0 , 1 , 0],
               [0 , 0 , 0 , 0 , 0 , 1 , 0 , 1],
               [1 , 0 , 0 , 0 , 0 , 0 , 1 , 0]])

A_3 = np.matrix([[0 , 1 , 1 , 0 , 0 , 0 , 0 , 1],
               [1 , 0 , 1 , 1 , 0 , 0 , 0 , 0],
               [1 , 1 , 0 , 1 , 0 , 0 , 0 , 0],
               [0 , 1 , 1 , 0 , 1 , 0 , 0 , 0],
               [0 , 0 , 0 , 1 , 0 , 1 , 1 , 0],
               [0 , 0 , 0 , 0 , 1 , 0 , 1 , 1],
               [0 , 0 , 0 , 0 , 1 , 1 , 0 , 1],
               [1 , 0 , 0 , 0 , 0 , 1 , 1 , 0]])

A_4 = np.matrix([[0 , 1 , 1 , 0 , 0 , 0 , 1 , 1],
               [1 , 0 , 1 , 1 , 0 , 0 , 0 , 1],
               [1 , 1 , 0 , 1 , 1 , 0 , 0 , 0],
               [0 , 1 , 1 , 0 , 1 , 1 , 0 , 0],
               [0 , 0 , 1 , 1 , 0 , 1 , 1 , 0],
               [0 , 0 , 0 , 1 , 1 , 0 , 1 , 1],
               [1 , 0 , 0 , 0 , 1 , 1 , 0 , 1],
               [1 , 1 , 0 , 0 , 0 , 1 , 1 , 0]])

def Sym(A):
  """
  To check if matrix A is symmetric,
  as to fulfill the property of Adjacency matrix
  """
  B = A.T #Defining a matrix which is transpose of matrix A
  if B.all()==A.all():    #Condition for checking symmetry
    print("Matrix is symmetric")

#To check if the three example matrix are symmetric
Sym(A_2)
Sym(A_3)
Sym(A_4)

#Eigenvalues calculation and rounding off
A2_eig = np.round(np.linalg.eigvals(A_2))
A3_eig = np.round(np.linalg.eigvals(A_3))
A4_eig = np.round(np.linalg.eigvals(A_4))

def Mul(A_eig):
    """
    To get the multiplicity of each eigenvalues in A_eig list
    """
    uni = np.unique(A_eig) #To get the unique eigenvalues
    lst = dict.fromkeys(uni, []) #Extract keys from the dictionary of unique eigenvalues
    for i in range(len(uni)):
        c = 0
        for j in range(len(A_eig)):
            if (A_eig[j] == uni[i]):
              c = c + 1
        lst[uni[i]] = c
    return lst

#Printing the results
print('\nEigenvalues of 2-regular graph:', A2_eig)
print('Multiplicity of eigenvalues of 2-regular graph:',Mul(A2_eig))
print('\nEigenvalues of matrix of 3-regular:', A3_eig)
print('Multiplicity of eigenvalues of 3-regular graph:',Mul(A3_eig))
print('\nEigenvalues of matrix of 4-regular:', A4_eig)
print('Multiplicity of eigenvalues of 4-regular graph:',Mul(A4_eig))
```

    Matrix is symmetric
    Matrix is symmetric
    Matrix is symmetric

    Eigenvalues of 2-regular graph: [-2. -1.  0.  2. -1. -0.  1.  1.]
    Multiplicity of eigenvalues of 2-regular graph: {-2.0: 1, -1.0: 2, 0.0: 2, 1.0: 2, 2.0: 1}

    Eigenvalues of matrix of 3-regular: [-2.  3.  2.  1. -1. -1. -1. -1.]
    Multiplicity of eigenvalues of 3-regular graph: {-2.0: 1, -1.0: 4, 1.0: 1, 2.0: 1, 3.0: 1}

    Eigenvalues of matrix of 4-regular: [ 4. -0. -1.  1.  1. -2. -2. -1.]
    Multiplicity of eigenvalues of 4-regular graph: {-2.0: 2, -1.0: 2, -0.0: 1, 1.0: 2, 4.0: 1}

**c)** Show that if $G$ is $k$-regular, then $k$ is an eigenvalue of $G$.

**Ans.c)**
Let's take an eigenvalue $\lambda$ of graph G whose adjacency matrix is $A_G$ and the corresponding eigenvector be $x = (x_1, x_2, ...,x_n)$.

Now, let's assume $x_k$ is the largest co-ordinate of the vector $x$.

And, $\Delta(G)$ be the maximum degree of nodes in the graph.

Using linear algebra's basic equation, we have,
$$\lambda x = Ax$$
Then, $$=)|\lambda.x_k| = \left|(a_{k1},a_{k2},...,a_{kn})(x_1,x_2...,x_n)\right|$$\
$$=)|\lambda.x_k|=|\lambda||x_k| = \left|\sum_{i=1}^{n} a_{ki}x_i \right|$$

As we know that, $x_k$ is the largest co-ordinate in vector $x$,
$$|\lambda.x_k| = \left|\sum_{i=1}^{n} a_{ki}x_i\right| \leq deg(G) \left|x_k\right| $$

Now, the following conditions,
$$deg(G) = \Delta(G) \; \&\; x_i = x_k \forall i,$$ needs to be true, for the given equality to hold.

We know that, for any k-regular graph,
$$deg(G) = k = \Delta(G)-----(1)$$

Also, for graph G to be regular, (1,1,1,..1) must be an eigenvector of the adjacency matrix $A_G$ which shows that,
$$x_i = x_k \;\forall\; i-----(2)$$

From (1) and (2) we can say that the above conditions for equality holds.
Thus, if $G$ is $k$-regular, then $k$ is an eigenvalue of $G$.

```python
#Ans.c) From Q(a) graphs:
print('Eigenvalues of 2-regular matrix:', A2_eig)
print('Eigenvalues of 3-regular matrix:', A3_eig)
print('Eigenvalues of 4-regular matrix:', A4_eig)
print('\nFrom the above list, its clear that k is present in the eigenvalue list')
print('Thus, if G is k-regular, then k is an eigenvalue of G')
```

    Eigenvalues of 2-regular matrix: [-2. -1.  0.  2. -1. -0.  1.  1.]
    Eigenvalues of 3-regular matrix: [-2.  3.  2.  1. -1. -1. -1. -1.]
    Eigenvalues of 4-regular matrix: [ 4. -0. -1.  1.  1. -2. -2. -1.]

    From the above list, its clear that k is present in the eigenvalue list
    Thus, if G is k-regular, then k is an eigenvalue of G

**d)** Show that $G$ is $k$-regular and connected, then the eigenvalue $k$ of $G$ has multiplicity one.

**Ans.d)**
From (c), we know that for $k$-regular graph with eigenvalue $k$ and eigenvector $x = (x_1,x_2,..,x_n)$ the equality $x_1 = x_2 = ... = x_n$ holds.

Here, $k$ is eigenvalue of the graph $G$.

Now, the space of the eigenvectors of eigenvalue $k$ is of dimension 1 and it proves that $k$ has a multiplicity of 1.

Also, for a graph to be connected, the maximum value of spectral radius must be the eigenvalue of the graph and it satisfies the condition of multiplicity to be 1. Here, for a connected k-regular graph the maximum value of spectral radius is k and thus its multiplicity will be 1.

```python
#Ans.d) From Q(a) graphs:
print('Eigenvalues of 2-regular matrix:', A2_eig)
print('Multiplicty of eigenvalue 2:', Mul(A2_eig)[2])
print('\nEigenvalues of 3-regular matrix:', A3_eig)
print('Multiplicty of eigenvalue 3:', Mul(A3_eig)[3])
print('\nEigenvalues of 4-regular matrix:', A4_eig)
print('Multiplicty of eigenvalue 4:', Mul(A4_eig)[4])
print('\nIf G is k-regular and connected, then the eigenvalue k of G has multiplicity 1')
```

    Eigenvalues of 2-regular matrix: [-2. -1.  0.  2. -1. -0.  1.  1.]
    Multiplicty of eigenvalue 2: 1

    Eigenvalues of 3-regular matrix: [-2.  3.  2.  1. -1. -1. -1. -1.]
    Multiplicty of eigenvalue 3: 1

    Eigenvalues of 4-regular matrix: [ 4. -0. -1.  1.  1. -2. -2. -1.]
    Multiplicty of eigenvalue 4: 1

    If G is k-regular and connected, then the eigenvalue k of G has multiplicity 1

**e)** Show that $G$ is $k$-regular then $|\lambda|\leq k$ for any eigenvalue $\lambda$ of $G$.

**Ans.e)**
Let's take an eigenvalue $\lambda$ of graph G whose adjacency matrix is $A_G$ and the corresponding eigenvector be $x = (x_1, x_2, ...,x_n)$.

Now, let's assume $x_k$ is the largest co-ordinate of the vector $x$.

And, $\Delta(G)$ be the maximum degree of nodes in the graph.

To prove the above statement, we have to show that  $\lambda ≤ Δ(G)$.

Using linear algebra's basic equation, we have,
$$\lambda x = Ax$$
Then, $$=)|\lambda.x_k| = \left|(a_{k1},a_{k2},...,a_{kn})(x_1,x_2...,x_n)\right|$$\
$$=)|\lambda.x_k|=|\lambda||x_k| = \left|\sum_{i=1}^{n} a_{ki}x_i \right|$$

As we know that, $x_k$ is the largest co-ordinate in vector $x$,
$$=)|\lambda.x_k| = \left|\sum_{i=1}^{n} a_{ki}x_i  \right| \leq \left| \sum_{i=1}^{n} a_{ki}x_k\right| $$\
$$=)|\lambda.x_k| \leq \left| \sum_{i=1}^{n} a_{ki}x_k\right| \leq \left| x_k \sum_{i=1}^{n} a_{ki}\right| -----(1)$$

As the graph is $k$-regular, the summation of elements in each row of adjacency matrix will be the $\Delta(G)=k -----(2)$

Using $(1),(2)$ and substitutuing $|x_k| = 1$ in (1), we can prove that, if $G$  is $k$-regular then $|\lambda| \leq  k $ for any eigenvalue $\lambda$ of $G$.

```python
#Ans.e) From Q(a) graphs:
print('Absolute value of Eigenvalues of 2-regular matrix:', np.abs(A2_eig))
print('Maximum value of Eigenvalues of 2-regular matrix:', np.max(np.abs(A2_eig)))
print('\nAbsolute value of Eigenvalues of 3-regular matrix:', np.abs(A3_eig))
print('Maximum value of Eigenvalues of 2-regular matrix:', np.max(np.abs(A3_eig)))
print('\nAbsolute value of Eigenvalues of 4-regular matrix:', np.abs(A4_eig))
print('Maximum value of Eigenvalues of 2-regular matrix:', np.max(np.abs(A4_eig)))

print('\nIt shows if G is k-regular then |λ|≤k for any eigenvalue λ of G')
```

    Absolute value of Eigenvalues of 2-regular matrix: [2. 1. 0. 2. 1. 0. 1. 1.]
    Maximum value of Eigenvalues of 2-regular matrix: 2.0

    Absolute value of Eigenvalues of 3-regular matrix: [2. 3. 2. 1. 1. 1. 1. 1.]
    Maximum value of Eigenvalues of 2-regular matrix: 3.0

    Absolute value of Eigenvalues of 4-regular matrix: [4. 0. 1. 1. 1. 2. 2. 1.]
    Maximum value of Eigenvalues of 2-regular matrix: 4.0

    It shows if G is k-regular then |λ|≤k for any eigenvalue λ of G

**f)** Let $J$ be the matrix of all ones and $A$ be the adjacency matrix of a $k$-regular graph. Show that $AJ = JA=kJ$.

**Ans.f)**
We have the following facts-
1. $G$ is the $k$-regular graph.
2. Adjacency Matrix  $A$ is symmetric.
3. $J$ is a matrix of all ones and is of the same size of $A$ and also symmetric.

As both matrix $A$ and $J$ are symmetric, then their product by interchanging positions will be same. Thus, $$AJ = JA----(1)$$

Now, in $kJ$ resultant matrix, each element will be equal to $k$. From the  definition of $k$, it corresponds to the number of neighbors of each nodes.

Using (1) and comparing it with $kJ$, we have, for every row $J_i$ in matrix $J$, and multipying with column $A_i$ the output will be count of the number of 1's in column $A_i$ of the matix $A$. So, it will be equal to $k$. We can show it as follows-

$$=)J_i.A_i =  (1,1,...,1).(a_{i1},a_{i2},...a_{in})^T $$\
$$=)J_i.A_i =  \sum_{j = 1}^{n}(a_{ij}) $$\
$$=)J_i.A_i =  k $$

This shows that each element of the resultant matrix $AJ$ and $JA$ is equal to $k$. Hence, it proves that, $$AJ = JA = kJ$$

```python
#f)
J = np.ones((4,4)) #To construct a matrix of all ones
A = np.matrix([[0 , 1 , 0 , 1], #To construct a 2-regular adjacency matrix of node 4
               [1 , 0 , 1 , 0],
               [0 , 1 , 0 , 1],
               [1 , 0 , 1 , 0]])
k = np.max(np.round(np.linalg.eigvals(A))) #To get the maximum eigenvalue
a = A*J
b = J*A
c = k*J
display(a)
display(b)
display(c)
if (a.all()==b.all()==c.all()): #Given condition to prove
  print('Given equation holds')
else:
  print('Given equation does not hold')
```

    matrix([[2., 2., 2., 2.],
            [2., 2., 2., 2.],
            [2., 2., 2., 2.],
            [2., 2., 2., 2.]])

    matrix([[2., 2., 2., 2.],
            [2., 2., 2., 2.],
            [2., 2., 2., 2.],
            [2., 2., 2., 2.]])

    array([[2., 2., 2., 2.],
           [2., 2., 2., 2.],
           [2., 2., 2., 2.],
           [2., 2., 2., 2.]])

    Given equation holds

**g)** Show by construction that there exists regular graph with least eigenvalue equal to  −2 .

```python
#Ans.g) #Graph parameters to draw a 2-regular graph of node 5
G = nx.Graph()
G.add_nodes_from([1,2,3,4,5])
G.add_edges_from([(1,2),(2,3),(3,4),(4,5),(5,1)])

#Plotting parameters
basic_graph,ax = plt.subplots(1,1, figsize= (5,5))
nx.draw(G,
        ax=ax,
        pos=nx.kamada_kawai_layout(G),
        with_labels=True,
        node_color='#444444',
        font_color="white",
        )
ax.set_title('2-regular graph with 5 nodes');
```

```python
#Ans.g)
#Constructing the adjacency matrix of the above graph
A_g = np.matrix([[0 , 1 , 0 , 0 , 1],
               [1 , 0 , 1 , 0 , 0],
               [0 , 1 , 0 , 1 , 0],
               [0 , 0 , 1 , 0 , 1],
               [1 , 0 , 0 , 1 , 0]])

Sym(A_g) #To check if the matrix is symmetric

#To calculate the eigenvalue and find the minimum
Ag_eig = np.round(np.linalg.eigvals(A_g))
print('Eigenvalues:',Ag_eig)
Ag_eig_min = np.min(Ag_eig)
print('Minimum eigenvalue:', Ag_eig_min)
print('It shows that the minimum eigenvalue of the regular graph is -2')
```

    Matrix is symmetric
    Eigenvalues: [-2.  1.  2. -2.  1.]
    Minimum eigenvalue: -2.0
    It shows that the minimum eigenvalue of the regular graph is -2

**h)** Show that the following graph, called the Petersen Graph, is $3$-regular by finding its eigenvalues.

```python
#Ans.h) Petersen Graph
#Graph parameters to draw Petersen Graph
G = nx.Graph()
G.add_nodes_from([1,2,3,4,5,6,7,8,9,10])
G.add_edges_from([(1,2),(1,5),(1,6),(2,3),(2,7),(3,4),(3,8),(4,5),
                  (4,9),(5,10),(6,8),(6,9),(7,9),(7,10),(8,10)])

#Plotting parameters
basic_graph,ax = plt.subplots(1,1, figsize= (5,5))
seq = [[10,6,7,8,9],[1,2,3,4,5]]
nx.draw(G,
        ax=ax,
        pos=nx.shell_layout(G,seq),
        with_labels=True,
        node_color='#444444',
        font_color="white",
        )
ax.set_title('Petersen Graph');
```

```python
#Ans.h)
#Representing the adjacency matrix for Petersen Graph
A_h = np.matrix([[0,1,0,0,1,1,0,0,0,0],
                [1,0,1,0,0,0,1,0,0,0],
                [0,1,0,1,0,0,0,1,0,0],
                [0,0,1,0,1,0,0,0,1,0],
                [1,0,0,1,0,0,0,0,0,1],
                [1,0,0,0,0,0,0,1,1,0],
                [0,1,0,0,0,0,0,0,1,1],
                [0,0,1,0,0,1,0,0,0,1],
                [0,0,0,1,0,1,1,0,0,0],
                [0,0,0,0,1,0,1,1,0,0]])

#To find the eigenvalues and the maximum of them
Ah_eig = np.round(np.linalg.eigvals(A_h))
print('Eigenvalues:',Ah_eig)
Ah_eig_max = np.max(Ah_eig)
print('Maximum eigenvalue:', Ah_eig_max)
print('As maximum eigenvalue is 3, thus Petersen Graph is 3-regular graph')
```

    Eigenvalues: [-2.  1.  3. -2. -2.  1. -2.  1.  1.  1.]
    Maximum eigenvalue: 3.0
    As maximum eigenvalue is 3, thus Petersen Graph is 3-regular graph

**i)**
Show that if both  n≥k+1  and  nk  is even, then there exists a  k -regular graph of size  n .

**Ans.i)**
As $nk$ is even, and if $n$ depends on $k$, we have following cases:
1. $n$ = 0 or $k$ = 0
2. if $k$ is even then $n$ is even
3. if $k$ is odd then $n$ is even or odd

Considering the above options, we have the following cases-

**i)**
If $n = 0$

Then a graph doesn't exit. Also, it means that $k$ will be negative.This is not possible.

If $k = 0$

Then $n \geq 1$, and the graph will have nodes without any neighbors. This type of graphs are called $0$-regular graphs. Thus, a $k$-regular graph of size $n$ exists.

**ii)**
If $k$ is even and $k \geq 2$

Let's assume that $G_{n}(k)$ is a connected $k$-regular graph with a vertex set of $\{1,2,3,...,n\}$, for some $n \geq k+1 $.

Now, by using the property that if minimum degree of a vertex in a graph $X$ is $\delta \geq 2$, then there exists a path in $G$ containing $\delta$ edges.

Let's consider, path $P$ to be a path in $G_{n}(k)$ containing $k$ edges. As $k$ is even, there will be $\frac{k}{2}$ vertex disjoint edges, $e_i = (u_i,v_i), 1\leq i \leq \frac{k}{2}$ in the path $P$.

Thus, the set, $\bigcup_{1\leq i \leq \frac{k}{2}} \left\{ u_i,v_i \right\}\;$ has k distict verices.

By removing the edges, $e_i$, $1\leq i\leq \frac{k}{2}$ and adding $k$ new edges, the new set becomes-

$$\bigcup_{i=1}^{\frac{k}{2}}\left\{(n+1,u_i)\right\} \bigcup \left\{(n+1,v_i)\right\} $$

The resulting graph is a connected $k$-regular graph with vertices $\{1,2,...,n+1\}$ and is denoted as $G_{n+1}(k)$ graph.

**iii)**
If $k$ is odd,

And if $k = 1$, then, $n \geq 2$ and has to be even. We can denote it as-

$n = 2z,$ where $z$ is natural number.

As number of nodes are always even, they can be distributed in two disjoint sets and each node can be connected one element of set A and to exactly one element of set B.

For $k \geq 3$

Let's assume that $G_{n}(k)$ is a connected $k$-regular graph with a vertex set of $\{1,2,3,...,n\}$, for some even $n \geq k+1 $.
The resulting graph is a connected $k$-regular graph with vertices $\{1,2,...,n+1\}$ and is denoted as $G_{n+1}(k)$ graph.

Now, by using the property that if minimum degree of a vertex in a graph $X$ is $\delta \geq 2$, then there exists a path in $G$ containing $\delta$ edges.

Let's consider, path $Q_{n}(k) = (q_1,q_2,...,q_{k+1})$ to be a path in $G_{n}(k)$ containing $k$ edges as $\{(q_j,q_{j+1})\}_{1\leq j \leq k}$.

By removing $k-1$ edges, $\{(q_j,q_{j+1})\}_{1\leq j \leq k-1}$ and adding the following edges-

1. For $1\leq j \leq k-1, j $ odd, add the edges $\{(n+1, q_j),(n+1, q_{j+1})\}$

2. For $1\leq j \leq k-1, j $ even, add the edges $\{(n+2, q_j),(n+2, q_{j+1})\}$

3. And add the edge $(n+1, n+2)$

Since k is odd, the total number of edges added to $1$ is $k-1$ and thus, there are $k-1$ edges with $n+1$ as the end vertex after $1$. Similarly, after step $2\;$ there are $k-1$ edges with $n+2$ as the end vertex and the resulting graph, is a connected $k$-regular graph with vertex set $\{1,2,...,n+1,n+2\}$ and denoted as $G_{n+2}(k)$.

Thus, from the above cases we can state that if $nk$  is even and  $n≥k+1$, then there exists a  $k$-regular graph of size  $n$.

**Reference:** https://arxiv.org/abs/1801.08345, https://www.math.kit.edu/iag6/lehre/graphtheo2013w/media/how_to_write_a_proof.pdf

Graphs and Linear Algebra/Graphs and Linear Algebra.ipynb


# Logistic Regression

```python
import numpy as np
from matplotlib import pyplot as plt
from scipy.optimize import minimize
from sklearn.preprocessing import PolynomialFeatures
from scipy import optimize

```

In this part, we will solve a`classification problem`.
Classification ploblems are just like the regression problems except that the values we want to predict is only a small number of discrete values.

### Examples:
- spam / not spam
- Tumor:Malignant/Benign

---------------
## Part One
---------------

#### Firstly, we will define the Basic equations of Logistic Regression model.

### 1- Hypothesis

### $ h_\theta(x) = g(\theta^T x)$
        where   0 <$ h_\theta(x) <1 \\ $

### $g(\theta^T x)$= $\frac{1}{1-e^(\theta^T x)}$

And this's what called `sigmoid fuction`, or logistic function.

The concept of hypothesis also can be described by conditional probability:

### $ h_\theta(x) = P(y=1 | x;\theta)$

### $So, when$    $ h_\theta(x) = P(y=1 | x;\theta) =0.7, then$   $ h_\theta(x) = P(y=0 | x;\theta) = 0.3$

```python
def sigmoid(z):
    return(1 / (1 + np.exp(-z)))

```

```python
x = np.array([np.arange(-10., 10., 0.2)])
sig = sigmoid(x)

fig = plt.figure(figsize=(15,8))
fig.suptitle('Sigmoid Function ', fontsize=14, fontweight='bold')

ax = fig.add_subplot(111)
fig.subplots_adjust(top=0.85)
ax.set_title('axes title')

ax.set_xlabel('h(x)')
ax.set_ylabel('y')
ax.plot(x, sig, 'o')

ax.text(5, 1.5, 'y=1 if h(x)>0.5', style='italic',bbox={'facecolor':'blue', 'alpha':0.6, 'pad':20})
ax.text(-5, 0.5, 'y=0 if h(x)<0.5', style='italic',bbox={'facecolor':'blue', 'alpha':0.6, 'pad':20})

plt.axhline(0, color='black')
plt.axvline(0, color='black')

ax.axis([-10, 10, -0.5, 2])
ax.grid(True)
plt.show()

```

### 2- Cost Function
The logistic model's cost function is different than linear regression one, because if we apply the same equation, the cost function will be non-convex.

so, the logistic regression new cost function is:

### $\begin{equation}
  cost(h_\theta(x),y) =
    \begin{cases}
        \text{ -log($h_\theta(x)$) ... if y=1}\\
        \text{-log($1-h_\theta(x)$) ... if y=0}\\
    \end{cases}
\end{equation}$

##### but the simple version written as :

### $ cost(h_\theta(x),y) = -y log(h_\theta(x)) - (1-y) log(1- h_\theta(x)) $

##### So, now we can write the overall structure of the equation:

### $ J(\theta) = \frac{-1}{m} \sum_{i=1}^{m} cost(h_\theta(x),y) $

```python
def costFunction(theta,x,y):
    m= y.size
    h= sigmoid(x.dot(theta))
    cost_= (-1/m)*(y.dot(np.log(h)) + (1-y).dot(np.log(1-h)))

    return cost_
```

### 3-Gradient Decsent / Minimization Function

```python
def gradientDescent(theta, x, y):
    m=y.size
    z = np.dot(x,theta)
    h= sigmoid(z)
    error = h.T - y
    grad = (1 / m) * np.dot(error, x)
    return grad
def optimization (theta,x,y):
    return minimize(costFunction, theta, args=(x,y), method=None, jac=gradientDescent, options={'maxiter':400})

```

### Now, lets start solving the exercise ...

Suppose that you are the administrator of an university department and you want to determine each applicant’s chance of admission based on their results on two exams.

So, we aim to build a classification model that estimates an applicant’s probability of admission based the scores from those two exams.

##### a- Uploading and Visualization

```python
data = np.loadtxt('Datasets/ex2data1.txt', delimiter=',')
X = np.array(data[:,0:2])
X = np.insert(X,0,1,axis=1)
y= data[:,2].T
theta =np.zeros([X.shape[1],1])
```

```python
def plotData(data, label_x, label_y, label_pos, label_neg, axes=None):
    # Get indexes for class 0 and class 1
    neg = data[:,2] == 0
    pos = data[:,2] == 1

    # If no specific axes object has been passed, get the current axes.
    if axes == None:
        axes = plt.gca()
    axes.scatter(data[pos][:,0], data[pos][:,1], marker='+', c='k', s=60, linewidth=2, label=label_pos)
    axes.scatter(data[neg][:,0], data[neg][:,1], c='y', s=60, label=label_neg)
    axes.set_xlabel(label_x)
    axes.set_ylabel(label_y)
    axes.legend(frameon= True, fancybox = True);

```

```python
plotData(data, 'Exam 1 score', 'Exam 2 score', 'Admitted', 'Not admitted')
```

#### b- Fit the parameters theta to estimate the decision boundary line equation
Now, lets us firstly compare between the cost of using gradient descent and the minimize function, then plot the decision boundary.

```python
grad_theta = gradientDescent(theta,X,y)
opt_theta= optimization(theta, X,y).x

print ('The fitted parameters', grad_theta[0,:], 'have a cost', costFunction(theta,X,y))

print ('The fitted parameters', opt_theta, 'have a cost', costFunction(opt_theta,X,y))

```

    The fitted parameters [ -0.1        -12.00921659 -11.26284221] have a cost [0.69314718]
    The fitted parameters [-25.16133284   0.2062317    0.2014716 ] have a cost 0.20349770158944375

    /opt/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:4: RuntimeWarning: divide by zero encountered in log
      after removing the cwd from sys.path.

```python
plotData(data, 'Exam 1 score', 'Exam 2 score', 'Admitted', 'Not admitted')
x1_min, x1_max = X[:,1].min(), X[:,1].max(),
x2_min, x2_max = X[:,2].min(), X[:,2].max(),
xx1, xx2 = np.meshgrid(np.linspace(x1_min, x1_max), np.linspace(x2_min, x2_max))
h = sigmoid(np.c_[np.ones((xx1.ravel().shape[0],1)), xx1.ravel(), xx2.ravel()].dot(opt_theta))
h = h.reshape(xx1.shape)
plt.contour(xx1, xx2, h, [0.5])
```

#### c- Prediction:

As we know from the sigmoid function, y=1 when $h_\theta(x)$ > 0.5, and vise versa.

so from this rule, we can predict where the new inputs are belong, based on the new fitted parameters $\theta$

```python
def prediction(theta, x):
    threshold=0.5
    result= sigmoid(np.dot(x,theta.T)) >= threshold
    return result.astype('int')
```

#### d- Case Study

```python
"""""
We'll assume that we have two students, the first student's score at the first exam= 30, and the second exam = 50
while the second student's score at the first exam= 40, and the second exam = 100
"""""
case1= np.array([1,30,50])
case2= np.array([1,40,100])

def result(result):
    if result == 0:
        result='not admitted'
    else:
        result= 'admitted'
    return result

print( 'case 1 -with score1 = 30 and score2=50- is ', result(prediction(opt_theta,case1)))
print( 'case 2 -with score1 = 40 and score2=100- is ', result(prediction(opt_theta,case2)))
```

    case 1 -with score1 = 30 and score2=50- is  not admitted
    case 2 -with score1 = 40 and score2=100- is  admitted

```python
plotData(data, 'Exam 1 score', 'Exam 2 score', 'Admitted', 'Not admitted')
plt.scatter(case1[1], case1[2], s=100, c='g', marker='x', label='case1')
plt.scatter(case2[1], case2[2], s=100, c='r', marker='v', label='case2')
plt.legend(frameon= True, fancybox = True);

x1_min, x1_max = X[:,1].min(), X[:,1].max()
x2_min, x2_max = X[:,2].min(), X[:,2].max()
xx1, xx2 = np.meshgrid(np.linspace(x1_min, x1_max), np.linspace(x2_min, x2_max))
h = sigmoid(np.c_[np.ones((xx1.ravel().shape[0],1)), xx1.ravel(), xx2.ravel()].dot(opt_theta))
h = h.reshape(xx1.shape)
plt.contour(xx1, xx2, h, [0.5])
```

---------------
## Part Two
---------------

### Overfitting problem

If we have too many features, the learned hypothesis may fit the training set very well, with almost zero cost. but fail to generalize to new examples.
otherwise, if we have a lot of features, and very little training data, overfitting can occurs.

- ###### Adressing overfitting:

1- Reduce number of features

2- Regularization

### Regularization Method

The prupose of regularization, is to reduce the affect -or the wieght- of theta parameters, by adding thetas with reverse signs, and multiplying them by a specific factor.

### $ J(\theta) = \frac{-1}{m}[\sum_{i=1}^{m}-y log(h_\theta(x)) - (1-y) log(1- h_\theta(x))]  +  \frac{\gamma }{2m} [\sum_{i=1}^{n} \theta_j^2]$

```python
def Reg_costFunction(theta,x,y, regPar):
    m= y.size
    h= sigmoid(x.dot(theta))
    cost_= (-1/m)*(y.dot(np.log(h)) + (1-y).dot(np.log(1-h)))+(regPar/(2*m))*np.sum(np.square(theta[1:]))

    return cost_
```

#### And we should apply the same rule on gradient descent and/or minimization functions

```python
def Reg_gradientDescent(theta, x, y, regPar ):
    m=y.size
    z = np.dot(x,theta)
    h= sigmoid(z)
    error = h.T - y
    grad = (1 / m) * np.dot(error, x)
    reg_term= regPar/(m)*np.sum(np.square(theta[1:]))
    reg_grad= grad + reg_term

    return reg_grad

def optimization (theta,x,y, regPar):
    result = optimize.minimize(Reg_costFunction, theta, args=(x, y, regPar),  method='BFGS', options={"maxiter":500, "disp":False} )
    return result

```

### *The Exercise:*
we will implement regularized logistic regression; to predict whether microchips from a fabrication plant passes quality assurance (QA).

Suppose you are the product manager of the factory and you have the
test results for some microchips on two different tests. From these two tests,
you would like to determine whether the microchips should be accepted or
rejected. To help you make the decision, you have a dataset of test results
on past microchips, from which you can build a logistic regression model.

```python
data2 = np.loadtxt('Datasets/ex2data2.txt', delimiter=',')
X2 = np.array(data2[:,0:2])
ones= np.ones([data2.shape[0]])
X2 = np.insert(X2,0,1,axis=1)
y2 = data2[:,2].T
theta2 = np.zeros([X2.shape[1],1])
```

```python
plotData(data2, 'Exam 1 score', 'Exam 2 score', 'Admitted', 'Not admitted')
```

#### Because the distribution of the data implies that it needs a polynomial(non-linear) decision boundary, such as:
### $ g(\theta_0 + \theta_1 + \theta_1^2+\theta_1 \theta_2+ \theta_2^3 ...)$
#### we need to apply the following function:

```python
def map_feature( feature1, feature2 ):

    degrees = 6
    out = np.ones( (feature1.shape[0], 1) )

    for i in range(1,degrees+1):
        for j in range(0,i+1):
            term1 = feature1 ** (i-j)
            term2 = feature2 ** (j)
            term  = (term1 * term2).reshape( term1.shape[0], 1 )
            out   = np.hstack(( out, term ))
    return out
```

```python
theta2 = np.zeros(XX.shape[1])
cost= Reg_costFunction(theta2,XX, y2, 1)
print ('The cost value =', cost)
```

    The cost value = 0.6931471805599453

## Non-linear Decision Boundary

```python
XX= map_feature(X2[:,1], X2[:,2])
theta2 = np.zeros([XX.shape[1],1])

num=[0,1,50,100]
thetas= np.zeros([theta2.shape[0],4])

for i in range(len(num)):
    op_theta= optimization(theta2.T, XX, y2 ,num[i]).x
    thetas[:,i]= op_theta

def Reg_Boundary(theta):
    xvals = np.linspace(-1,1.5,50)
    yvals = np.linspace(-1,1.5,50)
    zvals = np.zeros((len(xvals),len(yvals)))

    for i in range(len(xvals)):
        for j in range(len(yvals)):
            myfeaturesij = map_feature(np.array([xvals[i]]),np.array([yvals[j]]))
            zvals[i][j] = np.dot(theta,myfeaturesij.T)
    zvals = zvals.transpose()

    u, v = np.meshgrid( xvals, yvals )
    plt.contour( xvals, yvals, zvals, [0])
```

```python
plt.figure(figsize=(12,10))
plt.subplot(221)
plotData(data2, 'Microchip Test 1', 'Microchip Test 2', 'y = 1', 'y = 0')
plt.title('Lambda = %d'%num[0])
Reg_Boundary(thetas[:,0])

plt.figure(figsize=(12,10))
plt.subplot(222)
plotData(data2, 'Microchip Test 1', 'Microchip Test 2', 'y = 1', 'y = 0')
plt.title('Lambda = %d'%num[1])
Reg_Boundary(thetas[:,1])

plt.figure(figsize=(12,10))
plt.subplot(223)
plotData(data2, 'Microchip Test 1', 'Microchip Test 2', 'y = 1', 'y = 0')
plt.title('Lambda = %d'%num[2])
Reg_Boundary(thetas[:,2])

plt.figure(figsize=(12,10))
plt.subplot(224)
plotData(data2, 'Microchip Test 1', 'Microchip Test 2', 'y = 1', 'y = 0')
plt.title('Lambda = %d'%num[3])
Reg_Boundary(thetas[:,3])
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


# XGBoost & LightGBM

## XGBoost

mathematically, we can write our ensemble tree model in the form

$$\hat{y}_{i} = \sum_{k=1}^{K}f_{k}(x_{i}), f_{k}\in\mathcal{F}$$

where $K$ is the number of trees, $f$ is a function in the function space $\mathcal{F}$, and $\mathcal{F}$ is the set of all possible CARTs.

the objective function to be optimized is given by

$$\mbox{obj}(\theta) = \sum_{i=1}^{n}l(y_{i}, \hat{y}_{i}) + \sum_{k=1}^{K}\Omega(f_{k})$$

where $l$ is the loss term, $\Omega$ is the regularization term.

### Additive Learning

what are the parameters of trees? you can find that what we need to learn are those functions $f_{i}$, each containing the structure of the tree and the leaf scores.

learning tree strcture is much harder than traditional optimization problem where you can simply take the gradient, instead, we use an additive strategy: fix what we learned, and add one new tree at a time. we write the prediction value at step $t$ as $\hat{y}_{i}^{(t)}$, then

$$
\begin{equation}
\begin{split}
\mbox{obj}^{(t)} =& \sum_{i=1}^{n}l(y_{i}, \hat{y}_{i}^{(t)}) + \sum_{k=1}^{t}\Omega(f_{k}) \\
=& \sum_{i=1}^{n}l(y_{i}, \hat{y}_{i}^{(t - 1)} + f_{t}(x_{i})) + \Omega(f_{t}) + \mbox{constant}
\end{split}
\end{equation}
$$

$$\hat{y}_{i}^{(0)} = 0$$

in general case($l$ arbitrary), we take the taylor expansion of the loss function up to the second order

$$\mbox{obj}^{(t)} \approx \sum_{i=1}^{n}[l(y_{i}, \hat{y}_{i}^{(t - 1)}) + g_{i}f_{t}(x_{i}) + \frac{1}{2}h_{i}f_{t}(x_{i})^{2}] + \Omega(f_{t}) + \mbox{constant}$$

where $g_{i}$ and $h_{i}$ are defined as

$$g_{i} = \frac{\partial l(y_{i}, \hat{y}_{i}^{(t - 1)})}{\partial \hat{y}_{i}^{(t-1)}}, h_{i} = \frac{\partial^{2} l(y_{i}, \hat{y}_{i}^{(t - 1)})}{{\partial \hat{y}_{i}^{(t-1)}}^2}$$

after removing all the constants, the specific objective at step $t$ becomes

$$\sum_{i=1}^{n}[g_{i}f_{t}(x_{i}) + \frac{1}{2}h_{i}f_{t}(x_{i})^{2}] + \Omega(f_{t})$$

this becomes our optimization goal for the new tree.

### Model Complexity

we need to define the complexity of the tree $\Omega(f)$, in order to do so, let us first refine the definition of the tree $f(x)$ as

$$f(x) = w_{q(x)}, w \in \mathbb{R}^{T}, q: \mathbb{R}^{d} \to \{1,2,...,T\}$$

where $w$ is the vector of scores on leaves, $q$ is a function assigning each data point to the corresponding leaf, and $T$ is the number of leaves. in XGBoost, we define the complexity as

$$\Omega(f) = \gamma{T} + \frac{1}{2}\lambda\sum_{j=1}^{T}w_{j}^{2}$$

this works well in practice.

### The Structure Score

now we can write the objective value with the $t$-th tree as:

$$
\begin{equation}
\begin{split}
\mbox{obj}^{(t)} = & \sum_{i=1}^{n}[g_{i}f_{t}(x_{i}) + \frac{1}{2}h_{i}f_{t}(x_{i})^{2}] + \gamma{T} + \frac{1}{2}\lambda\sum_{j=1}^{T}w_{j}^{2} \\
=& \sum_{j=1}^{T}[(\sum_{i\in{I_{j}}}g_{i})w_{j} + \frac{1}{2}(\sum_{i\in{I_j}}h_{i} + \lambda)w_{j}^2] + \gamma{T}
\end{split}
\end{equation}
$$

where $I_{j} = \{i|q_{i}=j\}$ is the set of indices of data-points assign to the $j$-th leaf.

we could further compress the expression by defining $G_{j} = \sum_{i\in{I_{j}}}g_{i}, H_{j} = \sum_{i\in{I_j}}h_{i}$:

$$\mbox{obj}^{(t)} = \sum_{j=1}^{T}[G_{j}w_{j} + \frac{1}{2}(H_{j} + \lambda)w_{j}^2] + \gamma{T}$$

in this equation, $w_{j}$ are independent with respect to each other, the form $G_{j}w_{j} + \frac{1}{2}(H_{j} + \lambda)w_{j}^2$ is quadratic and the best $w_{j}$ for a given $q(x)$ and the best objective reduction we can get is:

$$w_{j}^{\ast} = -\frac{G_{j}}{H_{j} + \lambda}$$

$$\mbox{obj}^{\ast} = -\frac{1}{2}\sum_{i=1}^{T}\frac{G_{j}^2}{H_{j} + \lambda} + \gamma{T}$$

the last equation measures how good a tree structure $q(x)$ is.

### Learn the Tree Structure

now we have a way to measure how good a tree is, ideally we would enumerate all possible trees and pick the best one, but not practical.

instead we will try to optimize **one level** of the tree at a time, specifically we try to split a leaf into two leaves, and the score gain is:

$$Gain = \frac{1}{2}\left [\frac{G_L^2}{H_L + \lambda} + \frac{G_R^2}{H_R + \lambda} - \frac{(G_L + G_R)^2}{H_L + H_R + \lambda}\right ] - \gamma$$

if the first part of $Gain$ is smaller than $\gamma$, we would do better not add that branch, this is exactly pruning!

for real valued data, we places all instances in sorted order(by the split feature), then a left to right scan is sufficient to calculate the structure score of all possible split solutions, and we can find the best split efficiently.

in practice, since it is intractable to enumerate all possible tree structures, we add one split at a time, this approach works well at most of the time.

### XGBoost Practice

```python
"""quadratic dataset"""
import numpy as np
from sklearn.model_selection import train_test_split

np.random.seed(42)
X = np.random.rand(100, 1) - 0.5
y = 3*X[:, 0]**2 + 0.05 * np.random.randn(100)
X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=42)
```

```python
"""basic xgboost"""
import xgboost
from sklearn.metrics import mean_squared_error

xgb_reg = xgboost.XGBRegressor()
xgb_reg.fit(X_train, y_train)
y_pred = xgb_reg.predict(X_val)
mean_squared_error(y_pred, y_val)
```

    0.0030701301701716146

```python
"""xgboost automatically taking care of early stopping"""
xgb_reg.fit(X_train, y_train, eval_set=[(X_val, y_val)], early_stopping_rounds=2)
y_pred = xgb_reg.predict(X_val)
mean_squared_error(y_pred, y_val)
```

    [0]	validation_0-rmse:0.19678
    [1]	validation_0-rmse:0.14325
    [2]	validation_0-rmse:0.10835
    [3]	validation_0-rmse:0.08482
    [4]	validation_0-rmse:0.07044
    [5]	validation_0-rmse:0.06255
    [6]	validation_0-rmse:0.05927
    [7]	validation_0-rmse:0.05698
    [8]	validation_0-rmse:0.05519
    [9]	validation_0-rmse:0.05513
    [10]	validation_0-rmse:0.05473
    [11]	validation_0-rmse:0.05463
    [12]	validation_0-rmse:0.05427
    [13]	validation_0-rmse:0.05376
    [14]	validation_0-rmse:0.05377
    [15]	validation_0-rmse:0.05363
    [16]	validation_0-rmse:0.05358
    [17]	validation_0-rmse:0.05387

    0.0028706534131390338

## LightGBM

XGBoost uses level-wise tree growth:

while LightGBM uses leaf-wise tree growth:

we can formalize leaf-wise tree growth as:

$$(p_m, f_{m}, v_{m}) = \underset{(p,f,v)}{argmin}\ L(T_{m-1}(X).split(p, f, v), Y)$$

$$T_{m}(X) = T_{m-1}(X).split(p_m, f_{m}, v_{m})$$

finding the best split is costly, in XGBoost, we enumerate all features and all thresholds to find the best split.

LightGBM optimize that by using the histogram algorithm:

```python

```


```python
from sklearn import metrics
import numpy as np
import sympy
from sympy.matrices import Matrix
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
```

```python
y_true = np.round(np.random.sample(10))
y_pred = np.round(np.random.sample(10))
equals = 0
for true, pred in zip(y_true, y_pred):
  if true == pred: equals +=1
equals
```

    5

```python
# Alternative way of 0-1 sample generation
n, p = 1, .5  # number of trials, probability of each trial
s = np.random.binomial(n, p, size=10)
s
```

    array([0, 0, 0, 1, 0, 0, 0, 0, 0, 1])

Try ours from the quiz

```python
y_true = np.array([1, 1, 0, 1, 1, 1])
y_pred = np.array([1, 1, 1, 1, 0, 0])

```

```python
df = pd.DataFrame(data={"y_true": y_true, "y_pred": y_pred})
df.transpose()
```

      0
      1
      2
      3
      4
      5

      y_true
      1
      1
      0
      1
      1
      1

      y_pred
      1
      1
      1
      1
      0
      0

```python
allCount = len(y_true)
oneCount = int(np.sum(y_true))
zeroCount = allCount - oneCount
allCount, oneCount, zeroCount
```

    (6, 5, 1)

```python

```

```python
cm = metrics.confusion_matrix(y_true, y_pred)
cm
```

    array([[0, 1],
           [2, 3]])

```python
display = metrics.ConfusionMatrixDisplay(cm, ["0's or negative", "1's or positive"])
display.plot()
plt.show()
```

```python
metrics.accuracy_score(y_true, y_pred)
```

    0.5

## By hand, in Python terminology (non-statistical)
Basic cells of confusion matrix.

```python
(TN, FP), (FN, TP) = cm
print("TN =", TN)
print("FP =", FP)
print("FN =", FN)
print("TP =", TP)
```

    TN = 0
    FP = 1
    FN = 2
    TP = 3

Marginals

```python

```

```python
diagonal = TP +  TN
print("diagonal =", diagonal)
all = TP + TN + FP + FN
print("all =", all)
accByFormula = diagonal / all
print("accByFormula =", accByFormula)
```

    diagonal = 3
    all = 6
    accByFormula = 0.5

```python
metrics.precision_score(y_true, y_pred)
```

    0.75

```python
metrics.recall_score(y_true, y_pred)
```

    0.6

```python
%%script false
plt.plot(*metrics.precision_recall_curve(y_true, y_pred))
plt.show()
```

```python
metrics.f1_score(y_true, y_pred)
```

    0.6666666666666665

```python
print(metrics.classification_report(y_true, y_pred))
```

                  precision    recall  f1-score   support

               0       0.00      0.00      0.00         1
               1       0.75      0.60      0.67         5

        accuracy                           0.50         6
       macro avg       0.38      0.30      0.33         6
    weighted avg       0.62      0.50      0.56         6

```python

```

------------------

```python
y_true = np.array([1, 1, 0, 1, 1, 1])
y_pred = np.array([1, 1, 1, 1, 0, 0])
```

```python
print(metrics.classification_report(y_true, y_pred))
```

                  precision    recall  f1-score   support

               0       0.00      0.00      0.00         1
               1       0.75      0.60      0.67         5

        accuracy                           0.50         6
       macro avg       0.38      0.30      0.33         6
    weighted avg       0.62      0.50      0.56         6

--------------------------

```python
y_true = np.array([1, 1, 0, 1, 1, 1])
y_pred = np.array([1, 0, 0, 1, 1, 1])
```

```python
print(metrics.classification_report(y_true, y_pred))
```

                  precision    recall  f1-score   support

               0       0.50      1.00      0.67         1
               1       1.00      0.80      0.89         5

        accuracy                           0.83         6
       macro avg       0.75      0.90      0.78         6
    weighted avg       0.92      0.83      0.85         6

```python

```

#ROC

```python
import matplotlib.pyplot as plt
from sklearn import datasets, metrics, model_selection, svm
X, y = datasets.make_classification(random_state=0)
X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y, random_state=0)
clf = svm.SVC(random_state=0)
clf.fit(X_train, y_train)

metrics.plot_roc_curve(clf, X_test, y_test)
plt.show()
```

```python
import numpy as np
from sklearn import metrics
#y = np.array([1, 1, 2, 2])
y = np.array([0, 0, 1, 1])
scores = np.array([0.1, 0.4, 0.35, 0.8])
fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=1)
fpr

```

    array([0. , 0. , 0.5, 0.5, 1. ])

```python
tpr
```

    array([0. , 0.5, 0.5, 1. , 1. ])

```python
thresholds
```

    array([1.8 , 0.8 , 0.4 , 0.35, 0.1 ])

```python
plt.plot(fpr, tpr)
```

```python
fig = plt.figure()
ax = fig.add_subplot(projection="3d")
ax.plot(fpr, tpr, thresholds)
```

```python
from plotly import graph_objects as go
```

```python
line = go.Scatter3d(x=fpr, y=tpr, z=thresholds, mode="lines")
fig = go.Figure(data=line)
fig.show()
```

<meta charset="utf-8" />

Chapter 2/python/classification metrics/2-class metrics.ipynb


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## Graded Homework 6

For this homework, we'll focus on:
- integrals in 2D polar and 3D spherical space
- probability (including integrating continuous distributions)
---

As a reminder, you do not need to use Python to solve the problems. If you want, you can use other methods, just put your answers in the appropriate places.

To turn in, either download as Notebook (.ipynb) or Print to PDF and upload to Gradescope.

Make sure you fill in any place that says YOUR CODE HERE or "YOUR ANSWER HERE", as well as your name and collaborators (i.e., anyone you discussed this with) below:

```python
NAME = ""
COLLABORATORS = ""
```

### Cartesian to Spherical Integrals

Consider the Cartesian integral:
$$
\iiint z^{2} d x d y d z
$$

Evaluation is fairly simple over a rectangular region, but if we wish to evaluate across a spherical volume, the limits of integration become messy.

Instead, we can transform $z^2$, resulting in the integral:

$$
\int_{0}^{a} \int_{0}^{\pi} \int_{0}^{2 \pi}(r \cos \theta)^{2} r^{2} \sin \theta d \varphi d \theta d r
$$

Integrate across a sphere of size "a"

```python
from sympy import init_session
init_session()
```

```python
a, r, theta, phi = symbols("N r theta phi")
# technically, we're integrating psi**2 but let's not worry about that now
# z => (r*cos(theta)) in spherical coordinates
f = (r*cos(theta)**2)

integrate(# something)
```

### Normalizing

We often wish to normalize functions. Calculate a normalization constant $N^2$ for the following integral (e.g., evaluate it, set it equal to one, etc.)

$$
\int_{0}^{\infty} \int_{0}^{\pi} \int_{0}^{2 \pi} N^2 \mathrm{e}^{-2 r} \cos ^{2} \theta r^{2} \sin \theta d \varphi d \theta d r
$$

(This is related to a $2p_z$ hydrogen atomic orbital.)

```python
# if you have an error, make sure you run the cell above this
N, r, theta, phi = symbols("N r theta phi")
# technically, we're integrating psi**2 but let's not worry about that now
f = N**2 * exp(-2*r) * cos(theta)**2

integrate(# something)
```

### Gaussian Distribution and Probability

You may have heard much about the Gaussian "normal" distribution ("Bell curve").

If we flip a coin enough, the binomial distribution becomes essentially continuous. Moreover, the "law of large numbers" (i.e., the [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem)) indicates that by taking the enough data points, the sum - and average will tend towards a Gaussian distribution.

In short, even if our underlying data comes from some different distribution, the average will become normal (e.g. consider the average velocities in an ideal gas - a mole is a big number).

We will spend some time on the Gaussian distribution.
- mean $x_0$
- standard deviation $\sigma$, variance $\sigma^2$

A normalized Gaussian distribution is then:

$$
p(x) =\frac{1}{\sqrt{2 \pi \sigma^{2}}} \exp \left[-\frac{\left(x-x_{0}\right)^{2}}{2 \sigma^{2}}\right]
$$

If the mean $x_0 = 0$, what fraction of data will fall:
- within $\pm 0.5\sigma$
- within one standard deviation
- within $\pm 1.5 \sigma$
- within $\pm 2.0 \sigma$

(In other words, change the limits of integration on the probability function.)

```python
sigma = symbols('sigma')

p = (1/sqrt(2*pi*sigma**2))*exp((-x**2)/(2*sigma**2))
simplify(integrate(p, (x, -0.5*sigma, +0.5*sigma)))
```

### If we want to include "error bars" with 95% confidence, what intervals do we use?

YOUR ANSWER HERE

What about 99% confidence intervals?


# Legendre Functions

Series expansions are used in a variety of circumstances:
- When we need a tractable approximation to some ugly equation
- To transform between equivalent ways of looking at a problem (e.g. time domain vs frequency domain)
- When they are (part of) a solution to a particular class of differential equation

For approximations, there is an important divide between getting the best fit *near a point* (e.g. Taylor series) and getting the best fit *over an interval*. This notebook deals with one example of the latter; there is a separate notebook for Taylor expansions and others for Fourier, Bessel, etc.

## Fitting over an interval

What is the best (tractable) series approximating my function across some range of values? What matters is an overall best fit (e.g. least-squares deviation) across the range, and we can't tolerate wild divergences as with the Taylor series.

There are various series which are useful in different contexts, but a common property is that the terms are *orthogonal* over some interval $[a,b]$. If $f(t)$ is a real-valued function their *inner product* is defined as

$$ \langle f(m t),f(n t) \rangle   \colon =\int _a^b f(m t) f(n t) \,  dt $$

For orthogonal functions, this is non-zero if $m=n$ and zero if $m \ne n$, i.e.

$$\langle f(m t),f(n t) \rangle = a \delta_{mn}$$

where $\delta$ is the Kronecker delta.  If $a = 1$ the functions are said to be orthonormal.

## The Legendre differential equation

This is of the form

$$ (1 - x^2)y'' -2x y' + l(l+1)y = 0 $$

where $l$ is a constant. The most useful solutions are the Legendre polynomials, where $y = P_l(x)$.

## Legendre Polynomials

These are "just" polynomials, so maybe conceptually simpler than, for example, Bessel functions. Their special feature is that the coefficients are chosen so that they are mutually orthogonal over the range $[-1,1]$.

They are given by the formula

$$ P_n(x) = \frac{1}{2^n n!} \frac{d^n}{dx^n} (x^2 -1)^n $$

They tend to crop up in the sort of problems which naturally use spherical coordinates and/or spherical harmonics, such as fluctuations in the CMB, "sunquakes" in our local star or (at the other end of the scale range) electron orbitals in the hydrogen atom.

## Associated Legendre Functions

the function $P_n^m(x)$ is of degree $n$ and order $m$. It is related to the $n$th order polynomial $P_n(x)$ by

$$ P_n^m(x) = (-1)^m (1-x^2)^{m/2}\ \frac{d^m P_n(x)}{dx^m} $$

Order zero functions are just the corresponding Legendre polynomials: $P_n^0(x) \equiv P_n(x)$.

## Software

Start with a few basics, then we can get mathematical.

```python
import numpy as np
import matplotlib.pyplot as plt

plt.rcParams.update({'font.size': 16})
```

How to work with Legendre functions in Python? A quick Google search turns up quite a few possibilities, though it may not be immediately obvious how these relate to one another:
- `scipy.special.legendre()`
- `scipy.special.lpmn()` and `.lpmv()`
- `numpy.polynomial.legendre`
- `sympy.functions.special.polymomials.legendre()`
- `sympy.functions.special.polymomials.assoc_legendre()`
- `sympy.polys.orthopolys.legendre_poly()`
- `mpmath.legendre()`

### scipy.special

This one is relatively simple. Calling `legendre(n)` returns the nth-order polynomial as a function which can then itself be called with one or more x-values.

```python
import scipy.special as sp
P_3_sp = sp.legendre(3)
display(P_3_sp)
```

    poly1d([ 2.5,  0. , -1.5,  0. ])

```python
x10 = np.linspace(-1, 1, 10)
display(P_3_sp(x10))
```

    array([-1.        , -0.00960219,  0.40466392,  0.40740741,  0.16323731,
           -0.16323731, -0.40740741, -0.40466392,  0.00960219,  1.        ])

For the associated Legendre functions there are a couple of related SciPy functions which take different approaches to vector input and output. `scipy.special.lpmn(m, n, x)` will only take a single scalar $x$, but returns an $(m+1, n+1)$ array of results for all orders $0 \dots m$ and degrees $0 \dots n$. In contrast, `scipy.special.lpmv(m, n, x)` accepts arrays of $x$ and returns results for just the specified $m$ and $n$.

```python
xs = np.linspace(0, 1, 100)
m = 1
n = 2
P_lm, _ = sp.lpmn(m, n, xs[5])
display(P_lm.shape)

P_lmv = sp.lpmv(m, n, xs)
display(P_lmv.shape)
```

### numpy.polynomial

This is less simple and needs more exploration. Start like this, then read whatever documentation you can find.

```python
from numpy.polynomial import Legendre as P
P_3_npl = P([3])
display(P_3_npl)
```

    Legendre([3.], domain=[-1,  1], window=[-1,  1])

### sympy.functions.special.polymomials

This is symbolic math, which will give you differentiation, integration, etc, as well as nice $LaTeX$ output. Not so convenient for plotting.

```python
from sympy import legendre, assoc_legendre, init_printing
init_printing()
from sympy.abc import x

display(legendre(3, x))
display(assoc_legendre(3, 2, x))
```

### sympy.polys.orthopolys

Sort of like sympy.functions.special.polymomials, but with some different options.

```python
from sympy import legendre_poly
display(legendre_poly(3))
display(legendre_poly(3, polys=True))
```

### mpmath

This is aimed at arbitrary-precision floating point arithmetic. It doesn't seem to do symbolic math like SymPy or (more surprisingly?) handle array input like SciPy.

If you don't have the `mpmath` package installed, don't worry: this is the only cell that tries to use it.

```python
import mpmath as mp
for x1 in np.arange(0, 1, 0.2):
    display(mp.legendre(3, x1))
```

    mpf('0.0')

    mpf('-0.28000000000000003')

    mpf('-0.44')

    mpf('-0.35999999999999988')

    mpf('0.08000000000000014')

### Provisional conclusions

It seems like `sympy.functions.special.polymomials` offers the simplest way to do symbolic math, and `scipy.special` the easiest way to do numerical calculations. Other packages no doubt have more sophisticated capabilities but I'm not the right person to judge.

The first few __Legendre polymomials__ look like this. Note that they are alternately odd/even functions.

```python
from IPython.display import Math
from sympy import latex
from sympy.abc import x

for i in range(6):
    l_i = latex(legendre(i, x))
    display(Math('P_{} = {}'.format(i, l_i)))
```

$\displaystyle P_0 = 1$

$\displaystyle P_1 = x$

$\displaystyle P_2 = \frac{3 x^{2}}{2} - \frac{1}{2}$

$\displaystyle P_3 = \frac{5 x^{3}}{2} - \frac{3 x}{2}$

$\displaystyle P_4 = \frac{35 x^{4}}{8} - \frac{15 x^{2}}{4} + \frac{3}{8}$

$\displaystyle P_5 = \frac{63 x^{5}}{8} - \frac{35 x^{3}}{4} + \frac{15 x}{8}$

The first few __associated Legendre functions__:

```python
for i in range(4):
    for j in range(i):
        l_ij = latex(assoc_legendre(i, j, x))
        display(Math('P_{}^{} = {}'.format(i, j, l_ij)))
```

$\displaystyle P_1^0 = x$

$\displaystyle P_2^0 = \frac{3 x^{2}}{2} - \frac{1}{2}$

$\displaystyle P_2^1 = - 3 x \sqrt{- x^{2} + 1}$

$\displaystyle P_3^0 = \frac{5 x^{3}}{2} - \frac{3 x}{2}$

$\displaystyle P_3^1 = - \sqrt{- x^{2} + 1} \left(\frac{15 x^{2}}{2} - \frac{3}{2}\right)$

$\displaystyle P_3^2 = 15 x \left(- x^{2} + 1\right)$

__Plotting__ the first few Legendre polymomials over the range where they are orthogonal:

```python
import scipy.special as sp

xlims = (-1, 1)
x = np.linspace(xlims[0], xlims[1], 1000)

plt.figure(figsize=(9, 9))
for v in range(0, 6):
    plt.plot(x, sp.legendre(v)(x))

plt.xlim(xlims)
plt.ylim((-1.1, 1.1))
plt.legend(('$\mathcal{P}_0(x)$', '$\mathcal{P}_1(x)$', '$\mathcal{P}_2(x)$',
           '$\mathcal{P}_3(x)$', '$\mathcal{P}_4(x)$', '$\mathcal{P}_5(x)$'),
           loc = 0)
plt.xlabel('$x$')
plt.ylabel('$\mathcal{P}_n(x)$')
plt.title('Plots of the first six Legendre Polynomials')
plt.grid(True)
```

## Spherical coordinates

An interesting use of the associated Legendre functions has $x = \cos(\theta)$. The resulting functions are a component in the spherical harmonics $Y_l^m(\theta, \phi)$, described in another Jupyter notebook in this folder.

We can make polar plots showing the magnitude of $P_l^m(\cos \theta)$ in the direction $\theta$. Here $\theta$ is the angle down from the $+z$ axis. There is no $\phi$ dependency in $P_l^m(\cos \theta)$ so think of these plots as being radially symmetric around the $z$-axis (i.e. rotate them about the vertical axis).

TODO - color-code the plots by the sign of $P_l^m(\cos \theta)$. This would make the nodes clearer to see.

```python
thetas = np.linspace(0, np.pi, 200)
theta_x = np.sin(thetas)
theta_y = np.cos(thetas)

fig = plt.figure(figsize = (15,15))

for n in range(3):
    for m in range(n+1):
        P_lm = sp.lpmv(m, n, np.cos(thetas))

        x_coords = theta_x*np.abs(P_lm)
        y_coords = theta_y*np.abs(P_lm)
        ax = fig.add_subplot(3, 3, m+1+3*n)
        ax.plot(x_coords, y_coords, 'b-', label='$P_{}^{}$'.format(n,m))
        # reflect the plot across the z-axis
        ax.plot(-x_coords, y_coords, 'b-')
        ax.axis('equal')
#         ax.set_title('$P_{}^{}$'.format(n,m))
        ax.legend()
```

## References

- Boas, "Mathematical methods in the physical sciences", 3rd ed, chapter 12
- MathWorld, http://mathworld.wolfram.com/LegendrePolynomial.html and http://mathworld.wolfram.com/AssociatedLegendrePolynomial.html
- Wikipedia, https://en.wikipedia.org/wiki/Legendre_polynomials
- Binney & Tremaine, "Galactic Dynamics", 2nd ed, appendix C.5
- Griffiths & Schroeter, "Introduction to Quantum Mechanics", 3rd ed, section 4.1.2
- Mathews & Walker, "Mathematical Methods of Physics", 2nd ed, section 7.1

```python

```


# Machine Learning Implementation

## Imports

```python
import json

import numpy as np
import pandas as pd
import plotly.offline as py
from plotly import graph_objects as go
```

## Logistic regression

### The maths

The logistic model aims to predict the discrete y variable a.k.a the target variable (e.g. whether something will happen) based on a collection of features. It does this by transforming a linear combination of the features into a curve and fitting this curve to the data.

The curve used in logistic regression is the sigmoid function

$$
\sigma(x) = \frac{1}{1+e^{-x}}
$$

Define y as

$$
\begin{align}
\hat{y} &= h_{\boldsymbol{\beta}}(\mathbf{x})\\
\hat{y}&= \sigma\left(\beta_0x_0+\cdots+\beta_nx_n\right)\quad &n\in \mathbb{N},x_0=1 \\
\hat{y}&=\sigma\left(\sum^{n}_{i=0}\beta_ix_i\right) \\
\hat{y}&=\sigma\left(\mathbf{\boldsymbol{\beta}^Tx}\right)\quad&\boldsymbol{\beta},\mathbf{x}\in\mathbb{R}^{n\times1}\\
\hat{y}&=\sigma\left(\boldsymbol{\beta}^T\mathbf{x}\right)
\end{align}
$$

notice

$$
\hat{y} = \frac{1}{1+e^{-\boldsymbol{\beta}^T\mathbf{x}}}
$$

so

$$
\begin{align}
\hat{y} + \hat{y}e^{-\boldsymbol{\beta}^T\mathbf{x}} &= 1\\
\hat{y}e^{-\boldsymbol{\beta}^T\mathbf{x}} &= 1 - \hat{y}\\
\frac{\hat{y}}{1 - \hat{y}} &= e^{\boldsymbol{\beta}^T\mathbf{x}}\\
\ln\left(\frac{\hat{y}}{1 - \hat{y}}\right)&=\boldsymbol{\beta}^T\mathbf{x}
\end{align}
$$

This above is the logit form of logistic regression. We model the logit as a linear combination of the x variables

We define the cost function as follows for each y and corresponding x

$$
\begin{align}
J(\mathbf{x})
&= \begin{cases}
-\log\left(h_{\boldsymbol{\beta}}(\mathbf{x})\right) &\text{if y=1}\\
-\log\left(1-h_{\boldsymbol{\beta}}(\mathbf{x})\right) &\text{if y=0}\\
\end{cases}
\end{align}
$$

$$
\begin{align}
J(\mathbf{x})
&= -\frac{1}{m}\sum_{j=1}^my^j\log\left(h_{\boldsymbol{\beta}}(\mathbf{x}^j)\right)
+(1-y^j)\log\left(1-h_{\boldsymbol{\beta}}(\mathbf{x}^j)\right)\\
&= -\frac{1}{m}\sum_{j=1}^my^j\log\left(\frac{1}{1+e^{-\boldsymbol{\beta}^T\mathbf{x}}}\right)
+(1-y^j)\log\left(1-\frac{1}{1+e^{-\boldsymbol{\beta}^T\mathbf{x}}}\right)\\
&= -\frac{1}{m}\sum_{j=1}^my^j\log\left(\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}\right)
+(1-y^j)\log\left(1-\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}\right)
\end{align}
$$

note

$$
\begin{align}
h_{\boldsymbol{\beta}}(\mathbf{x}^j)&=\frac{1}{1+e^{-\boldsymbol{\beta}^T\mathbf{x}^j}}\\
&=\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\end{align}
$$

so

$$
\begin{align}
\frac{\partial h}{\partial \beta_k} &= -\left(1+e^{-\sum^{n}_{i=0}\beta_ix_i}\right)^{-2}e^{-\sum^{n}_{i=0}\beta_ix_i} (-x_k^j)\\
&=\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\frac{-e^{-\sum^{n}_{i=0}\beta_ix_i} (-x_k^j)}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}\\
&=\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\frac{(1-1-e^{-\sum^{n}_{i=0}\beta_ix_i})(-x_k^j)}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}\\
&=\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\left(
\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}-
\frac{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\right)(-x_k^j)\\
&=\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\left(
\frac{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}-
\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\right)(x_k^j)\\
&=\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\left(
1-
\frac{1}{1+e^{-\sum^{n}_{i=0}\beta_ix_i}}
\right)(x_k^j)\\
&=h_{\boldsymbol{\beta}}(\mathbf{x}^j)(1-h_{\boldsymbol{\beta}}(\mathbf{x}^j))x_k^j
\end{align}
$$

We need to differentiate the cost function i.e. find the gradient

$$
\begin{align}
\frac{\partial J}{\partial\beta_k}\left(\boldsymbol{\beta}\right)
&=\frac{\partial}{\partial\beta_k}\left(
-\frac{1}{m}\sum_{j=1}^my^j\log\left(h_{\boldsymbol{\beta}}(\mathbf{x}^j)\right)
+(1-y^j)\log\left(1-h_{\boldsymbol{\beta}}(\mathbf{x}^j)\right)
\right)\\
&=-\frac{1}{m}\sum_{j=1}^m\frac{y^j}{h_{\boldsymbol{\beta}}(\mathbf{x}^j)}\frac{\partial h}{\partial \beta_k}
+\frac{-(1-y^j)}{1-h_{\boldsymbol{\beta}}(\mathbf{x}^j)}\frac{\partial h}{\partial \beta_k}\\
&=-\frac{1}{m}\sum_{j=1}^m\frac{y^j}{h_{\boldsymbol{\beta}}(\mathbf{x}^j)}
h_{\boldsymbol{\beta}}(\mathbf{x}^j)(1-h_{\boldsymbol{\beta}}(\mathbf{x}^j))x_k^j
+\frac{-(1-y^j)}{1-h_{\boldsymbol{\beta}}(\mathbf{x}^j)}
h_{\boldsymbol{\beta}}(\mathbf{x}^j)(1-h_{\boldsymbol{\beta}}(\mathbf{x}^j))x_k^j\\
&=-\frac{1}{m}\sum_{j=1}^my^j(1-h_{\boldsymbol{\beta}}(\mathbf{x}^j))x_k^j
-(1-y^j)
h_{\boldsymbol{\beta}}(\mathbf{x}^j)x_k^j\\
&=\frac{1}{m}\sum_{j=1}^m
\left(h_{\boldsymbol{\beta}}(\mathbf{x}^j)-y^j\right)x_k^j
\end{align}
$$

hence

$$
\nabla_{\boldsymbol{\beta}} J
=
\begin{bmatrix}
       \frac{\partial J}{\partial\beta_1} \\
       \vdots \\
       \frac{\partial J}{\partial\beta_n}
\end{bmatrix}
=
\begin{bmatrix}
       \frac{1}{m}\sum_{j=1}^m
            \left(h_{\boldsymbol{\beta}}(\mathbf{x}^j)-y^j\right)x_1^j\\
       \vdots \\
       \frac{1}{m}\sum_{j=1}^m
           \left(h_{\boldsymbol{\beta}}(\mathbf{x}^j)-y^j\right)x_n^j
\end{bmatrix}
$$

Define the design matrix and column representation of y. Here each row of X and y are training examples hence there are m rows

$$\mathbf{X}\in\mathbb{R}^{m\times n},
\quad \mathbf{y}\in\mathbb{R}^{m\times 1}
$$

$$
\mathbf{X}=\begin{bmatrix}
       \dots & (\mathbf{x}^1)^T & \dots\\
       \dots & (\mathbf{x}^2)^T & \dots\\
       \dots & \vdots  & \dots\\
       \dots & (\mathbf{x}^m)^T & \dots
\end{bmatrix}\quad
\mathbf{y}=\begin{bmatrix}
    y_1\\y_2\\\vdots\\y_m
\end{bmatrix}
$$

$$
\begin{align}
\nabla_{\boldsymbol{\beta}} J
=
\begin{bmatrix}
       \frac{1}{m}\sum_{j=1}^m
            \left(h_{\boldsymbol{\beta}}(\mathbf{x}^j)-y^j\right)x_1^j\\
       \vdots \\
       \frac{1}{m}\sum_{j=1}^m
           \left(h_{\boldsymbol{\beta}}(\mathbf{x}^j)-y^j\right)x_n^j
\end{bmatrix}
=
\frac{1}{m}
\begin{bmatrix}
       \sum^{n}_{i=0}h_{\boldsymbol{\beta}}(\mathbf{x}^j)^jx^j_1\\
       \vdots \\
       \sum^{n}_{i=0}h_{\boldsymbol{\beta}}(\mathbf{x}^j)x^j_n
\end{bmatrix}
-
\frac{1}{m}
\begin{bmatrix}
       \sum^{m}_{j=1}y^jx^j_1\\
       \vdots \\
       \sum^{m}_{j=1}y^jx^j_n\\
\end{bmatrix}
\end{align}
$$

$$
h_{\boldsymbol{\beta}}(\mathbf{x}^j) = \sigma({\mathbf{x}^j}^T\boldsymbol{\beta})
$$

so

$$
\begin{align}
\nabla_{\boldsymbol{\beta}} J
&=\frac{1}{m}\left(
\mathbf{X}^T\sigma(\mathbf{X}\mathbf{\boldsymbol{\beta}})-\mathbf{X}^T\mathbf{y}
\right)\\
&=\frac{1}{m}\mathbf{X}^T\left(
\sigma(\mathbf{X}\mathbf{\boldsymbol{\beta}})-\mathbf{y}
\right)\\
&=\frac{1}{m}\mathbf{X}^T\left(
\mathbf{\hat{y}}-\mathbf{y}
\right)
\end{align}
$$

where

$$
\mathbf{\hat{y}} = \sigma(\mathbf{X}\mathbf{\boldsymbol{\beta}})
$$

We could have derived the same thing using matrix calculus

### Example sigmoid

The curve used in logistic regression is the sigmoid function

$$
\sigma(x) = \frac{1}{1+e^{-x}}
$$

```python
sigmoid_fig = go.FigureWidget()
demo_x = np.arange(-10,10,0.1)
demo_y = 1 / (1 + np.exp(-demo_x))
sigmoid_fig.add_scatter(
    x=demo_x,
    y=demo_y)
sigmoid_fig.layout.title = 'Sigmoid Function'
sigmoid_fig
```

    FigureWidget({
        'data': [{'type': 'scatter',
                  'uid': 'fc81bfbe-f9f8-419c-9923-4d127962d5e2',
     …

### Make fake data

```python
m = 100
x0 = np.ones(shape=(m, 1))
x1 = np.linspace(0, 10, m).reshape(-1, 1)
X = np.column_stack((x0, x1))

# let y = 0.5 * x + 1 + epsilon
epsilon =  np.random.normal(scale=2, size=(m, 1))
y = x1 + epsilon
y = (y > 5).astype(int)
```

```python
fig = go.FigureWidget()
fig = fig.add_scatter(
    x=X[:,1],
    y=y[:,0],
    mode='markers',
    name='linear data + noise')
fig
```

    FigureWidget({
        'data': [{'mode': 'markers',
                  'name': 'linear data + noise',
                  'ty…

### Logistic regression class

```python
import json

import numpy as np

class LogisticRegression():

    def __init__(self, learning_rate=0.05):
        """
        Logistic regression model

        Parameters:
        ----------
        learning_rate: float, optional, default 0.05
            The learning rate parameter controlling the gradient descent
            step size
        """
        self.learning_rate = learning_rate
        print('Creating logistic model instance')

    def __repr__(self):
        return (
            f'<LogisticRegression '
            f'learning_rate={self.learning_rate}>')

    def fit(self, X, y, n_iter=1000):
        """
        Fit the logistic regression model

        Updates the weights with n_iter iterations of batch gradient
        descent updates

        Parameters:
        ----------
        X: numpy.ndarray
            Training data, shape (m samples, (n - 1) features + 1)
            Note the first column of X is expected to be ones (to allow
            for the bias to be included in beta)
        y: numpy.ndarray
            Target values - class label {0, 1}, shape (m samples, 1)
        n_iter: int, optional, default 1000
            Number of batch gradient descent steps
        """
        m, n = X.shape
        print(f'fitting with m={m} samples with n={n-1} features\n')
        self.beta = np.zeros(shape=(n, 1))
        self.costs = []
        self.betas = [self.beta]
        for iteration in range(n_iter):
            y_pred = self.predict_proba(X)
            cost = (-1 / m) * (
                (y.T @ np.log(y_pred)) +
                ((np.ones(shape=y.shape) - y).T @ np.log(
                    np.ones(shape=y_pred.shape) - y_pred))
            )
            self.costs.append(cost[0][0])
            gradient = (1 / m) * X.T @ (y_pred - y)
            self.beta = self.beta - (
                self.learning_rate * gradient)
            self.betas.append(self.beta)

    def predict_proba(self, X):
        """
        Predicted probability values for class 1

        Note this is calculated as the sigmoid of the linear combination
        of the feature values and the weights.

        Parameters:
        ----------
        X: numpy.ndarray
            Training data, shape (m samples, (n - 1) features + 1)
            Note the first column of X is expected to be ones (to allow
            for the bias to be included in beta)

        Returns:
        -------
        numpy.ndarray:
            Predicted probability of samples being in class 1
        """
        y_pred = self.sigmoid(X @ self.beta)
        return y_pred

    def predict(self, X, descision_prob=0.5):
        """
        Predict the class values from sample X feature values

        Parameters:
        ----------
        X: numpy.ndarray
            Training data, shape (m samples, (n - 1) features + 1)
            Note the first column of X is expected to be ones (to allow
            for the bias to be included in beta)

        Returns:
        -------
        numpy.ndarray:
            Prediceted class values, shape (m samples, 1)
        """
        y_pred = self.sigmoid(X @ self.beta)
        return (y_pred > descision_prob) * 1

    def sigmoid(self, x):
        """
        Sigmoid function

        f(x) = 1 / (1 + e^(-x))

        Parameters:
        ----------
        x: numpy.ndarray

        Returns:
        -------
        numpy.ndarray:
            sigmoid of x, values in (0, 1)
        """
        return 1 / (1 + np.exp(-x))

```

```python
logistic_regression = LogisticRegression()
logistic_regression.fit(X, y)
```

    Creating logistic model instance
    fitting with m=100 samples with n=1 features

```python
example_X = np.array([[1,1],[1,4],[1,7]])
logistic_regression.predict(example_X)
```

    array([[0],
           [0],
           [1]])

### Plot the best fit

```python
fig = fig.add_scatter(
    x=X[:,1],
    y=logistic_regression.predict_proba(X)[:,0],
    mode='markers',
    name='logistic best fit')
fig
```

    FigureWidget({
        'data': [{'mode': 'markers',
                  'name': 'linear data + noise',
                  'ty…

### Plot the cost function

```python
# Haven't got round to this yet - see linear regression for an example error
# surface decent.
```

## Logisitc regression - Titanic example

### Load data

```python
X_train = pd.read_feather('../data/titanic/processed/X_train.feather')
X_test = pd.read_feather('../data/titanic/processed/X_test.feather')
y_train = pd.read_feather('../data/titanic/processed/y_train.feather')
y_test = pd.read_feather('../data/titanic/processed/y_test.feather')
```

### Train model

```python
titanic_logistic_model = LogisticRegression()
```

    Creating logistic model instance

```python
titanic_logistic_model.fit(X=X_train.values, y=y_train.values, n_iter=4000)
```

    fitting with m=712 samples with n=29 features

### Plot the cost

```python
titanic_cost_fig = go.FigureWidget()

titanic_cost_fig.add_scatter(
    x=list(range(len(titanic_logistic_model.costs))),
    y=titanic_logistic_model.costs,
)

titanic_cost_fig.layout.title = 'Cost Vs gradient descent iterations'
titanic_cost_fig.layout.xaxis.title = 'Iterations'
titanic_cost_fig.layout.yaxis.title = 'Cost'
titanic_cost_fig
```

    FigureWidget({
        'data': [{'type': 'scatter',
                  'uid': 'a3a5dfba-fddd-428e-87a9-4a7f4461bffc',
     …

### Error analysis

```python
y_pred = titanic_logistic_model.predict(X_test.values)
test_accuracy = (y_pred == y_test.values).sum() / len(y_pred)

print(f'Test accuracy is with my implementation {test_accuracy:.2%}')
```

    Test accuracy is with my implementation 79.89%

## End


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.0 (Python 3.6.3-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/

```python
r, u, v, w, E, e, p = symbols("rho u v w E e p")
dedr, dedp = symbols(r"\left.\frac{\partial{}e}{\partial\rho}\right|_p \left.\frac{\partial{}e}{\partial{}p}\right|_\rho")
Bx, By, Bz = symbols("B_x B_y B_z")
```

```python
A = Matrix(
    [[1, 0, 0, 0, 0, 0, 0, 0],
     [u, r, 0, 0, 0, 0, 0, 0],
     [v, 0, r, 0, 0, 0, 0, 0],
     [w, 0, 0, r, 0, 0, 0, 0],
     [e + r*dedr + (u**2 + v**2 + w**2)/2, r*u, r*v, r*w, r*dedp, Bx, By, Bz],
     [0, 0, 0, 0, 0, 1, 0, 0],
     [0, 0, 0, 0, 0, 0, 1, 0],
     [0, 0, 0, 0, 0, 0, 0, 1]])
```

```python
A
```

```python
A.inv()
```

```python
Fr, Fu, Fv, Fw, FE, FBx, FBy, FBz = symbols(r"F_{\rho} F_{\rho{}u} F_{\rho{}v} F_{\rho{}w} F_{\rho{}E} F_{B_x} F_{B_y} F_{B_z}")
```

```python
F = Matrix([[Fr], [Fu], [Fv], [Fw], [FE], [FBx], [FBy], [FBz]])
```

```python
A.inv() * F
```

```python

```