# Exercise: beam bending with the subdomain method

The differential equation for beam bending
$$
(EI w'')'' - q = 0
$$
can be integrated analytically by specifying the linear loading as $q = q_0\frac{x}{L}$ and the boundary conditions
$$
w(0) = 0\\
w(L) = 0\\
w''(0)=0\\
w''(L) = 0
$$
to yield the deflection solution ("Biegelinie"):
$$
w(x) = \frac{q_0 L^4}{360 EI} \left[ 3\left(\frac{x}{L}\right)^5 - 10 \left(\frac{x}{L}\right)^3 + 7 \left(\frac{x}{L}\right)\right]
$$
We now seek to approximate this solution numerically using the subdomain method.

```python

```

```python
import numpy as np #numerical methods
import sympy as sp #symbolic operations
import matplotlib.pyplot as plt #plotting
sp.init_printing(use_latex='mathjax') #makes sympy output look nice

#Some plot settings
plt.style.use('seaborn-deep')
plt.rcParams['lines.linewidth']= 2.0
plt.rcParams['lines.color']= 'black'
plt.rcParams['legend.frameon']=True
plt.rcParams['figure.figsize'] = (8, 6)
plt.rcParams['font.family'] = 'serif'
plt.rcParams['legend.fontsize']=14
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.left'] = True
plt.rcParams['axes.spines.bottom'] = True
plt.rcParams['axes.axisbelow'] = True
```

```python
#Defining the geometric an material characteristics as symbolic quantities
L,q,EI,x = sp.symbols('L q_0 EI x')
```

```python
#Analytical solution to deflection
def deflection_analytical():
    a = x/L
    f = q*L**4/(360 * EI)
    return f*(3*a**5 - 10*a**3 + 7*a)
```

```python
deflection_analytical() #check definition
```

$$\frac{L^{4} q_{0}}{360 EI} \left(\frac{7 x}{L} - \frac{10 x^{3}}{L^{3}} + \frac{3 x^{5}}{L^{5}}\right)$$

Now, let's plot the analytical solution. For that purpose, we use some Python magic ("lambdify"). We sample the analytical solution for $x \in [0,L]$ at 100 points and plot the dimensionless deflection over the dimensionless length.

```python
lam_x = sp.lambdify(x, deflection_analytical(), modules=['numpy'])
#For the variable x the function deflection_analytical() will obtain something

x_vals = np.linspace(0, 1, 100)*L #This something is x from 0 to L
analytical = lam_x(x_vals) #We calculate the solution by passing x = [0,...,L] to deflection_analytical

plt.plot(x_vals/L, analytical/(L**4*q)*EI)
plt.xlabel('$x\ /\ L$')
plt.ylabel('$w\ /\ L^4 q_0  EI^{-1}$')
```

## Trigonometric Ansatz

Let's try the approximation
$$
    \tilde{w} = a_1 \sin \left(\pi \frac{x}{L}\right) + a_2 \sin \left(2\pi\frac{x}{L}\right)
$$

```python
a1, a2 = sp.symbols('a_1 a_2')#Defining the free values as new symbols
```

```python
def deflection_ansatz():
    return a1*sp.sin(sp.pi/L*x) + a2*sp.sin(2*sp.pi/L*x) #defining the approximate solution with the unknown coefficients
```

By domain-wise integration of the ODE with domain-wise weighting by unity
$$
EI w^\text{IV} - q_0 \frac{x}{L} = 0 \text{ with } EI = \text{const.}
$$
we arrive at the boundary flux-based formulation
$$
    0 = EI (w'''|_{x_{i+1}} - w'''|_{x_{i}}) - \frac{q_0}{2L}(x_{i+1}^2 - x_i^2)
$$
In our case, to determine $a_1$ and $a_2$, we define the two domains:

```python
domain_1 = [0,L/2]
domain_2 = [L/2,L]
```

Hence, we get the system of equations

```python
#get w'''
d3w_dx3 = deflection_ansatz().diff(x,3)
#flux balance in domain 1
flux_domain_1 = EI*(d3w_dx3.subs(x,domain_1[1]) - d3w_dx3.subs(x,domain_1[0])) - q/(2*L)*(domain_1[1]**2 - domain_1[0]**2)
#flux balance in domain 2
flux_domain_2 = EI*(d3w_dx3.subs(x,domain_2[1]) - d3w_dx3.subs(x,domain_2[0])) - q/(2*L)*(domain_2[1]**2 - domain_2[0]**2)
#combining both domains into an equation system
flux_conditions = (flux_domain_1,flux_domain_2)
```

```python
coefficients = sp.solve(flux_conditions,a1,a2)
```

```python
coefficients
```

$$\left \{ a_{1} : \frac{L^{4} q_{0}}{4 \pi^{3} EI}, \quad a_{2} : - \frac{L^{4} q_{0}}{128 \pi^{3} EI}\right \}$$

```python
deflection_ansatz().subs([(a1,coefficients[a1]),(a2,coefficients[a2])]).simplify()
```

$$\frac{L^{4} q_{0}}{128 \pi^{3} EI} \left(32 \sin{\left (\frac{\pi x}{L} \right )} - \sin{\left (\frac{2 \pi}{L} x \right )}\right)$$

Now we're ready to plot the result and compare it to the analytical solution.

```python
#We first substite the now known Ansatz free values into our Ansatz
z = sp.symbols('z')
w_numerical = deflection_ansatz().subs([(a1,coefficients[a1]),(a2,coefficients[a2]),(x,z*L)])
#We also made the coordinate dimensionless (x/L --> z) because of sympy problems

lam_x_num = sp.lambdify(z, w_numerical, modules=['numpy'])
#For the variable x the expression w_numerical will be given something
z_vals = np.linspace(0, 1,100) #This something is z from 0 to 1
numerical = lam_x_num(z_vals) #We calculate the solution by passing x = [0,...,L] to deflection_analytical
plt.plot(x_vals/L, analytical/(L**4*q)*EI,label='analytical')
plt.plot(z_vals, numerical/(L**4*q)*EI,label='numerical')
plt.legend()
plt.xlabel('$x\ /\ L$')
plt.ylabel('$w\ /\ L^4 q_0  EI^{-1}$')
plt.tight_layout()
plt.savefig('beam_subdomain.pdf')
```

```python
print("Maximum absolute error: ", np.max(np.abs(analytical/(L**4*q)*EI - numerical/(L**4*q)*EI)))
```

    Maximum absolute error:  0.00155626045741949

We can also plot and compare the bending moment. Let's first find the analytical expression by symbolically differentiating the deflection expression twice to obtain $M(x) = -EI w''(x)$:

```python
#analytical bending moment
moment_analytical = -deflection_analytical().diff(x,2)*EI
#numerical bending moment
moment_numerical = -deflection_ansatz().subs([(a1,coefficients[a1]),(a2,coefficients[a2])]).diff(x,2)*EI

#create lambdas for plotting along dimensionless length z
lam_x_analyt = sp.lambdify(z, moment_analytical.subs(x,z*L), modules=['numpy'])
lam_x_num = sp.lambdify(z, moment_numerical.subs(x,z*L), modules=['numpy'])
z_vals = np.linspace(0, 1,100)
analytical = lam_x_analyt(z_vals)
numerical = lam_x_num(z_vals)

#plot
plt.plot(x_vals/L, analytical/(L**2*q),label='analytical')
plt.plot(z_vals, numerical/(L**2*q),label='numerical')
plt.xlabel('$x\ /\ L$')
plt.ylabel('$M\ /\ L^2 q_0$')
plt.legend()
plt.tight_layout()
plt.savefig('beam_subdomain_moment.pdf')
```

```python
print("Maximum absolute error in bending moment: ", np.max(np.abs(analytical/(L**2*q) - numerical/(L**2*q))))
```

    Maximum absolute error in bending moment:  0.0177679314001814

*Tasks:*
- What happens if the interfal boundaries are shifted?
- How could you improve the solution?

```python

```


# Simplified Linear VMM (Vessel Maneuvering Model)

# Purpose
Implementing according to:
Matusiak, Jerzy. Dynamics of a Rigid Ship. Aalto University, 2017. https://aaltodoc.aalto.fi:443/handle/123456789/24408.

# Methodology
Define the problem using SymPy

# Setup

```python
# %load imports.py
%matplotlib inline
%load_ext autoreload
%autoreload 2
%config Completer.use_jedi = False  ## (To fix autocomplete)

## External packages:
import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)

import numpy as np
import os
import matplotlib.pyplot as plt

import plotly.express as px
import plotly.graph_objects as go

import seaborn as sns
import sympy as sp
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify

import pyro

import sklearn
import pykalman
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import statsmodels.api as sm

from scipy.integrate import solve_ivp

## Local packages:
#from src.data import mdl
from src import symbols
from src import prime_system
from src.symbols import *
from src.linear_vmm_equations import *

import src.linear_vmm_simplified_equations as eq
import src.models.linear_vmm as model
from src.visualization.plot import track_plot
import src.nonlinear_vmm_equations as nonlinear_vmm_equations

```

```python
symbols.df_parameters
```

## Linearized simplified equation of motion ($x_G=0$)

### X

Nonlinear equation of motion in X-direction:

```python
Math(vlatex(nonlinear_vmm_equations.X_eom))
```

this equation is linearized by assuming : $v=0$ and dropping $r^2$

```python
Math(vlatex(eq.X_eom))
```

### Y

Nonlinear equation of motion in Y-direction:

```python
Math(vlatex(nonlinear_vmm_equations.Y_eom))
```

this equation is linearized by assuming that $u \approx U $ where $U$ is the initial total velocity.

```python
Math(vlatex(eq.Y_eom))
```

### N

Nonlinear equation of motion in N-direction:

```python
Math(vlatex(nonlinear_vmm_equations.N_eom))
```

this equation is linearized by assuming that $u \approx U $ where $U$ is the initial total velocity.

```python
Math(vlatex(eq.N_eom))
```

```python
Math(vlatex(eq.fx_eq))
```

```python
Math(vlatex(eq.fy_eq))
```

```python
Math(vlatex(eq.mz_eq))
```

```python
Math(vlatex(eq.X_eq))
```

```python
Math(vlatex(eq.Y_eq))
```

```python
Math(vlatex(eq.N_eq))
```

```python
r1d_eq = sp.Eq(r1d, sp.solve(N_eq, r1d)[0])
```

## Steady state solution

```python
X_steady = X_eq.subs([
    (u1d,0),
    (v1d,0),
    (r1d,0),
          ])
X_steady
```

```python
Y_steady = Y_eq.subs([
    (u1d,0),
    (v1d,0),
    (r1d,0),
          ])
Y_steady
```

```python
N_steady = N_eq.subs([
    (u1d,0),
    (v1d,0),
    (r1d,0),
          ])
N_steady
```

```python
eqs = [Y_steady,N_steady]
solution = sp.solve(eqs,v,r, dict=True)
solution[0][v]
```

```python
solution[0][r]
```

```python
subs = {value:key for key,value in eq.p.items()}
#u_lambda = lambdify(solution[0][u].subs(subs))
v_lambda = lambdify(solution[0][v].subs(subs))
r_lambda = lambdify(solution[0][r].subs(subs))
```

## Prime system

```python
prime_system.df_prime.loc[['denominator']]
```

## Ship parameters

```python
T_ =10
L_ = 200
CB_ = 0.7
B_ = 30
rho_ = 1025
m_ = T_*B_*L_*CB_*rho_

ship_parameters = {
        'T' : T_,
        'L' : L_,
        'CB' :CB_,
        'B' : B_,
        'rho' : rho_,
        'x_G' : 0,
        'm' : m_,
        'I_z': 0.2*m_*L_**2,
    }

ps = prime_system.PrimeSystem(**ship_parameters)

```

```python
def calculate_prime(row, ship_parameters):
    return run(function=row['brix_lambda'], inputs=ship_parameters)

mask = df_parameters['brix_lambda'].notnull()
df_parameters.loc[mask,'prime'] = df_parameters.loc[mask].apply(calculate_prime, ship_parameters=ship_parameters, axis=1)
df_parameters.loc['Ydelta','prime'] = 0.0005  # Just guessing
df_parameters.loc['Ndelta','prime'] = -df_parameters.loc['Ydelta','prime']/4  # Just guessing
```

```python
df_parameters['prime'].fillna(0, inplace=True)
df_parameters
```

## Steady turning

```python
inputs = dict(df_parameters['prime'])
inputs.update(ps.prime(ship_parameters))

U = 10
delta = np.deg2rad(35)
inputs['delta'] = delta
inputs['u'] = inputs['U'] =  ps._prime(U,'linear_velocity', U=U)

v_steady_prime = run(function=v_lambda, inputs=inputs)
r_steady_prime = run(function=r_lambda, inputs=inputs)

v_steady = ps._unprime(v_steady_prime,'linear_velocity', U=U)
r_steady = ps._unprime(r_steady_prime,'angular_velocity' ,U=U)

```

```python
v_steady
```

```python
r_steady
```

```python
model.u1d_lambda
```

```python
model.v1d_lambda
```

```python
states = {
    'u' : ps._prime(U,'linear_velocity',U=U),
    'v' : ps._prime(v_steady,'linear_velocity',U=U),
    'r' : ps._prime(r_steady,'angular_velocity',U=U),
    #'u1d' : 0,
    #'v1d' : 0,
    #'r1d' : 0,

}

control = {
    'delta' : delta,
    'U' : ps._prime(U,'linear_velocity',U=U)
}

inputs = dict(df_parameters['prime'])
inputs.update(ps.prime(ship_parameters))

inputs.update(states)
inputs.update(control)

run(function=model.v1d_lambda, inputs=inputs)
```

```python
run(function=model.u1d_lambda, inputs=inputs)
```

```python
run(function=model.v1d_lambda, inputs=inputs)
```

```python
run(function=model.r1d_lambda, inputs=inputs)
```

```python
t_max = 2*np.pi/r_steady  # Turn one lap
t = np.arange(0,t_max,0.01)

control = {
    'delta' : delta,
}

parameters = dict(df_parameters['prime'])

u_steady = np.sqrt(U**2 - v_steady**2)
y0 = {
    'u' : u_steady,
    'v' : v_steady,
    'r' : r_steady,
    'x0' : 0,
    'y0' : 0,
    'psi' : 0,
    }

#solution = model.simulate(y0=y0, t=t, df_parameters=df_parameters, df_ship_parameters=df_ship_parameters, control=control, rtol=1e-3, atol=1e-3)
solution = model.simulate(y0=y0, t=t, df_parameters=df_parameters, ship_parameters=ship_parameters, control=control)

columns = ['u','v','r','x0','y0','psi']
df_result_prime = pd.DataFrame(data=solution.y.T, columns=columns)
df_result_prime.index=t[0:len(df_result_prime)]

#df_result_prime['U'] = np.sqrt(df_result_prime['u']**2 + df_result_prime['v']**2)

df_result_prime.plot(y='u')
df_result_prime.plot(y='v')
df_result_prime.plot(y='r')

```

```python
y0
```

```python
solution
```

```python
#units = {key:value[1] for key,value in zip(columns,y0)}
df_result = ps.unprime(values=df_result_prime, U=U)

df_result['beta'] = -np.arctan2(df_result['v'],df_result['u'])
```

```python
df_result.head()
```

```python
track_plot(df=df_result, lpp=ship_parameters['L'], beam=ship_parameters['B'])

df_result.plot(y='u')
df_result.plot(y='v')
df_result.plot(y='r')
df_result.plot(y='psi')
df_result.plot(y='beta')

```

## Steady turnings

```python
inputs = dict(df_parameters['prime'])
inputs.update(ps.prime(ship_parameters))

U = 10
delta = np.deg2rad(35)
inputs['delta'] = delta
inputs['u'] = inputs['U'] =  ps._prime(U,'linear_velocity', U=U)

s = pd.Series(inputs)
N=10
data = np.tile(s.values,(N,1))
df_variation = pd.DataFrame(data, columns=s.index)
df_variation.head()
df_variation['delta'] = np.linspace(0.01,0.3,N)

df_variation['v'] = run(function=v_lambda, inputs=df_variation)
df_variation['r'] = run(function=r_lambda, inputs=df_variation)

```

```python
df_variation.plot(x='delta', y='r')
df_variation.plot(x='delta', y='v')

```

```python
sp.solve(eq.Y_eq,v1d)[0]
```

```python
sp.solve(eq.N_eq,r1d)[0]
```

```python
sp.solve(eq.X_eq,u1d)[0]
```


# Kernel Method

## Polynomial Regression

basic linear regression with one variable:

$$y=\theta_{0} + \theta_{1}x$$

what if linear model could not nicely fit training examples?

we can naturally extend linear model to polynomial model, for example:

$$y=\theta_{0} + \theta_{1}x + \theta_{2}x^{2} + \theta_{3}x^{3}$$

this method can be conclude as: map original attibutes x to some new set of quantities $\phi(x)$ (called features), and use the same set of model.

### least mean squares with features
let $\phi : \mathbb{R}^{d} \to \mathbb{R}^{p}$ be a feature map, then original batch gradient descent:

$$\theta := \theta + \alpha\sum_{i=1}^{n}(y^{(i)} - \theta^{T}x^{(i)})x^{(i)}$$

using features:

$$\theta := \theta + \alpha\sum_{i=1}^{n}(y^{(i)} - \theta^{T}\phi(x^{(i)}))\phi(x^{(i)})$$

the above becomes computationally expensive when $\phi(x)$ is high dimensional.

but we can observe that, if at some point , $\theta$ can be represented as:

$$\theta = \sum_{i=1}^{n}\beta_{i}\phi(x^{(i)})$$

then in the next round:

$$
\begin{equation}
\begin{split}
\theta &:= \theta + \alpha\sum_{i=1}^{n}(y^{(i)} - \theta^{T}\phi(x^{(i)}))\phi(x^{(i)}) \\
&=\sum_{i=1}^{n}\beta_{i}\phi(x^{(i)}) + \alpha\sum_{i=1}^{n}(y^{(i)} - \theta^{T}\phi(x^{(i)}))\phi(x^{(i)}) \\
&=\sum_{i=1}^{n}(\beta_{i} + \alpha(y^{(i)} - \theta^{T}\phi(x^{(i)})))\phi(x^{(i)})
\end{split}
\end{equation}
$$

$\theta$ can be also represented as a linear representation of $\phi(x^{(i)})$

we can then derive $\beta$'s update rule:

$$\beta_{i} := \beta_{i} + \alpha(y^{(i)} - \sum_{j=1}^{n}\beta_{j}\phi(x^{(j)})^{T}\phi(x^{(i)}))$$

we only need to compute $\left \langle \phi(x^{(j)}), \phi(x^{(i)}) \right \rangle = \phi(x^{(j)})^{T}\phi(x^{(i)}))$ to update parameters no matter how high the feature dimension p is.

### kernel

we define the kernel corresponding to the feature map $\phi$ as a function that satisfying:

$$K(x, z) := \left \langle \phi(x), \phi(z) \right \rangle$$

define the kernel matrix:

$$K_{ij} = K(x^{(i)},x^{(j)})$$

properties of kernel matrix:

1. symmetric, since $\phi(x)^{T}\phi(z) = \phi(z)^{T}\phi(x)$.
2. positive semidefinite:

$$
\begin{equation}
\begin{split}
z^{T}Kz=&\sum_{i}\sum_{j}z_{i}K_{ij}z_{j}\\
=&\sum_{i}\sum_{j}z_{i}\phi(x^{(i)})^{T}\phi(x^{(j)})z_{j}\\
=&\sum_{i}\sum_{j}z_{i}\sum_{k}\phi_{k}(x^{(i)})\phi_{k}(x^{(j)})z_{j}\\
=&\sum_{k}\sum_{i}\sum_{j}z_{i}\phi_{k}(x^{(i)})\phi_{k}(x^{(j)})z_{j}\\
=&\sum_{k}\left(\sum_{i}z_{i}\phi_{k}(x^{(i)})\right)^2\\
\ge&\ 0
\end{split}
\end{equation}
$$

in the other hand, we have sufficient conditions for valid kernels:

(mercer): let $ K: \mathbb{R}^{d} \times \mathbb{R}^{d} \mapsto \mathbb{R}$. then for K be a valid kernel, it is necessary and sufficient that for any $\left \{ x^{(1)},...,x^{(n)} \right \} $, the corresponding kernel matrix is symmetric positive semi-definite.

proof t.b.c.

## support vector machine

### Margins

consider logistic regression, where the probability $p(y=1|x;\theta)$ is modeled by $h_{\theta}(x)=\sigma(\theta^{T}x)$. we predict 1 on an input x if and only if $\theta^{T}x >= 0.5$, the larger $\theta^{T}x$ is, the more confidence we are.

the distance from the hyperplane is important.

functional margin:

$$\hat{\gamma}^{(i)}=y^{(i)}(w^{T}x^{(i)} + b)$$

geometric margin:

$$\gamma^{(i)}=\frac{y^{(i)}(w^{T}x^{(i)} + b)}{\left \| w \right \| }$$

geometric margin is the euclid distance.

### the optimal margin classifier

we want to maximize geometic margin:

$$
\begin{equation}
\begin{split}
\underset{\gamma, w, b}{max}\ &\gamma \\
s.t\quad &\frac{y^{(i)}(w^{T}x^{(i)} + b)}{\left \| w \right \| } >= \gamma
\end{split}
\end{equation}
$$

without loss of generality, we can set $\gamma\left \| w \right \|=1$, then the above is equivalent to:

$$
\begin{equation}
\begin{split}
\underset{w, b}{min}\ &\frac{1}{2}{\left \| w \right \|}^2 \\
s.t\quad &{y^{(i)}(w^{T}x^{(i)} + b)} >= 1
\end{split}
\end{equation}
$$

### lagrange duality

consider the following primal optimization problem:

$$
\begin{equation}
\begin{split}
\underset{w}{min}\quad &f(w)\\
s.t\quad &g_{i}(w) \le 0,i=1,...,k\\
&h_{i}(w)=0, i=1,...,l
\end{split}
\end{equation}
$$

we define the lagrangian of this optimization problem:

$$L(w,\alpha,\beta)=f(w) + \sum_{i=1}^{k}\alpha_{i}g_{i}(w) + \sum_{i=1}^{l}\beta_{i}h_{i}(w)$$

here $\alpha_{i}, \beta_{i}$ are the lagrange multipliers. consider the quantity:

$$\theta_{P}(w) = \underset{\alpha,\beta:\alpha_{i}\ge{0}}{max}L(w,\alpha,\beta) $$

here $P$ stands for "primal".

let some $w$ be given, if $w$ violates any primal constraints, i.e., if either $g_{i}(w) > 0$ or $h_{i}(w) \ne 0$, then:

$$\theta_{P}(w) = \underset{\alpha,\beta:\alpha_{i}\ge{0}}{max}f(w) + \sum_{i=1}^{k}\alpha_{i}g_{i}(w) + \sum_{i=1}^{l}\beta_{i}h_{i}(w) = \infty$$

conversely, if the constraints are saitistied for particular $w$, then $\theta_{P}(w) = f(w)$, hence:

$$
\theta_{P}(w) =
\begin{cases}
f(w)\ &\text{if w satisfy constraints}\\
\infty\ &\text{otherwise.}
\end{cases}
$$

with this equation, we have:

$$\underset{w}{min}f(w),\ \text{satisfy constraints} \Leftrightarrow \underset{w}{min}\theta_{P}(w)=\underset{w}{min}\underset{\alpha,\beta:\alpha_{i}\ge{0}}{max}L(w,\alpha,\beta)$$

we define the optimal value of the objective function to be $p^{\ast} = \underset{w}{min}\theta_{P}(w)$.

reverse the min max oder, we define:

$$\theta_{D}(\alpha, \beta) = \underset{w}{min}L(w,\alpha,\beta)$$

D stands for "dual", we can pose the dual optimization problem:

$$\underset{\alpha,\beta:\alpha_{i}\ge{0}}{max}\theta_{D}(\alpha, \beta) = \underset{\alpha,\beta:\alpha_{i}\ge{0}}{max}\underset{w}{min}L(w,\alpha,\beta)$$

we define the optimal $d^{\ast}=\underset{\alpha,\beta:\alpha_{i}\ge{0}}{max}\theta_{D}(\alpha, \beta)$, it can easily be shown that:

$$d^{\ast}=\underset{\alpha,\beta:\alpha_{i}\ge{0}}{max}\theta_{D}(\alpha, \beta) \le \underset{w}{min}\theta_{P}(w) = p^{\ast}$$

however, under certain conditions, we have:

$$d^{\ast} = p^{\ast}$$

so that we can solve the dual problem instead of the primal problem.

(KKT conditions)suppose $f$ and $g_{i}$ are convex, $h_{i}$ are affine, and there exits some $w$ so that $g_{i}(w) < 0$ for all $i$.

under these assumptions, there must exists $w^{\ast}, \alpha^{\ast}, \beta^{\ast}$ so that $w^{\ast}$ is the solution to the primal problem, $\alpha^{\ast},\beta^{\ast}$ are the solutions to the dual problem, and $p^{\ast}=d^{\ast}=L(w^{\ast}, \alpha^{\ast}, \beta^{\ast})$.

moreover, $w^{\ast}, \alpha^{\ast}, \beta^{\ast}$ iff satisfy the KKT conditions:

$$
\begin{equation}
\begin{split}
\frac{\partial}{\partial w_{i}}L(w^{\ast},\alpha^{\ast},\beta^{\ast}) =& 0,\ i=1,...,d\\
\frac{\partial}{\partial \beta_{i}}L(w^{\ast},\alpha^{\ast},\beta^{\ast}) =& 0,\ i=1,...,l\\
\alpha_{i}^{\ast}g_{i}(w) =& 0,\ i=1,...,k\\
g_{i}(w^{\ast}) \le& 0,\ i=1,...,k\\
\alpha_{i}^{\ast} \ge& 0,\ i=1,...,k
\end{split}
\end{equation}
$$

proof t.b.c.

## 2.4 use lagrange duality

previous optimal margin problem:

$$
\begin{equation}
\begin{split}
\underset{w, b}{min}\ &\frac{1}{2}{\left \| w \right \|}^2 \\
s.t\quad &{y^{(i)}(w^{T}x^{(i)} + b)} >= 1
\end{split}
\end{equation}
$$

we can write the constraints as:

$$g_{i}(w) = 1 - y^{(i)}(w^{T}x^{(i)} + b) \le 0$$

this problem satisfy the KKT prerequisites:

1. $\frac{1}{2}{\left \| w \right \|}^2$ is convex.
2. $1 - y^{(i)}(w^{T}x^{(i)} + b)$ is convex.
3. there exits $w, b$, such that $1 - y^{(i)}(w^{T}x^{(i)} + b) < 0$, just increase the order.

write the lagrangian of this problem:

$$
L(w,b,\alpha )=\frac{1}{2}\left \| w \right \|^{2} - \sum_{i=1}^{n}\alpha_{i}\left [ y^{(i)}(w^{T}x^{(i)} + b) - 1 \right ]
$$

just solve the dual form of the problem, to do so, we need to first minimize $L(w,\alpha,\beta)$ with respect to $w,b$, setting derivatives to 0:

$$\nabla_{w}L(w, b, \alpha) = w - \sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)} = 0$$

this implies:

$$w = \sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)}$$

derivative with respect to b:

$$\nabla_{b}L(w, b, \alpha) = \sum_{i=1}^{n}\alpha_{i}y^{(i)}=0$$

plug these equations back to lagrangian:

$$
\begin{equation}
\begin{split}
L(w, b, \alpha) =& \frac{1}{2}\left \| w \right \|^{2} - \sum_{i=1}^{n}\alpha_{i}\left [ y^{(i)}(w^{T}x^{(i)} + b) - 1 \right ]\\
=& \frac{1}{2}(\sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)})^{T}(\sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)}) - \sum_{i=1}^{n}\alpha_{i}y^{(i)}(\sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)})^{T}x^{(i)} + b\sum_{i=1}^{n}\alpha_{i}y^{(i)} + \sum_{i=1}^{n}\alpha_{i}\\
=& \sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}(x^{(i)})^{T}x^{(i)}
\end{split}
\end{equation}
$$

we thus obtain the following dual optimization problem:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{max}\quad &W(\alpha)=\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle x^{(i)},x^{(j)} \right \rangle \\
s.t\quad &\alpha_{i}\ge{0},\ i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

this is easier to solve(we'll talk about it later). after abtained $\alpha_{i}^{\ast}$, we have:

$$w^{\ast} = \sum_{i=1}^{n}\alpha_{i}^{\ast}y^{(i)}x^{(i)}$$

go back to the original problem, we get:

$$b^{\ast} = -\frac{max_{i: y^{(i)}=-1}w^{\ast{T}}x^{(i)} + min_{i: y^{(i)}=1}w^{\ast{T}}x^{(i)}}{2} $$

when making predictions, we have:

$$w^{T}x + b = \left ( \sum_{i=1}^{n}\alpha_{i}y^{(i)}x^{(i)}\right )^{T}x + b = \sum_{i=1}^{n}\alpha_{i}y^{(i)}\left \langle x^{(i)},x \right \rangle + b$$

we only needed inner product.

### non-separable case

so far, we assumed the data is linearly separable.

while mapping data to a high dimensional feature space via $\phi$ does not increase the likelihood that the data is separable.

so we need to make the algorithm work for non-linearly separable datasets.

we reformulate our optimization as follows:

$$
\begin{equation}
\begin{split}
\underset{w, b}{min}\ &\frac{1}{2}{\left \| w \right \|}^2 + C\sum_{i=1}^{n}\xi_{i}\\
s.t\quad &{y^{(i)}(w^{T}x^{(i)} + b)} >= 1 - \xi_{i},\ i=1,...,n\\
&\xi_{i} \ge 0,\ i=1,...,n.
\end{split}
\end{equation}
$$

the cost of outlier $C\xi_{i}$.

the lagrangian:

$$
    L(w,b,\xi,\alpha,r )=\frac{1}{2}\left \| w \right \|^{2} + C\sum_{i=1}^{n}\xi_{i} - \sum_{i=1}^{n}\alpha_{i}\left [ y^{(i)}(w^{T}x^{(i)} + b) - 1 + \xi_{i}\right ] -\sum_{i=1}^{n}r_{i}\xi_{i}
$$

here $\alpha_{i},r_{i}$ are our lagrange multipliers.

the dual form of the problem:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{max}\quad &W(\alpha)=\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle x^{(i)},x^{(j)} \right \rangle \\
s.t\quad &0 \le \alpha_{i}\le{C},\ i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

the same as before except $\alpha_{i}$'s constraints, the calculation for $b^{\ast}$ has to be modified, t.b.c.

```python
import numpy as np
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import LinearSVC
```

```python
iris = datasets.load_iris()
X = iris["data"][:, 2:]
y = (iris["target"] == 2).astype(np.int)

svm_clf = Pipeline([("scaler", StandardScaler()),
                    ("linear_svc", LinearSVC(C=1, loss="hinge"))])

svm_clf.fit(X, y)
```

    Pipeline(steps=[('scaler', StandardScaler()),
                    ('linear_svc', LinearSVC(C=1, loss='hinge'))])

### SMO algorithm

consider when trying to solve the unconstrained optimization problem:

$$\underset{\alpha}{max}\ W(\alpha_{1},...,\alpha_{n})$$

we have the coordinate ascent algorithm:

$$
\begin{equation}
\begin{split}
&\text{loop until convergence:}\\
&\qquad \text{for }i=1,...,n:\\
&\qquad\qquad \alpha_{i}:=\underset{\hat{\alpha_{i}}}{argmax}
W(\alpha_{1},...,\alpha_{i-1},\hat{\alpha_{i}},\alpha_{i+1},...,\alpha_{n}).
\end{split}
\end{equation}
$$

we can not directly use coordinate ascent facing SVM dual problem:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{max}\quad &W(\alpha)=\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle x^{(i)},x^{(j)} \right \rangle \\
s.t\quad &0 \le \alpha_{i}\le{C},\ i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

if we want to update some $\alpha_{i}$, we must update at least two of them simutaneously.

coordinate ascent change to:

$$
\begin{equation}
\begin{split}
&\text{loop until convergence:}\\
&\qquad 1.\text{select some pair } \alpha_{i}, \alpha_{j}.\\
&\qquad 2.\text{optimize } W(\alpha) \text{ with respect to } \alpha_{i}, \alpha_{j}, \text{while holding other } \alpha_{k} \text{ fixed}.
\end{split}
\end{equation}
$$

the reason that SMO is an efficient is that the update to $\alpha_{i}, \alpha_{j}$ can be computed efficiently.

take $i,j=1,2$ as example, SMO step is like:

$$max\ W(\alpha_{1},\alpha_{2},...,\alpha_{n}) \text{ while } $$
$$\alpha_{3},...,\alpha_{n} \text{ fixed}$$
$$\alpha_{1}y^{(1)} + \alpha_{1}y^{(1)} = \zeta$$
$$0 \le \alpha_{1} \le{C},0 \le \alpha_{2} \le{C},$$

this can be change to:

$$max\ W(\alpha_{1},(\zeta - \alpha_{1}y^{(1)})y^{(2)},...,\alpha_{n}) \text{ while } L\le \alpha_{1}\le H$$

this is direct quadratic optimization, easy to solve.

remained questions:

1. the choice of $\alpha_{i},\alpha_{j}$ , this is heuristic.
2. how to update b.

t.b.c according to Platt's paper.

## SVM with kernels

let $\phi : \mathbb{R}^{d} \to \mathbb{R}^{p}$ be a feature map, the original dual form of the problem:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{max}\quad &W(\alpha)=\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle x^{(i)},x^{(j)} \right \rangle \\
s.t\quad &\alpha_{i}\ge{0},\ i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

now change to:

$$
\begin{equation}
\begin{split}
\underset{\alpha}{max}\quad &W(\alpha)=\sum_{i=1}^{n}\alpha_{i} - \sum_{i,j=1}^{n}y^{(i)}y^{(j)}\alpha_{i}\alpha_{j}\left \langle \phi(x^{(i)}),\phi(x^{(j)}) \right \rangle \\
s.t\quad &\alpha_{i}\ge{0},\ i=1,...,n \\
&\sum_{i=1}^{n}\alpha_{i}y^{(i)}=0
\end{split}
\end{equation}
$$

we only need to know $\left \langle \phi(x^{(i)}),\phi(x^{(j)}) \right \rangle$ to optimize.

when predicting:

$$w^{T}\phi(x) + b = \left ( \sum_{i=1}^{n}\alpha_{i}y^{(i)}\phi(x^{(i)})\right )^{T}x + b = \sum_{i=1}^{n}\alpha_{i}y^{(i)}\left \langle \phi(x^{(i)}),\phi(x) \right \rangle + b$$

```python
"""PolynomialFeatures + LinearSVC"""
from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures

polynomial_svm_clf = Pipeline([
        ("poly_features", PolynomialFeatures(degree=3)),
        ("scaler", StandardScaler()),
        ("svm_clf", LinearSVC(C=10, loss="hinge", random_state=42))
    ])

polynomial_svm_clf.fit(X, y)
```

    Pipeline(steps=[('poly_features', PolynomialFeatures(degree=3)),
                    ('scaler', StandardScaler()),
                    ('svm_clf', LinearSVC(C=10, loss='hinge', random_state=42))])

```python
"""poly SVC"""
from sklearn.svm import SVC

poly_kernel_svm_clf = Pipeline([
        ("scaler", StandardScaler()),
        ("svm_clf", SVC(kernel="poly", degree=3, coef0=1, C=5))
])
poly_kernel_svm_clf.fit(X, y)
```

    Pipeline(steps=[('scaler', StandardScaler()),
                    ('svm_clf', SVC(C=5, coef0=1, kernel='poly'))])

```python
"""rbf SVC"""
from sklearn.svm import SVC

rbf_kernel_svm_clf = Pipeline([
    ("scaler", StandardScaler()),
    ("svm_clf", SVC(kernel="rbf", gamma=5, C=0.001))
])
rbf_kernel_svm_clf.fit(X, y)
```

    Pipeline(steps=[('scaler', StandardScaler()),
                    ('svm_clf', SVC(C=0.001, gamma=5))])

```python
"""SVM can support regression"""
from sklearn.svm import SVR

svm_poly_reg = SVR(kernel="poly", degree=2, C=100, epsilon=0.1, gamma="scale")
svm_poly_reg.fit(X, y)
```

    SVR(C=100, degree=2, kernel='poly')

```python

```


# Conditioning
Due to the finite precision in our computers and probably some other practical reasons, there will be some errors in the input data $x$, which lead to the error in the solution $y$.

so for a problem $y = f(x):X \longmapsto Y$, we define its **conditional number** as:

$$\kappa = \max_{\Delta x} \left( \frac{\left\| \Delta f \right\|} {\left\| f(x) \right\|} \Big/ \frac{\left\| \Delta x \right\|} {\left\| x \right\|} \right)$$

Here $x$ and $f(x)$ are the exact input and solution, and $\Delta x$ and $\Delta f$ are absolute errors, and $\kappa$ represents the *sensitivity* of the solution to perturbations in the input.

If $\kappa$ is small as $10^2$ or less, then the solution of the problem is not sensitive to the perturbations in the input data. We say this problem is **well-conditioned**, otherwise it is **ill-conditioned**, like its $\kappa$ has reached $10^6$ or higher.

## Condition number of functions
Now we assume that $y = f(x):\mathbb{R} \longmapsto \mathbb{R}$ and $f(x)$ is differentiable. Then we have

$$\kappa = \max_{\Delta x} \left( \frac{\left\| \Delta f \right\|} {\left\| f(x) \right\|} \Big/ \frac{\left\| \Delta x \right\|} {\left\| x \right\|} \right) = \frac{ \lim_{\Delta x \to 0}\big( \left| \Delta f \right| \big/ \left| \Delta x \right| \big)} {\left| f (x) \right| \big/ \left| \,x\, \right|} = \Bigg|\, \frac{f'(x)} {f(x)/x} \, \Bigg| $$

为什么可以从max直接换到lim？？？

>**e.g.**
>
>$f(x) = \frac{1} {x}$ at mathbb{R} except for $0$. Here the condition number is
>
>$$\kappa = \Bigg|\, \frac{f'(x)} {f(x)/x} \, \Bigg| = \Bigg|\, \frac{x^{-2}} {x^{-1}/x} \, \Bigg| = 1$$
>
>So it is well-conditioned, even near zero. See $f(0.001)$ and $f(0.001001)$. The relative error of input is $10^{-3}$, and the relative error of output is $0.999/1000 \approx 10^{-3}$, and there quotient is just as the condition number, $1$.

***
>**e.g.**
>
>Another example is the *subtraction* operation. Let the inputs be $x_1, x_2 \in \mathbb{R}$, and the output be $y = f(x_1, x_2) = x_1 - x_2$.
>
>Consider a perturbation in the input as $\Delta x_1$ and $\Delta x_2$ , and the relative perturbation is then defined by $\max \left\{ \left| \frac{\Delta x_1} {x_1} \right|, \left| \frac{\Delta x_2} {x_2} \right| \right\}$, with assumption that $x_1, x_2 \neq 0$. And then the relative output error is:
>
>$$\left| \frac{\big((x_1 + \Delta x_1) - (x_2 + \Delta x_2) \big) - (x_1 - x_2)} {x_1 - x_2} \right| = \left| \frac{\Delta x_1 - \Delta x_2} {x_1 - x_2} \right|$$
>
>And this can be huge if $x_1 \approx x_2$ . This implies that the subtraction of two values is a ill-conditioned problem when the two values are very close.

## Condition number of finding the roots of polynomials
Finding the roots of polynomials is an ill-conditioned problem.

>**e.g.**
>
>The roots of $x^2 − 2x + 1 = 0$, which are $x = 1$. A perturbation would be like $x^2 − 2x + 0.99999999 = 0$, and now the roots are $0.99999$ and $1.00001$. That is a relative error in the coefficient in the order of $10^{-10}$ leads to a relative error of roots in the order of $10^{-5}$.

## Condition number of matrices
Given a $m \times m$ nonsingular matrix $A$, with input data $\vec{x}$ and then we have: $\vec{y} = A\vec{x}: \mathbb{R} \longmapsto \mathbb{R}$

And since we have

$$\begin{align*}
\kappa &= \max_{\Delta x} \left( \frac{\left\| A(\vec{x} + \Delta \vec{x}) - A\vec{x} \right\|} {\left\| A\vec{x} \right\|} \middle/ \frac{\left\| \Delta \vec{x} \right\|} {\left\| \vec{x} \right\|} \right) \\
&= \max_{\Delta x} \left( \frac{\left\| A \Delta \vec{x} \right\|} {\left\| A\vec{x} \right\|} \middle/ \frac{\left\| \Delta \vec{x} \right\|} {\left\| \vec{x} \right\|} \right) \\
&= \max_{\Delta x} \left( \frac{\left\| A \Delta \vec{x} \right\|} {\left\| \Delta \vec{x} \right\|} \right) \cdot \frac{\left\| \vec{x} \right\|} {\left\| A\vec{x} \right\|} \\
&= \left\| A \right\| \cdot \frac{\left\| A^{-1}A\vec{x} \right\|} {\left\| A\vec{x} \right\|} \leq \small\| A \small\| \small\| A^{-1} \small\|
\end{align*}$$

So we define the condition number here as: $\DeclareMathOperator{\cond}{Cond}
\cond(A)= \small\| A \small\|_2 \small\| A^{-1} \small\|_2$

Later we can see that actually $\left\| A\right\|_2$ is the maximum singular values奇异值 of the matrix $A$, $\sigma_1$, and $\small\| A^{-1} \small\|_2$ is the inverse of the minimum singular values, $\frac{1} {\sigma_m}$.

*Singular value is the square roots of the eigenvalues of $A^{\mathrm{H}}A$, here $A^{\mathrm{H}}$ is the Conjugate Transpose* (from [MathWorld](http://mathworld.wolfram.com/SingularValue.html))

But for the case that $A$ is a **real symmetric matrix**, the singular values of $A$ are just the absolute values of its eigenvalues.

## Application
And if the matrix is singular, then $\sigma_m = 0$, and so that $\cond(A) = \infty$. Same in MATLAB with `cond` command.

And when solving the linear equation: $A\vec{\mathstrut x} = \vec{\mathstrut b}$, this problem has the condition number bounded by $\small\| A \small\|_2 \small\| A^{-1} \small\|_2$

The following example demonstrates that how the accuracy of multiplying a matrix with a vector depends on the condition number of the matrix.

>**e.g.**
>
>Consider $A =
\left[
\begin{array} {cc}
1 & 2 \\
1 & 1
\end{array}
\right], B=
\left[
\begin{array} {cc}
1 & 1 + 10^{-14} \\
1 &  1
\end{array}
\right]$
>
>Using MATLAB, we can see that $\cond(A) \approx 6.8541, \cond(B) \approx 4.0191\times 10^{10}$. Now let us compute $A\vec{x}$ and $B\vec{x}$ for $\vec{x} = \left[
\begin{array} {c}
1 \\
-1
\end{array}
\right]$ and consider their accuracies for the vector
>
>Of course the exact answer is
>
>$$ A\vec{x} = \left[ \begin{array} {c}
1 \\
0
\end{array} \right], B\vec{x} = \left[ \begin{array} {c}
-10^{-14} \\
0
\end{array} \right]$$
>
>However in MATLAB, we can see that
>
>$$ \widetilde{ A\vec{x}} = \left[ \begin{array} {c}
1 \\
0
\end{array} \right], \widetilde{ B\vec{x}} = \left[ \begin{array} {c}
-0.992 \times 10^{-14} \\
0
\end{array} \right]$$
>
>Since matrix $A$ has small condition number, MATLAB can give out a relatively accurate result, but for $B$, even the relative error of input is in the order of $10^{-16}$, the relative error of output is still of $10^{-3}$.

# Stability
To solve a problem accurately, first of all the **problem** has to be well-conditioned. Otherwise any input error will be amplified enormously in the solution. But that's not enough. The accuracy also depends on the **algorithm** used to solve the problem. Here we discuss the stability of algorithms used to solve problems.

We learn this through an example, consider solving

$$A\vec{x} = \left[ \begin{array} {cc}
10^{-20} & 1 \\
1 & 1
\end{array} \right] \left[ \begin{array} {c}
x_1 \\
x_2
\end{array} \right] = \left[ \begin{array} {c}
1 \\
0
\end{array} \right]$$

Here the condition number of the matrix $A$ is $2.61$, so it's well-conditioned.

## Gaussian elimination and Pivoting step
$
\left\{
\begin{array}{ccrcll}
10^{-20} \, x_1 &+&  x_2 & =& 1 & (1)\\[1ex]
(1- 10^{20} \times 10^{-20}) \, x_1 &+&  (1- 10^{20} \times 1) \, x_2 & =& 0- 10^{20} \times 1 & (2)
\end{array}
\right.
$

And this is an upper triangular matrix, so that from $\textrm{(2)}$ we have $ -10^{20} \, x_2 = -10^{20}$, because in computer, $ 1- 10^{20} = - 10^{20}$, even in double precision. Therefore $x_2 = 1$ and $x_1 = 0$. And that's just not the answer.

So we use **pivioting** step: before each step of the Gauss elimination, the rows need to be exchanged such that the value appearing at the pivot diagonal position be the largest one among all entries below the pivot position. So now what we're going to solve is:

$$B\vec{x} = \left[ \begin{array} {cc}
1 & 1 \\
10^{-20} & 1
\end{array} \right] \left[ \begin{array} {c}
x_1 \\
x_2
\end{array} \right] = \left[ \begin{array} {c}
0 \\
1
\end{array} \right]$$

Now then we have the Gaussian Elimination like below:

$
\left\{
\begin{array}{ccrcll}
x_1 &+&  x_2 & =& 0 & (3)\\[1ex]
(10^{-20}- 10^{-20} \times 1) \, x_1 &+&  (1- 10^{-20} \times 1) \, x_2 & =& 1- 10^{-20} & (4)
\end{array}
\right.
$

So we can get that $x_2 = 1$ from $\textrm{(2)}$, Then plug in $x_2 = 1$ to the first equations, we have $x_1 = −1$, which is in fact the solution to a problem close to the original system.

## General condition
When we are solving $x \longrightarrow f(x): X \longmapsto Y$ in a computer, firstly the input $x$ is transformed to say $\tilde{x}$, and then the output is given and we denote this process as

$$\tilde{f}(x) : X \longmapsto Y$$

Here, $\tilde{f}(x)$ represents the computed result of the problem in the computer by the given algorithm.

$Def$

We say an algorithm $\tilde{f}(x)$ implemented on a computer to solve the mathematical problem $f(x)$ is **stable** (backward stable), if for any input $x \in X$, the computed result$\tilde{f}(x)$ is the true solution to a nearby input, $i.e.$, there exists a nearby input $\tilde{x} = x(1 + O(\varepsilon_{\text{machine}} ))$, such that

$$\tilde{f}(x) = f(\tilde{x})$$

And a not very surprising result is that for operation $+$, $-$, $\cdot$, $\div$ are all stable. Now we check the stability of $x_1 - x_2$.

Given real numbers $x_1, x_2$, then $fl(x_1) = x_1(1+\varepsilon_1),fl(x_2) = x_2(1+\varepsilon_2)$, where $\varepsilon_1, \varepsilon_2$ are in the order of $O(\varepsilon_{\text{machine}})$, so in computer:

$\begin{align}
fl(x_1) \ominus fl(x_2) &= \big(fl(x_1) - fl(x_2)\big)\cdot(1+ \varepsilon_3) \\
&= x_1/,(1+\varepsilon_1)(1+\varepsilon_3) - x_2/,(1+\varepsilon_2)(1+\varepsilon_3) \\
&= x_1/,(1+\varepsilon_4) - x_2/,(1+\varepsilon_5)
\end{align}$

This means that the computed value is the true difference of the two nearby values and therefore this operation is backward stable.

And an example of an unstable is outerproduct $\times$. Given $\vec{x} \in \mathbb{R}^n$, $\vec{y} \in \mathbb{R}^n$, compute $A = \vec{x} \times \vec{y}^{\mathrm{T}}$. And the obvious is to compute entry by entry as $x_iy_j$, denoted as $\tilde{A}$.

Due to the rounding error, it is almost unlike for $\tilde{A}$ to have a rank of 1, but the true value of $(x + \Delta x)^{\mathrm{T}} \times (y + \Delta y)^{\mathrm{T}}$ is always with rank of 1.

# Accuracy

$Def$

Given a mathematical problem $y = f(x):X \longmapsto Y$ and the algorithm to solve it in computer $\tilde{f}(x) : X \longmapsto Y$, we say that it is **accurate** if for any input $x \in X$, we have

$$\frac{\left\| \tilde{f}(x) - f(x) \right\|} {\left\| f(x) \right\|} = O(\varepsilon_{\text{machine}} )$$

Following Theorems shows that when a stable algorithm $\tilde{f}(x)$ is used to solve a problem $f(x)$, the accuracy of the algorithm is only dependent on the condition number of the problem $f(x)$.

$Theorem$

Let a stable algorithm $\tilde{f}(x)$ be used to solve a problem $f(x)$, whose condition number is $\kappa(x)$. Then, for any input x, the relative error of the computed solution satisfies:

$$\frac{\left\| \tilde{f}(x) - f(x) \right\|} {\left\| f(x) \right\|} = O(\kappa(x) \cdot \varepsilon_{\text{machine}} )$$

$Proof$

Since the algorithm is stable, we can know that $\tilde{f}(x) = f(\tilde{x})$, for some $\tilde{x}$ with $\displaystyle \frac{\left\| \tilde{x} - x \right\|} {\left\| x \right\|} = O(\varepsilon_{\text{machine}} )$

And we also have:

$$\kappa = \max_{\Delta x} \left( \frac{\left\| \Delta f \right\|} {\left\| f(x) \right\|} \Big/ \frac{\left\| \Delta x \right\|} {\left\| x \right\|} \right) \geq \left( \frac{\left\| f(\tilde{x}) - f(x) \right\|} {\left\| f(x) \right\|} \Big/ \frac{\left\| \tilde{x} - x \right\|} {\left\| x \right\|} \right)$$

So that we can finally get that:

$$\frac{\left\| \tilde{f}(x) - f(x) \right\|} {\left\| f(x) \right\|} = \frac{\left\| f(\tilde{x}) - f(x) \right\|} {\left\| f(x) \right\|} \leq \kappa(x) \cdot \frac{\left\| \tilde{x} - x \right\|} {\left\| x \right\|} = O(\kappa(x) \cdot \varepsilon_{\text{machine}} )$$

$\square$

So for accuracy:
1. condition number of the problem
2. the stability of the algorithm

# Singular Value Decomposition of Matrices (SVD)
## Hyperellipse
The singular value decomposition of matrix $A$ is motivated by the following geometric fact: *the image of
a unit sphere in the space $\mathbb{R}^n$ under any $m \times n$ matrix is a hyperellipse in thespace $\mathbb{R}^m$.*

Here the *hyperellipse* means an ellipse with $m$ orthogonal directions $\vec{u}_1, \vec{u}_2, \dots,\vec{u}_m \in \mathbb{R}^m$ corresponding to its principle semiaxes, with the length of the each principle semiaxes represented by $\sigma_1, \sigma_2 , \dots, \sigma_m$. And we assume that $\| u_i \|_2 = 1$ for $i = 1,2,\dots,m$ and $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_m \geq 0$. Besides, some $\sigma_i$ can equal $0$. For example only r number of principle vector has norm greater than $0$, then the hyperellipse is located within a $r\textrm{-dimension}$ hypersurface of the space $\mathbb{R}^m$ spanned by orthogonal vector $\vec{u}_1, \vec{u}_2, \dots,\vec{u}_r$.

Then given matrix $A \in \mathbb{R}^{m \times n}$ ($m \geq n$), the image of the unit sphere in space $\mathbb{R}^n$ under the map $\vec{y} = A\vec{x}$ is a hyperellipse of $n\text{-dimension}$ hypersurface of the space $\mathbb{R}^m$. We denote the $n$ principle semiaxes of the hyperellipse as $\sigma_1 \vec{u}_1, \sigma_2 \vec{u}_2, \dots,\sigma_n \vec{u}_n$, which are the images of $n$ orthonormal vectors $\vec{v}_1, \vec{v}_2, \cdots, \vec{v}_n$ in the space $\mathbb{R}^n$ , $i.e.$,

$$A\vec{v_i} = \sigma_i \vec{u_i}, \;\; i = 1, 2, 3, \dots, n$$

This is called the singular value decomposition of the matrix $A$, and we call $\sigma_1, \sigma_2 , \dots, \sigma_n$ the singular values of the matrix $A$.

## Conclusions
1. If the rank of $A$ is $r$ and $r \leq n$, then only the first $r$ singular values $\sigma_1, \sigma_2 , \dots, \sigma_r$ of $A$ are nonzero and the others $\sigma_{r+1} = \sigma_{r+2} = \cdots = \sigma_{n} = 0$. The hyperellipse
is therefore located in the hypersurface formed by $\vec{u_1}, \vec{u_2}, \dots,\vec{u_r}$, which in fact form a **basis of the range space of matrix** $A$.
2. The **kernel space** of $A$ is the space spanned by the vectors $\vec{u}_{r+1}, \vec{u}_{r+2}, \dots,\vec{u}_{n}$. 3. $\vec{v}_1$ is amplified the **most** and $\vec{v}_r$ is amplified the least, seeing from the magnitude relationship of $\sigma_1$ and $\sigma_r$.

We also write the SVD as:

$$ AV = A\cdot\left[
\begin{array}{c|c|c|c}
& & & \\[0.3ex]
\vec{v_1} & \vec{v_2} & \cdots & \vec{v_n} \\[0.3ex]
& & &
\end{array}
\right] = \left[
\begin{array}{c|c|c|c}
& & & \\[0.3ex]
\vec{u_1} & \vec{u_2} & \cdots & \vec{u_n} \\[0.3ex]
& & &
\end{array}
\right] \cdot \left[
\begin{array}{ccccccc}
\sigma_1 & 0 & \cdots & 0 & 0 & \cdots & 0 \\
0 & \sigma_2 & \cdots & 0 & 0 & \cdots & 0 \\
0 & 0 & \ddots & 0 & 0 & \cdots & 0 \\
0 & 0 & \cdots & \sigma_n & 0 & \cdots & 0 \\
0 & 0 & \cdots & 0 & 0 & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & 0 & 0 & \cdots & 0
\end{array}
\right] = \widehat{U}\widehat{\Sigma}$$

where $V \in \mathbb{R}^{n \times n}, \widehat{U} \in \mathbb{R}^{m \times n}, \widehat{\Sigma} \in \mathbb{R}^{n \times n}$.

And we can also add an additional $m-n$ orthonormal vectors $\vec{u}_{n+1}, \vec{u}_{n+2}, \dots,\vec{u}_{m}$ to $\vec{u}_1, \vec{u}_2, \dots,\vec{u}_{n}$, such that a whole orthogonal basis of space $\mathbb{R}^m$ is formed. Anyway they will be multiplied by 0, no effects on our results. So now we can write:

$$AV = \left[
\begin{array}{c|c|c|c|c|c}
& & & & &\\[0.3ex]
\vec{u}_1 & \vec{u}_2 & \cdots & \vec{u}_n & \vec{u}_{n+1} & \cdots & \vec{u}_m\\[0.3ex]
& & & & &
\end{array}
\right] \cdot \left[
\begin{array}{ccccccc}
\sigma_1 & 0 & \cdots & 0 & 0 & \cdots & 0 \\
0 & \sigma_2 & \cdots & 0 & 0 & \cdots & 0 \\
0 & 0 & \ddots & 0 & 0 & \cdots & 0 \\
0 & 0 & \cdots & \sigma_n & 0 & \cdots & 0 \\
0 & 0 & \cdots & 0 & 0 & \cdots & 0 \\
\vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots \\
0 & 0 & \cdots & 0 & 0 & \cdots & 0 \\
0 & 0 & \cdots & 0 & 0 & \cdots & 0 \\
\vdots & \vdots & & \vdots & \vdots & & \vdots \\
0 & 0 & \cdots & 0 & 0 & \cdots & 0
\end{array}
\right] = U \Sigma$$

And now $V \in \mathbb{R}^{n \times n}, U \in \mathbb{R}^{m \times m}, \Sigma \in \mathbb{R}^{m \times n}$. And since $V$ is orthogonal, we finally get that:

$$\boxed{A = U \Sigma V^{\mathrm{T}}}$$

which is the matrix form of the SVD of the matrix $A$.

$Theorem 1$

*Let $r$ be the number of nonzero singular values of the matrix $A$. Then the **range space** of $A$ is spanned by the vectors $\vec{u}_1, \vec{u}_2, \dots,\vec{u}_r$, and the **null space** of *A* is spanned by $\vec{\upsilon}_{r+1}, \vec{\upsilon}_{r+2}, \cdots, \vec{\upsilon}_n$.

$Theorem 2$

Let $r$ be the number of **nonzero** singular values of the matrix $A$. Then the *rank* of $A$ is $r$, $i.e.$, the dimension of the **column space** of $A$ is $r$.

$Theorem 3$

Let $A$ be a $m \times m$ square matrix. Then the determinant of $A$ equals
$$ \prod _{i = 1} ^{m} \sigma_1$$

***
Now let $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_r > 0$ be the nonzero singular values of the matrix, $A$. Then the SVD can be also written as

$$A = \sum _{i = 1} ^{r} \sigma_i \vec{u}_i \vec{\upsilon}_i^{\mathrm{T}}$$

$\dagger$
This is the **sum of matrix**!

Also for $\nu = 1,2,\dots,r$ define $\displaystyle A_{\nu} = \sum _{i = 1} ^{\nu} \sigma_i \vec{u}_i \vec{\upsilon}_i^{\mathrm{T}}$, and after this definition we have

$Theorem 4$

$$\left\| A - A_{\nu}\right\| = \sigma_{\nu + 1}$$

$Proof$

Since $\vec{\upsilon}_1, \vec{\upsilon}_2, \cdots, \vec{\upsilon}_n$ is an orthonormal basis of the space $\mathbb{R}^n$, we denote the vector $\vec{x}$ by $x_1 \vec{\upsilon}_1 + x_2 \vec{\upsilon}_2 + \cdots + x_n \vec{\upsilon}_n$, where $x_{1}^{2} + x_{2}^{2} + \cdots +  x_{n}^{2} = 1$ since $\|x\|_2 = 1$. So we have

$$(A - A_{\nu})\vec{x} = \left( \sum _{i = \nu + 1} ^{n} \sigma_i \vec{u}_i \vec{\upsilon}_i ^{\mathrm{T}} \right) \big(x_1 \vec{\upsilon}_1 + x_2 \vec{\upsilon}_2 + \cdots + x_n \vec{\upsilon}_n \big) = \sum _{i = \nu + 1} ^{n} \sigma_i \vec{u}_i x_i$$

So that we have

$$\begin{align}
\big\| (A - A_{\nu})\vec{x} \big\|_{2} ^{2} &= \big( (A - A_{\nu})\vec{x} \big)^{\mathrm{T}} (A - A_{\nu})\vec{x} \\
&= \left( \sum _{i = \nu + 1} ^{n} \sigma_i \vec{u}_i^{\mathrm{T}} x_i \right) \cdot \left( \sum _{j = \nu + 1} ^{n} \sigma_j \vec{u}_j x_j \right) \\
&= \sum _{i = \nu + 1} ^{n} \sigma_i^{2}  x_i^{2} \leq \sum _{i = \nu + 1} ^{n} \sigma_{\nu + 1}^{2}  x_i^{2} = \sigma_{\nu + 1}^{2}
\end{align}$$

$i.e.$, $\big\| (A - A_{\nu})\vec{x} \big\|_{2} \leq \sigma_{\nu + 1}$. Now we have an upper limit, and at the same time, if we take $\vec{x} = \vec{\upsilon}_{\nu + 1}$, obviously we have

$$\begin{align}
\big\| (A - A_{\nu})\vec{x} \big\|^{2}_{2} &= \big\| \sigma _{\nu + 1} \vec{u} _{\nu + 1} \vec{\upsilon} ^{\mathrm{T}} _{\nu + 1} \vec{\upsilon} _{\nu + 1} + \sigma _{\nu + 2} \vec{u} _{\nu + 2} \vec{\upsilon} ^{\mathrm{T}} _{\nu + 2} \vec{\upsilon} _{\nu + 1} + \cdots + \sigma_{r} \vec{u}_{r} \vec{\upsilon} ^{\mathrm{T}} _{r} \vec{\upsilon} _{\nu + 1} \big\|^{2}_{2} \\
&=\big\| \sigma _{\nu + 1} \vec{u} _{\nu + 1} \cdot 1 + \sigma _{\nu + 2} \vec{u} _{\nu + 2} \cdot 0 + \cdots + \sigma_{r} \vec{u}_{r} \cdot 0 \big\|^{2}_{2} = \sigma^{2}_{\nu + 1}
\end{align}$$

$\dagger$
Other value for $\vec{x}$ would be like for $i = 1,2,\dots,r$, take $\vec{x} = \vec{\upsilon}_{i}$. When $i \leq \nu$, $\big\| (A - A_{\nu})\vec{x} \big\|^{2}_{2} = 0$, and when $i > \nu$, $\big\| (A - A_{\nu})\vec{x} \big\|^{2}_{2} = \sigma^{2}_{i}$

That is to say such upper limit is reachable. Therefore, $\left\| A - A_{\nu}\right\| = \sigma_{\nu + 1}$.

$\square$
***
$Theorem 5$

For any $m \times n$ matrix $A$,

$$\left\|A\right\|_2 = \sigma_1$$,

$i.e.$, the **largest singular value** of the matrix $A$. And if it is a nonsingular $m \times m$ matrix, then

$$\left\| A^{-1}\right\|_2 = \frac{1} {\sigma_m}$$

$Proof$

The first part is just the special case in $Theorem 4$ for $\nu = 0$. We will just skip that, and focus on the rest part. We notice that the singular value of nonsingular matrix $A^{-1}$ are $\displaystyle \frac{1} {\sigma_m} \geq \frac{1} {\sigma_{m-1}} \geq \cdots \geq \frac{1} {\sigma_2} \geq \frac{1} {\sigma_1} > 0$. And so that we can use the first result and get the result we want! All Done!

$\square$

Computational/Intro to Numerical Computing/Note_Chap04_Conditioning, Stability, and Accuracy.ipynb


# Kelly Strategy vs All-In Strategy

## Objective

Repeated gambles on a favourable bet have the features that:
* Feature 1: The all-in strategy achieves the maximal expected value for a given number of plays.
* Feature 2: The Kelly strategy achieves higher value than any other strategy a.e.

My intuitive reaction is that these are in contradiction. The goal of this lab is to build a better intuition of the distinction.

## Background

### Setup

This follows the presentation of \[Thorp, 1969\].

In particular, we assume binary bets offered at evens and take:

* Times: $i \in {1, 2, ...}$
* Trials: $T_i \in {-1, 1}$, with $P(T_i = 1) = p = 1 - q$
* Bet size: $B_i$, such that our earning for the $n$th bet is $B_n T_n$

Our wealth after the result of the $n$th bet is: $X_n = X_0 + \displaystyle\sum_{i=1}^n B_i T_i$

We further assume:
* $0 \leq B_i \leq X_{i-1}$ (roughly: this is one-sided market with no leverage)
* $X_i = 0 => X_j = 0 \, \forall \, j > i$ (roughly: we have fixed capital)
* $p > \frac{1}{2}$ (we are only interested in favourable bets)

### Feature 1: Maximal Expectation

The strategy maximizing the expectation of Wealth to step $n$ (for any $n$) is that of betting your entire wealth at each stage ($B_i = X_{i-1}$).

To see this, let $B_i := f_i X_{i-1}$ for some $0 \leq f_i \leq 1$ and condition on $X_{n-1}$:

$$
\begin{align}
\mathbb{E}[X_n] &= \mathbb{E}[X_{n-1}] + \mathbb{E}[B_n T_n] \\
    &= \mathbb{E}[X_{n-1}] + \mathbb{E}[B_n] \mathbb{E}[T_n] \\
    &= \mathbb{E}[X_{n-1}] + (p-q) \mathbb{E}[B_n] \\
    &= \mathbb{E}[X_{n-1}] + (p-q) \mathbb{E}[f_n X_{n-1}] \\
    &= \mathbb{E}\big[X_{n-1} \big(1 + (p-q) f_n \big) \big] \\
    &= \mathbb{E}\big[X_{n-1} \mathbb{E}[1 + (p-q) f_n | X_{n-1}] \big] \\
    &= \mathbb{E}\big[X_{n-1} \big(1 + (p-q)\mathbb{E}[f_n | X_{n-1}]\big) \big]
\end{align}
$$

From this, the optimal $f_n = 1$ (or $B_n = X_{n-1}$). That is, bet all-in. And similarly, by induction, $B_i = X_{i-1}$.

For a given $n$, this strategy is worth

$$
X_n = \left\{
     \begin{array}{11}
         2^n & \mbox{with probability }p^n \\
         0   & \mbox{otherwise}
     \end{array}
\right.
$$

giving an expected value $\mathbb{E}[X_n] = (2p)^n \to \infty$, but ruin probability $1-p^n \to 1$

### Feature 2: Kelly Criterion

The Kelly Criterion maximizes $\mathbb{E}[\log(\frac{X_n}{X_{n-1}})] = \mathbb{E}[\log(1+f_n T_n)] = p \log(1+f_n) + q \log(1-f_n)$.

Letting $G(f) := p \log(1+f) + q \log(1-f)$, we see that $G$ maximized at $f^* = p-q$. So the Kelly Criterion bets a constant fraction $p-q$ on each trial.

Intuitively, this maximizes the rate of growth. But more formally, this satisfies:

* $\mathbb{P}(\lim \inf X^*_n > M) = 1 \, \forall M$ \[Thorp, 1969\]
* $\mathbb{P}(X^*_n/X'_n \to \infty) = 1$ where $X'$ follows an "essentially different" strategy \[Brieman, 1961\]
* $\lim_{x \to \infty} (\mathbb{E}[T(x)] - \mathbb{E}[T^*(x)]) > 0$ where $T(x)$ is the first hitting time of $X_n = x$ \[Brieman, 1961\]

## Discussion

On explicating the nature of the maximal expectation strategy, it's now clear that the apparent contradiction between (Feature 1) and (Feature 2) is not a contradiction at all. And in fact that the maximal expectation strategy really isn't of any interest. The maximal expectation strategy exceeds the Kelly strategy only on the event of all successful trials up to $n$. This even has probability $p^n \to 0$. But the value on this event grows quickly enough in $n$ such that the expectation of this strategy exceeds all others. Like the St Petersburg paradox, this only works with unbounded (and indeed almost immediately infeasibly large) capital.

Noting that both strategies are fixed fractional strategies, we can consider $X_n = X_n(f)$ as a function of that fraction $f$:

$$
\begin{align}
    X_n(f) &= X_{n-1} + f X_{n-1} * T_n \\
        &= X_{n-1} (1 + f T_n) \\
        &= \ldots \\
        &= X_0 \displaystyle\prod_{i=1}^n(1 + f T_i) \\
\end{align}
$$

From which:

$$
\begin{align}
    \mathbb{E}[X_n(f)] &= X_0 \big( 1+(p-q)f \big)^n \\
    Var\big( X_n(f) \big) &= X_0^2 \Big( \big( 1 + (p-q)f + f^2 \big)^n - \big( 1 + (p-q)f \big)^n \Big)
\end{align}
$$

More interesting, and again pointing to the importance of the log expectation, is the tricotomy shown in \[Thorp 1969\] for $G(f) := \mathbb{E}\big[ \log \big( \frac{X_n(f)}{X_{n-1}(f)} \big) \big] = \mathbb{E}\Big[ \log \Big( \big( \frac{X_n(f)}{X_0(f)} \big)^\frac{1}{n} \Big) \Big]$. He shows that there is a unique $f_c$ with $0 < f^* < f_c < 1$ such that:

$$
G(f) \left\{
    \begin{array}{11}
        > 0  & \mbox{  if } 0 \leq f < f_c \\
        = 0  & \mbox{  if } f = f_c \\
        < 0  & \mbox{  if } f_c < f \leq 1 \\
    \end{array}
\right.
$$

which drives:

$$
\left\{
    \begin{array}{11}
        X_n(f) \to \infty & \mbox{  if } G(f) > 0 \\
        \lim \sup X_n(f) = \infty, \lim \inf X_n(f) = 0  & \mbox{  if } G(f) = 0 \\
        X_n(f) \to 0 & \mbox{  if } G(f) < 0 \\
    \end{array}
\right.
$$

We can see $G(f)$ here:

```python
import altair as alt
import numpy as np
import pandas as pd

pgrid = np.linspace(0.5, 1, num=10)
fgrid = np.linspace(0, 1, num=1000, endpoint=False)
p, f = np.meshgrid(pgrid, fgrid)
G = p*np.log(1+f)+(1-p)*np.log(1-f)

alt.data_transformers.disable_max_rows()

(
    alt
    .Chart(
        pd.DataFrame(
            data=np.array([p, f, G]).reshape(3, -1).T,
            columns=["p", "f", "G"]
        )
    )
    .mark_line()
    .encode(
        x="f",
        y="G",
        color="p"
    )
)

```

## References

[\[Brieman, 1961\]](https://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-05.pdf) Breiman, L. Optimal Gambling Systems for Favorable Games. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics, 65--78, University of California Press, Berkeley, Calif., 1961. https://projecteuclid.org/euclid.bsmsp/1200512159

[\[Thorp, 1969\]](https://www.jstor.org/stable/1402118) Thorp, E. O. "Optimal Gambling Systems for Favorable Games." Revue De L'Institut International De Statistique / Review of the International Statistical Institute 37, no. 3 (1969): 273-93. Accessed November 28, 2020. doi:10.2307/1402118.

notebooks/1. Kelly Strategy vs All-In Strategy.ipynb


# Numbers

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
```

```python
import numpy as np
```

## The `ndarray`: Vectors, matrices and tenosrs

dtype, shape, strides

### Vector

```python
x = np.array([1,2,3])
x
```

    array([1, 2, 3])

```python
type(x)
```

    numpy.ndarray

```python
x.dtype
```

    dtype('int64')

```python
x.shape
```

    (3,)

```python
x.strides
```

    (8,)

### Matrix

```python
x = np.array([[1,2,3], [4,5,6]], dtype=np.int32)
x
```

    array([[1, 2, 3],
           [4, 5, 6]], dtype=int32)

```python
x.dtype
```

    dtype('int32')

```python
x.shape
```

    (2, 3)

```python
x.strides
```

    (12, 4)

### Tensor

```python
x = np.arange(24).reshape((2,3,4))
```

```python
x
```

    array([[[ 0,  1,  2,  3],
            [ 4,  5,  6,  7],
            [ 8,  9, 10, 11]],

           [[12, 13, 14, 15],
            [16, 17, 18, 19],
            [20, 21, 22, 23]]])

## Creating `ndarray`s

### From a file

```python
%%file numbers.txt
a,b,c # can also skip headers
1,2,3
4,5,6
```

    Overwriting numbers.txt

```python
np.loadtxt('numbers.txt', dtype='int', delimiter=',',
           skiprows=1, comments='#')
```

    array([[1, 2, 3],
           [4, 5, 6]])

### From Python lists or tuples

```python
np.array([
    [1,2,3],
    [4,5,6]
])
```

    array([[1, 2, 3],
           [4, 5, 6]])

### From ranges

arange, linspace, logspace

```python
np.arange(1, 7).reshape((2,3))
```

    array([[1, 2, 3],
           [4, 5, 6]])

```python
np.linspace(1, 10, 4)
```

    array([  1.,   4.,   7.,  10.])

```python
np.logspace(0, 4, 5, dtype='int')
```

    array([    1,    10,   100,  1000, 10000])

### From a function

fromfunciton

```python
np.fromfunction(lambda i, j: i*3 + j + 1, (2,3))
```

    array([[ 1.,  2.,  3.],
           [ 4.,  5.,  6.]])

```python
np.fromfunction(lambda i, j: (i-2)**2 + (j-2)**2, (5,5))
```

    array([[ 8.,  5.,  4.,  5.,  8.],
           [ 5.,  2.,  1.,  2.,  5.],
           [ 4.,  1.,  0.,  1.,  4.],
           [ 5.,  2.,  1.,  2.,  5.],
           [ 8.,  5.,  4.,  5.,  8.]])

```python
np.fromfunction(lambda i, j: np.where(i<j, 1, np.where(i==j,0, -1)), (5,5))
```

    array([[ 0,  1,  1,  1,  1],
           [-1,  0,  1,  1,  1],
           [-1, -1,  0,  1,  1],
           [-1, -1, -1,  0,  1],
           [-1, -1, -1, -1,  0]])

Adapted from [Circle Squared](http://igpphome.ucsd.edu/~shearer/COMP233/SciAm_Mandel.pdf)

```python
xs = np.fromfunction(lambda i, j:  0.27**2*(i**2 + j**2) % 1.5,
                     (400, 400),)
plt.figure(figsize=(8,8))
plt.imshow(xs, interpolation='nearest', cmap=plt.cm.YlOrBr)
plt.xticks([])
plt.yticks([])
pass
```

### From special construcotrs

zeoros, ones, eye

```python
np.zeros((2,3))
```

    array([[ 0.,  0.,  0.],
           [ 0.,  0.,  0.]])

```python
np.ones((2,3))
```

    array([[ 1.,  1.,  1.],
           [ 1.,  1.,  1.]])

```python
np.eye(3)
```

    array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

```python
np.eye(3, 4)
```

    array([[ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.]])

```python
np.eye(4, k=-1)
```

    array([[ 0.,  0.,  0.,  0.],
           [ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.]])

### From random variables

#### Convenience functions

rand, randn

```python
np.random.rand(2,3)
```

    array([[ 0.04737547,  0.82228651,  0.22224204],
           [ 0.96886745,  0.54149529,  0.94853004]])

```python
np.random.randn(2,3)
```

    array([[-0.80516331,  0.9086527 , -1.21134755],
           [-0.08444392,  1.00515737,  0.94762797]])

#### Distributions

uniform, normal, randint, poisson, multinomial, multivariate_ normal

```python
np.random.uniform(0, 1, (2,3))
```

    array([[ 0.12880127,  0.32643222,  0.52082391],
           [ 0.35404701,  0.54964578,  0.456206  ]])

```python
np.random.normal(0, 1, (2,3))
```

    array([[ 0.8195548 ,  0.43940477, -0.24901744],
           [ 2.6248388 ,  0.16150201,  0.12905971]])

```python
np.random.randint(0, 10, (4,5))
```

    array([[5, 0, 2, 3, 9],
           [8, 5, 1, 1, 9],
           [5, 5, 2, 0, 2],
           [2, 0, 5, 7, 4]])

```python
np.random.poisson(10, (4,5))
```

    array([[ 8,  9, 17,  8, 15],
           [ 8,  9, 12, 12, 11],
           [ 6,  8, 10, 13,  6],
           [ 7, 12,  8, 16,  8]])

```python
np.random.multinomial(n=5, pvals=np.ones(5)/5, size=8)
```

    array([[2, 0, 0, 2, 1],
           [4, 0, 0, 1, 0],
           [3, 0, 0, 1, 1],
           [1, 1, 1, 1, 1],
           [0, 1, 2, 1, 1],
           [1, 1, 2, 1, 0],
           [2, 0, 1, 1, 1],
           [0, 1, 1, 1, 2]])

```python
np.random.multivariate_normal(mean=[10,20,30], cov=np.eye(3), size=4)
```

    array([[  9.34907429,  20.28742704,  31.20954084],
           [ 10.61473257,  19.60538925,  29.09199082],
           [  9.10517086,  19.82157961,  28.57954932],
           [  8.35947889,  22.60010629,  29.05471801]])

## Indexing

```python
x = np.arange(20).reshape((4,5))
x
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19]])

### Extracing a scalar

```python
x[1,1]
```

    6

### Extracting a vector

```python
x[1]
```

    array([5, 6, 7, 8, 9])

### Using slices

```python
x[1,:]
```

    array([5, 6, 7, 8, 9])

```python
x[:,1]
```

    array([ 1,  6, 11, 16])

```python
x[1:3,1:3]
```

    array([[ 6,  7],
           [11, 12]])

### Using slices with strides

```python
x[::2,::2]
```

    array([[ 0,  2,  4],
           [10, 12, 14]])

### Extrcting blocks with arbitrary row and column lists (fancy indexing)

`np.ix_`

```python
x[:, [0,3]]
```

    array([[ 0,  3],
           [ 5,  8],
           [10, 13],
           [15, 18]])

Warning: Fancy indexing can only be used for 1 dimension.

```python
x[[0,2],[0,3]]
```

    array([ 0, 13])

Use the helper `np.ix_` to extract arbitrary blocks.

```python
x[np.ix_([0,2], [0,3])]
```

    array([[ 0,  3],
           [10, 13]])

### A slice is a view, not a copy

```python
x
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19]])

```python
y = x[1:-1, 1:-1]
y
```

    array([[ 6,  7,  8],
           [11, 12, 13]])

```python
y *= 10
```

```python
y
```

    array([[ 60,  70,  80],
           [110, 120, 130]])

```python
x
```

    array([[  0,   1,   2,   3,   4],
           [  5,  60,  70,  80,   9],
           [ 10, 110, 120, 130,  14],
           [ 15,  16,  17,  18,  19]])

Use the copy method to convert a view to a copy

```python
z = x[1:-1, 1:-1].copy()
```

```python
z
```

    array([[ 60,  70,  80],
           [110, 120, 130]])

```python
z[:] = 0
```

```python
z
```

    array([[0, 0, 0],
           [0, 0, 0]])

```python
x
```

    array([[  0,   1,   2,   3,   4],
           [  5,  60,  70,  80,   9],
           [ 10, 110, 120, 130,  14],
           [ 15,  16,  17,  18,  19]])

### Boolean indexing

```python
x[x % 2 == 0]
```

    array([  0,   2,   4,  60,  70,  80,  10, 110, 120, 130,  14,  16,  18])

```python
x [x > 3]
```

    array([  4,   5,  60,  70,  80,   9,  10, 110, 120, 130,  14,  15,  16,
            17,  18,  19])

### Functions that return indexes

```python
idx = np.nonzero(x)
idx
```

    (array([0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3]),
     array([1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4]))

```python
x[idx]
```

    array([  1,   2,   3,   4,   5,  60,  70,  80,   9,  10, 110, 120, 130,
            14,  15,  16,  17,  18,  19])

```python
idx = np.where(x > 3)
idx
```

    (array([0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3]),
     array([4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4]))

```python
x[idx]
```

    array([  4,   5,  60,  70,  80,   9,  10, 110, 120, 130,  14,  15,  16,
            17,  18,  19])

## Margins and the `axis` argument

```python
x
```

    array([[  0,   1,   2,   3,   4],
           [  5,  60,  70,  80,   9],
           [ 10, 110, 120, 130,  14],
           [ 15,  16,  17,  18,  19]])

The 0th axis has 4 items, the 1st axis has 5 items.

```python
x.shape
```

    (4, 5)

```python
x.mean()
```

    35.149999999999999

### Marginalizing out the 0th axis = column summaries

```python
x.mean(axis=0)
```

    array([  7.5 ,  46.75,  52.25,  57.75,  11.5 ])

### Marginalizing out the 1st axis = row summaries

```python
x.mean(axis=1)
```

    array([  2. ,  44.8,  76.8,  17. ])

Note marginalizing out the last axis is a common default.

```python
x.mean(axis=-1)
```

    array([  2. ,  44.8,  76.8,  17. ])

### Marginalization works for higher dimensions in the same way

```python
x = np.random.random((2,3,4))
x
```

    array([[[ 0.25091919,  0.97511897,  0.6520031 ,  0.56832053],
            [ 0.97444408,  0.72376855,  0.15067567,  0.10752009],
            [ 0.20627991,  0.0234196 ,  0.70143528,  0.87408988]],

           [[ 0.95505841,  0.79364654,  0.76668484,  0.47971008],
            [ 0.34780788,  0.44169709,  0.63645331,  0.96215426],
            [ 0.92889169,  0.52710155,  0.22280321,  0.04262088]]])

```python
x.shape
```

    (2, 3, 4)

```python
x.mean(axis=0).shape
```

    (3, 4)

```python
x.mean(axis=1).shape
```

    (2, 4)

```python
x.mean(axis=2).shape
```

    (2, 3)

```python
x.mean(axis=(0,1)).shape
```

    (4,)

```python
x.mean(axis=(0,2)).shape
```

    (3,)

```python
x.mean(axis=(1,2)).shape
```

    (2,)

## Broadcasting

```python
x1 = np.arange(12)
```

```python
x1
```

    array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])

```python
x1 * 10
```

    array([  0,  10,  20,  30,  40,  50,  60,  70,  80,  90, 100, 110])

```python
x2 = np.random.randint(0,10,(3,4))
```

```python
x2
```

    array([[3, 0, 1, 2],
           [2, 4, 0, 7],
           [9, 9, 3, 0]])

```python
x2 * 10
```

    array([[30,  0, 10, 20],
           [20, 40,  0, 70],
           [90, 90, 30,  0]])

```python
x2.shape
```

    (3, 4)

### Column-wise broadcasting

```python
mu = np.mean(x2, axis=0)
mu.shape
```

    (4,)

```python
x2 - mu
```

    array([[-1.66666667, -4.33333333, -0.33333333, -1.        ],
           [-2.66666667, -0.33333333, -1.33333333,  4.        ],
           [ 4.33333333,  4.66666667,  1.66666667, -3.        ]])

```python
(x2 - mu).mean(axis=0)
```

    array([ -2.96059473e-16,   2.96059473e-16,   7.40148683e-17,
             0.00000000e+00])

### Row wise broadcasting

```python
mu = np.mean(x2, axis=1)
mu.shape
```

    (3,)

```python
try:
    x2 - mu
except ValueError as e:
    print(e)
```

    operands could not be broadcast together with shapes (3,4) (3,)

### We can add a "dummy" axis using None or `np.newaxis`

```python
mu[:, None].shape
```

    (3, 1)

```python
x2 - mu[:, None]
```

    array([[ 1.5 , -1.5 , -0.5 ,  0.5 ],
           [-1.25,  0.75, -3.25,  3.75],
           [ 3.75,  3.75, -2.25, -5.25]])

```python
x2 - mu[:, np.newaxis]
```

    array([[ 1.5 , -1.5 , -0.5 ,  0.5 ],
           [-1.25,  0.75, -3.25,  3.75],
           [ 3.75,  3.75, -2.25, -5.25]])

```python
np.mean(x2 - mu[:, None], axis=1)
```

    array([ 0.,  0.,  0.])

#### Reshaping works too

```python
x2 - mu.reshape((-1,1))
```

    array([[ 1.5 , -1.5 , -0.5 ,  0.5 ],
           [-1.25,  0.75, -3.25,  3.75],
           [ 3.75,  3.75, -2.25, -5.25]])

#### Broadcasting examples

Creating a 12 by 12 multiplication table

```python
x = np.arange(1, 13)
x[:,None] * x[None,:]
```

    array([[  1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12],
           [  2,   4,   6,   8,  10,  12,  14,  16,  18,  20,  22,  24],
           [  3,   6,   9,  12,  15,  18,  21,  24,  27,  30,  33,  36],
           [  4,   8,  12,  16,  20,  24,  28,  32,  36,  40,  44,  48],
           [  5,  10,  15,  20,  25,  30,  35,  40,  45,  50,  55,  60],
           [  6,  12,  18,  24,  30,  36,  42,  48,  54,  60,  66,  72],
           [  7,  14,  21,  28,  35,  42,  49,  56,  63,  70,  77,  84],
           [  8,  16,  24,  32,  40,  48,  56,  64,  72,  80,  88,  96],
           [  9,  18,  27,  36,  45,  54,  63,  72,  81,  90,  99, 108],
           [ 10,  20,  30,  40,  50,  60,  70,  80,  90, 100, 110, 120],
           [ 11,  22,  33,  44,  55,  66,  77,  88,  99, 110, 121, 132],
           [ 12,  24,  36,  48,  60,  72,  84,  96, 108, 120, 132, 144]])

Scaling to have zero mean and unit standard devation for each feature.

```python
x = np.random.normal(10, 5,(3,4))
x
```

    array([[  7.65396872,   8.74106757,   7.26659687,   9.68096491],
           [  9.27132075,   7.89349589,   9.47813473,   7.04088855],
           [ 13.78187972,   7.39243968,  10.90034325,   4.27477767]])

Standardize column-wise

```python
(x - x.mean(axis=0))/x.std(axis=0)
```

    array([[-0.99566783,  1.31524702, -1.30321654,  1.21511731],
           [-0.37192708, -0.20751913,  0.17598238,  0.01903327],
           [ 1.36759491, -1.10772789,  1.12723416, -1.23415058]])

Standardize row-wise

```python
(x - x.mean(axis=1)[:, None])/x.std(axis=1)[:, None]
```

    array([[-0.72037796,  0.42843253, -1.12973985,  1.42168528],
           [ 0.84786879, -0.52591836,  1.0540767 , -1.37602714],
           [ 1.31012381, -0.47301018,  0.50595856, -1.34307219]])

## Combining `ndarray`s

```python
x1 = np.zeros((3,4))
x2 = np.ones((3,5))
x3 = np.eye(4)
```

```python
x1
```

    array([[ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.]])

```python
x2
```

    array([[ 1.,  1.,  1.,  1.,  1.],
           [ 1.,  1.,  1.,  1.,  1.],
           [ 1.,  1.,  1.,  1.,  1.]])

```python
x3
```

    array([[ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  1.]])

### Binding rows when number of columns is the same

```python
np.r_[x1, x3]
```

    array([[ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.],
           [ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  1.]])

### Binding columns when number of rows is the same

```python
np.c_[x1, x2]
```

    array([[ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.],
           [ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.],
           [ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.]])

### You can combine more than 2 at a time

```python
np.c_[x1, x2, x1]
```

    array([[ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.,  0.,  0.,  0.,  0.]])

### Stacking

```python
np.vstack([x1, x3])
```

    array([[ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.],
           [ 1.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.],
           [ 0.,  0.,  0.,  1.]])

```python
np.hstack([x1, x2])
```

    array([[ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.],
           [ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.],
           [ 0.,  0.,  0.,  0.,  1.,  1.,  1.,  1.,  1.]])

```python
np.dstack([x2, 2*x2, 3*x2])
```

    array([[[ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.]],

           [[ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.]],

           [[ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.]]])

### Generic stack with axis argument

```python
np.stack([x2, 2*x2, 3*x2], axis=0)
```

    array([[[ 1.,  1.,  1.,  1.,  1.],
            [ 1.,  1.,  1.,  1.,  1.],
            [ 1.,  1.,  1.,  1.,  1.]],

           [[ 2.,  2.,  2.,  2.,  2.],
            [ 2.,  2.,  2.,  2.,  2.],
            [ 2.,  2.,  2.,  2.,  2.]],

           [[ 3.,  3.,  3.,  3.,  3.],
            [ 3.,  3.,  3.,  3.,  3.],
            [ 3.,  3.,  3.,  3.,  3.]]])

```python
np.stack([x2, 2*x2, 3*x2], axis=1)
```

    array([[[ 1.,  1.,  1.,  1.,  1.],
            [ 2.,  2.,  2.,  2.,  2.],
            [ 3.,  3.,  3.,  3.,  3.]],

           [[ 1.,  1.,  1.,  1.,  1.],
            [ 2.,  2.,  2.,  2.,  2.],
            [ 3.,  3.,  3.,  3.,  3.]],

           [[ 1.,  1.,  1.,  1.,  1.],
            [ 2.,  2.,  2.,  2.,  2.],
            [ 3.,  3.,  3.,  3.,  3.]]])

```python
np.stack([x2, 2*x2, 3*x2], axis=2)
```

    array([[[ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.]],

           [[ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.]],

           [[ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.],
            [ 1.,  2.,  3.]]])

### Repetition and tiling

#### For a vector

```python
x = np.array([1,2,3])
```

```python
np.repeat(x, 3)
```

    array([1, 1, 1, 2, 2, 2, 3, 3, 3])

```python
np.tile(x, 3)
```

    array([1, 2, 3, 1, 2, 3, 1, 2, 3])

#### For a matrix

```python
x = np.arange(6).reshape((2,3))
x
```

    array([[0, 1, 2],
           [3, 4, 5]])

```python
np.repeat(x, 3)
```

    array([0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5])

```python
np.repeat(x, 3, axis=0)
```

    array([[0, 1, 2],
           [0, 1, 2],
           [0, 1, 2],
           [3, 4, 5],
           [3, 4, 5],
           [3, 4, 5]])

```python
np.repeat(x, 3, axis=1)
```

    array([[0, 0, 0, 1, 1, 1, 2, 2, 2],
           [3, 3, 3, 4, 4, 4, 5, 5, 5]])

```python
np.tile(x, 3)
```

    array([[0, 1, 2, 0, 1, 2, 0, 1, 2],
           [3, 4, 5, 3, 4, 5, 3, 4, 5]])

## Splitting `ndarray`s

```python
x = np.arange(32).reshape((4,8))
```

```python
x
```

    array([[ 0,  1,  2,  3,  4,  5,  6,  7],
           [ 8,  9, 10, 11, 12, 13, 14, 15],
           [16, 17, 18, 19, 20, 21, 22, 23],
           [24, 25, 26, 27, 28, 29, 30, 31]])

```python
np.split(x, 4)
```

    [array([[0, 1, 2, 3, 4, 5, 6, 7]]),
     array([[ 8,  9, 10, 11, 12, 13, 14, 15]]),
     array([[16, 17, 18, 19, 20, 21, 22, 23]]),
     array([[24, 25, 26, 27, 28, 29, 30, 31]])]

```python
np.split(x, 4, axis=1)
```

    [array([[ 0,  1],
            [ 8,  9],
            [16, 17],
            [24, 25]]), array([[ 2,  3],
            [10, 11],
            [18, 19],
            [26, 27]]), array([[ 4,  5],
            [12, 13],
            [20, 21],
            [28, 29]]), array([[ 6,  7],
            [14, 15],
            [22, 23],
            [30, 31]])]

## Vectorization

### Example 1

The operators and functions (ufuncs) in Python are vectorized, and will work element-wise over all entries in an `ndarray`.

```python
xs = np.zeros(10, dtype='int')
for i in range(10):
    xs[i] = i**2
xs
```

    array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])

```python
xs = np.arange(10)**2
xs
```

    array([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])

Using ufuncs

```python
np.sqrt(xs)
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])

```python
np.log1p(xs)
```

    array([ 0.        ,  0.69314718,  1.60943791,  2.30258509,  2.83321334,
            3.25809654,  3.61091791,  3.91202301,  4.17438727,  4.40671925])

### Example 2

```python
n = 10

xs = np.random.rand(n)
ys = np.random.rand(n)

s = 0
for i in range(n):
    s += xs[i] * ys[i]
s
```

    2.6868481407430282

```python
np.dot(xs, ys)
```

    2.6868481407430282

```python
xs @ ys
```

    2.6868481407430282

### Example 3

\begin{align}
y_0 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
y_1 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
y_2 &= \alpha + \beta_1 x_1 + \beta_2 x_2 \\
\end{align}

```python
m = 3
n = 2

alpha = np.random.rand(1)
betas = np.random.rand(n,1)
xs = np.random.rand(m,n)
```

```python
alpha
```

    array([ 0.05007867])

```python
betas
```

    array([[ 0.06399532],
           [ 0.37993848]])

```python
xs
```

    array([[ 0.79258489,  0.20951971],
           [ 0.89978267,  0.044122  ],
           [ 0.34222118,  0.06629664]])

### Using loops

```python
ys = np.zeros((m,1))
for i in range(m):
    ys[i] = alpha
    for j in range(n):
        ys[i] += betas[j] * xs[i,j]
ys
```

    array([[ 0.180405  ],
           [ 0.1244242 ],
           [ 0.09716787]])

### Removing inner loop

```python
ys = np.zeros((m,1))
for i in range(m):
    ys[i] = alpha + xs[i,:].T @ betas
ys
```

    array([[ 0.180405  ],
           [ 0.1244242 ],
           [ 0.09716787]])

### Removing all loops

```python
ys = alpha + xs @ betas
ys
```

    array([[ 0.180405  ],
           [ 0.1244242 ],
           [ 0.09716787]])

### Alternative approach

The calculaiton with explicit intercepts and coefficients is common in deep learning, where $\alpha$ is called the bias ($b$) and $\beta$ are called the weights ($w$), and each equation is $y[i] = b + w[i]*x[i]$.

It is common in statisiics to use an augmented matrix in which the first column is all ones, so that all that is needed is a single matrix multiplicaiotn.

```python
X = np.c_[np.ones(m), xs]
X
```

    array([[ 1.        ,  0.79258489,  0.20951971],
           [ 1.        ,  0.89978267,  0.044122  ],
           [ 1.        ,  0.34222118,  0.06629664]])

```python
alpha
```

    array([ 0.05007867])

```python
betas
```

    array([[ 0.06399532],
           [ 0.37993848]])

```python
betas_ = np.concatenate([[alpha], betas])
betas_
```

    array([[ 0.05007867],
           [ 0.06399532],
           [ 0.37993848]])

```python
ys = X @ betas_
ys
```

    array([[ 0.180405  ],
           [ 0.1244242 ],
           [ 0.09716787]])

### Simulating diffusion

```python
w = 100
h = 100
x = np.zeros((w+2,h+2), dtype='float')
x[(w//2-1):(w//2+2), (h//2-1):(h//2+2)] = 1

wts = np.ones(5)/5

for i in range(41):
    if i % 10 == 0:
        plt.figure()
        plt.imshow(x[1:-1, 1:-1], interpolation='nearest')

    center = x[1:-1, 1:-1]
    left = x[:-2, 1:-1]
    right = x[2:, 1:-1]
    bottom = x[1:-1, :-2]
    top = x[1:-1, 2:]
    nbrs = np.dstack([center, left, right, bottom, top])
    x = np.sum(wts * nbrs, axis=-1)
```

```python

```